How to use ‘flow traceoptions’ and the ‘security datapath-debug’

Understand the configuration and use of flow traceoptions and datapath-debug for SRX series.

In order to debug the flow processing in SRX platforms, it is necessary to configure traceoptions. Flow traceoptions are configured under the “security flow traceoptions” stanza. It requires a configuration change instead of just an operational command as parameters need to be specified.

This article discusses the following:

  • Flow Traceoptions Feature – trace (debug) only capability – supported on all SRX platforms
  • Security datapath-debug Feature – system component tracing, flow trace (debug) and packet capture capabilities – supported on high-end SRX platforms

Flow Traceoptions Feature (all SRX platforms)

For all SRX platforms running all Junos versions, flow traceoptions are configured under the “security flow traceoptions” stanza.

The configuration can be done by specifying three components:
1.Trace file, fize size, number of files:
It is necessary to specify the file to which the trace will be logged. It is also possible to choose the file size and how many files will be saved, in case the output goes over the file size:

2.Trace flags:
The flags indicate the level of tracing required. For flow traceoptions, the “basic-datapath” flag is recommended:

3.Packet filters:
By default, flow traceoptions logs all packets processed by the system. The recommendation is to configure packet filters to capture only the desired traffic. This is to avoid system overload as traffic load can be very high. It is not necessary to configure filters for both directions because reverse traffic matches the same filter. Packet filters can be configured with source and destination prefix and port (including ranges), interface and protocol.

Below is a configuration example:

Note that part of the configuration is inactive. This can be very useful as the configuration doesn’t need to be deleted, the user can activate it whenever needed.
Warning: It is not recommended to deactivate all packet-filters but instead deactivate traceoptions as a whole, using ‘deactivate security flow traceoptions’.

Below are the corresponding configuration commands:

In order to see the trace output, use the command “show log <filename>”.
Match, trim and other show options can be used to help read the output.

Security datapath-debug Feature (SRX1400, SRX3000, and SRX5000 platforms)

Specifically for SRX1400, SRX3000 and SRX5000 platforms, flow traceoptions can also be configured using the “security datapath-debug” stanza. This advanced feature includes system component tracing, flow trace (debug) and packet capture capabilities.

This stanza objective provides tracing capability for packet’s entire life cycle in datapath, from ingress to egress (not only flow processing). It augments the tracing capabilities with the following features:
1.End-to-end debug:
Ingress IOC:
– mac-ingress (only SRX3000)
– np-ingress *
Ingress flow (lbt) *
Flow processing *
Route lookup (jexec)
Egress flow (pot) *

Egress IOC:
– np-egress *
– mac-egress (only SRX3000)
2.Packet Dump: captures the first 100 bytes of the packets including internal headers used by the system
3.Packet Summary: prints a summary of the packet, including the following headers: *
Meta header (used internally by the system)
Flow header (used internally by the system)
IP header of the actual packet
4.Trace: specific debugging messages for product engineering use *
5.preserve-trace-order: due to multi-threaded processing, traces can be written to the trace file out of order. This option adds an extra debug header with a sequence number that can be used by the Routing Engine (RE) to keep trace order.
6.record-pic-history: adds an extra debug header to record each PIC through which the packets are processed

Configuration

The configuration is also done in three steps:
1.Trace file, file size, number of files:
It is necessary to specify the file to which the trace will be logged. It is also possible to choose the file size and how many files will be saved in case the output goes over the file size:

2.Action profile:
With a custom action profile, it is strongly recommended to configure the following. All are optional, it depends on the type of trace desired.

a.Specify the type of events to be traced and the action for each of them.

b.Enable preserve-trace-order, record-pic-history options:

c.Configure capture file if packet-dump option is used:

d.Enable flow module trace:

3.Packet filters:
If the default action is deny, that means no packets are captured if a packet filter is not configured. The recommendation is to configure the filter as specific as possible in order to avoid system overload as traffic load can be very high. It is necessary to configure filters for both directions. Packet filters can be configured with source and destination prefix and port (including ranges), and protocol.

Configuration Example:

Below is an example of the configuration commands:

Verify Configuration:

Use “show security datapath-debug action-profile” command to see the active action-profiles. The output below shows an example with two packet filters:

Display Trace Output:

In order to see the flow module trace, use the command “show log <filename>”. Match, trim and other show options can be used to help read the output;Below is an example with the preserve-trace-order and record-pic-history options enabled. Note that the output was filtered with “except” option to remove the PIC history and thread id information. This is very usefull if the objective is to check only the flow processing:

For packet summary and trace actions, use also “show log <filename>”. Below is an example with “packet-summary” and “record-pic-history” options enabled. This information is useful when debugging packet loss inside the system. The recommendation is to use np-egress event with packet-summary action, together with record-pic-history option; this is because logging on the egress side shows the complete path the packet had through the system.

Display Counters:

In order to see the counters, use “show security datapath-debug counters”. Below is an example showing counters for np-ingress and np-egress, for a system with only one IOC on FPC 3 in both nodes of the cluster. The output is shown for all packet filters configured and active. This is usefull to see if traffic that is entering is also leaving the system:

Start and Stop Packet Capture:

In order to execute packet capture, it is necessary to enable it via an operational command:

In order to stop the packet capture, use the operational command:

View Packet Capture:

In order to see the capture, use “show security datapath-debug capture”. The example below is a capture done at the first ingress point (np-ingress). The recommendation is either to use capture at the ingress and/or egress point to confirm packets have correct header information before and after being processed by the system. Other capture points could be used when these specific points need to be investigated.

Configuration Example Recommendation from JTAC

In summary, below is the configuration recommendation from JTAC using the options shown above and that would provide good basic information for troubleshooting.

  • flow module is enabled
  • np-ingress and np-egress events are enabled with count action
  • packet-summary is enabled on egress side (np-egress), in combination with “record-pic-history”
  • packet-dump is configured in both ingress and egress sides, but deactivated so it can be used only when specific need exists.
  • two packet filters are configured, one for each traffic direction
  • preserve-trace-order is enabled to ensure correct order of the messages
Warning: When the troubleshooting is finished, make sure to remove or deactivate all the traceoptions configurations (not limited to flow traceoptions) and the complete ‘security datapath-debug’ configuration stanza. If any of the debugging configurations remains active, it will continue utilizing resources of the device (CPU/memory).

About the author

Prasanna

Leave a Comment