Config Router

  • Google Sheets
  • CCNA Online training
    • CCNA
  • CISCO Lab Guides
    • CCNA Security Lab Manual With Solutions
    • CCNP Route Lab Manual with Solutions
    • CCNP Switch Lab Manual with Solutions
  • Juniper
  • Linux
  • DevOps Tutorials
  • Python Array
You are here: Home / Juniper / High CPU occurs in all FPC when datapath debug is enabled

High CPU occurs in all FPC when datapath debug is enabled

June 4, 2016 by James Palmer

On an SRX1K/3K, despite CPU in both SPU and RE is low, High CPU in ‘show chassis fpc’ is observed when datapath debug is enabled.

On an SRX1K/3K device, low CPU is seen in both SPU and RE. However, when datapath debug is enabled, the CPU in ‘show chassis fpc’ is high.
In the following example, 100% CPU is seen in all FPC:
Note: Slot 7 and 8 is SPC

SRX3600.HK> show chassis fpc 
node0:
--------------------------------------------------------------------------
                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Online            33    100          0       1024        5         26
  1  Online            28    100          0       1024        5         26
  2  Online            28    100          0       1024        5         26
  3  Online            28    100          0       1024        5         26
  4  Online            35    100          0       1024        5         26
  5  Empty           
  6  Empty           
  7  Online            44    100          0       1024        5         26
  8  Online            42    100          0       1024        5         26
  9  Empty           
10  Empty           
11  Online            31    100          0       1024        5         26
12  Online            31    100          0       1024        5         26

RE and SPU CPU is very low:

SRX3600.HK> show chassis routing-engine 
node0:
--------------------------------------------------------------------------
Routing Engine status:
  Slot 0:
    Current state                  Master
    Election priority              Master (default)
    DRAM                      1023 MB
    Memory utilization          42 percent
    CPU utilization:
      User                       1 percent
      Background                 0 percent
      Kernel                    11 percent
      Interrupt                  2 percent
      Idle                      85 percent
    Model                          RE-PPC-1200-A
    Start time                     2015-07-02 00:41:53 ICT
    Uptime                         153 days, 9 hours, 54 minutes, 39 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.05       0.15       0.20

SRX3600.HK> show security monitoring performance spu 
node0:
--------------------------------------------------------------------------
fpc  7  pic  0
Last 60 seconds:
0:   1   1:   1   2:   1   3:   1   4:   1   5:   1
6:   1   7:   1   8:   1   9:   1  10:   1  11:   1
12:   1  13:   1  14:   1  15:   1  16:   1  17:   1
18:   1  19:   1  20:   1  21:   1  22:   1  23:   1
24:   1  25:   1  26:   1  27:   1  28:   1  29:   1
30:   1  31:   1  32:   1  33:   1  34:   1  35:   1
36:   1  37:   1  38:   1  39:   1  40:   1  41:   1
42:   1  43:   1  44:   1  45:   1  46:   1  47:   1
48:   1  49:   1  50:   1  51:   1  52:   1  53:   1
54:   1  55:   1  56:   1  57:   1  58:   1  59:   1
fpc  8  pic  0
Last 60 seconds:
0:   1   1:   1   2:   1   3:   1   4:   1   5:   1
6:   1   7:   1   8:   1   9:   1  10:   1  11:   1
12:   1  13:   1  14:   1  15:   1  16:   1  17:   1
18:   1  19:   1  20:   1  21:   1  22:   1  23:   1
24:   1  25:   1  26:   1  27:   1  28:   1  29:   1
30:   1  31:   1  32:   1  33:   1  34:   1  35:   1
36:   1  37:   1  38:   1  39:   1  40:   1  41:   1
42:   1  43:   1  44:   1  45:   1  46:   1  47:   1
48:   1  49:   1  50:   1  51:   1  52:   1  53:   1
54:   1  55:   1  56:   1  57:   1  58:   1  59:   1

Less than 6K concurrent sessions in SRX:

SRX3600.HK> show security monitoring performance session 
node0:
--------------------------------------------------------------------------
fpc  7  pic  0
Last 60 seconds:
0:    1846   1:    1841   2:    1850   3:    1848   4:    1853   5:    1850
6:    1857   7:    1853   8:    1855   9:    1852  10:    1849  11:    1847
12:    1851  13:    1847  14:    1853  15:    1842  16:    1838  17:    1831
18:    1836  19:    1834  20:    1843  21:    1839  22:    1844  23:    1840
24:    1843  25:    1839  26:    1839  27:    1834  28:    1834  29:    1829
30:    1831  31:    1827  32:    1833  33:    1823  34:    1826  35:    1823
36:    1823  37:    1820  38:    1833  39:    1830  40:    1829  41:    1828
42:    1848  43:    1843  44:    1843  45:    1841  46:    1837  47:    1833
48:    1840  49:    1835  50:    1826  51:    1824  52:    1827  53:    1823
54:    1830  55:    1825  56:    1825  57:    1822  58:    1832  59:    1829
fpc  8  pic  0
Last 60 seconds:
0:    3659   1:    3662   2:    3652   3:    3675   4:    3661   5:    3664
6:    3655   7:    3679   8:    3656   9:    3666  10:    3655  11:    3661
12:    3654  13:    3669  14:    3660  15:    3674  16:    3630  17:    3646
18:    3625  19:    3633  20:    3619  21:    3639  22:    3631  23:    3643
24:    3631  25:    3649  26:    3640  27:    3651  28:    3642  29:    3645
30:    3638  31:    3644  32:    3661  33:    3636  34:    3642  35:    3630
36:    3634  37:    3618  38:    3643  39:    3628  40:    3635  41:    3621
42:    3656  43:    3642  44:    3641  45:    3626  46:    3642  47:    3632
48:    3635  49:    3624  50:    3633  51:    3594  52:    3619  53:    3608
54:    3617  55:    3604  56:    3617  57:    3606  58:    3617  59:    3606

Based on the vty output of CPP0 below, high CPU in CPP is mostly due to the thread bcmCNTR.X, which is a period thread to scan the counters and state of the BCM Chip. Whenever datapath debug is enabled, CPU in CPP is used by this thread to collect counter for datapath debug output.

bcmCNTR.X use up CPU from “show thread” in node0.cpp0:

CPP0(SRX3600.HK vty)# show thread cpu

PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  2 L  running   Idle                   280/2048   0/5/1484635090 ms 36%
 61 M  asleep    bcmCNTR.4             1056/16392  0/70/431289560 ms 10%
 59 M  asleep    bcmCNTR.3             1056/16384  0/70/422013360 ms 10%
 57 M  asleep    bcmCNTR.2             1056/16392  0/70/416447860 ms 10%
 55 M  asleep    bcmCNTR.1             1056/16392  0/70/292533985 ms  7%
 53 M  asleep    bcmCNTR.0             1056/16384  0/70/283324765 ms  7%
 17 L  ready     E2E Packet Handler Thread  3552/8200   0/5/280746175 ms  6%
 30 M  asleep    PIC Periodic          2192/4104   0/45/194068860 ms  4%
  8 M  asleep    SFP                   1248/4104   0/230/124587125 ms  3%
 42 M  asleep    PFE Manager           4056/8192   0/45/48336635 ms  1%
 32 M  asleep    CPP CM                6632/16392  0/915/12181450 ms  0%
 46 M  asleep    USP Trace              792/16392  0/5/3637995 ms  0%
 27 H  asleep    RDP Input              968/2048   0/10/1911015 ms  0%

=== CPP0, node0.cpp0> show sched

Total system uptime 46+11:16:42, (4015002485 ms), 1742367292 thread dispatches
CPU load is 99% (1 second), 98% (5 second)
            99% (10 second), 100% (1 minute)
Total interrupt time 17007295 (ms)

 CPU       Name      Time(ms)
 36%       Idle      1484597085
 62%    Threads      2513398105
  0%        ISR      17007295
  0%    Level 1      15075
  0%    Level 2      1848
  0%    Level 5      83

 Top Thread:
   pid      = 61
   name     = bcmCNTR.4
   time     = 430941775 ms
   cpu      = 10%

Note that CPP is the control CPU monitoring status of SPC/IOC/NPC in SRX1K/3K. The command, ‘show chassis fpc’ displays CPU usage of CPP, whereas ‘show security monitoring performance spu’ displays the CPU usage of SPU which processes network traffic. Hence, high CPU in CPP (show chassis fpc) should not impact traffic.

It is expected for the CPU in CPP to be as high as 100%. Also, this thread’s priority is medium. When the system has the higher priority threads, it will give out the CPU time to other higher priority threads. Hence, despite CPU in CPP is 100%, it does not affect the whole system or network traffic (which is handled by SPU).

Note: In SRX5K, there is individual control CPU on each SPC/IOC card. Therefore, ‘show chassis fpc’ may display different CPU usage for different cards. But for SRX1K/SRX3K, there is only one control CPU in CPP for all SPC/NPC/IOC cards. The CPU usage should be from one single value and it should be same.

Related

Filed Under: Juniper Tagged With: datapath debug, FPC, High CPU, SRX

Recent Posts

  • How do I give user access to Jenkins?
  • What is docker volume command?
  • What is the date format in Unix?
  • What is the difference between ARG and ENV Docker?
  • What is rsync command Linux?
  • How to Add Music to Snapchat 2021 Android? | How to Search, Add, Share Songs on Snapchat Story?
  • How to Enable Snapchat Notifications for Android & iPhone? | Steps to Turn on Snapchat Bitmoji Notification
  • Easy Methods to Fix Snapchat Camera Not Working Black Screen Issue | Reasons & Troubleshooting Tips to Solve Snapchat Camera Problems
  • Detailed Procedure for How to Update Snapchat on iOS 14 for Free
  • What is Snapchat Spotlight Feature? How to Make a Spotlight on Snapchat?
  • Snapchat Hack Tutorial 2021: Can I hack a Snapchat Account without them knowing?

Copyright © 2023 · News Pro Theme on Genesis Framework · WordPress · Log in