On an SRX1K/3K, despite CPU in both SPU and RE is low, High CPU in ‘show chassis fpc’ is observed when datapath debug is enabled.
On an SRX1K/3K device, low CPU is seen in both SPU and RE. However, when datapath debug is enabled, the CPU in ‘show chassis fpc’ is high.
In the following example, 100% CPU is seen in all FPC:
Note: Slot 7 and 8 is SPC
SRX3600.HK> show chassis fpc
node0:
--------------------------------------------------------------------------
Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online 33 100 0 1024 5 26
1 Online 28 100 0 1024 5 26
2 Online 28 100 0 1024 5 26
3 Online 28 100 0 1024 5 26
4 Online 35 100 0 1024 5 26
5 Empty
6 Empty
7 Online 44 100 0 1024 5 26
8 Online 42 100 0 1024 5 26
9 Empty
10 Empty
11 Online 31 100 0 1024 5 26
12 Online 31 100 0 1024 5 26
RE and SPU CPU is very low:
SRX3600.HK> show chassis routing-engine
node0:
--------------------------------------------------------------------------
Routing Engine status:
Slot 0:
Current state Master
Election priority Master (default)
DRAM 1023 MB
Memory utilization 42 percent
CPU utilization:
User 1 percent
Background 0 percent
Kernel 11 percent
Interrupt 2 percent
Idle 85 percent
Model RE-PPC-1200-A
Start time 2015-07-02 00:41:53 ICT
Uptime 153 days, 9 hours, 54 minutes, 39 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
0.05 0.15 0.20
SRX3600.HK> show security monitoring performance spu
node0:
--------------------------------------------------------------------------
fpc 7 pic 0
Last 60 seconds:
0: 1 1: 1 2: 1 3: 1 4: 1 5: 1
6: 1 7: 1 8: 1 9: 1 10: 1 11: 1
12: 1 13: 1 14: 1 15: 1 16: 1 17: 1
18: 1 19: 1 20: 1 21: 1 22: 1 23: 1
24: 1 25: 1 26: 1 27: 1 28: 1 29: 1
30: 1 31: 1 32: 1 33: 1 34: 1 35: 1
36: 1 37: 1 38: 1 39: 1 40: 1 41: 1
42: 1 43: 1 44: 1 45: 1 46: 1 47: 1
48: 1 49: 1 50: 1 51: 1 52: 1 53: 1
54: 1 55: 1 56: 1 57: 1 58: 1 59: 1
fpc 8 pic 0
Last 60 seconds:
0: 1 1: 1 2: 1 3: 1 4: 1 5: 1
6: 1 7: 1 8: 1 9: 1 10: 1 11: 1
12: 1 13: 1 14: 1 15: 1 16: 1 17: 1
18: 1 19: 1 20: 1 21: 1 22: 1 23: 1
24: 1 25: 1 26: 1 27: 1 28: 1 29: 1
30: 1 31: 1 32: 1 33: 1 34: 1 35: 1
36: 1 37: 1 38: 1 39: 1 40: 1 41: 1
42: 1 43: 1 44: 1 45: 1 46: 1 47: 1
48: 1 49: 1 50: 1 51: 1 52: 1 53: 1
54: 1 55: 1 56: 1 57: 1 58: 1 59: 1
Less than 6K concurrent sessions in SRX:
SRX3600.HK> show security monitoring performance session node0: -------------------------------------------------------------------------- fpc 7 pic 0 Last 60 seconds: 0: 1846 1: 1841 2: 1850 3: 1848 4: 1853 5: 1850 6: 1857 7: 1853 8: 1855 9: 1852 10: 1849 11: 1847 12: 1851 13: 1847 14: 1853 15: 1842 16: 1838 17: 1831 18: 1836 19: 1834 20: 1843 21: 1839 22: 1844 23: 1840 24: 1843 25: 1839 26: 1839 27: 1834 28: 1834 29: 1829 30: 1831 31: 1827 32: 1833 33: 1823 34: 1826 35: 1823 36: 1823 37: 1820 38: 1833 39: 1830 40: 1829 41: 1828 42: 1848 43: 1843 44: 1843 45: 1841 46: 1837 47: 1833 48: 1840 49: 1835 50: 1826 51: 1824 52: 1827 53: 1823 54: 1830 55: 1825 56: 1825 57: 1822 58: 1832 59: 1829 fpc 8 pic 0 Last 60 seconds: 0: 3659 1: 3662 2: 3652 3: 3675 4: 3661 5: 3664 6: 3655 7: 3679 8: 3656 9: 3666 10: 3655 11: 3661 12: 3654 13: 3669 14: 3660 15: 3674 16: 3630 17: 3646 18: 3625 19: 3633 20: 3619 21: 3639 22: 3631 23: 3643 24: 3631 25: 3649 26: 3640 27: 3651 28: 3642 29: 3645 30: 3638 31: 3644 32: 3661 33: 3636 34: 3642 35: 3630 36: 3634 37: 3618 38: 3643 39: 3628 40: 3635 41: 3621 42: 3656 43: 3642 44: 3641 45: 3626 46: 3642 47: 3632 48: 3635 49: 3624 50: 3633 51: 3594 52: 3619 53: 3608 54: 3617 55: 3604 56: 3617 57: 3606 58: 3617 59: 3606
Based on the vty output of CPP0 below, high CPU in CPP is mostly due to the thread bcmCNTR.X, which is a period thread to scan the counters and state of the BCM Chip. Whenever datapath debug is enabled, CPU in CPP is used by this thread to collect counter for datapath debug output.
bcmCNTR.X use up CPU from “show thread” in node0.cpp0:
CPP0(SRX3600.HK vty)# show thread cpu
PID PR State Name Stack Use Time (Last/Max/Total) cpu
--- -- ------- --------------------- --------- ---------------------
2 L running Idle 280/2048 0/5/1484635090 ms 36%
61 M asleep bcmCNTR.4 1056/16392 0/70/431289560 ms 10%
59 M asleep bcmCNTR.3 1056/16384 0/70/422013360 ms 10%
57 M asleep bcmCNTR.2 1056/16392 0/70/416447860 ms 10%
55 M asleep bcmCNTR.1 1056/16392 0/70/292533985 ms 7%
53 M asleep bcmCNTR.0 1056/16384 0/70/283324765 ms 7%
17 L ready E2E Packet Handler Thread 3552/8200 0/5/280746175 ms 6%
30 M asleep PIC Periodic 2192/4104 0/45/194068860 ms 4%
8 M asleep SFP 1248/4104 0/230/124587125 ms 3%
42 M asleep PFE Manager 4056/8192 0/45/48336635 ms 1%
32 M asleep CPP CM 6632/16392 0/915/12181450 ms 0%
46 M asleep USP Trace 792/16392 0/5/3637995 ms 0%
27 H asleep RDP Input 968/2048 0/10/1911015 ms 0%
=== CPP0, node0.cpp0> show sched
Total system uptime 46+11:16:42, (4015002485 ms), 1742367292 thread dispatches
CPU load is 99% (1 second), 98% (5 second)
99% (10 second), 100% (1 minute)
Total interrupt time 17007295 (ms)
CPU Name Time(ms)
36% Idle 1484597085
62% Threads 2513398105
0% ISR 17007295
0% Level 1 15075
0% Level 2 1848
0% Level 5 83
Top Thread:
pid = 61
name = bcmCNTR.4
time = 430941775 ms
cpu = 10%
Note that CPP is the control CPU monitoring status of SPC/IOC/NPC in SRX1K/3K. The command, ‘show chassis fpc’ displays CPU usage of CPP, whereas ‘show security monitoring performance spu’ displays the CPU usage of SPU which processes network traffic. Hence, high CPU in CPP (show chassis fpc) should not impact traffic.
It is expected for the CPU in CPP to be as high as 100%. Also, this thread’s priority is medium. When the system has the higher priority threads, it will give out the CPU time to other higher priority threads. Hence, despite CPU in CPP is 100%, it does not affect the whole system or network traffic (which is handled by SPU).
Note: In SRX5K, there is individual control CPU on each SPC/IOC card. Therefore, ‘show chassis fpc’ may display different CPU usage for different cards. But for SRX1K/SRX3K, there is only one control CPU in CPP for all SPC/NPC/IOC cards. The CPU usage should be from one single value and it should be same.