If an EX Series switch receives unwanted Network Time Protocol (NTP) requests, it will be too busy to serve them. The NTP and Software Forwarding Infrastructure Daemon (SFID) processes will use too many resources, which could result in protocol flaps. This article describes the technical background, the symptoms, and the proper filter rule for the EX Series (except the EX9200).
An EX Series switch receives unwanted NTP requests, which could result in protocol flaps.
The NTP server in UNIX and Linux supports the monlist request feature. With this feature, you can request the NTP client list that the NTP server is serving. Recent versions of the NTP server only support this request from localhost. However, older versions of the NTP server allow this request not only from localhost but also from the remote client.
Users can see what information is provided by the request using the following command:
(172.27.14.17 is the IP address of EX3200) [root@HK-CENTOS ~]# ntpdc -c monlist 172.27.14.17 remote address port local address count m ver code avgint lstint =============================================================================== 172.27.122.90 55387 0.0.0.0 1 7 2 0 0 0 172.27.14.23 123 0.0.0.0 5639 4 4 0 1024 203 172.27.122.41 52465 0.0.0.0 4 3 4 0 0 1634400
Hackers can use this feature to launch a UDP volume attack on the victim by:
- Sending a small number and volume of request packets
- Triggering a large number and volume of response packets toward the victim
This is an evolved form of DDoS attack, using an NTP feature without zombie.
Symptoms in EX Switches
1. The following log messages are observed:
Jan 5 22:37:11 bb2-sej4-sel xntpd: too many recvbufs allocated (40) Jan 5 22:57:15 bb2-sej4-sel xntpd: too many recvbufs allocated (40) Jan 5 23:02:11 bb2-sej4-sel xntpd: too many recvbufs allocated (40) Jan 5 23:07:07 bb2-sej4-sel xntpd: too many recvbufs allocated (40) Jan 5 23:12:10 bb2-sej4-sel xntpd: too many recvbufs allocated (40) Jan 5 23:42:06 bb2-sej4-sel xntpd: too many recvbufs allocated (40)
2. Under a DDoS attack, the CPU utilization of sfid and ntpd will be high:
root@XXXX> show system processes extensive last pid: 14101; load averages: 2.44, 2.38, 2.18 up 559+05:29:33 10:09:02 123 processes: 8 running, 94 sleeping, 21 waiting Mem: 301M Active, 48M Inact, 90M Wired, 93M Cache, 110M Buf, 449M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 1090 root 2 44 -52 70828K 17712K ucond 458.8H 46.04% sfid 1106 root 1 132 0 2832K 2680K RUN 43.3H 32.62% ntpd 1089 root 1 132 0 15452K 5576K RUN 1002.8 6.05% chassism
3. Even though the NTP configuration was removed, the CPU utilization of sfid is still high:
root@acc4-sej4-sel> show system processes extensive last pid: 14467; load averages: 1.95, 2.00, 2.06 up 559+05:49:04 10:28:33 119 processes: 4 running, 94 sleeping, 21 waiting Mem: 294M Active, 48M Inact, 90M Wired, 94M Cache, 110M Buf, 456M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 1090 root 2 44 -52 70828K 17712K select 458.9H 41.46% sfid
Differences Between EX and M/T/MX
1. The lo0 filter of EX does not cover the me0/vme interface. If the me0/vme is configured and used, the filter must be configured for me0/vme as well.
2. The lo0 filter does not affect communication between lo0 and lo0. However, the EX Series (except EX9200) works differently from the other platforms.
There was no problem when tested in EX8208 with 11.4R8.5 and 12.3R4.6, even without any accept rule for the IP address of the lo0 interface for the lo0 filter. The lo0 filter mechanism of the EX Series is different from that of the other platforms.
3. The EX Series does not support the port matching condition.
- EX CLI allows the port matching condition entered in the firewall filter as a hidden command but it does not work. This confuses operators.
- “Commit” does not show any warning or error, only the show command displays a warning such as the following:
root@EX8208_2_RE0# show firewall
family inet {
filter ntp_test {
term 1 {
from {
##
## Warning: statement ignored: unsupported platform (ex8208)
##
port ntp;
}
}
}
}
In many cases, however, switch operators prefer to use the command show | display set, which does not show the warning.
Recommended Filter Rule Set
Add these firewall filter terms to the lo0 interface filter and also to the me0/vme interface filter, if it is configured.
term 1 {
from {
source-address {
<trusted-addresses>;
}
protocol udp;
source-port ntp;
}
then accept;
}
term 2 {
from {
protocol udp;
destination-port ntp;
}
then {
discard;
}
}
term default {
then accept;
}
Future Enhancement for this Issue
From Junos OS:
11.4R12, 12.1R10, 12.2R8, 12.3R2-S9, 12.3R3-S10, 12.3R7,
13.1R4-S2, 13.1R5, 13.2R4, 13.3R2, 14.1R1, 14.2R1, 15.1R1,
12.1X44-D35, 12.1X45-D25, 12.1X46-D15, 12.1X46-D20, 12.1X47-D10,
13.1X49-D42, 13.1X50-D30, 13.2X50-D20, 13.2X51-D25, 13.2X52-D20
- NTPd will be upgraded to 4.2.7p26 or later
- monlist response will be disabled by NTPd
- CLI knob for restricting response to NTP control packets will be introduced
Thus the above firewall filter will not be necessary in the future, but for now, it is mandatory in case switches have a public IP address.