This article describes the issue of being unable to assign BGP recursion (the use of BGP to solve the BGp next-hop) labels to the link-local address under the forwarding plane.
As the kernel does not support the insertion of labels in the link-local address on the PFE engine , the KRT queue will be stuck with the Operation not supported error message.
For example:
root@router-name> show krt queue
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop add/change: 1 queued
ADD indirect to 2001:e68:402e::39 index 0 (0x8f51c00) MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 1 queued
ADD gf 6 inst id 0 ::ffff:x.x.x.x/128 type 3 (403) error 'Operation not supported' Routing table delete queue: 0 queued
Also, a lot of messages, such as the following, are generated:
rpd[1304]: %DAEMON-3: Cannot perform nh operation ADDANDGET nhop fe80::280:4224:426:b8ed type unicast index 0 errno 45
Usually, to solve the BGP protocol next-hop, if the protocol next-hop was learnt from IGP in inet6.0 and MPLS inet6.3 (from LDP or setup a LSP), the LSP in inet6.3 will be selected and installed in the forwarding table.
For example:
root@router-name> show route 2001:e68:4005:8000::1 extensive
inet6.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
2001:e68:4005:8000::/49 (1 entry, 1 announced)
State: <Record Pending>
TSI:
KRT in-kernel 2001:e68:4005:8000::/49 -> {indirect(262149)}
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f9ba30
Next-hop reference count: 3
Source: <RR IP address>
Next hop type: Router, Next hop index: 975
Next hop: via so-2/2/0.0 weight 0x1, selected
Label-switched-path LSP:02.igw01.jrc--core01.klj
Label operation: Push 2, Push 299792(top)
Label TTL action: no-prop-ttl, no-prop-ttl(top)
Protocol next hop: ::ffff:x.x.x.x
Push 2
Indirect next hop: 8f51c00 262149
State: <Active Int Ext>
Local AS: 4788 Peer AS: 4788
Age: 13 Metric2: 241
Task: BGP_4788.<RR IP address>+54335
Announcement bits (3): 0-KRT 2-RT 6-Resolve tree 5
AS path: I (Originator) Cluster list: <RR IP address> <Originator IP address>
AS path: Originator ID: x.x.x.x
Accepted
Route Label: 2
Localpref: 100
Router ID: <RR IP address>
Indirect next hops: 1
Protocol next hop: ::ffff:x.x.x.x Metric: 241
Push 2
Indirect next hop: 8f51c00 262149
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: via so-2/2/0.0 weight 0x1
::ffff:x.x.x.x/128 Originating RIB: inet6.3
Metric: 241 Node path count: 1
Forwarding nexthops: 1
Nexthop: via so-2/2/0.0
The protocol next-hop should be solved to the 2001:e68:4005:8000::/49 prefix and lookup from both inet6.0 and inet6.3 in RIB as indirect next-hop; but the inet6.3 next-hop is installed in FIB.
root@router-name> show route forwarding-table destination 2001:e68:4005:8000::1 Routing table: default.inet6 Internet6: Destination Type RtRef Next hop Type Index NhRef Netif 2001:e68:4005:8000::/49 user 0 indr 262149 2 Push 2, Push 299792(top) 975 2 so-2/2/0.0
Check the protocol next-hop, which is learnt from IS-IS and LDP; as the route was not from BGP, inet6.3 will not be looked up:
root@router-name> show route ::ffff:x.x.x.x extensive
inet6.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
::ffff:x.x.x.x/128 (1 entry, 1 announced)
TSI:
KRT in-kernel ::ffff:x.x.x.x/128 -> {fe80::280:4224:426:b8ed}
*IS-IS Preference: 165 < If the route is learnt from IGP, the next-hop should be installed with the unicast type Level: 2
Next hop type: Router, Next hop index: 899
Address: 0x8f9b688
Next-hop reference count: 18
Next hop: fe80::280:4224:426:b8ed via so-2/2/0.0, selected
State: <Active Int Ext>
Local AS: 4788
Age: 8:03 Metric: 241 Tag: 10001010
Task: IS-IS
Announcement bits (3): 0-KRT 2-RT 6-Resolve tree 5
AS path: I
AS path: Recorded
inet6.3: 5 destinations, 6 routes (5 active, 0 holddown, 0 hidden)
::ffff:x.x.x.x/128 (1 entry, 1 announced)
State: <FlashAll>
*LDP Preference: 9
Next hop type: Router
Address: 0x90f6640
Next-hop reference count: 6
Next hop: via so-2/2/0.0 weight 0x1, selected < meanwhile the LSP was seleted and the label for the prefix is pushed; if shortcuts or FA was enabled, the next-hop can be used by inet6.0 Label-switched-path LSP:02.igw01.jrc--core01.klj
Label operation: Push 299792
Label TTL action: no-prop-ttl
State: <Secondary Active Int>
Local AS: 4788
Age: 47:34 Metric: 241
Task: LDP
Announcement bits (3): 1-RT 3-Resolve tree 2 4-Resolve tree 5
AS path: I
AS path: Recorded
Primary Routing Table inet.3
root@router-name> show route forwarding-table destination ::ffff:x.x.x.x
Routing table: default.inet6
Internet6:
Destination Type RtRef Next hop Type Index NhRef Netif
::ffff:x.x.x.x/128
user 0 fe80:: ucst 899 8 so-2/2/0.0
For the sake of comparison, the normal behavior was described above. Check for the issue as follows:
root@router-name> show route 2001:e68:4005:8000::1 extensive < This is the target BGP route inet6.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
2001:e68:4005:8000::/49 (1 entry, 1 announced)
State: <Record Pending>
TSI:
KRT in-kernel 2001:e68:4005:8000::/49 -> {indirect(262144)}
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f9b880
Next-hop reference count: 3
Source: <RR IP address>
Next hop type: Router, Next hop index: 981
Next hop: via so-2/2/0.0 weight 0x1, selected
Label-switched-path LSP:02.igw01.jrc--core01.klj
Label operation: Push 2, Push 299792(top)
Label TTL action: no-prop-ttl, no-prop-ttl(top)
Protocol next hop: ::ffff:x.x.x.x < The protocol next-hop was learnt from both inet6.0 and inet6.3 Push 2
Indirect next hop: 8f51f00 262144
State: <Active Int Ext>
Local AS: 4788 Peer AS: 4788
Age: 35 Metric2: 241
Task: BGP_4788.<RR IP address>+54335
Announcement bits (3): 0-KRT 2-RT 6-Resolve tree 5
AS path: I (Originator) Cluster list: <RR IP address> <Originator IP address>
AS path: Originator ID: x.x.x.x
AS path:
AS path: Recorded
Accepted
Route Label: 2
Localpref: 100
Router ID: <RR IP address>
Indirect next hops: 1
Protocol next hop: ::ffff:x.x.x.x Metric: 241
Push 2
Indirect next hop: 8f51f00 262144
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: via so-2/2/0.0 weight 0x1
::ffff:x.x.x.x/128 Originating RIB: inet6.3
Metric: 241 Node path count: 1
Forwarding nexthops: 1
Nexthop: via so-2/2/0.0
root@router-name> show route ::ffff:x.x.x.x extensive < the protocol next-hop inet6.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
::ffff:x.x.x.x/128 (1 entry, 1 announced)
State: <Record Pending>
TSI:
KRT queued (deferred) add
::ffff:x.x.x.x/128 -> {indirect(-)}
*BGP Preference: 170/-101
Next hop type: Indirect
Address: 0x8f9ba30
Next-hop reference count: 3
Source: <RR IP address>
Next hop type: Router
Next hop: fe80::280:4224:426:b8ed via so-2/2/0.0, selected < If the route is still learnt from BGP and LDP, it will keep the lookup protocol next-hop Label operation: Push 2
Label TTL action: no-prop-ttl
Protocol next hop: 2001:e68:402e::39
Push 2
Indirect next hop: 90e8000 -
State: <Active Int Ext>
Local AS: 4788 Peer AS: 4788
Age: 1:04 Metric2: 360
Task: BGP_4788.<RR IP address>+54335
Announcement bits (3): 0-KRT 2-RT 6-Resolve tree 5
AS path: I (Originator) Cluster list: <RR IP address> 58.27.127.26
AS path: Originator ID: x.x.x.x
AS path:
AS path: Recorded
Accepted
Route Label: 2
Localpref: 100
Router ID: <RR IP address>
Indirect next hops: 1
Protocol next hop: 2001:e68:402e::39 Metric: 360
Push 2
Indirect next hop: 90e8000 -
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: fe80::280:4224:426:b8ed via so-2/2/0.0
2001:e68:402e::38/127 Originating RIB: inet6.0
Metric: 360 Node path count: 1
Forwarding nexthops: 1
Nexthop: fe80::280:4224:426:b8ed via so-2/2/0.0
inet6.3: 5 destinations, 6 routes (5 active, 0 holddown, 0 hidden)
::ffff:x.x.x.x/128 (1 entry, 1 announced)
State: <FlashAll>
*LDP Preference: 9
Next hop type: Router
Address: 0x90f6640
Next-hop reference count: 6
Next hop: via so-2/2/0.0 weight 0x1, selected < Normally, the inet6.3 should be selected as high priority Label-switched-path LSP:02.igw01.jrc--core01.klj
Label operation: Push 299792
Label TTL action: no-prop-ttl
State: <Secondary Active Int>
Local AS: 4788
Age: 30:11 Metric: 241
Task: LDP
Announcement bits (3): 1-RT 3-Resolve tree 2 4-Resolve tree 5
AS path: I
AS path: Recorded
Primary Routing Table inet.3
root@router-name> show route forwarding-table destination 2001:e68:4005:8000::1 < For the subnet, it works by inet6.3 labeled Routing table: default.inet6
Internet6:
Destination Type RtRef Next hop Type Index NhRef Netif
2001:e68:4005:8000::/49
user 0 indr 262149 2
Push 2, Push 299792(top) 980 2 so-2/2/0.0
root@router-name> ping 2001:e68:4005:8000::1
PING6(56=40+8+8 bytes) fd7d:2c4f:bc9c:0:10:233:32:19 --> 2001:e68:4005:8000::1
16 bytes from 2001:e68:4005:8000::1, icmp_seq=0 hlim=62 time=1.381 ms
16 bytes from 2001:e68:4005:8000::1, icmp_seq=1 hlim=62 time=1.499 ms
16 bytes from 2001:e68:4005:8000::1, icmp_seq=2 hlim=62 time=1.586 ms
16 bytes from 2001:e68:4005:8000::1, icmp_seq=3 hlim=62 time=1.116 ms
^C
--- 2001:e68:4005:8000::1 ping6 statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 1.116/1.396/1.586/0.177 ms
The issue occurs, when the protocol next-hop is resolved twice, which is due to BGP recursion and a day one software issue:
root@router-name> show route forwarding-table destination ::ffff:x.x.x.x < But the protocol next-hop cannot be installed to PFE as it is stuck in krt Routing table: default.inet6
Internet6:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 rjct 44 1
Routing table: __master.anon__.inet6
Internet6:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 rjct 539 1
root@router-name> show krt queue
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop add/change: 1 queued
ADD indirect to 2001:e68:402e::39 index 0 (0x8f51c00) MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 1 queued
ADD gf 6 inst id 0 ::ffff:x.x.x.x/128 type 3 (58) error 'Operation not supported' Routing table delete queue: 0 queued
root@router-name> ping ::ffff:x.x.x.x
PING6(56=40+8+8 bytes) ::ffff:10.55.37.20 --> ::ffff:x.x.x.x
ping: sendmsg: No route to host
ping6: wrote ::ffff:x.x.x.x 16 chars, ret=-1
ping: sendmsg: No route to host
ping6: wrote ::ffff:x.x.x.x 16 chars, ret=-1
ping: sendmsg: No route to host
ping6: wrote ::ffff:x.x.x.x 16 chars, ret=-1
^C
--- ::ffff:x.x.x.x ping6 statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
liutao@m20-2-re0> show route 2001:e68:402e::39 extensive < As the next-hop was 'fe80::' which is a link-local address, it can not be labeled towards the point-to-point interface for IPv6 tunneling
inet6.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
2001:e68:402e::38/127 (1 entry, 1 announced)
TSI:
KRT in-kernel 2001:e68:402e::38/127 -> {fe80::280:4224:426:b8ed}
*IS-IS Preference: 18
Level: 2
Next hop type: Router, Next hop index: 899
Address: 0x8f9b688
Next-hop reference count: 15
Next hop: fe80::280:4224:426:b8ed via so-2/2/0.0, selected
State: <Active Int>
Local AS: 4788
Age: 3:26:31 Metric: 360
Task: IS-IS
Announcement bits (3): 0-KRT 2-RT 6-Resolve tree 5
AS path: I
AS path: Recorded
The following outputs can be examined to determine the cause of this message:
request support information | no-more show route show route x.x.x.x extensive show route forwarding-table destination x.x.x.x
As the issue is due the kernel not supporting the assignment of the label to the IPv6 link-local address, assigning the label to the prefix will resolve this issue. Perform the following procedure:
- To enable FA or shortcuts in the next-hop of the forwarding path towards the P2P interface, such as sonet , frame relay, and MLPPP, and so on, run the following command:
set protocols isis traffic-engineering family inet6 shortcuts
- Replacing the P2P interface with the ethernet interface will work.
- Avoid the BGP recursion towards one prefix, as only BGP can lookup the label table.