This article outlines the problem of BGP route selection broken in certain code version and how could one find that they are affected.
In 9.3 and 9.4, due to an uninitialized variable being used to select the best route among all routes for the same prefix, random route selection may happen.
This may lead a bgp running router to select the route with lowest local preference.
Following output is one symptom that would arise
[email protected]> show route 8.6.19.0/24 detail | no-more inet.0: 318921 destinations, 3155019 routes (318113 active, 10 holddown, 1880 hidden) Restart Complete 8.6.19.0/24 (17 entries, 1 announced) *BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 249450 Source: 64.125.0.210 Next hop type: Router, Next hop index: 1048668 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-mpr4.sjc7.us Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.26.126 Protocol next hop: 64.125.0.210 Indirect next hop: 124f25e8 1048650 State: <Active Int Ext> Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 3 Task: BGP_6461.64.125.0.210+61872 Announcement bits (5): 0-KRT 2-RT 6-Aggregate 9-BGP RT Background 10-Resolve tree 4 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2501 6461:2601 6461:2722 6461:2863 Accepted Localpref: 100 Router ID: 64.125.0.210 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 177766 Source: 64.125.0.233 Next hop type: Router, Next hop index: 1048773 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-mpr1.lax12.us Label operation: Push 379178 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.24.17 Label operation: Push 483253, Push 685360(top) Protocol next hop: 64.125.0.233 Indirect next hop: 11a2e348 1048644 State: <NotBest Int Ext> Inactive reason: Not Best in its group - IGP metric Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 522 Task: BGP_6461.64.125.0.233+179 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2501 6461:2601 6461:2713 6461:2869 Accepted Localpref: 100 Router ID: 64.125.0.233 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 96204 Source: 64.125.0.208 Next hop type: Router, Next hop index: 1048665 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er2.ord7.us Label operation: Push 370746 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.28.18 Label operation: Push 630338, Push 685312(top) Protocol next hop: 64.125.0.208 Indirect next hop: 17d6ad20 1048726 State: <NotBest Int Ext> Inactive reason: Not Best in its group - IGP metric Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 2278 Task: BGP_6461.64.125.0.208+50988 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2502 6461:2601 6461:2718 6461:2860 Accepted Localpref: 100 Router ID: 64.125.0.208 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 121366 Source: 64.125.0.202 Next hop type: Router, Next hop index: 1048612 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er2.dfw2.us Label operation: Push 334122 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.24.17 Label operation: Push 471285, Push 685360(top) Protocol next hop: 64.125.0.202 Indirect next hop: 169a3f18 1048730 State: <NotBest Int Ext> Inactive reason: Not Best in its group - IGP metric Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 2325 Task: BGP_6461.64.125.0.202+59686 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2501 6461:2601 6461:2707 6461:2814 Accepted Localpref: 100 Router ID: 64.125.0.202 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 79123 Source: 64.125.0.220 Next hop type: Router, Next hop index: 1048784 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-mpr4.atl6.us Label operation: Push 379050 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.24.17 Label operation: Push 483237, Push 685360(top) Protocol next hop: 64.125.0.220 Indirect next hop: 140429d8 1048685 State: <NotBest Int Ext> Inactive reason: Not Best in its group - IGP metric Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 3031 Task: BGP_6461.64.125.0.220+63498 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2503 6461:2601 6461:2702 6461:2857 Accepted Localpref: 100 Router ID: 64.125.0.220 BGP Preference: 170/-201 Next hop type: Indirect Next-hop reference count: 2 Source: 64.124.201.186 Next hop type: Router, Next hop index: 936 Next hop: 64.124.201.186 via ge-11/0/0.0, selected Protocol next hop: 64.124.201.186 Indirect next hop: b9acb28 1048880 State: <Ext> Inactive reason: Local Preference Local AS: 6461 Peer AS: 13535 Age: 3d 15:15:41 Metric2: 0 Task: BGP_13535.64.124.201.186+61095 AS path: 13535 I AS path: Recorded Communities: 6461:1001 6461:2001 6461:2101 6461:2222 6461:2463 Accepted Localpref: 200 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Route is inactive Router ID: 208.82.17.254 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 121357 Source: 64.125.0.194 Next hop type: Router, Next hop index: 1048599 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er2.lga5.us Label operation: Push 788489 Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.25.5 Label operation: Push 390441, Push 685408(top) Protocol next hop: 64.125.0.194 Indirect next hop: 12841d20 1048766 State: <NotBest Int Ext> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inactive status Inactive reason: Local Preference Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 3179 Task: BGP_6461.64.125.0.194+179 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2503 6461:2601 6461:2714 6461:2829 Accepted Localpref: 100 Router ID: 64.125.0.194 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 178738 Source: 64.125.0.205 Next hop type: Router, Next hop index: 1048871 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er1.iad10.us Label operation: Push 376778 Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.25.5 Label operation: Push 594457, Push 685408(top) Protocol next hop: 64.125.0.205 Indirect next hop: 14ffc0a8 1048687 State: <NotBest Int Ext> Inactive reason: Local Preference Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 3618 Task: BGP_6461.64.125.0.205+179 AS path: 7018 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2503 6461:2601 6461:2711 6461:2858 Accepted Localpref: 100 Router ID: 64.125.0.205 BGP Preference: 170/-101 Next hop type: Router, Next hop index: 772 Next-hop reference count: 261436 Source: 64.125.13.242 Next hop: 64.125.13.242 via xe-1/3/0.0, selected State: <Ext> Inactive reason: Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 3356 Age: 8w4d 13:17:16 Metric: 4294967294 Task: BGP_3356.64.125.13.242+53449 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1666 6461:2501 6461:2601 6461:2722 6461:2863 Accepted Localpref: 100 Router ID: 4.68.185.244 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 164080 Source: 64.125.0.201 Next hop type: Router, Next hop index: 1048756 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er1.dfw2.us Label operation: Push 343946 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.24.17 Label operation: Push 473973, Push 685360(top) Protocol next hop: 64.125.0.201 Indirect next hop: 14b42b28 1048758 State: <NotBest Int Ext> Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 2324 Task: BGP_6461.64.125.0.201+61936 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2501 6461:2601 6461:2707 6461:2814 Accepted Localpref: 100 Router ID: 64.125.0.201 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 165035 Source: 64.125.0.251 Next hop type: Router, Next hop index: 1048663 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-mpr1.lhr1.uk Label operation: Push 380010 Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.25.5 Label operation: Push 600681, Push 685408(top) Protocol next hop: 64.125.0.251 Indirect next hop: 12f25150 1048643 State: <NotBest Int Ext> Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:40 Metric: 4294967294 Metric2: 7313 Task: BGP_6461.64.125.0.251+179 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2504 6461:2607 6461:2715 6461:2833 Accepted Localpref: 100 Router ID: 64.125.0.251 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 164858 Source: 64.125.0.187 Next hop type: Router, Next hop index: 1048852 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er1.ams1.nl Label operation: Push 378650 Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.25.5 Label operation: Push 598009, Push 685408(top) Protocol next hop: 64.125.0.187 Indirect next hop: 234f9d20 1048738 State: <NotBest Int Ext> Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 7347 Task: BGP_6461.64.125.0.187+179 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2505 6461:2602 6461:2700 6461:2800 Accepted Localpref: 100 Router ID: 64.125.0.187 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 152440 Source: 64.125.0.9 Next hop type: Router, Next hop index: 1048669 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-mpr1.cdg11.fr Label operation: Push 379946 Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.25.5 Label operation: Push 600521, Push 685408(top) Protocol next hop: 64.125.0.9 Indirect next hop: 14951d20 1048676 State: <NotBest Int Ext> Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:40 Metric: 4294967294 Metric2: 8285 Task: BGP_6461.64.125.0.9+51227 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2504 6461:2603 6461:2704 6461:2873 Accepted Localpref: 100 Router ID: 64.125.0.9 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 170419 Source: 64.125.0.8 Next hop type: Router, Next hop index: 1048579 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-mpr1.fra4.de Label operation: Push 377098 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.28.18 Label operation: Push 633410, Push 685312(top) Protocol next hop: 64.125.0.8 Indirect next hop: 14b42150 1048678 State: <NotBest Int Ext> Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:40 Metric: 4294967294 Metric2: 8329 Task: BGP_6461.64.125.0.8+62578 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2504 6461:2606 6461:2710 6461:2872 Accepted Localpref: 100 Router ID: 64.125.0.8 BGP Preference: 170/-101 Next hop type: Router, Next hop index: 590 Next-hop reference count: 59506 Source: 64.125.12.50 Next hop: 64.125.12.50 via ge-11/0/9.0, selected State: <Ext> Inactive reason: Interior > Exterior > Exterior via Interior Local AS: 6461 Peer AS: 3561 Age: 1w5d 17:31:05 Metric: 4294967294 Task: BGP_3561.64.125.12.50+179 AS path: 3561 13535 I AS path: Recorded Communities: 6461:1666 6461:2501 6461:2601 6461:2722 6461:2863 Accepted Localpref: 100 Router ID: 206.24.210.98 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 193527 Source: 64.125.0.207 Next hop type: Router, Next hop index: 1048692 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er1.ord7.us Label operation: Push 377738 Next hop: 64.125.26.126 via xe-1/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.28.18 Label operation: Push 633666, Push 685312(top) Protocol next hop: 64.125.0.207 Indirect next hop: 14ffc1f8 1048743 State: <Int Ext> Inactive reason: IGP metric Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 2277 Task: BGP_6461.64.125.0.207+50560 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2502 6461:2601 6461:2718 6461:2860 Accepted Localpref: 100 Router ID: 64.125.0.207 BGP Preference: 170/-101 Next hop type: Indirect Next-hop reference count: 153332 Source: 64.125.0.206 Next hop type: Router, Next hop index: 1048788 Next hop: 64.125.27.90 via xe-1/1/0.0 weight 0x1, selected Label-switched-path mpr3.sjc7.us-to-er2.iad10.us Label operation: Push 373050 Next hop: 64.125.27.86 via xe-0/0/0.0 weight 0x8001 Label-switched-path Bypass->64.125.27.90->64.125.25.5 Label operation: Push 585801, Push 685408(top) Protocol next hop: 64.125.0.206 Indirect next hop: 17a72b28 1048746 State: <NotBest Int Ext> Inactive reason: Not Best in its group - IGP metric Local AS: 6461 Peer AS: 6461 Age: 3d 15:15:39 Metric: 4294967294 Metric2: 3619 Task: BGP_6461.64.125.0.206+179 AS path: 3356 13535 I AS path: Recorded Communities: 6461:1114 6461:1666 6461:2503 6461:2601 6461:2711 6461:2858 Accepted Localpref: 100 Router ID: 64.125.0.206