Troubleshooting BGP Neighbor Relationships
This section discusses the most common issues in forming a neighbor relationship between two BGP-speaking routers. BGP speakers exchange routing information only after they successfully become neighbors with each other. Troubleshooting neighbor relationship issues should follow the OSI reference model. First, you should check Layer 2 connectivity; then check IP connectivity (Layer 3), TCP connections (Layer 4), and finally the BGP configuration in Cisco IOS Software.
The section is arranged to discuss external BGP neighbors’ issues, internal BGP neighbors, and then problems that are common in both external and internal BGP neighbor relationships.
The following is the list of problems most commonly seen when forming BGP neighbor relationships.
- Directly connected external BGP neighbors not initializing
- Nondirectly connected external BGP neighbors not initializing
- Internal BGP neighbors not initializing
- BGP neighbors (external and internal) not initializing
Problem: Directly Connected External BGP Neighbors Not Initializing
This section discusses issues when a directly connected EBGP neighbor relationship is unsuccessful. The autonomous system (AS) will not send or receive any IP prefix updates to or from a neighboring AS unless the neighbor relationship reaches the Established state, which is the final stage of BGP neighbor establishment, as described in Chapter 14, “Understanding Border Gateway Protocol Version 4 (BGP-4).” When an AS has a single EBGP connection, no IP connectivity can occur until BGP finalizes its operation of sending and receiving IP prefixes.
Figure 15-1 shows a network in which an external BGP neighbor relationship is configured between AS 109 and AS 110.
The most common possible causes of this problem are as follows:
- Layer 2 is down, preventing communication with a directly connected EBGP neighbor.
- An incorrect neighbor IP address is in the BGP configuration.
Directly Connected External BGP Neighbors Not Coming Up—Cause: Layer 2 Is Down, Preventing Communication with Directly Connected BGP Neighbor
IP connectivity cannot occur until Layer 2 in the OSI reference model is up. Whether this Layer 2 information is learned dynamically or is configured statically, each router must have a correct Layer 2 rewrite information of adjacent routers. Ethernet, Frame Relay, ATM, and so on are most commonly used Layer 2 technologies. Most network administrators configure Layer 2 parameters in router configurations correctly; sometimes, basic cabling issues also can cause Layer 2 issues. Among cabling issues, misconfiguration in router configuration can cause ARP, DLCI mapping, and VPI/VIC encapsulation failures, which are the most common Layer 2 failures. It is beyond the scope of this book to address how this Layer 2 information is obtained. Case(s) in this section try to address how to troubleshoot BGP problems when the cause of the EBGP neighbor relationship failure is Layer 2.
Figure 15-2 shows the flowchart to follow to fix this problem.
Debugs and Verification
Example 15-1 shows the relevant configuration of R1 and R2, respectively.
Example 15-1 R1 and R2 Configuration
R1#router bgp 109 neighbor 131.108.1.2 remote-as 110 interface Ethernet0 ip address 131.108.1.1 255.255.255.0 _____________________________________________________________________________________ R2#router bgp 110 neighbor 131.108.1.1 remote-as 109 interface Ethernet0 ip address 131.108.1.2 255.255.255.0
You can verify the BGP neighbor relationship on Cisco IOS Software by using the commands in Example 15-2.
Example 15-2 Verifying BGP Neighbor Relationship
R1#show ip bgp summary BGP router identifier 206.56.89.6, local AS number 109 BGP table version is 1, main routing table version 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 131.108.1.2 4 110 3 7 0 0 0 00:03:14 Activ R1#show ip bgp neighbors 131.108.1.2 BGP neighbor is 131.108.1.2, remote AS 110, external link BGP version 4, remote router ID 0.0.0.0 BGP state = Active Last read 00:04:23, hold time is 180, keepalive interval is 60 seconds Received 3 messages, 0 notifications, 0 in queue Sent 7 messages, 1 notifications, 0 in queue Route refresh request: received 0, sent 0 Minimum time between advertisement runs is 30 second For address family: IPv4 Unicast BGP table version 1, neighbor version 0 Index 1, Offset 0, Mask 0x2 0 accepted prefixes consume 0 bytes Prefix advertised 0, suppressed 0, withdrawn 0 Connections established 1; dropped 1 Last reset 00:04:44, due to BGP Notification sent, hold time expired No active TCP connection
The output in Example 15-2 shows that the BGP neighbor is in the Active state. This state indicates that no successful communication between the neighbors has taken place and that BGP has failed to form neighbor relationship.
You can use ping to verify IP connectivity between R1 and R2. Example 15-3 shows that R1 cannot ping R2’s IP address.
Example 15-3 R1 Ping of R2’s IP Address Fails
R1#ping 131.108.1.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 131.108.1.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5)
Solution
In this example, the ping failure from R1 to R2 was the result of Layer 2 on R2 being down. Example 15-4 shows the output indicating a Layer 2 problem.
Example 15-4 show interface Command Output Pinpoints That This Is a Layer 2 Problem
R2# show interface ethernet 0 Ethernet0 is down, line protocol is down
This might be because of cable issues or a hardware problem.
Apart from the interface being down, as in Example 15-4, Layer 2 encapsulation failure can also cause IP connectivity to break. Layer 2 encapsulation failure can occur because of corruption in the ARP table in case of Ethernet or an incorrect DLCI–VPI/VCI mapping in cases of Frame Relay and ATM, respectively. Fixing these should enable basic IP connectivity, and the BGP neighbor relationship should initialize.
Directly Connected External BGP Neighbors Not Coming Up—Cause: Incorrect Neighbor IP Address in BGP Configuration
Figure 15-3 shows the flowchart to follow to fix this problem.
Debugs and Verification
Example 15-5 shows the relevant configuration of R1 and R2, respectively.
Example 15-5 R1 and R2 Configuration
R1# router bgp 109 neighbor 131.108.1.2 remote-as 110 interface Ethernet0 ip address 131.108.1.1 255.255.255.0 _____________________________________________________________________________________ R2# router bgp 110 neighbor 131.108.1.11 remote-as 109 interface Ethernet0 ip address 131.108.1.2 255.255.255.0
Misconfiguration of the neighbor address is a fairly common mistake, and it can be caught with visual inspection of the configuration. However, in a large IP network, this might not be a trivial task. Example 15-6 shows how to capture this mistake using debugs in Cisco IOS Software.
Example 15-6 debug ip bgp Command Output Helps Pinpoint Incorrectly Configured Neighbor Addresses
R2#debug ip bgp BGP debugging is on R2# Nov 28 13:25:12: BGP: 131.108.1.11 open active, local address 131.108.1.2 Nov 28 13:25:42: BGP: 131.108.1.11 open failed: Connection timed out; remote host not responding
The output in Example 15-6 clearly points out that R2 is having difficulty communicating with host 131.108.1.11.
Solution
The correct neighbor address should be configured when establishing BGP neighbor relationship. Therefore, R2’s BGP configuration must look like Example 15-7.
Example 15-7 Correcting R2’s BGP Configuration
R2# router bgp 110 neighbor 131.108.1.1 remote-as 109
A similar problem can occur when the incorrect AS number is configured.
Problem: Nondirectly Connected External BGP Neighbors Not Coming Up
As discussed in Chapter 14, in some cases, EBGP neighbors are not directly connected. BGP neighbor relationships can be established in the following situations as well:
- Between loopback interfaces of two routers.
- Between routers trying to make EBGP neighbor relationship that are separated by one or more routers. Such a neighbor relationship is termed EBGP multihop in Cisco IOS Software.
EBGP multihop can be used for several reasons. Peering between loopbacks between EBGP typically is done when multiple interfaces exist between the routers, and IP traffic needs to be load-shared among those interfaces. Another scenario might be one in which an edge router cannot run BGP and, therefore, EBGP must be run between a nonedge device in one AS and an edge router in another.
A neighbor relationship must be established before any BGP updates and IP traffic can flow from one AS to another. This section addresses most of the common causes in which nondirectly connected EBGP neighbor relationships won’t establish.
Figure 15-4 shows that AS 109 and AS 110 are forming an EBGP neighbor relationship between the loopback interfaces. Such a connection will be considered nondirectly connected.
The most common possible causes of this problem are as follows:
- The route to the nondirectly connected peer address is missing from the routing table.
- The ebgp-multihop command is missing in BGP configuration.
- The update-source interface command is missing.
Nondirectly Connected External BGP Neighbors Not Coming Up—Cause: Route to the Nondirectly Connected Peer Address Is Missing from the Routing Table
Figure 15-5 shows the flowchart to follow to fix this problem.
When BGP tries to peer the neighbor relationship with IP addresses that are not directly connected, as shown in Figure 15-4, the IP routing table must have the route to that IP address.
In Figure 15-4, R1 is configured to peer with Loopback 0 of R2, and R2 is configured to peer with Loopback 0 of R1. This is a common practice when multiple connections exist between R1 and R2 and load sharing over these multiple lines is required.
Debugs and Verification
Example 15-8 shows the relative configuration of both Routers R1 and R2.
Example 15-8 Configurations for R1 and R2 in Figure 15-4
R1# router bgp 109 neighbor 131.108.10.2 remote-as 110 neighbor 131.108.10.2 ebgp-multihop 2 neighbor 131.108.10.2 update-source Loopback0 _____________________________________________________________________________________ R2# router bgp 110 neighbor 131.108.10.1 remote-as 109 neighbor 131.108.10.1 ebgp-multihop 2 neighbor 131.108.10.1 update-source Loopback0
In Example 15-8, Routers R1 and R2 are configured to peer with each other’s loopback IP address. ebgp-multihop 2 means that R1 and R2 peering addresses could be two hops away. update-source means that the source of BGP packets is the Loopback the 0 IP address because both routers accept only BGP packets sourced with the Loopback 0 IP address of other router.
The output in Example 15-9 shows the neighbor relationship between R1 and R2.
Example 15-9 show ip bgp Command Output Displays the BGP Neighbor Relationship Between R1 and R2
R1#show ip bgp summary BGP router identifier 131.108.10.1, local AS number 109 BGP table version is 1, main routing table version 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 131.108.10.2 4 110 3 3 0 0 0 00:03:21 Active R1#show ip bgp neighbors 131.108.10.2 BGP neighbor is 131.108.10.2, remote AS 110, external link BGP version 4, remote router ID 0.0.0.0 BGP state = Active Last read 00:04:20, hold time is 180, keepalive interval is 60 seconds Received 3 messages, 0 notifications, 0 in queue Sent 3 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Minimum time between advertisement runs is 30 seconds For address family: IPv4 Unicast BGP table version 1, neighbor version 0 Index 2, Offset 0, Mask 0x4 0 accepted prefixes consume 0 bytes Prefix advertised 0, suppressed 0, withdrawn 0 Connections established 1; dropped 1 Last reset 00:04:21, due to User reset External BGP neighbor may be up to 2 hops away. No active TCP connection
The highlighted output in Example 15-9 shows that R1’s neighbor relationship with R2 is in the Active state and that the BGP relationship is not complete.
The configuration shown in Example 15-8 is a required configuration when configuring an EBGP connection to a nondirectly connected peer; however, one thing that is not controlled by BGP configuration is the reachability to peer addresses.
Example 15-10 shows that R1 cannot reach the loopback interface of R2 because R1 does not have the route to reach R2.
Example 15-10 R1 Cannot Ping R2’s Loopback Interface and Has No Route to Reach R2
R1#ping 131.108.10.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 131.108.10.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) R1#show ip route 131.108.10.2 % Subnet not in table
In R1, BGP will send its packets to peer 131.108.10.2, but packets will be dropped by R1 because no routing reachability is available for 131.108.10.2.
Solution
BGP relies on an IP routing table to reach a peer address. In the example in Figure 15-4, R1 must have a route to the loopback interface of R2, and R2 must have a route to the loopback interface of R1. It is irrelevant how the route to the peer address is learned, as long as the route is present in the routing table. Administrators of R1 and R2 can choose to run a dynamic IP routing (an IGP) between each other (for example, using OSPF), or they could nail a static route to each other. Using a static route is a common practice. A simple rule of thumb is that R1 and R2 must have most specific routes for each other’s loopback addresses through any other protocol other than BGP.
To configure a static route on R1 to reach multihop EBGP neighbors, you would enter the following:
ip route 131.108.10.2 255.255.255.255 131.108.1.2
R1 has a host static route for R2’s loopback interface with a next hop of 131.108.1.2, which is R2’s Ethernet interface IP address. Similarly, R2 should have a static route for R1’s loopback interface. This will ensure that both routers have reachability to multihop EBGP neighbors.
Nondirectly Connected External BGP Neighbors Not Coming Up—Cause: ebgp-multihop Command Is Missing in BGP Configuration
Figure 15-6 shows the flowchart to follow to fix this problem.
By default, in Cisco IOS Software, BGP packets sent to an external BGP neighbor have their IP Time To Live (TTL) set to 1. If an EBGP neighbor is not directly connected, the first device in the path will drop BGP packets with TTL equal to 1 to that EBGP neighbor.
Debugs and Verification
Returning to the network in Figure 15-4, R1 is trying to peer EBGP to R2’s Loopback 0 interface, which is considered more than one hop away. Example 15-11 shows the configuration of R1.
Example 15-11 R1’s Configuration to Form EBGP Multihop Neighbor Relationship
R1# router bgp 109 neighbor 131.108.10.2 remote-as 110 neighbor 131.108.10.2 update-source Loopback0
Example 15-12 shows the output to verify that nondirectly connected EBGP neighbors are either coming up or not coming up.
Example 15-12 show ip bgp Command Output Verifies if the EBGP Neighbors Are Coming Up
R1#show ip bgp summary BGP router identifier 131.108.10.1, local AS number 109 BGP table version is 1, main routing table version 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 131.108.10.2 4 110 25 25 0 0 0 00:00:51 Idle R1#show ip bgp neighbors 131.108.10.2 BGP neighbor is 131.108.10.2, remote AS 110, external link BGP version 4, remote router ID 0.0.0.0 BGP state = Idle Last read 00:00:15, hold time is 180, keepalive interval is 60 seconds Received 25 messages, 0 notifications, 0 in queue Sent 25 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Minimum time between advertisement runs is 30 seconds For address family: IPv4 Unicast BGP table version 1, neighbor version 0 Index 2, Offset 0, Mask 0x4 0 accepted prefixes consume 0 bytes Prefix advertised 0, suppressed 0, withdrawn 0 Connections established 4; dropped 4 Last reset 00:02:18, due to User reset External BGP neighbor not directly connected. No active TCP connection
The highlighted output shows that BGP neighbor is in the Idle state, in which no resources are allocated to BGP neighbor. This might be because the other side has not received any BGP negotiation from R1 or because R1 has not received anything from R2.
Solution
Use the ebgp-multihop command to increase the IP TTL value to the desired number. Example 15-13 shows the required configuration on R1 to bring up the EBGP neighbor R2.
Example 15-13 Using ebgp-multihop to Increase the IP TTL Value
R1# router bgp 109 neighbor 131.108.10.2 remote-as 110 neighbor 131.108.10.2 ebgp-multihop 2 neighbor 131.108.10.2 update-source Loopback0
The ebgp-multihop 2 command sets the IP TTL value to 2 rather than the default of 1. This way, if a BGP speaker is two hops away, the BGP packet will reach it; otherwise, it would have been dropped by the intermediate device because of the IP TTL expiration.
Example 15-15 shows the debug ip packet output and sniffer capture in R2 of BGP packets from R1 to R2. ebgp-multihop is set to 5, as shown in the configuration of Example 15-14.
Example 15-14 Setting ebgp-multihop to 5
R1# router bgp 109 neighbor 131.108.10.2 remote-as 110 neighbor 131.108.10.2 ebgp-multihop 5 neighbor 131.108.10.2 update-source Loopback0
Example 15-15 debug ip packet and Sniffer Capture in R2 of BGP Packets from R1 to R2
IP: s=131.108.10.1 (Ethernet0), d=131.108.10.2, len 59, rcvd 4 TCP src=179, dst=13589, seq=1287164041, ack=1254239588, win =16305 ACK 04009210: 0000 0C47B947 00000C09 ...G9G.... 04009220: 9FEA0800 45C00028 00060000 04 069B2F .j..E@.(......./ 04009230: 836C0A01 836C0A02 00B33515 4CB89089 .l...l...35.L8.. 04009240: 4AC22D64 50103FB1 CA170000 00000000 JB-dP.?1J....... 04009250: 0000C8 ..H
The debug shows that R2 is receiving a BGP packet on TCP port 179 from source 131.108.10.1 (R1). In the sniffer capture, the highlighted hex value 04 is the IP TTL value of 4. The IP TTL value is 4 because R2 decremented the TTL by 1. This example of capturing packets through sniffer is shown to illustrate the use of the ebgp-multihop command in BGP to increase the IP TTL of a BGP packet.
Nondirectly Connected External BGP Neighbors Not Coming Up—Cause: update-source interface Command Is Missing
By default in Cisco IOS Software, the source of the BGP packet is the outgoing interface IP address as taken from the routing table.
In BGP, the neighbor’s IP address must be statically defined in configuration. If an EBGP speaker does not receive a BGP update from a IP source that is identical to what it has configured, it rejects that update. The update-source command in BGP changes the source address of the IP packet. Instead of picking the outgoing interface as a source IP address, BGP packets will be sourced with the interface IP address configured with the update-source command.
Figure 15-7 shows the flowchart to follow to fix this problem.
Debugs and Verification
Example 15-16 displays output from R1 to shows R1’s IP routing table entry for R2’s loopback 131.108.10.2 and R1’s outgoing interface address to reach R2’s loopback interface.
Example 15-16 R1 IP Routing Table Entry for R2’s Loopback Interface
R1#show ip route 131.108.10.2 Routing entry for 131.108.10.2/32 Known via "static", distance 1, metric 0 Routing Descriptor Blocks: * 131.108.1.2 Route metric is 0, traffic share count is 1 R1#show ip route 131.108.1.2 Routing entry for 131.108.1.0/24 Known via "connected", distance 0, metric 0 (connected, via interface) Routing Descriptor Blocks: * directly connected, via Ethernet0 Route metric is 0, traffic share count is 1 R1#show interfaces ethernet 0 Ethernet0 is up, line protocol is up Hardware is Lance, address is 0000.0c09.9fea (bia 0000.0c09.9fea) Internet address is 131.108.1.1/24
All IP packets destined for 131.108.10.2 have a source address of 131.108.1.1, which is the outgoing interface address of R1, as shown in Example 15-16.
Example 15-17 shows a simple BGP configuration of R1 and R2 (as depicted in the network in Figure 15-4), peering with each other’s loopback. The configuration in Example 15-17 does not result in an EBGP neighbor relationship because the IP source address of BGP packets won’t be what R2 is configured to expect. The working configuration is shown in the “Solution” section.
Example 15-17 R1/R2 BGP Configuration with Peering with Loopback Interfaces
R1# router bgp 109 neighbor 131.108.10.2 remote-as 110 neighbor 131.108.10.2 ebgp-multihop 2 _____________________________________________________________________________________ R2# router bgp 110 neighbor 131.108.10.1 remote-as 109 neighbor 131.108.10.1 ebgp-multihop 2
The problem comes in when R1 sends its BGP packets to its configured neighbor 131.108.10.2. The source IP address of those BGP packets will be 131.108.1.1, the out-going interface address. R2 is expecting BGP packets from R1 with the source address of R1’s loopback 131.108.10.1, the peering address, so it will reject these BGP packets.
Example 15-18 shows the output of the debug ip bgp that shows how R2 rejects packets from R1.
Example 15-18 debug ip bgp Command Output Reveals R2 Rejecting Packets from R1
R1#debug ip bgp 04:42:10: BGP: 131.108.10.2 open active, local address 131.108.1.1 04:42:10: BGP: 131.108.10.2 open failed: Connection refused by remote host
Solution
R1 should be configured with the update-source command, using the neighbor statement for R2 to accept any BGP packets from R1. The update-source command ensures that the source address is that of Loopback 0, which R2 expects. Example 15-19 shows the necessary configuration addition in R1 and R2 to form an EBGP multihop neighbor relationship.
Example 15-19 Correct Configuration of R1 and R2 to Form EBGP Multihop Neighbor Relationship
R1# router bgp 109 neighbor 131.108.10.2 remote-as 110 neighbor 131.108.10.2 ebgp-multihop 2 neighbor 131.108.10.2 update-source Loopback0 _____________________________________________________________________________________ R2# router bgp 110 neighbor 131.108.10.1 remote-as 109 neighbor 131.108.10.1 ebgp-multihop 2 neighbor 131.108.10.1 update-source Loopback0
Problem: Internal BGP Neighbors Not Coming Up
IBGP can experience issues similar to EBGP in neighbor relationship. IBGP is an important piece of overall BGP-running networks. Chapter 14 discusses the importance and usage of IBGP. This section addresses some commonly seen issues exclusive to IBGP neighbor relationship problems. The causes of this problem are identical to the previous problem of nondirectly connected external BGP neighbors not coming up:
- The route to the nondirectly connected IBGP neighbor address is missing.
- The update-source interface command is missing in BGP configuration.
You can use the same troubleshooting and configuration techniques as those used for the EBGP problem.
Problem: BGP Neighbors (External and Internal) Not Coming Up—Cause: Interface Access List Blocking BGP Packets
Interface access list/filters are another common cause of BGP neighbor activation problems. If an interface access list unintentionally blocks TCP packets that carry BGP protocol packets, the BGP neighbor will not come up.
Figure 15-8 shows the flowchart to follow to fix this problem.
Debugs and Verification
Example 15-20 shows sample access list 101 that explicitly blocks TCP. Example 15-20 shows access list 102 that has an implicit deny of BGP because Cisco IOS Software has an implicit deny at the end of each access list.
Both access lists 101 and 102 will prevent a BGP neighbor relationship from coming up.
Example 15-20 Access List Configuration Blocking BGP Neighbors
R1#access-list 101 deny tcp any any access-list 101 deny udp any any access-list 101 permit ip any any interface ethernet 0 ip access-group 101 in access-list 102 permit udp any any access-list 102 permit ospf any any interface ethernet 0 ip access-group 102 in
Solution
An interface access list must permit the BGP port (TCP port 179) explicitly or implicitly to allow neighbor relationships.
Example 15-21 shows the revised access list configuration that allows BGP.
Example 15-21 Access List Configuration Permitting BGP
R1#no access-list 101 access-list 101 deny udp any any access-list 101 permit tcp any any eq bgp access-list 101 permit ip any any
All BGP packets will be permitted because of the second line in access list 101.