Troubleshooting IS-IS

Troubleshooting IS-IS

Troubleshooting IS-IS Adjacency Problems

IS-IS adjacency-related problems normally are caused by link failures and configuration errors. On Cisco routers, link failures easily can be identified by inspection of a show interface command output. Also, because IS-IS routing is not required to establish IP connectivity to directly attached routers, it is easy to discern whether the problem is media-related or specific to the IS-IS configuration.

The show clns neighbors command is usually the starting point for troubleshooting IS-IS adjacency problems. Chapter 10 provides a preview of this command in the coverage of basic configuration and verification of IS-IS operation. The output of this command should list all neighbors expected to be adjacent to the router being investigated. The command show clns is-neighbors provides similar output, but it is intended to list only neighbor routers or IS-IS adjacencies; the previous command lists all types of adjacencies, both for IS-IS and for ES-IS.

Before proceeding with the troubleshooting scenarios, take a look at this command again. The output in Example 11-1 was obtained from RT1, which is connected, as shown in Figure 11-1. Also shown in Example 11-1 is a variation of this command with an additional keyword, detail.

Troubleshooting IS-ISfig11.1

Example 11-1 show clns neighbors Command Output

The show clns neighbors command provides a summary of known neighbors, the connecting interface, and the state of the adjacency. The show clns neighbors detail command provides more information about each neighbor, such as the area that it belongs to and how long it has been known (uptime). Explanation of the fields in this output is as follows:

  • System ID— System identifier of the neighbor.
  • Interface— Physical interface where the neighbor is connected.
  • SNPA— Subnetwork point of attachment. This is the data link type or address (HDLC or PPP for serial, and MAC address for LANs).
  • State— State of the adjacency—up, down, or init.
  • Holdtime— The time interval until expiration of the adjacency. The holdtime is reset to the product of the hello interval and hello multiplier for corresponding neighbors every time that a hello is received from them. The default values of the hello interval and the hello multiplier are 10 seconds and 3, respectively; hence, the holdtime is reset to 30 seconds on receipt of a hello packet under default conditions of operation.
  • Type— Type of adjacency—Level 1, Level 2, or both.
  • Protocol— The type of adjacency—IS-IS, ISO IGRP, or ES-IS.

Problems with IS-IS adjacency formation can be registered by the presence of fewer neigh-bors than are expected or a situation in which the status of an expected adjacency is not up. Another symptom could be that the neighbor is known through the ES-IS protocol instead of IS-IS.

You might recall from Chapter 10 that the ES-IS protocol is one of three protocols that supports the ISO CLNS environment. All ISO devices run the ES-IS protocol to facilitate mutual discovery and communication between end systems and routers in the CLNS environment. End systems and routers exchange end-system hellos (ESHs) and intermediate-system hellos (ISHs) within the ES-IS framework. Connected routers also receive each other’s ISHs and form ES-IS adjacencies. Therefore, it is possible that ES-IS adjacencies might still be formed between two routers even if there are problems with the IS-IS adjacency.

In the output of show clns displayed in Example 11-1, RT1 has properly formed adjacencies with its directly connected neighbors, RT2 (over Serial 0/0) and RT5 (over Ethernet 0/0). By using a three-way handshake to complete the adjacency process, RT1 can be certain that both RT2 and RT5 also have normally established corresponding adjacencies in the reverse direction. Example 11-2 shows similar output from RT1 featuring problems with the adjacencies formed with RT2 and RT5.

Example 11-2 show clns neighbors Command Output

In this example, the IS-IS adjacency with RT2 is in INIT state instead of UP. The protocol is correctly shown as IS-IS. However, the adjacency with RT5 is in UP state; however, the protocol is ES-IS instead of IS-IS. As explained previously, the ES-IS protocol runs independently of IS-IS; therefore, the ES-IS adjacency formed between RT1 and RT5 has nothing to do with IS-IS. These routers cannot form an IS-IS adjacency with each other, apparently because of a problem in the configuration or the IS-IS environment. When you determine that an IS-IS adjacency problem is not the result of link problems, you can use the show clns interface command to display anomalies, such as contention over the role of designated intermediate system (DIS). Most adjacency problems related to the IS-IS environment can be debugged with the debug isis adj-packets command. The output of this command can be daunting if the router under inspection has many neighbors because the display shows all the hellos transmitted and received by the local routers. Consecutive hellos sent and received between two stations are spaced at approximately 10 seconds, in the default setting, so information fills the screen quickly.

The objective of the sections that follow is to assist you in understanding the nature of adjacency problems, uncover the causes, and correct them. Networks are critical in today’s business environment, and the ability to troubleshoot problems quickly is a virtue that helps save businesses from the high costs associated with network outages and earns the network operator years of employability and accompanying benefits.

The following sections discuss adjacency problems and suggest possible causes.

Table 11-1 summarizes the adjacency problems and possible causes covered in this section. This abbreviated representation is designed to facilitate reference to the material. Each problem listed is subsequently elaborated, and pointers are provided on how to identify the symptoms and the necessary actions required to correct a specific problem.

Troubleshooting IS-IStb11.1

Problem 1: Some or All of the Adjacencies Are Not Coming Up

The absence of some expected IS-IS adjacencies means that the affected routers will not be capable of exchanging routing information, effectively creating reachability problems to certain destinations in the network.

Figure 11-2 shows a simple network in which four daisy-chained routers are grouped in twos and placed in separate areas.

Troubleshooting IS-ISfig11.2

The output of the show clns neighbors command captured on RT1 and shown in Example 11-3 displays only one neighbor instead of the expected two. RT2 is listed, but RT5 is missing. Because RT5 also is expected to be adjacent, this signals an adjacency problem that needs to be further investigated.

Example 11-3 show clns neighbors Command Output Reveals a Missing Neighbor

The flowchart in Figure 11-3 illustrates the logical steps for troubleshooting the problem. These steps also are elaborated in the text that follows.

Troubleshooting IS-ISfig11.3

Step 1: Checking for Link Failures
The first step toward addressing this problem is to make sure that all the interfaces on which adjacencies should be formed are in the up/up state. On Cisco routers, this can be done quickly with the show ip interfaces brief command, which displays a summary of all the interfaces, as shown in Example 11-4. The example is based on Figure 11-2.

Example 11-4 show ip interfaces brief Command Output Displays Physical State of Interfaces

The Status column of the show ip interfaces brief command tells whether an interface is up, down, or administratively down at the physical level. The Protocol column also needs to be up in order to confirm normal operation at the data link. Verify that the interface is working by pinging across to the other end, as demonstrated on Example 11-5. From Figure 11-2, you know that RT2 is on the other end of serial0/0 and has an IP address of 192.168.1.2 on the corresponding interface. Therefore, a ping to this interface will verify whether packets can go over the link. A successful ping test is confirmed by exclamations, as shown in Example 11-5. When the ping test fails, dots instead of exclamations are displayed. If the ping fails, the physical connectivity prob-lem first must be resolved before IS-IS operation is checked any further.

Example 11-5 Using ping to verify connectivity

Step 2: Verifying Basic Configuration
If the link is fine, the next step involves verifying the IS-IS configuration. In general, IS-IS is enabled in two steps. First, the IS-IS process is configured, as shown in Example 11-6. Make sure that the IS-IS process is defined and that an NSAP or NET is configured.

Example 11-6 IS-IS Process Configuration

Unlike other IP routing protocols, such as RIP and OSPF, the IS-IS configuration doesn’t use network statements to enable IS-IS routing on the router’s interfaces. For example, in OSPF, if the IP subnet configured on an interface falls in the range of a network statement, OSPF will send Hellos over that interface to form adjacencies and therefore exchange IP routing inform-ation over that interface.

To enable IS-IS routing for IP on a Cisco router, you must configure the ip router isis command on the appropriate interfaces. The IP subnets on interfaces enabled for IS-IS routing in this manner automatically are put into the locally generated LSP. The only network statement required in IS-IS is the ISO NSAP, which also commonly is referred to as the network entity title (NET). However, it should be noted that misconfigured NETs are a common cause of IS-IS adjacency problems. This is further described later in Step 4, “Checking for Area Misconfiguration.”

For interfaces on which the ip router isis command is missing, make sure that the router-level passive-interface command is not being used to disable IS-IS routing on them. When an inter-face is made passive, the ip router isis command automatically is removed from the interface. An interface is made passive if it is desirable to advertise its associated IP subnet without form-ing any adjacencies over it. Loopback interfaces normally are configured this way.

Step 3: Checking for Mismatched Level 1 and Level 2 Interfaces
IS-IS supports a two-level routing hierarchy in which routing within an area is designated as Level 1 and routing between areas is designated as Level 2. An IS-IS router can be configured to participate in Level 1 routing only (Level 1 router), Level 2 routing only (Level 2 router), or both (Level 1–2 router). Level 1–2 routers act as border routers between IS-IS areas and facil-itate interarea communication.

In the default mode of operation, Cisco routers have Level 1–2 capability. Two directly con-nected routers with a common area ID will form a Level 1–2 adjacency by default, even though only a Level 1 adjacency is necessary for them to communicate. You can use the router-level is-type command to change this behavior.

Using Figure 11-2 as an example, it might be desirable to make RT5 a Level 1–only router while RT1 remains capable of Level 1–2. This requires RT5 to be configured with the is-type level-1 command, but nothing needs to be done on RT1. If RT1 is made a Level 2–only router with the command is-type level-2-only, it will not be capable of forming a Level 1 adjacency with RT5. As shown in Figure 11-2, the proper setup is to make RT5 a Level 1 router only if it has limited resources (memory and CPU capacity); RT1 should be a Level 1–2 router for it to communicate with RT5 at Level 1 and with RT2 at Level 2 because RT2 is in another area. Just as with RT5, RT6 can be a Level 1–only router, if necessary.

Step 4: Checking for Area Misconfiguration
In the CLNP addressing overview provided in Chapter 10, the three main components of an NSAP address are discussed. With 1 byte at the farthest right end of the NSAP reserved for the NSEL and the following 6 bytes for the system ID, the rest of the address defines the area ID (see Figure 10-8 in Chapter 10).

Just as in the case of Step 3, two routers in different areas with different area IDs conse-quently are assigned to different areas and, therefore, form only a Level 2 adjacency. Using Figure 11-2 as an example, if RT5 is configured as Level 1 only but its area ID is miscon-figured to be different from the area ID of RT1, these two routers will not form any adjacency. The configuration in Example 11-7, even though RT1 is expected to be in area 49.0001, has been configured with an area ID of 49.0005 placing it in a different area from RT5. Therefore, RT5 must be Level 2–capable to form adjacencies with RT1. However, it has been made a Level 1–only router with the command is-type level-1. Therefore, no IS-IS adjacency will be formed between RT1 and RT5.

Example 11-7 RT1 and RT5 Configurations Show Area IDs and Adjacency Capabilities

Step 5: Checking for Misconfigured IP Subnets
In recent releases of Cisco IOS Software, particularly in 12.0S, 12.0ST, and 12.0T release trains, adjacencies will not be formed between two neighbors if their directly connected interfaces are not in the same IP subnet. In Example 11-8, the address of RT1 is changed to that of another subnet. Again using Figure 11-2 for illustration, Example 11-9 shows RT1 rejecting the hello received from RT5 because the interface address 10.1.1.5 advertised by the latter is not on subnet 10.1.8.0/24.

Example 11-8 Verifying the Effect of an IP Subnet Mismatch on IS-IS Adjacencies

Example 11-9 Debugging Adjacency Problems Caused by an IP Subnet Mismatch

In earlier Cisco IOS Software releases, it did not matter whether the routers belonged to different IP subnets because IS-IS adjacency formation occurs primarily within the CLNP framework, where IP addresses are irrelevant. However, in IP applications, directly connected routers must be on the same subnet, except when IP unnumbered is used. Therefore, the recent behavior provides an extra check for the IP configuration while introducing sanity into IS-IS data structures for tracking IP information.

In summary, it is important to make sure that directly connected routers that need to form IS-IS adjacencies for IP routing are on the same IP subnet.

Step 6: Check for Duplicate System IDs
If the previous steps checked out okay but a specific neighbor is not in the show clns neighbor output, it is possible that adjacency is not being formed because that neighbor has a duplicate system ID with the local router. An IS-IS router will not form an adjacency with a router in the same area that has a duplicate system ID. It also logs a duplicate system ID error, as shown in Example 11-10. Use the show logging command to display entries in the log. If duplicate system ID errors are found, the source of the conflict can be determined from the output of debug adj-packets, which points to the interface where the hellos with the duplicate system ID are coming from. See Example 11-11.

Example 11-10 Duplicate System ID Error Logs

Example 11-11 Detecting Duplicate System IDs with debug isis adj-packet

Problem 2: Adjacency in INIT State

The most common causes for an adjacency getting stuck in INIT state are mismatched interface MTU and misconfigured authentication parameters. The output in Example 11-12 shows what an adjacency would look like when stuck in INIT.

Example 11-12 IS-IS Adjacency Stuck in an INIT State

To guarantee that adjacent IS-IS routers can receive and process LSPs of manageable sizes from each other, ISO 10589 specifies that hellos, which are used to form and maintain adjacencies, should be padded up to maxsize – 1, where maxsize is the maximum LSP buffer size or the MTU of the outgoing interface of the source node. The objective of this requirement is to ensure that an adjacency will be formed only between routers capable of handling all IS-IS packets sent to each other. The maximum LSP buffer size is specified as 1492 bytes; however, Cisco IOS Software defaults to the MTU, meaning that hellos are padded to MTU – 1 bytes. Therefore for Cisco routers, every-thing being equal, adjacencies are formed only when MTUs of directly connected interfaces match.

Figure 11-4 illustrates a situation for normal adjacency formation.

Troubleshooting IS-ISfig11.4

In Example 11-13, the output of show clns neighbors from RT1 shows that the adjacency state is up and that the protocol type is IS-IS. Example 11-13 also shows excerpts of the show clns interface command from the two routers involved, RT1 and RT2. The MTU is 1500 bytes on either side.

Example 11-13 Correctly Formed IS-IS Adjacency

Often, an adjacency will be stuck in INIT because only one side is enabled for authentication. For example, when basic IS-IS authentication is enabled, a simple clear-text password is carried in a Type Length Value (TLV) field (Type 10) in the hello packets. If the hello received by a peer does not contain the authentication TLV or the password in the TLV is not as expected, the peer ignores the hello. On the other hand, if a router is not enabled for authentication, it doesn’t care about the existence of the authentication TLV in a neighbor’s hello. This latter situation leads to a router processing a neighbor’s hellos, registering that neighbor, advancing the status of the adjacency to INIT, and not going further. This is because the other router ignores any received hellos from the first and, therefore, never lists the first router as a detected IS neighbor. Therefore, the two routers are unable to complete the three-way adjacency process to form a complete adjacency. This is further elaborated later in this chapter, in the “Misconfigured Authentication” section. Another reason why an adjacency might be stuck in INIT is that the local routers’ hello might not be getting corrupted before reaching the other end.

Figure 11-5 is a flowchart representing troubleshooting steps for problems in which an IS-IS adjacency is stuck in INIT.

Troubleshooting IS-ISfig11.5

Step 1. If IS-IS authentication is configured, the first step in tackling this problem is to address any potential issues in this area. If no authentication is in use, the potential cause of the problem could be mismatched MTUs.

Step 2. In this step, the configuration for authentication is verified. The Cisco implementation allows authentication to be configured in three ways: at the domain, area, or interface levels. Make sure that the appropriate method is properly configured and that the passwords used are consistent. Authentication is further elaborated in a subsequent section, “Misconfigured Authentication.”

Step 3. In this step, it is assumed that there are no authentication issues. Hence, the next possibility is mismatched MTUs. The show clns interface command can quickly verify the MTUs on either side of the link. The “Mismatched MTU” section provides further details on debugging and verification pro-cedures for troubleshooting MTU mismatched problems.

Step 4. Recent 12.0S and 12.0ST Cisco IOS Software releases allow hello padding to be disabled to reduce significant and unnecessary bandwidth consumption in some network environments. Hello padding is disabled with the assump-tion that the MTUs match. This step suggests making sure that hello padding is configured consistently on either side. If the MTUs match on either side, disabling hello padding on only one side will not have any operational con-sequences. Issues relating to the disabling of hello padding are explored further in the later section “IS-IS Hello Padding.”

Step 5. If no issues are found with authentication and MTUs also match, debugging of the IS-IS adjacency-formation process should be enabled with the command debug isis adj-packets. The output should be scrutinized for clues that could provide more insight into the init state of the adjacency. Debugging should be enabled at both ends with timestamps to help correlate events at both sides.

Step 6. In this decision step, you determine whether you have enough information to diagnose the problem and implement a solution, or whether you need to ask for expert assistance.

Step 7. If nothing can be discerned about the problem at this time, it most likely is caused by a software malfunction and should be turned over for technical assistance.

Step 8. This step is the solution stage. The troubleshooting process ends at this step if the problem is figured out. If it is determined that the problem is with authentication, the necessary configuration changes can be made to enable the adjacency to complete. On the other hand, if there is an MTU mismatch, the MTU must be changed on one side to match the other. Disabling the hello padding does not remove MTU verification at either end; it just allows smaller hellos to be advertised to save bandwidth.

As mentioned previously, a possible cause of the adjacency getting stuck in INIT is when one side’s hellos are reaching the other side but corrupted in transition. This situation is comparable to the misconfigured adjacency scenario. To determine if packet corruption might be the cause, you need to check for packet corruption errors in the logs of the remote router.

Mismatched MTU

Examples 11-14 and 11-15 demonstrate the effect of the default behavior of mismatched MTUs. The demonstration is based on Figure 11-4.

Example 11-14 Debugging MTU Mismatch on RT2

As shown in Example 11-14, the MTU on RT2 is changed to 2000 bytes, and the debug isis adj-packet command is enabled to observe what goes on in the background. Notice that be-cause the MTU on the serial interface is now 2000 bytes, the hello packets are padded to 1999 bytes. The debug output shows RT2 sending out serial IIHs of 1999 bytes.

This output also shows that RT2 continues to receive and process IIHs from RT1. However, at this time, RT1 cannot accept and process the 1999-byte IIHs from RT2 and, therefore, deletes the adjacency, removing RT2 from the list of known IS neighbors advertised in its hello. Con-sequently, RT2 changes the state of the adjacency to INIT and logs an adjacency change error. From this time on, the adjacency gets stuck in INIT. RT2 receives 1499-byte hellos from RT1, which are not larger than the 1999 bytes expected. However, the IS-IS adjacency is stuck in INIT state because RT1 cannot complete the three-way adjacency process.

Example 11-15 features show clns neighbors command output from RT2, confirming the debugging output.

Example 11-15 Init State Confirmed on RT2

Example 11-16 Debugging MTU Mismatch on RT1

The logging timestamps are almost synchronized to match related events as they occur on the separate routers. RT1 does not report receipt of any corresponding hellos from RT2 for the three highlighted consecutive hellos that it advertises; consequently, it drops the adjacency to RT2 and also logs an adjacency change error. RT1 silently discards the hellos from RT2 and doesn’t log any corresponding errors until it drops the adjacency. Even though RT1 does not process any of the IIHs from RT2 because they are larger than expected, resulting in deletion of the IS-IS adjacency, it retains an ES-IS adjacency to RT2. The ES-IS adjacency is sustained independently by ISHs under the ES-IS protocol.

Example 11-17 shows the corresponding output of show clns neighbors on RT1 after the IS-IS adjacency is dropped.

Example 11-17 ES-IS Adjacency Retained on RT1 Without IS-IS Adjacency

Example 11-18 demonstrates reinstatement of the adjacency after the MTU is corrected on RT2. As highlighted in the debugging output, RT2 begins to send 1499-byte IIHs to RT1. These obviously are accepted and processed by RT1, which then sends IIHs with RT2 listed as an IS nieghbor to complete the three-way handshake, thus restoring the adjacency.

Example 11-18 Correcting the MTU Mismatch Restores IS-IS Adjacency Between RT1 and RT2

IS-IS Hello Padding

Recent releases of Cisco IOS Software from the 12.0S and 12.0ST trains allow padding of hellos to be disabled. Older releases follow ISO 10589, which requires hellos to be padded automatically, as described earlier. The motivation behind disabling hello padding is based on the notion that this process is effectively not an MTU discovery mechanism and designed to check only MTU consistency between adjacent neighbors. Because most media have default MTUs, this check is, therefore, redundant and costly. Network operators are also well aware of their networking environments, so padding hellos to the MTU size of outgoing interfaces results in unnecessary waste of network bandwidth.

A couple of commands exist for disabling and re-enabling IS-IS hello padding. Hellos are padded by default on Cisco routers.

The following is a router-level command and applies globally to all IS-IS–enabled interfaces. The multipoint and point-to-point keywords restrict the effect of the command to only multipoint or point-to-point interfaces, respectively:

The following is an interface-level command that can be applied to disable padding on a specific interface:

The status of hello padding on an interface can be verified with the show clns interface command, as shown in Example 11-19. Examples 11-20 and 11-21 display debug isis adj-packet outputs for RT1 and RT2, respectively (see Figure 11-4). Example 11-20 shows RT1 sending hellos of only 38 bytes because padding has been disabled. On the other hand, RT2 is sending 1499 bytes, 1 byte short of the MTU on Serial 0/0, because hello padding is still enabled. Because the MTU on RT2 has not changed, its expected maximum hello size from RT1 remains 1499. The adjacency will fail if RT1 is configured with an MTU of 2000, for example, and starts sending hellos of 1999.

In general, suppressing hello padding only should not affect the adjacency, as long as the hellos sent out on the transmitting side are smaller than the MTU on the receiving side. Also, disabling hello padding does not disable verification of the maximum acceptable size of received hello packets.

Example 11-19 Verifying Status of Hello Padding on an Interface

Example 11-20 Debugging IS-IS Hello Packets on RT1

Example 11-21 Debugging IS-IS Hello Packets on RT2

The no hello padding command is effective only after the adjacency is activated (see Example 11-22). Therefore, before adjacency is formed, the size of hello packets transmitted to solicit adjacencies is MTU – 1. This also implies that an adjacency stays up when the MTU size is changed to a larger value after the hello padding is disabled; the hello packets transmitted are deceptively small. However, if packets larger than the MTU on the receiving side are transmitted, they are dropped. In this situation, if the link flaps, the adjacency fails be-cause the MTU mismatch is detected during the initial exchange of hellos after the flap.

Example 11-22 shows that RT1 stops sending 38-byte unpadded hellos and starts sending 1499-byte hellos after the interface is shut down to deactivate the IS-IS adjacency.

Example 11-22 Debugging Hello Packets with Interface Shut Down

Misconfigured Authentication

Authentication can be misconfigured in two situations, resulting in incomplete adjacencies:

  • Authentication passwords on either side of the adjacency do not match.
  • One side of the adjacency is not enabled for authentication.

Recall from Chapter 10 that currently (at the time of writing) only simple passwords are supported on Cisco routers. Three methods exist for enabling IS-IS authentication:

  • The first method is at the interface level with the isis password command. This configuration can be for routing at Level 1 or Level 2 or both.
  • The second method uses the router-level area-password command and applies to only Level 1 routing on all active IS-IS interfaces on the router.
  • The third method is for Level 2 only and uses the domain-password command at the router level.

When there is a password mismatch, the IS-IS adjacency does not come up at all for the applicable level of routing on either side of the connection, and the show clns neighbors command displays only ES-IS adjacency, as shown in Example 11-17.

When only one side is enabled for authentication, that side does not process hellos from a neigh-bor that does not contain the appropriate passwords. The side without authentication does not check for passwords, so it receives and processes hellos as usual; however, the adjacency will not advance beyond INIT because the local side is never listed in the IS neighbor’s TLV of the remote router. The output of show clns neighbors looks like the output in Example 11-15. The router configured for authentication does not process IS-IS hellos from the remote side, so IS-IS adjacency is not formed, even though ES-IS adjacencies are formed, as shown in Example 11-17.

Example 11-23 shows the debugging output from the debug isis adj-packets command, providing an indication of authentication problems. The authentication errors are high-lighted.

Example 11-23 Debugging Authentication Problems

Problem 3: Only ES-IS Adjacency Instead of IS-IS Adjacency Formed

Cisco routers running IS-IS in IP environments still listen to ISHs generated by ES-IS protocol, in conformance with ISO 10589 requirements. Hence, when the physical and data link layers are operational, an ES-IS adjacency can be formed even if appropriate conditions do not exist for establishing an IS-IS adjacency. Example 11-24 shows what the output for the show clns neighbors command looks like when an ES-IS adjacency is formed instead of an IS-IS adjacency. This is most likely because IS-IS hellos are not being processed, as a result of interface MTU mismatch or misconfigured authentication. Both of these causes are covered extensively in the previous section.

Example 11-24 Forming an ES-IS Adjacency

Troubleshooting IS-IS Routing Update Problems

Configuration in IS-IS is fairly simple and straightforward. In the two-stage process discussed in Chapter 10, the routing process is enabled globally, and IS-IS adjacency formation and LSP flooding is enabled on an interface by applying the command ip router isis. This command also puts the IP subnet information of the interface into the router’s LSP that is generated and flooded to adjacent neighbors. This section covers IS-IS routing update problems on the premise that there are no adjacency problems. This essentially implies that troubleshooting any routing update problems should start with verifying the appropriate adjacencies. Adjacency problems and related troubleshooting methodology are discussed extensively in the previous sections.

The following routing update problems are covered in this section:

  • Route advertisement problems
  • Route flaps
  • Route redistribution problems

One important method for troubleshooting routing problems in IS-IS is by direct inspection of the contents of link-state packets (LSPs). Depending on its configuration, an IS-IS router generates an LSP for each of the levels of routing that it participates in—Level 1 LSP, Level 2 LSP, or both. Inspection of the LSP contents of the expected source of a route can help diagnose routing advertisement problems in cases when no obvious adjacency problems exist. The show isis database detail command displays the contents of a specific LSP. Example 11-25 shows sample output from this command.

Example 11-25 Displaying the Routing Information in an LSP

RT2.00-00 is the LSP ID of RT2. Detail output of the LSP, with ID RT2.00-00, shows the IP subnets for directly connected links, together with their metric information.

Another interesting command is show isis topology, which displays a list of all known routers. For example, Example 11-26 shows the IS-IS topology for Figure 11-6 as captured on RT1.

Troubleshooting IS-ISfig11.6

The backbone (Level 2) is the shaded area. Example 11-26 shows the Level 1 and Level 2 data-bases of RT1. The Level 1 database features RT3 and RT5, confirming that these routers are indeed in the same area as RT1. The Level 2 database describes the backbone, which features RT2 and RT3. RT2 is not in the same area as RT1, but it is connected to the backbone, whereas RT3 is a Level 1–2 router in the same area as RT1.

All this information gleaned from Example 11-26 agrees with the layout in Figure 11-6; this makes the show isis topology command an invaluable command for troubleshooting routing-related problems.

Example 11-26 Obtaining IS-IS Topology Information

Route Advertisement Problems

Most route advertisement problems can be narrowed down to configuration problems at the source or LSP propagation issues.

Suppose that there is concern that a specific route is missing on RT5 (see Figure 11-6). You can start troubleshooting the problem by obtaining more information on the topology of the network. This should help you determine which router is the original source of the route. Suppose that the route 11.1.1.2/32 turns out to be the loopback address of RT2; the flowchart in Figure 11-7 can be followed to narrow the cause of the problem.

Troubleshooting IS-ISfig11.7

Using knowledge about the topology, you know that RT2 and RT5 are not in the same area. In this case, if route leaking is not enabled in the network, RT5, which is a Level 1–only router, should not see any route from RT2. Hence, tracking the problem takes you along the flowchart through boxes 0-1-7-10-5, where you determine that this is normal behavior. As a Level 1–only router, RT5 should follow a default router to the nearest Level 1 router to get to remote areas.

If you end up in Box 9 or Box 11 along the flowchart, the next case studies and flowcharts might help narrow the problem.

The procedure covered by the flowchart in Figure 11-7 provides a generic method for trouble-shooting missing route problems. The following sections discuss procedures for specific scenarios.

Local Routes Not Being Advertised to Remote

Because IS-IS is a link-state protocol, IS-IS routers depend on LSP flooding to obtain topology and routing information. For example, during stable conditions, each IS-IS router in an area will have the same Level 1 link-state database, which contains the LSPs generated by every router in the area. Dijkstra’s algorithm is run over the LS database to obtain the best path to every advertised route. Therefore, if a route is missing at a section of the area, it’s because the routers in that section did not receive the original LSP, or the LSP was received corrupted and, there-fore, was purged. An even simpler reason could be that the route was not even put into the LSP at the source. The flowchart in Figure 11-8 provides the troubleshooting methodology for such problems. The output of debug isis update-packets and debug isis snp-packets, as demon-strated in Example 11-27, could help decipher this sort of problem if it is related to LSP flooding or issues with link-state database synchronization.

Troubleshooting IS-ISfig11.8

Step 8 of the flowchart for troubleshooting route advertisement problems calls for enabling the debugging of LSP updates if it is determined that an LSP is not making it to certain remote locations of the IS-IS domain. Example 11-27 shows an output of the debug isis update-packets command. The highlighted lines show RT1 flooding its LSP and also receiving an LSP from RT2. Because the adjacency was just brought up, the output also shows the one-time exchange of CSNPs on point-to-point links between RT1 and RT2.

Example 11-27 Debugging IS-IS Routing Update Problems

The debug isis snp-packets command enables debugging of database synchronization on broadcast interfaces and can troubleshoot route update problems over media such as LANs.

The highlighted line indicates receipt of a CSNP by RT5 from the DIS (RT1). By comparing the contents of the CSNP with the local Level 1 database, RT5 determines that no changes occurred in all known LSPs.

Solution Summary
As depicted by the flowchart in Figure 11-8, there could be many potential causes for problems in which routes are not reaching remote locations in the network. In the extreme case, the route might not be making it into the routing table because of a software glitch relating to IS-IS data structures (Steps 4 and 5). Such situations might need to be reported to the Cisco Technical Assistance Center (TAC) for further assistance.

In other cases, the LSPs might not be reaching some parts of the network because they are getting corrupted in flight (Step 9). Such problems can be determined by a combination of activities, such as debugging the update process and monitoring router logs for LSP errors. If the culprit device is located, it can be isolated or replaced to address the problem.

In most cases, however, the problem could be caused by a trivial configuration error, such as IS-IS not being enabled on an interface (Step 11). Applying the appropriate interface command (ip router isis) to the interface should be sufficient to address the problem.

Situations involving route redistribution or route-leaking problems are addressed in the next section.

Route Redistribution and Level 2-to-Level 1 Route-Leaking Problems

Cisco IOS Software allows routes from other routing sources, such as static, connected interfaces, and other routing protocols, to be redistributed into IS-IS Level 1 routing, Level 2 routing, or both as external routes. Technically, external routes should exist only in the Level 2 routing environment, but Cisco provides a configuration attribute that allows redistribution into Level 1 for practical operational purposes. For example, service providers using IS-IS as IGP with flat Level 1-only domains might need this capability. The existence of the IP External Reachability TLV in a Level 1 LSP should not pose any interoperability issues because IS-IS routers are required to skip TLVs that they don’t understand when they parse an IS-IS packet.

Redistribution in IS-IS is straightforward with hardly any potential pitfalls except occasional, unexpected software bugs. Sometimes, effective metric assignment becomes a challenge in redistribution scenarios. When configuring redistribution, the operator is asked to specify internal or external metrics. The default is the external metric type, which bumps up the metric by 128 on Cisco routers. The increase by 128 instead of 64 is the result of an incorrect bit setting in the default metric field when using narrow metrics. This issue is discussed in the configur-ation section as a Cisco IOS Software implementation issue that has been retained for backward compatibility.

The rather simple flowchart in Figure 11-9 should provide guidance for troubleshooting redistribution problems.

Troubleshooting IS-ISfig11.9

Route-Flapping Problem

Route flaps normally are caused by unstable links between the source of the route and the location where the flap is being observed. Typically, multiple routes are impacted at the same time because of an adjacency change affecting many LSPs, all of which might carry numerous IP prefixes. Also, route flaps could induce consistent running of the SPF process, causing potentially dangerous high CPU utilization on route processors that might crash affected routers. If the advertised LSP changes affect only IP prefixes, only partial route calculation (PRC) is run instead of full SPF calculations. PRC is less costly in terms of CPU cycles than full SPF runs. The flowchart in Figure 11-10 provides guidance for troubleshooting route-flapping problems.

Troubleshooting IS-ISfig11.10

Apart from certain destinations not being reachable—obviously implying routing problems—high CPU utilization by the SPF process certainly should flag instabilities in the network. High CPU utilization can be observed through the IOS command line interface (CLI), network-management applications, or special network-performance analysis tools, such as CiscoWorks 2000, HP Openview, and so on.

From the Cisco IOS Software CLI, the show process cpu command can obtain CPU utilization information. If any indication of high CPU utilization caused by the SPF process is determined, the show isis spf-log command can determine SPF-related events that might cause the high CPU utilization. Example 11-28 provides sample output of this command.

Example 11-28 Tracking SPF Process-Related Events

The output in Example 11-28 lists SPF process runs by time, trigger, and duration. You might recall that LSPs are refreshed every 15 minutes, triggering periodic SPF calculations. Such events are labeled periodic in the show isis spf-log output. It also shows that at 01:25:25, the process was run over an insignificant period of time because of receipt of a new LSP from RT5. Table 11-2 lists and describes common SPF triggers.

Troubleshooting IS-IStb11.2

The following IS-IS debugging commands also can narrow SPF-related problems:

Use caution when enabling debugging in situations in which the route processor’s CPU already is overtaxed. Example 11-29 provides sample output of the debug isis spf-events command, which displays the events following an interface shut to simulate a link flap. Lines indicating LSPs flagged for recalculation as a result of this event are highlighted. Also, events flagging computation of the Level 1 and Level 2 SPF trees are highlighted.

Example 11-29 debug isis spf-events Command Output Displays Events Following an Interface Shut

Solution Summary
As shown in the flowchart for troubleshooting route flapping problems (see Figure 11-10), most route-flapping problems can be traced to unstable links in the network (see Step 2). Physical connectivity problems are addressed by troubleshooting the physical and data link layers.

In more complicated scenarios, however, the flaps could be caused by LSP corruption storms or even a routing loop. This might be the case when there are no unstable links in the network but CPU utilization is high, indicating continuous running of the SPF process (see Step 4). The show isis spf-log command can provide indication of which LSPs are changing the most frequently and are triggering the SPF calculations. Similar clues can be gleaned by enabling debugging of the update process with debug isis update-packets. This should be done with care so that the CPU is not overloaded. The logs also can be observed for LSP error loggings, which could give an indication of packet corruption by a culprit device (see Step 7). Zeroing in on any continuously changing LSPs is critical for determining and addressing the problem.

IS-IS Errors

This section reviews just a couple of typical errors encountered in IS-IS routing environments. Example 11-30 describes a situation in which the IS-IS process receives a hello packet that is only 51 bytes instead of the expected 53 bytes ATM cell. This is caused by packet corruption most likely because of malfunction of some interface hardware. This might result in adjacency failures if too many consecutive hellos are corrupted in this manner.

Example 11-30 Unexpected Hello Packet Size

Example 11-31 indicates an incorrectly formatted link-state packet in which an NSAP address length appears longer than expected. This could be caused by software implementation bugs and might have an effect on the dissemination of routing information.

Example 11-31 Unexpected NSAP Address Length in Incorrectly Formatted LSP

The router-level command log-adjacency-changes causes logging of adjacency changes, as shown in Example 11-32.

Example 11-32 Tracking Adjacency Changes

Example 11-33 shows the type of message logged when a router determines that there is another router in the same area or backbone with a duplicate of its system ID.

Example 11-33 Duplicate System ID Message

CLNS ping and traceroute

Cisco IOS Software provides ping and traceroute tools for ISO CLNP, which are analogous to the all-too-familiar IP version. ping clns and traceroute clns apparently were designed for use in ISO CLNP environments, yet they can be useful for troubleshooting IS-IS operation problems in IP environments. Contrary to popular belief, the clns router isis command is not required to enable the ping clns and traceroute clns commands to work. You might recall that, in addition to the IS-IS process, only the ip router isis command is required to activate IS-IS routing for IP only on a router’s interface. Examples 11-34 through 11-38 demonstrate the operation of the CLNS-based ping clns and traceroute clns commands. These examples are based on Figure 11-11. There is an extended option for each of these commands, just as in the case of the corresponding IP versions.

Troubleshooting IS-ISfig11.11

Example 11-34 Operation of the ping clns Command

Example 11-35 Operation of the ping clns Command in Extended Mode

Example 11-36 shows the packet debugs during operation of the ping clns command (see Examples 11-34 and 11-35). The debugs show the source and destination NSAPs, as well as the outgoing interface for each outgoing packet.

Example 11-36 CLNS Packet Debugs During CLNS pings

Examples 11-37 and 11-38 illustrate the operation of the traceroute clns command in standard and extended modes from RT5 to RT6. As in the case of the ping clns command, operation in extended mode allows customization of the command, including explicit specification of the source NSAP address of the traceroute packets.

Example 11-37 Operation of traceroute clns Command

Example 11-38 Operation of traceroute clns Command in Extended Mode

Case Study: ISDN Configuration Problem

This case study explores a problem that involves setting up IS-IS routing over an ISDN link. The objective is to put the troubleshooting knowledge acquired in this chapter to immediate use by trying to figure out any potential problems in the setup.

RTA and RTB are connected over an ISDN link, as shown in Figure 11-12. Standard configuration is employed, as demonstrated in Example 11-39.

Troubleshooting IS-ISfig11.12

Example 11-39 Configurations for RTA and RTB in Figure 11-12

In the configurations in Example 11-39, the dialer-list command is used to specify “interesting” packets, which could force a connection setup and bring up the ISDN link. Both ip and clns keywords are specified on either side of the link; however, as shown in Example 11-40 and the corresponding debugs in Example 11-41, using CLNS ping from RTB cannot bring up the circuit.

Example 11-40 Attempting to Bring Up the ISDN Connection with CLNS ping from RTB

Example 11-41 shows how debugging of CLNs packets determines the root cause of the problem. The highlighted text indicates encapsulation failure because the BRI interface is an unknown data link type.

Example 11-41 Using debugs clns packets to Troubleshoot the Problem

Experimentation with the various options in the dialer-list command, however, confirms that the appropriate keyword required is clns_is instead of clns, as demonstrated in Example 11-42.

Example 11-42 CLNS Related Keyword Options for the dialer-list Command

Example 11-43 shows that the ping clns can now bring up the links after appropriate change is made in the dialer list configuration.

This case study elaborates how the ping clns command is combined with CLNP packet debugging to troubleshoot a basic connectivity problem.

Example 11-43 Retesting the Connection with the clns is option in the dialer-list statement

IS-IS Troubleshooting Command Summary

Troubleshooting IS-IStb11.3

About the author

Prasanna

Leave a Comment