On MX routers, this article addresses how to detect CRC errors on High Speed Link (HSL) interconnection between Dense Port Concentrator (DPC)’s Packet Forwarding Engine (PFE) or Flexible Port Concentrator (FPC), midplane connector, and a Switch Control Board (SCB – MX series) or Switch Interface Boards (SIB – T Series) or Forwarding Engine Board (FEB – M120).
The
CHASSISD_FASIC_HSL_LINK_ERROR
means the chassis process (chassisd) detected an error for the indicated high speed link (HSL) for the indicated F chip on the indicated Control Board (CB).
SYMPTOMS:
When a
CHASSISD_FASIC_HSL_LINK_ERROR
event occurs, a message similar to the following is reported:
CHASSISD_FASIC_HSL_LINK_ERROR: Fchip (CB 1, ID 0): link 25 failed because of crc errors
Depending on the severity and frequency of errors, a forwarding plane may be brought in check state or even be off-lined.
CHASSISD_FASIC_HSL_LINK_ERROR messages are reported when there is a link failure due to packet corruption. This is usually caused by a hardware error, and may or may not indicate faulty hardware. To determine which component is causing the link failure, check outputs from the forwarding plane, fabric chips, controls boards and FPCs.
To determine what part is causing the issue, first look at the
show chassis fabric fpcs
Note: For T-series routers, change “fpcs” to “sibs”; and for a M120, substitute “feb” for “fpcs”) output.
FPC 3
PFE #0
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Plane enabled
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
PFE #1
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Link error
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
In the following show chassis fabric fpcs output, all PFE planes show a link error. This is likely a DPC issue.
FPC 3
PFE #0
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Plane enabled
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
PFE #1
Plane 0: Link error
Plane 1: Link error
Plane 2: Link error
Plane 3: Link error
Plane 4: Link error
Plane 5: Link error
In the following show chassis fabric fpcs output, there is an error on one plane for multiple PFEs. This is likely SCB issue.
FPC 3
PFE #0
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Link error
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
PFE #1
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Link error
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
FPC 4
PFE #0
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Link error
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
PFE #1
Plane 0: Plane enabled
Plane 1: Plane enabled
Plane 2: Link error
Plane 3: Plane enabled
Plane 4: Links ok
Plane 5: Links ok
Then review the output of show chassis fabric plane. In the following example output, one DPC showing link errors on multiple planes. This is likely an issue with the DPC.
Fabric management PLANE state
Plane 0
Plane state: ACTIVE
FPC 0
PFE 0 :Link error
PFE 1 :Link error
PFE 2 :Link error
PFE 3 :Link error
FPC 1
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
Plane 1
Plane state: ACTIVE
FPC 0
PFE 0 :Link error
PFE 1 :Link error
PFE 2 :Link error
PFE 3 :Link error
FPC 1
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
In the following show chassis fabric plane, there are Link errors showing up on one plane for all DPCs. This likely an issue with the SCB.
Fabric management PLANE state
Plane 0
Plane state: ACTIVE
FPC 0
PFE 0 :Link error
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
FPC 1
PFE 0 :Link error
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
FPC 2
PFE 0 :Link error
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
Plane 1
Plane state: ACTIVE
FPC 0
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
FPC 1
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
FPC 2
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
In the following show chassis fabric plane, there are Link errors on one DPC and one plane. Further troubleshooting is needed to determine the part causing the errors.
Fabric management PLANE state
Plane 0
Plane state: ACTIVE
FPC 0
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
FPC 1
PFE 0 :Link error
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
Plane 1
Plane state: ACTIVE
FPC 0
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
FPC 1
PFE 0 :Links ok
PFE 1 :Links ok
PFE 2 :Links ok
PFE 3 :Links ok
For further troubleshooting, examine the following output to help determine the cause of this message:
show log messages | no-more show log chassisd | no-more show chassis fabric fpcs (or sibs or feb, depending on platform) show chassis fabric plane show chassis fabric plane-location Show chassis fabric map show chassis fabric summary show chassis fabric statistics totals <plane #> detail show chassis fabric fpcs
Look for any related events that occurred at or just before the CHASSISD_FASIC_HSL_LINK_ERROR message in the log messages and log chassisd outputs.
Perform the following steps:
1. Follow the instructions under “Confirm Causes”.
2. During a maintenance window, as it will impact transit traffic, try the following:
- In cases where DPC or FPC is determined to be the culprit, reseat the DPC or FPC.
- In cases where SCB, SIB, or FEB is determined to be the culprit, reseat the SCB, SIB, or FEB.
- if re-seating the component does not resolve the issue, or if the errors continually return shortly after re-seating, open a case with your technical support representative to investigate the issue further.
3. In cases where the part cannot be determined (for example, there is only one link error, or the errors exist across multiple dpcs/fpcs AND multiple planes), open a case with your technical support representative to investigate the issue further. Be warned that troubleshooting for such issues will likely include several hardware swaps, which will impact traffic on production routers.