Kernel Management Daemon (KMD) Internal Error during an interface connection failure.
Example of the kmd.core that was generated
Log Messages: Aug 18 09:38:16 XXXXXXX-xxxx-1 init: ipsec-key-management (PID 3633) terminated by signal number 11. Core dumped! Aug 18 09:38:16 XXXXXXX-xxxx-1 /kernel: tcp_auth_ok: Packet from 10.254.2.14:38567 unexpectedly has MD5 digest Aug 18 09:38:17 XXXXXXX-xxxx-1 (FPC Slot 1, PIC Slot 0) SERVICES_UI: Unable to handle message type 23 subtype 54 Aug 18 09:38:17 XXXXXXX-xxxx-1 kmd[4704]: KMD_INTERNAL_ERROR: kmd_pic_connect_failure_cb: Failed to connect PIC, ERR: Failed to connect PIC, ERR: Failed t Aug 18 09:38:17 XXXXXXX-xxxx-1 kmd[4704]: KMD_INTERNAL_ERROR: kmd_request_pic_for_sa_state: PIC sp-1/0/0 did not respond to stats re-sync query: Invalid argument Aug 18 09:38:17 XXXXXXX-xxxx-1 kmd[4704]: KMD_INTERNAL_ERROR: kmd_request_pic_for_sa_state: Couldn't request PIC: sp-1/0/0 to send sa state Aug 20 06:12:27 XXXXXXX-xxxx-1 init: ipsec-key-management (PID 12825) terminated by signal number 11. Core dumped! Aug 20 06:12:27 XXXXXXX-xxxx-1 (FPC Slot 1, PIC Slot 0) SERVICES_UI: Unable to handle message type 23 subtype 54 Aug 20 06:12:27 XXXXXXX-xxxx-1 kmd[13030]: KMD_INTERNAL_ERROR: kmd_pic_connect_failure_cb: Failed to connect PIC, ERR: Failed to connect PIC, ERR: Failed t Aug 20 06:12:27 XXXXXXX-xxxx-1 kmd[13030]: KMD_INTERNAL_ERROR: kmd_request_pic_for_sa_state: PIC sp-1/0/0 did not respond to stats re-sync query: Invalid argument Aug 20 06:12:27 XXXXXXX-xxxx-1 kmd[13030]: KMD_INTERNAL_ERROR: kmd_request_pic_for_sa_state: Couldn't request PIC: sp-1/0/0 to send sa state
The systems listed above are due to a kernel management daemon failing to connect to the service PIC. The KMD could not request the serivce PIC to send an sa (security-associations) state. This impacted the service PIC causing all related interfaces under that service PIC to bounce.
This issue was caused by configuring KMD traceoptions with the level set to warning. The results were the KMD coring during the rekey procedure.
kmd core dump regress@mobst960b# run show system core-dumps -rwxrwxrwx 1 root field 335089 Feb 13 07:17 /var/tmp/kmd.core-tarball.0.tgz*
Perform these steps:
- Review the following logs for KMD internal errors
show log messages
show log chassisd - If the failure messages continue and no root cause is found open a case with your technical support representative to investigate the issue further.