CTC EU – Delay in SCI device traffic 6/6/2024 – 6/7/2024

Incident Start Date: 6/6/2024
Started: 6/6/2024 09:12 pm PT
Corrective action: 6/7/2024 07:25 am PT
Event Cleared: 6/7/2024 07:25 am PT
Event declared over: 6/7/2024 07:45 am PT

Problem Statement

SCI device message processing stopped. The message gateway between SCI events and core CTC EU experienced a connection failure. There was no impact on messages from any other non-SCI devices.

Root Cause Analysis

The subsystem/gateway used to communicate SCI device data to the core CTC system experienced a connection failure after a planned deployment. The connection was tested during the deployment, however, a post-deployment step inadvertently caused the failure in the connection. The team restored the connectivity and processing returned to normal. No messages were lost but were delayed for processing.

Corrective Action and Follow-Up

1. CalAmp is reviewing the system deployment and validation process of the SCI processing application to prevent this issue in the future. Post-deployment steps are being modified.
2. Additional checks/alarms have been implemented to alert the team when SCI device processing stops.
3. CalAmp is reviewing the Architecture implementation to ensure best practices are being followed.