CTC US – Message Processing Delayed; Datapump, CTC Admin, CalAmp App, CTC API Access Degraded - 10/1/2024

‌

Incident Started: 10/1/2024 07:00 am PT
Service impacted: 10/1/2024 07:13 am PT (Message processing delayed. CTC Admin, CalAmp App, CTC APIs)
Corrective action: 10/1/2024 07:50 am PT (Message processing restored. Backlog of messages being processed. CTC Admin and CalAmp App still experiencing intermittent issues.)
Backlog Cleared: 10/1/2024 08:20 am PT (All data current)
UI Cleared: 10/1/2024 08:20 am PT (CTC Admin and CalAmp App fully functional)
Event declared over: 10/1/2024 09:00 am PT

Problem Statement

CTC message processing was delayed. Access to Datapump, CTC Admin, and CalAmp App experienced access failures.

Root Cause Analysis

CTC US messaging pipeline experienced a failure in processing messages. No messages were lost but a delay and backlog of messages occurred. This failure impacted users trying to get messages via datapump or using the API.

The issue was caused by degraded hardware in our Cloud provider's storage system. This required a CTC server restart to migrate to different hardware. Once the restart was performed, message processing was restored, and a backlog of messages needed to be processed. As part of the recovery, other services needed to be restarted to fully restore CTC Admin and CalAmp Application.

Once the services were restarted, all systems and applications were fully functional and returned to normal activity.

Corrective Action and Follow-Up

1. CalAmp had a review of the storage system infrastructure with the Cloud provider to understand the failure and identify ways for better monitoring and alerting of such issues.
2. CalAmp will implement additional monitoring and alarms specific to storage systems to improve the detection and recovery times.