Incident Started: 10/1/2024 07:00 am PT
Service impacted: 10/1/2024 07:13 am PT (Message processing delayed. CTC Admin, CalAmp App, CTC APIs)
Corrective action: 10/1/2024 07:50 am PT (Message processing restored. Backlog of messages being processed. CTC Admin and CalAmp App still experiencing intermittent issues.)
Backlog Cleared: 10/1/2024 08:20 am PT (All data current)
UI Cleared: 10/1/2024 08:20 am PT (CTC Admin and CalAmp App fully functional)
Event declared over: 10/1/2024 09:00 am PT
CTC message processing was delayed. Access to Datapump, CTC Admin, and CalAmp App experienced access failures.
CTC US messaging pipeline experienced a failure in processing messages. No messages were lost but a delay and backlog of messages occurred. This failure impacted users trying to get messages via datapump or using the API.
The issue was caused by degraded hardware in our Cloud provider's storage system. This required a CTC server restart to migrate to different hardware. Once the restart was performed, message processing was restored, and a backlog of messages needed to be processed. As part of the recovery, other services needed to be restarted to fully restore CTC Admin and CalAmp Application.
Once the services were restarted, all systems and applications were fully functional and returned to normal activity.