CTC US - Messaging delayed, iOn and DMCTC degraded

CTC US Message Processing, iOn, DMCTC – Cloud Provider outage - 5/7/2026

Incident Date: 05/07/2026
Issue Started: 05/07/2026 05:00 pm PT (iOn degraded performance)
05/07/2026 05:20 pm PT (Message processing degraded)
05/07/2026 06:00 pm PT (DMCTC degraded performance)
Recovery events: 05/07/2026 06:20 pm PT (Message processing restored, backlog being processed)
05/07/2026 06:25 pm PT (Message backlog cleared)
05/07/2026 09:30 pm PT (DMCTC recovered)
05/07/2026 10:00 pm PT (iOn recovered, backlog being processed)
05/07/2026 10:00 pm PT (iOn current, backlog cleared)
05/07/2026 11:59 pm PT (All systems up and current, continuing to monitor Cloud provider)
05/08/2026 07:01 am PT (BIS/Reports degraded)
05/08/2026 08:53 am PT (BIS/Reports online, backlog data being processed)
05/08/2026 11:00 am PT (BIS/Reports current)
Event declared over: 05/08/2026 08:15 pm PT (Cloud provider issue resolved)

Problem Statement

CTC services affected due to a failure in our Cloud Provider’s data center.

Root Cause Analysis

CalAmp CTC, DMCTC, iOn, and Reporting suffered degraded performance due to a data center cooling issue by our Cloud Provider. The Cloud Provider had earlier identified a potential degradation in their cooling system and was in the process of performing maintenance activities to correct the potential issues. However, during the hardware maintenance/replacement, cooling system failures occurred, causing the temperature in their data center to rise, forcing services to go offline.

As services failed over and restarted, CalAmp services were restored. CTC message processing was restored first and suffered up to a 42-minute delay. Some customers may have experienced a shorter delay as messages were being processed throughout but at a slower rate.

DMCTC and iOn continued to suffer impaired service until 4 hours later as the service was restored and backlog cleared.

On 5/8, CTC Reporting suffered degraded performance as the Cloud Provider continued to work on issues in restoring the caching service used by CalAmp

Corrective Action by Cloud Provider

1. Restore cooling capacity for all recovered hosts and replace all defective cooling hardware.
2. Update maintenance procedures for the specific cooling units to prevent issues during future hardware maintenance.
3. Improve detection, recovery, and failover of affected services.
4. Improve Transit Gateway networking distribution health system to reduce the amount of time for automatic detection and remediation of impaired host services

CalAmp Corrective Action and Follow-Up

1. CalAmp to review CTC, iOn, and DM system architecture along with the use of cloud provider’s services to identify areas to improve reliability and resiliency for faster recovery in such events.