CTC EU - Degraded Alert Service - Alerts not being sent

CTC EU Alert Services – Alert messages not being sent - 4/7/2026

Incident Date: 04/07/2026
Issue Started: 04/07/2026 04:00 am PT (Alert messages not going out)
Corrective action: 04/07/2026 08:05 am PT (System roll back completed)
Corrective action: 04/07/2026 08:15 am PT (Backlog being processed)
Backlog processed: 04/07/2026 02:11 pm PT (All alert messages current)
Event declared over: 04/07/2026 02:13 pm PT

Problem Statement

Alert Service messages stopped being sent.

Root Cause Analysis

CalAmp Alert Service stopped processing alert messages. The issue was caused after a system OS update was performed on 4/6/2026. The update process completed successfully, and all services were validated as processing normally. After several hours later, the Alert Services experienced a resource issue (memory exhaustion) that caused the services to stop processing.

The CalAmp team rolled back the OS upgrade and processing was restored. Additional Alert Service servers were added to increase the performance of processing through the backlog of alert messages.

All messages were processed. No messages were lost.

The root cause of the memory exhaustion was a misconfiguration of the OS allocation of memory to use for the service. Even though the machine had available resources, the memory allocated to the Alert Service was insufficient. This configuration should have been caught during the pre-production validation but was missed.

Corrective Action and Follow-Up

1. CalAmp rolled back to a previous version of the OS and configuration to restore services while further investigation was being done.
2. CalAmp is re-testing the OS upgrade to identify the issue and take corrective action to prevent any future occurrence of this issue. The incorrect configuration file was identified and corrected for future OS upgrade/deployment.
3. CalAmp is adding additional monitoring to identify and alert for this failure to allow for quicker identification and recovery should this occur in the future.
4. CalAmp will review the system testing process to ensure these updates are properly tested and prevent future occurrences. Changes and enhancements to the review of the configuration files are being implemented to catch these issues prior to deployment.