CTC US - Messaging delayed

CTC US – Message processing delayed - 5/23/2026

Incident Date: 05/23/2026
Issue Started: 05/23/2026 11:50 am PT (Message processing delayed)
Corrective action: 05/23/2026 12:15 pm PT (Processing performance restored)
Backlog processed: 05/23/2026 12:20 pm PT (All messages current)
Event declared over: 05/23/2026 12:33 pm PT

Problem Statement

Degraded message processing performance. Message backlog occurred.

Root Cause Analysis

One of the CTC messaging databases experienced resource exhaustion and automatically failed over to a secondary database. Message processing continued but at a reduced rate. The team took corrective action and restored the affected database, which resulted in message processing performance returning to normal and the backlog was quickly processed.

The issue was caused after the system OS ran out of disk space due to log file retention. The appropriate failover did occur, but message processing performance was degraded. The team cleaned up the file system, restarted the service, and restored normal processing.

All messages were processed. No messages were lost.

Corrective Action and Follow-Up

1. CalAmp cleaned up the file system of the affected database to restore service.
2. CalAmp implemented a log rotation/clean-up script to ensure that logs do not take excessive space on the system file system.
3. CalAmp added additional monitoring to identify and alert for this failure to allow for quicker identification and recovery should this occur in the future.
4. CalAmp will investigate to understand why, after the failover, the processing performance was degraded.