DM-CTC US - Device Management ID reports may be delayed

DM-CTC US – DM ID reports delayed 3/12/2025

Incident Start Date: 3/12/2025
Started: 3/12/2025 05:40 am PT
IDR comm stopped: 3/12/2025 11:50 am PT (ID report communication stopped to aid recovery)
IDR comm restored: 3/12/2025 01:10 pm PT (ID report communication restored)
Corrective action: 3/12/2025 03:25 pm PT
New IDR current: 3/12/2025 04:20 pm PT
Backlog IDR cleared: 3/13/2025 06:00 am PT

Problem Statement

ID report processing delayed.

Root Cause Analysis

A spike in ID reports caused some excessive reporting devices (hyper-reporting) caused the processing of ID reports to become slower than the rate of incoming reports. This caused a backlog of reports in the queue. As the queue continued to grow the processing of reports became even slower.

After investigation with our third-party cloud hosting provider, it was identified that our storage performance was hitting its max throughput and causing slow performance. To increase our required performance, CalAmp needed to move our message queues to faster storage.

A new message queuing system was put in place with faster storage. All new ID report traffic was routed to the new message system allowing for new ID reports to process. The older queues were paused to allow for new ID report to process first.

Once the message queue of new ID reports was current, the older message queues were turned back on to process the backlog. This took serval hours to complete.

Corrective Action and Follow Up

1. CalAmp reporting queues have been moved to faster performing storage.
2. Additional hyper-reporting checks have been further refined/adjusted to prevent devices from excessing reporting.
3. Additional alerts and monitoring have been added to identify queue performance issues earlier and in additional areas of report processing.
4. A review of devices that were causing excessive ID reporting is being done and communicated back to the appropriate owners.