CalAmp App EU - Degraded performance - Tracking not updating

CalAmp App – Event tracking delayed - 1/26/2026

Incident Date: 1/26/2026
Issue Started: 1/26/2026 03:27 am PT (Customer was not seeing the latest tracking events in the CalAmp App)
Card Reader issue: 1/22/2026 11:16 am PT (Card Reader Service stopped)
Corrective action: 1/26/2026 05:46 am PT (Service restored and backlog being processed)
Event Cleared: 1/26/2026 07:07 am PT (Backlog processed and data current)

Problem Statement

CalAmp App stopped message processing. Tracking of events was no longer current.

Root Cause Analysis

CalAmp App Messaging Pipeline was unable to finish processing messages from CTC due to the caching server exhausting available memory resources.

CalAmp App services use a caching service to stream for real-time event processing and communication among the microservices. There was a buildup of data in the stream after the Card Reader service stopped processing its messages on 1/22/2026 (11:16 am PT). When the Card Reader Server stopped processing, its messages continued to be queued in the cache server, causing memory resources to be exhausted. When the cache server memory reached its full capacity, all communications between the CalAmp App Services stopped.

Corrective action was taken to increase the memory resources, and services were restarted. Processing was restored, and messages were processed in a controlled fashion until the backlog was fully processed.

Corrective Action and Follow-Up

1. Additional alarms and notifications were added for better monitoring of the cache service and streams.
2. CalAmp to reprocess any messages from the logs to make sure there are no gaps in message data.
3. Engineering to review and improve the Card Reader Service to prevent the root cause of the service failure.
4. Engineering to review and improve the streaming service to prevent message communication failures between services.