CalAmp K-12 Message Processing Delay
Incident Report for CalAmp
Postmortem

CalAmp K-12 – Message Processing Delay – 11/8/2023

Incident Start Date: 11/8/2023 3:00 AM ET (System performance degradation start)

Incident Recovery: 11/8/2023 6:00 AM ET (Issue identified and message backlog started processing)

Incident End Date: 11/8/2023 8:00 AM ET (Full processing restored and data current for all customers)

Problem Statement

GPS and peripheral data (MDT/barcode/RFID/etc..) were delayed in the CalAmp K-12 system.  This impacted viewing real-time data in the CalAmp K-12 website, HereComesTheBus parent app, Time and Attendance, etc..

Business/Customer Impact

Device data stopped processing or was delayed. No data was lost, but messages were delayed until the system could be fully recovered. Delay in data/message processing impacted current data in the user interface and HereComesTheBus application.

Incident Resolution and Root Cause

CalAmp K-12 platform delay in message processing was caused by edge processing device failing to start collecting data.  Configuration issue was identified and corrected.  During this time, no data was lost and was recovered when the edge processing service started collecting data once again.

Corrective Action and Follow Up

  1. Improved alerting, metrics and incident escalation will be put in place in order to recover quicker.
Posted Nov 08, 2023 - 13:56 PST

Resolved
We are currently investigating a delay in message processing. As a result, the real-time location of vehicle(s) may not be available
Posted Nov 08, 2023 - 00:00 PST