CalAmp Status - CalAmp K-12 - Message Processing Delay

CalAmp K-12 - Message Processing Delay

Incident Report for CalAmp

Postmortem

CalAmp K-12 – Message Processing Delay – 11/2/2023 – 11/6/2023

Incident Start Date: 11/2/2023

Started: 11/2/2023 15:35 PM ET (System performance degradation start)

Incident End Date: 11/6/2023 6:00 PM ET (Full processing restored and data current for all customers)

Partial recovery: 11/2/2023 20:00 PM ET – 11/3/2023 10:00 AM ET (Message processing restored)

Partial recovery: 11/4/2023 12:30 AM ET – 11/6/2023 6:15 AM ET (Message processing restored)

Emergency Maint: 11/6/2023 - 09:00 PM ET (Server maintenance)

Event closed: 11/7/2023 – 10:00 AM ET (Status Page incident left open in monitoring status since 11/6/2023)

Problem Statement

GPS and peripheral data (MDT/barcode/RFID/etc..) were delayed in the CalAmp K-12 system. This impacted viewing real-time data in the CalAmp K-12 website, HereComesTheBus parent app, Time and Attendance, etc..

Business/Customer Impact

Device data stopped processing or was delayed. No data was lost, but messages were delayed until the system could be fully recovered. Delay in data/message processing impacted current data in the user interface and HereComesTheBus application.

Incident Resolution and Root Cause

CalAmp K-12 platform delay in message processing was caused by the inability of the message queueing system to send and receive data in a timely manner to/from GPS telematics devices in the field and other pieces of CalAmp infrastructure. After further investigation, it was decided that performance improvements and server maintenance was needed to alleviate stress. Additional improvements will be implemented to further load balance sending/receiving data in the coming days.

Corrective Action and Follow Up

Further investigation of system performance is being conducted. Appropriate system/data load balancing being reviewed and implemented.
CalAmp is reviewing its recovery process to allow for faster recovery.
Improved customer notification and communication being implemented.

Posted Nov 08, 2023 - 13:38 PST

Resolved

This incident has been resolved.

Posted Nov 07, 2023 - 07:06 PST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 06, 2023 - 08:58 PST

Investigating

We are currently investigating a delay in message processing. As a result, the real-time location of vehicle(s) may not be available.

Posted Nov 06, 2023 - 03:42 PST

This incident affected: Calamp K-12 (K-12 GPS Data Processing, K-12 Website, K-12 Reports, K-12 JSON API, K-12 XML API, K-12 MDT Processing, K-12 Student Scans, K-12 Time and Attendance Portal, K-12 Time and Attendance Payroll processing, K-12 Notification/Alerts, K-12 Address Processing, K-12 Mobile Application, K-12 Here Comes the Bus).