DM-CTC US – Delayed Device ID reports
Incident Report for CalAmp
Postmortem

DM-CTC US – Delayed Device ID reports 5/17/2024

Incident Start Date: 5/17/2024
Started: 5/17/2024 11:00 am PT
Corrective action: 5/17/2024 02:02 pm PT
Backlog cleared: 5/17/2024 08:30 pm PT

Problem Statement

A delay in ID report processing occurred. There was no impact to other Device Management functionality and no data was lost.

Root Cause Analysis

Long running queries in the database were identified that caused database locks. These locks caused a slowdown in processing of ID reports which lead to a backlog of queued ID reports.

Terminating the long running queries corrected the database locks. Some of the services were scaled up to speed up the processing of the ID reports and clear the backlog.

Corrective Action and Follow Up

  1. Additional alarms have been put in place to alert the team when ID Report queue metrics are exceeded.
  2. CalAmp is working on implementing process changes to prevent such queries from impacting the database in the future.
Posted May 21, 2024 - 07:41 PDT

Resolved
Then incident has been resolved. The backlog has cleared.
Posted May 17, 2024 - 20:18 PDT
Monitoring
A fix has been implemented and we are monitoring the results. There will still be a delay as we process the backlog.
Posted May 17, 2024 - 14:02 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted May 17, 2024 - 11:34 PDT
Investigating
We are currently investigating this issue.
Posted May 17, 2024 - 11:00 PDT
This incident affected: US CalAmp Telematics Cloud (US CTC Device Management (DM-CTC)).