DM-CTC US – Delayed Device ID reports

Incident Report for CalAmp

Postmortem

DM-CTC US – Delayed Device ID reports 5/17/2024

Incident Start Date: 5/17/2024
Started: 5/17/2024 11:00 am PT
Corrective action: 5/17/2024 02:02 pm PT
Backlog cleared: 5/17/2024 08:30 pm PT

Problem Statement

A delay in ID report processing occurred. There was no impact to other Device Management functionality and no data was lost.

Root Cause Analysis

Long running queries in the database were identified that caused database locks. These locks caused a slowdown in processing of ID reports which lead to a backlog of queued ID reports.

Terminating the long running queries corrected the database locks. Some of the services were scaled up to speed up the processing of the ID reports and clear the backlog.

Corrective Action and Follow Up

Additional alarms have been put in place to alert the team when ID Report queue metrics are exceeded.
CalAmp is working on implementing process changes to prevent such queries from impacting the database in the future.

Posted May 21, 2024 - 07:41 PDT

Resolved

Then incident has been resolved. The backlog has cleared.

Posted May 17, 2024 - 20:18 PDT

Monitoring

A fix has been implemented and we are monitoring the results. There will still be a delay as we process the backlog.

Posted May 17, 2024 - 14:02 PDT

Identified

The issue has been identified and a fix is being implemented.

Posted May 17, 2024 - 11:34 PDT

Investigating

We are currently investigating this issue.

Posted May 17, 2024 - 11:00 PDT

This incident affected: US CalAmp Telematics Cloud (US CTC Device Management (DM-CTC)).