CTC US – Core services not responding 9/21/2023

Incident Start Date: 9/21/2023
Started: 9/21//2023 11:14 pm PT
Corrective action: 9/22//2023 12:10 am PT
Event Cleared: 9/22//2023 12:19 am PT (All services restored, queues current)
Event declared over: 9/22//2023 12:33 am PT

Problem Statement

CTC Core Services stopped processing.

Business/Customer Impact

Customers could not log into the UI or API. Incoming device messages were not impacted and continued to flow throughout the duration of the incident. However, messages could not be pulled from the API or Datapump. No messages were lost.

Root Cause Analysis

Monitoring and alerting systems identified a connection issue in CalAmp systems. Upon investigation, it was found that a failover of the caching system of Cloud Infrastructure Provider caused CalAmp Core Service access to fail. A restart of Core Services was required to restore login function. This affected the UI, API, and Datapump. No messages were lost.

Corrective Action and Follow Up

1. CalAmp will review and adjust it’s operational process to prevent these maintenance activities by the Cloud Provider to impact CalAmp systems.
2. CalAmp will review the system architecture/environment for an automated recovery for a quicker recovery should this occur in the future.