Incident Start Date: 6/13/2024
Started: 6/13/2024 09:30 am PT (Intermittent delay and clear)
Server upgrade: 6/13/2024 05:40 pm PT (Servers in the cloud upgraded to improve processing)
Final Backlog Cleared: 6/13/2024 07:10 pm PT (All data current)
Event declared over: 6/13/2024 09:12 pm PT
CTC Data pump messages intermittently delayed. CalAmp App UI data delayed.
Customers using CTC Data pump in the US were intermittently not getting current messages. There were multiple times during the event where message processing would fall behind, then catch up and become current. This also impacted CalAmp Application, which would intermittently fall behind.
Restarting the message processing service initially cleared the delays and allowed the processing to become current. However, the pattern continued and CTC experience recurring delays requiring restart of the service for message processing to catch up. During the investigation and in collaboration with our Cloud provider, we identified that the cloud servers used to store the device messages had reached their maximum allowable network bandwidth. The CalAmp team identified the appropriate server configuration that supported higher network bandwidth and initiated an upgrade of the affected servers. Upon successful completion of the upgrade of the servers, all backlog was processed and data became current.
Below is a snapshot of the timeline:
09:30 am PT – Initial delay
10:25 am PT – backlog cleared and queue current
10:55 am PT – delay
12:00 pm PT – backlog cleared and queue current
12:50 pm PT – delay
Team continued working on the issue; intermittent delay and catch up continued
05:40 pm PT – Upgrade of cloud servers
07:10 pm PT – all backlog cleared and queues current