CTC US – CTC Admin, CalAmp App - Reports not responding 3/7/2024

Incident Start Date: 3/07/2024
Started: 3/07/2024 05:15 am PT
Corrective action: 3/07/2024 01:02 pm PT
Event Cleared: 3/07/2024 01:52 pm PT (Legacy reporting re-enabled)
Event declared over: 3/12/2024 (Patch applied for new reporting)

Problem Statement

Reports in CTC Admin and CalAmp Application timing out for subset of users.

Business/Customer Impact

Some customers/users could not access or run reports in CTC Admin or CalAmp Application. This did not affect any message processing but only the ability to access and run reports from the two applications. This did not impact all customers. However, several users were affected, especially any customer who had not run a report previously on the new reporting system.

Root Cause Analysis

CalAmp moved to a new reporting system and infrastructure. With the increased number of users moving from the old legacy reporting to the new system, the increased load caused a degradation in reporting availability.

As part of the recovery, the team implemented multiple work arounds to enable reporting for both CTC Admin and CalAmp Application. Services were scaled up within the reporting environment to handle the increase in load and legacy reporting was re-enabled to allow customers to access reports.

An additional patch has been deployed to Production to improve the access to the new reporting environment and improve performance. Users may experience a one-time slower performance the first time they use reporting, but additional usage should not cause any performance degradation. Ongoing monitoring of the Production environment is taking place to make sure no performance issues exist. Once completed, the legacy reporting will be disabled and removed.

Corrective Action and Follow Up

1. CalAmp is reviewing and modifying the new reporting infrastructure/architecture to insure there are no further performance issues. Extensive performance testing in pre-production environment has been put in place and will be exercised in all future deployments. This will prevent future occurrences of these types of issues.
2. CalAmp has deployed a Production patch and is monitoring performance. Once the team is sure of no issues, the legacy reporting system will be deprecated.