Completed -
π Deployment completed successfully. All services are up and running
Jul 11, 00:40 AEST
Update -
π Unmuting Datadog alerts
Jul 11, 00:20 AEST
Update -
π§ͺ Starting post-deployment testing to validate system stability and functionality.
Jul 11, 00:04 AEST
Verifying -
Verification is currently underway for the maintenance items.
Jul 10, 23:50 AEST
Update -
π Build completed. Starting the optimizer on all TMIs and grabber via admin portal.
Jul 10, 23:49 AEST
Update -
β
Deployment triggered. Monitoring progress on GitHub.
Jul 10, 23:04 AEST
Update -
π stopped optimization on all TMIs. Moving to deployment trigger
Jul 10, 23:02 AEST
Update -
β
Alerts on Datadog have been muted. Proceeding to stop optimization
Jul 10, 23:02 AEST
Update -
π Muting alerts on Datadog to avoid noise during deployment
Jul 10, 23:01 AEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 10, 23:00 AEST
Scheduled -
As part of our ongoing efforts to enhance system stability, an ad-hoc deployment is scheduled for tonight at 23:00 AEST.
This action follows a recent SEV2 incident where users experienced "Applying latest changes...". A detailed analysis using Java Flight Recorder identified inefficient garbage collection as the root cause.
β
Remediation Actions:
- Upgraded the Garbage Collector from CMS to G1GC
- Tuned JVM parameters to reduce GC pause times and improve session stability
These changes have been thoroughly validated in UAT, showing improved performance and no regressions.
π Deployment Details:
Component: asr-server
Expected Duration: ~1 hour
π Please Note:
Slot optimization on all TMIs will be paused and resumed post-deployment.
All frozen slot data will remain intact and unaffected.
We appreciate your cooperation and understanding as we work to improve system performance and reliability.
π’ A follow-up notification will be sent upon successful completion.
Thank you!
Jul 10, 21:04 AEST