Resolved -
We can now confirm that everything is working as expected now and executed necessary data recovery protocols. Users should not be facing any further issues to this end.
Mar 17, 14:37 UTC
Monitoring -
We can confirm that our systems are now running as expected. Our team is continuing to monitor the issue and also analyzing any eventual loss to initiate the appropriate recovery protocols.
Mar 17, 13:48 UTC
Identified -
Our team was able to identify the issue as being associated to the freezing our asynchronous message broker that was leading to us being unable to process any incoming traffic from our integrations as well as execute actions on the ticket pipeline.
We've now unblocked the processing of the queues and are actively monitoring the progress of the recovery.
Mar 17, 13:35 UTC
Update -
We're continuing to investigate the issue and can confirm that the impact also extends towards the synchronisation of data across our integrations (including messaging channels such as gmail).
We apologize for the inconvenience caused and are actively looking towards resolving the behavior.
Mar 17, 13:01 UTC
Investigating -
We are currently investigating a disruption affecting our ticketing workflow. This issue is causing delays in the ticket pipeline for the "europe-west4-e7f4" cluster and is preventing outbound customer messages from being delivered, and rules from being executed for some users.
Our engineering team is actively investigating and monitoring the system.
Mar 17, 12:31 UTC