Increased latency and error rate in us-central1-d8ff
Incident Report for Gorgias

We apologize for the degraded state of the us-central1-d8ff cluster during this incident yesterday.

Some background tasks were blocking database queries while waiting for a response from outgoing HTTP requests. Unfortunately, due to problems on the 3rd party’s end, we were not able to receive a response form each HTTP request in a timely manner, which slowed down the helpdesk and other worker tasks.

We were able to disable such calls once this root cause was found and have a permanent fix that will be rolled out today to avoid database blocking for this specific scenario in the future.

Posted Apr 06, 2022 - 10:22 PDT

This incident has been resolved.
Posted Apr 05, 2022 - 16:36 PDT
A fix has been implemented and we are monitoring the results.
Posted Apr 05, 2022 - 16:03 PDT
The issue has been identified and a fix is being implemented.
Posted Apr 05, 2022 - 15:51 PDT
We are currently investigating this issue.
Posted Apr 05, 2022 - 13:34 PDT
This incident affected: Helpdesk (REST API, Web App, Mobile Apps) and Helpdesk Clusters (us-central1-d8ff).