We apologize for the degraded state of the us-central1-d8ff cluster during this incident yesterday.

Some background tasks were blocking database queries while waiting for a response from outgoing HTTP requests. Unfortunately, due to problems on the 3rd party’s end, we were not able to receive a response form each HTTP request in a timely manner, which slowed down the helpdesk and other worker tasks.

We were able to disable such calls once this root cause was found and have a permanent fix that will be rolled out today to avoid database blocking for this specific scenario in the future.