Help Center is not working
Incident Report for Gorgias
Postmortem

On October 9th, we had an incident from 12:49 AM to 2:30 PM (UTC) that resulted in the Help Center feature being down. The underlying reason for this outage was an unexpected "hold" status placed on our domain, gorgias.help, by our domain registrar due to suspicions of malicious activity on one of the subdomains. Their policy is notably very strict; they reserve the right to place a "hold" status on the main domain in a very short amount of time, even if malicious activity is detected only on a single subdomain. While they did communicate their concerns to us before blocking the domain, the notification was sent over the weekend and unfortunately did not catch our immediate attention.

Unfortunately, even after we fixed the issue on our end, there was a delay from the registrar's side to lift the block, making the downtime for our Help Centers even longer. We understand this is unacceptable and every second of uptime on our platform is critical to our customers' business.

What are our future mitigation plans?

  • Implementing measures to improve communication with external partners
  • Adding additional metrics and alarms
  • Preparing more training exercises for on-call staff to improve incident resolution times

On behalf of the team at Gorgias, please accept our sincerest apology for this incident - we’ll be working hard to improve the stability and earn the trust you put in us

Posted Oct 09, 2023 - 07:27 PDT

Resolved
This incident has been resolved.
Posted Oct 09, 2023 - 07:23 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 09, 2023 - 05:41 PDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 09, 2023 - 02:44 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 09, 2023 - 02:02 PDT
This incident affected: Help Center (Customer Help Center website, Custom Domains).