On September 25th and 26th, 2025, we experienced a service disruption that impacted a subset of our customers. We want to be transparent about what happened, the steps we took to restore services, and how we are ensuring this does not occur again.

What Happened

In preparation for Black Friday and Cyber Monday, our team performed proactive tuning on our largest and oldest PostgreSQL database to improve performance and reliability during the upcoming peak season.

During this process, a rare and unexpected interaction between PostgreSQL vacuum and replication settings caused replication deadlocks at the buffer and storage level. These issues were not immediately apparent; the system operated normally for several hours before degradation began. At that point, the replication lag caused stale data to be served to some customers.

How We Responded

Once the problem became visible, our engineers:

• Investigated and identified the replication deadlocks.
• Performed additional maintenance operations to restore replication stability.
• Re-routed traffic to healthier database instances to minimize further disruption.

Despite these efforts, customers experienced disruptions and stale data before full service was restored.

Why This Happened

Individually, the database settings applied were not harmful. However, combined with the multi-terabyte scale of our datasets and the disk throughput required for cleanup operations, they created a failure condition that PostgreSQL could not automatically recover from.

What We’re Doing to Prevent This in the Future

We have launched initiatives to strengthen resilience and prevent this type of issue from recurring:

• Dataset optimization: Reducing the size of the datasets within this database to improve efficiency and recovery times.
• Resource scaling: Expanding available compute and disk throughput for large PostgreSQL instances.
• Resilience improvements: Investing in architectural changes that make the system more tolerant to replication failures.
• Enhanced monitoring: Implementing earlier detection for replication lag and vacuum anomalies.

Our Commitment

We sincerely apologize for the disruption this incident may have caused to your business. Reliability is our highest priority, and we are taking concrete steps to ensure the stability of our systems moving forward.

If you have any questions or concerns, please don’t hesitate to contact our support team.

Thank you for your understanding and trust.