On September 25th and 26th, 2025, we experienced a service disruption that impacted a subset of our customers. We want to be transparent about what happened, the steps we took to restore services, and how we are ensuring this does not occur again.
In preparation for Black Friday and Cyber Monday, our team performed proactive tuning on our largest and oldest PostgreSQL database to improve performance and reliability during the upcoming peak season.
During this process, a rare and unexpected interaction between PostgreSQL vacuum and replication settings caused replication deadlocks at the buffer and storage level. These issues were not immediately apparent; the system operated normally for several hours before degradation began. At that point, the replication lag caused stale data to be served to some customers.
Once the problem became visible, our engineers:
Despite these efforts, customers experienced disruptions and stale data before full service was restored.
Individually, the database settings applied were not harmful. However, combined with the multi-terabyte scale of our datasets and the disk throughput required for cleanup operations, they created a failure condition that PostgreSQL could not automatically recover from.
We have launched initiatives to strengthen resilience and prevent this type of issue from recurring:
We sincerely apologize for the disruption this incident may have caused to your business. Reliability is our highest priority, and we are taking concrete steps to ensure the stability of our systems moving forward.
If you have any questions or concerns, please don’t hesitate to contact our support team.
Thank you for your understanding and trust.