Database Migration Lock Causes 12-Hour Core Service Outage

A database migration estimated at five minutes locked a core customer-order table for 47 minutes, triggering a connection cascade that took 12 hours to fully resolve, the engineering team reported in a postmortem.

The product team needed a new nullable column on a table that had grown past 500 million rows. The migration tool ran an ALTER TABLE statement. PostgreSQL initiated a full table rewrite and held a write lock during the operation. The five-minute estimate had assumed a table half that size, the team said.

Read traffic kept hitting the table during the lock. Inserts and updates were blocked. Connections queued behind the migration transaction. Application servers exhausted the database connection pool. Kubernetes recycled pods, which opened more connections and worsened the exhaustion. The monitoring system never fired a critical alert because CPU and memory remained below thresholds. The incident was connection starvation, not a resource spike. The first human signal was a customer support ticket asking if the site was down.