Posts

Showing posts with the label Migrations

Your Dual-Write Migration Is Lying to You: Catching Silent Divergence Before Cutover

You flip the read path to the new database. Within a minute, a fraction of a percent of lookups start returning 404 , and a handful return data that's a few minutes stale. The write path has been dual-writing to both stores for three weeks. Every write returned 200 . The backfill job logged "complete." By every dashboard you had, the migration was done. It wasn't. The old store and the new store had been quietly drifting apart the entire time, and the cutover is exactly the moment you discover it — when the new store becomes the source of truth for reads and its gaps become user-visible. Why "both writes succeeded" doesn't mean "both stores match" The seductive thing about dual-write is that it looks correct locally. Your write path does something like this: def save_order(order): old_db.upsert(order) # legacy Postgres new_db.upsert(order) # Spanner return 200 Both calls return, you return 200 , and the r...