The best four minutes

The mistake

A spare/failover copy of the shop's environment has sat unused since 2019 and nobody has ever run a test cutover against it. When it's finally used to take over live traffic, it turns out to be a stale snapshot rather than a synced replica, so it serves the entire 2019 product catalog — including a discontinued item — to every current customer. There's no validation step before the migration that would have diffed the spare's contents against production and caught that it was six years out of date.