Ships It Friday
1 / 2
One digit wrong
Russ mistyped a digit of his own number when he set up the out-of-hours alarms. For years they went to a stranger called Brian, who rings in to pass them on.
The mistake
The on-call alerting system pages a phone number stored in its config, and back in 2019 whoever set it up fat-fingered one digit of Russ's number when entering it. Nobody validated the number against a real test page, so it silently routed every out-of-hours alarm to a stranger's phone instead of failing loudly or bouncing. The typo has never been caught or corrected since, because the alerts were still "being received" by someone — just not anyone on the team.
- alerting
- on-call
- sysadmin
- ops
- process