1 / 2
The left-hand box
The shop went down, so Russ restarted the grey box he was certain he had labelled in 2019. It was the wages machine, and everyone got paid twice.
The mistake
Two identical machines sit side by side with no reliable identification — the only labelling was done by hand years ago and has since fallen off, so the operator is working from memory and a colleague's equally unfounded confirmation rather than from any check against the running system. Nobody verifies what the box actually does before power-cycling it: no hostname lookup, no console, no glance at the machine's own status display, which is showing a payroll job in progress. The result is that a restart intended for the shop server is applied to the payroll server mid-run, re-triggering a completed pay cycle, while the shop stays down because it was never touched.
- sysadmin
- payroll
- ai-tooling
- on-call
- office