1 / 3
See you Monday
June had the app alert customers whenever the warehouse touched an order. The re-check touched every order, so 6,000 people heard their 2021 sofa was coming.
The mistake
The customer message fires on any write to an order record (the warehouse "touching" it), not on a real change of status to shipped. Nothing records whether a customer has already been told. The warehouse runs an hourly job that re-saves every order it has ever held, so every historical order counts as a fresh event and its customer gets a message. The off switch is a bulk update that writes a flag onto every order, and that write is itself a touch, so the still-subscribed handler messages everyone again. The fix is to trigger on an actual state change, such as status going from not shipped to shipped, and to keep an idempotency record of each notification sent.
- notifications
- mobile-app
- release
- warehouse
- customer-support