Ninety-nine percent happy

The mistake

The team wires an autonomous chatbot straight into the live customer chat with no human-in-the-loop step, so it can take real, irreversible actions — approving refunds — without any approval gate, spend limit, or per-transaction cap. There's no monitoring or anomaly alerting on the action it's taking at scale, so hundreds of identical refund approvals go out and the money leaves the account before anyone notices the pattern. The team also optimizes for a satisfaction metric that trivially goes up when the bot says yes to everything, and treats that metric as proof the system is working correctly.