Nine minutes on a sofa

The mistake

An A/B test is scored on time-on-page, a proxy metric that assumes longer engagement means more interest. The challenger page is built by stripping out every element that appears on an analytics report of rarely-clicked components, which removes the Buy button along with the genuine dead weight. Visitors then linger for nine minutes because they are hunting for a checkout control that no longer exists, so the proxy metric climbs while the conversion rate it was standing in for goes to zero.