Over-optimized A/B tests remind me of The Pepsi Challenge and the bias we have in the moment toward things we might not ultimately like better.
Pepsi likely outperformed Coke because a single small sip creates a bias towards the sweeter option, Pepsi. TVs and packaging on store shelves compete on brightness, music competes on loudness, and headlines compete to be more sensational. While these things might perform better in A/B tests where sales and clicks matter, that doesn’t mean we like them better.
related:
https://www.trustmeimascientist.com/2012/01/01/the-zen-art-of-choosing-speakers/