Whitepaper on Website A/B Testing

Dare blogged about an interesting whitepaper from the Microsoft Experimentation Team about the concept of A/B testing and applications for websites. 

"Practical Guide to Controlled Experiments on the Web: Listen to your Customers not to the HiPPO"

Note that HiPPO = Highest Paid Person's Opinion.

A/B testing (briefly, the concept of varying one or more settings/behaviors on a site for controlled subsets of users and analyzing resulting behaviors) was used extensively at my last two companies before joining Microsoft, so this was a good read.  Section 5, Lessons Learned, is alone very worthwhile.

Recommended reading even if you aren't running a web application since the concepts could be applied to a variety of systems (but the inherent flexibility of websites makes them primary candidates for such testing.)  Using A/B testing, you may be surprised to see how seemingly minor changes to a site can lead to major improvements in rates of use, clickthroughs, purchases, etc.

-Chris