Move Fast, Break Things and Collect Data

With the help of unit tests, BDD, Coverage reports, Continuous Integration, and Source code control it has become easier than ever to build code that is robust against regressions while letting you branch and play with ideas quickly and without risk.  However there are times when the scale of code use in production requires a different approach to testing.

Staged rollouts through a distributed system are one way to test the waters with new code in production, but what happens when you can’t risk even 0.001% of traffic experiencing particular errors on critical pieces of code?

Tests can have the effect of making developers over-confident that their changes will not break the system.  I have seen this happen on production systems.

In the world of sales and marketing, split testing is a core concept for optimizing a sales message.  You put out multiple advertisements and measure which performs better against a set of criteria (more opt-ins, sales, links clicked etc.).  When the experiment reaches a statistical conclusion you clean out the losers and try again to beat your best performing advertisement.

I’d like to see this approach used on more than just marketing.

Split testing code would let you verify the accuracy of a new algorithm in real world usage, or measure the performance of different code paths.

People usually read about code performance benchmarks and complain that they are too simple, too specific or missing something that makes it hard to apply the conclusions of one test to another implementation. Measuring and comparing the code in your actual application scenario gives you real performance in your actual application.