A Split Testing Conundrum

One of the big advantages you get in digital marketing and commerce is the ability to programatically deliver unique content to every individual user. With this dynamic nature you can test everything, and in it’s theoretical ideal state, each customer would see exactly the content that would make them buy at the highest price they would be happy to spend.

We don’t have a way to build this level of customization and targetting yet where everyone gets a custom price and message. Our technology is a bit more fragmented. The ads on the page might be customized to things you’ve visited before and your demographics. A lot of money has gone into building incredibly advanced ad serving platforms. The rest of the page is usually a lot dumber.

Advanced websites with lots of traffic might explore multi-variate tests. These typically are changing many individual elements on a page at once. Each person gets a unique page in the hope that you narrow down the set of colors, styles, images and text that optimizes for the best outcome. Multi-variate tests require a lot of traffic and can be difficult to setup. It’s a tool that people aspire to use and then usually fail to execute on because of the complexity.

Slightly easier is a split test with a smaller set of options. When testing one item at a time, it requires less traffic to get a statistically strong result. Doing a split test sounds like it should be an easy and great way to confirm that a design change is worth making – is the green buy button better than the blue one at driving sales? But human psychology makes this a harder thing to do in reality.

In real life, people ‘know’ the better price, the better color or the best photo to use. The designer hates the green button because it clashes with the navigation bar. The sales team ‘knows’ that the lowest price will drive the most sales. Everyone has an opinion on the best photo.

There are problems with this:

• The people with these opinions are not customers getting ready to open their wallet to buy. The goals are not aligned.
• For every potential test, people ‘know’ which will win so why test an inferrior option and lose sales to the people who are served that one?
• The person with the most convincing argument or with authority often wins

The scientific approach to business is based on hypothesis. This framing helps remove ego. Instead of statements like “I like this logo because it is simpler/cleaner/funky/fun/etc”, you propose a potential outcome: “I believe this logo could be recognizable 20% faster, and allow us to lift prices by 5% to luxury levels without impacting sales volumes.” Now you have a testable hypothesis. sometimes all you need is one person on the team to discuss things this way and elevate things beyond instinctual decisions and towards conscious and deliberate strategy.