Our User Experience practice at Astound uses a variety of tools to help our clients improve their eCommerce performance. They range from very broad (Personas), to very specific (A/B Tests). Solving a problem in the best way means both picking the appropriate tool, and being aware of possible shortcomings or side effects of that tool.
User Testing, Usability Testing and A/B Tests are some of my favorite methodologies for getting results in UX. They are all vulnerable to a problem called ‘local optimization.’
To illustrate this problem, imagine we’re trying to improve the performance of a call to action by experimenting with the text. We set up a series of tests with different text, alter the variations. After painstaking weeks of testing, we’ve improved the performance by 20%. Victory!
You can view this set of changes as climbing toward the top of the hill. The closer you are to the top, the closer you are to the optimum experience.
But, your laser-like focus on optimizing this text has caused you to miss the fact that there is a better solution that involves completely scrapping this call to action and using a different approach. You can imagine this as a mountain next to your hill. You’ve climbed the hill successfully, but you should have climbed the mountain. And you’ll never A/B test your way to the mountain. You’ve hit a local maximum that’s preventing you from finding better solutions.
So what does this mean? Well it does not mean you should stop A/B testing. It just means you should be aware of the pitfalls, and take steps to counter them. This could mean adopting a different UX approach (User Interviews instead of A/B testing), or it could mean that you need to broaden your A/B tests. (Instead of testing very narrow variations, test much larger variations). One issue with A/B tests is that the difficulty of setting them up and managing the number of variables in the test encourages testing small variations.
This is why we advocate incorporating multiple UX and analytic techniques into your UX practice, and making them a regular part of your cadence. Monthly moderated or unmoderated user testing, heuristic audits, best practices research, user interviews or other research, A/B testing and analytics can give you both broad and narrow perspectives to allow you to continuously improve the shopping experience on your eCommerce site.