A/B testing is the act of running two different versions of the same website as part of a controlled experiment and gather data on which performs or converts the best.
In an A/B test, half of the users landing on the website will see the original or 'control' version A, and the other half will see a 'variation' B that features a change or group of changes such as a different header, images, call to action, page structure, etc.
This methodology is used to test measurable elements of a website that can affect a visitor’s decision to convert—including page structure, registration and sign-up forms, calls to action.
After testing enough visitors to have a statistically significant sample size, you’ll know which version converts more users. You will then use the winner of the A/B test as the control in future versions of the website, and any new variations will have to perform even better to become the new control.
A/B testing removes gut-feel decision-making and guesswork from CRO: A/B testing lets the users decide, and they vote with their clicks and conversions.
Without the help of A/B testing, optimization teams are more likely to make changes based on so-called ‘best practices’ or intuition, or based on the Highest Paid Person’s Opinion (HiPPO).
What’s wrong with that?
Many people think that A/B testing and CRO are the same thing—that CRO is just a series of A/B tests that allow optimizing teams to stumble across new ideas—but that’s missing the point.
A/B testing is the last step in the CRO process, and it’s never about coming up with new ideas. It’s about testing ideas that came about by studying your market and your users. In fact, before you A/B test anything, you need to come up with some evidence-based hypotheses about how to improve user experience and boost conversions.
In other words, before you even start thinking about A/B testing, you need to answer questions like:
It's only after having collected some (or all) of this data that you and your optimization teams will be ready to formulate a hypothesis and start thinking about a testable B variation for your website that addresses the issues and offers potential solutions.