Multivariate testing is the act of combining and testing multiple variables on a website as part of a controlled experiment, to determine which combination produces the most conversions.
While an A/B test compares a standard ‘A’ version with a modified ‘B’ one, multivariate testing changes more than one variable, testing all resulting combinations against each other at the same time. For example, a multivariate test that studies two images and four calls-to-action (CTAs) would test eight versions simultaneously.
Use the following formula to calculate how many websites versions you need for a given multivariate test:
[# of variations for first element] x [# of variations for second element] = total number of versions to test.
Here’s a practical example: you're building a clickable button that requires an image and a line of text. Based on research, you have narrowed your options down to:
How many versions do you need?
[2 images] x [3 CTAs] = 6 versions
Just like A/B testing, multivariate testing removes the guesswork from Conversion Rate Optimization (CRO). Users let you know which website version is likely to produce the most conversions, and you can make changes to your website accordingly.
Multivariate testing has the following advantages:
Most A/B testing software allows you to run multivariate tests and easily calculate the results, but there is one potential disadvantage: multivariate tests require more traffic to achieve statistical significance than A/B tests simply because there are more pages to test.
Before running a multivariate test, use a sample size calculator to estimate how much traffic you’ll need per variation to reach statistically significant results. If the sample size you need makes your test impractical to run, reduce the number of variables and/or test only the most important changes.
CRO is not just a series of A/B or multivariate tests, and multivariate testing is not about discovering new ideas. In fact, testing is the last step in the CRO process.
Before you test anything, first come up with data-driven hypotheses about how to build a better experience and increase conversions. That means answering questions like:
Once you collect some solid data, it’s time to formulate a hypothesis and start thinking about testable elements. Focus your energy on the items that are likely to offer the biggest return, and balance your desire to test many variables with the traffic and resources at your disposal.