Learn / Guides / CRO glossary
Multivariate testing: a definition
What is multivariate testing? Multivariate testing, or MVT, is the act of combining and testing multiple variables on a website as part of a controlled experiment, to determine which combination produces the most conversions. Multivariate testing enables companies to validate hypotheses before implementing them at scale, reducing risk and improving upside.
While an A/B test compares a standard ‘A’ version of a landing page with a modified ‘B’ one, multivariate testing changes more than one variable, testing all resulting combinations against each other at the same time. For example, a multivariate test that studies two images and four calls-to-action (CTAs) would test eight versions simultaneously, letting. you determine which one is most highly optimized.
How many versions are required for a given test?
Use the following formula to calculate how many websites versions you need for a given multivariate test:
[# of variations for first element] x [# of variations for second element] = total number of versions to test.
Here’s a practical example: you're building a clickable button that requires an image and a line of text. Based on research, you have narrowed your options down to:
2 different images (a star and a smiley face)
3 different lines ('try now', 'buy now' and 'order now')
How many versions do you need?
[2 images] x [3 CTAs] = 6 versions
Why is multivariate testing important?
Just like A/B testing, multivariate testing removes the guesswork from Conversion Rate Optimization (CRO) and lets you test your optimization theories. Users let you know which website version is likely to produce the most conversions, and you can make changes to your website accordingly.
Two advantages of MVT over A/B testing
Multivariate testing has the following advantages:
There’s no need to run a lengthy series of sequential A/B tests.
Instead, the tests all run simultaneously: this enables rapid knowledge-gathering on a greater set of possible combinations.
You get to quickly see how a series of variables interact with one another.
For example, one version of a CTA may work better than another CTA as a standalone change, but its performance could drop when combined with a different image. Multivariate testing shows you how a set of page variations combine to increase or decrease conversions.
The disadvantage of multivariate testing
Most A/B split testing software allows you to run multivariate tests and easily calculate the results, but there is one potential disadvantage: multivariate tests require a higher amount of traffic to achieve statistical significance than A/B tests simply because there are more pages to test.
Before running a multivariate test, use a sample size calculator to estimate how much traffic you’ll need per variation to reach statistically significant results. If the sample size you need makes your test impractical to run, reduce the number of variables and/or test only the most important changes.
Multivariate testing is NOT the way to stumble across new ideas
CRO is not just a series of A/B or multivariate tests, and multivariate testing is not about discovering new ideas. In fact, testing is the last step in the CRO process.
Before you test anything (or even start brainstorming different versions of landing pages), first come up with data-driven hypotheses about how to build a better experience and increase conversions. That means answering questions like:
Where do visitors leave your website?
Which elements do visitors engage with?
Which elements do visitors ignore?
What do visitors want to achieve and how can your website help them accomplish those goals?
6 steps to finding variables to test
Before you commit to this testing method, it pays to do your due diligence and understand the prerequisites for a successful test that gets meaningful results:
Conduct informal research: study product reviews and look at Customer Support feedback to see what people are saying about you and your products. Speak with your designers, Sales, and Support staff.
Figure out where people leave: use tools (such as Google Analytics) and our to see where visitors leave your website.
Find out which page elements people engage with: show where the majority of users click, scroll, and hover their pointers (or tap their fingers on mobile devices). Spot trends that show how visitors interact with the most important pages, and zero-in on elements that might impact conversions if changed.
Gather customer feedback: use, feedback widgets, and surveys to collect open-ended feedback about how visitors see your website and what they really want from it.
Look at: see individual users (anonymized) as they work their way through your website. Note what they do right before leaving.
Explore usability testing: provide insight into how people use your site. Gather direct feedback about issues that visitors encounter and solutions they’d like to see.
Once you collect some solid data, it’s time to formulate a hypothesis and start thinking about testable elements. Focus your energy on the items that are likely to offer the biggest return, and balance your desire to test many variables with the traffic and resources at your disposal.
Understand your visitors
Get profound insights into visitor drives and behaviors—so you can study the elements that have the biggest impact on conversion.
