Last Updated Jul 29 2019

CRO glossary: multivariate testing

What is multivariate testing?

Multivariate testing is the act of combining and testing multiple variables on a website as part of a controlled experiment, to determine which combination produces the most conversions.

While an A/B test compares a standard ‘A’ version with a modified ‘B’ one, multivariate testing changes more than one variable, testing all resulting combinations against each other at the same time. For example, a multivariate test that studies two images and four calls-to-action (CTAs) would test eight versions simultaneously.

 

 

How many versions are required for a given test?

Use the following formula to calculate how many websites versions you need for a given multivariate test:

[# of variations for first element] x [# of variations for second element] = total number of versions to test. 

Here’s a practical example: you're building a clickable button that requires an image and a line of text. Based on research, you have narrowed your options down to:  

  • 2 different images (a star and a smiley face)
  • 3 different lines ('try now', 'buy now' and 'order now')  

How many versions do you need? 

[2 images] x [3 CTAs] = 6 versions

multivariate-testing

Why is multivariate testing important?

Just like A/B testing, multivariate testing removes the guesswork from Conversion Rate Optimization (CRO). Users let you know which website version is likely to produce the most conversions, and you can make changes to your website accordingly.

Two advantages of multivariate testing over A/B testing

Multivariate testing has the following advantages:

  • There’s no need to run a lengthy series of sequential A/B tests. Instead, the tests all run simultaneously.

  • You get to quickly see how a series of variables interact with one another. For example, one version of a CTA may work better than another CTA as a standalone change, but its performance could drop when combined with a different image. Multivariate testing shows you how a set of variables combine to increase or decrease conversions.

The disadvantage of multivariate testing

Most A/B testing software allows you to run multivariate tests and easily calculate the results, but there is one potential disadvantage: multivariate tests require more traffic to achieve statistical significance than A/B tests simply because there are more pages to test. 

Before running a multivariate test, use a sample size calculator to estimate how much traffic you’ll need per variation to reach statistically significant results. If the sample size you need makes your test impractical to run, reduce the number of variables and/or test only the most important changes.

Multivariate testing is NOT the way to stumble across new ideas

CRO is not just a series of A/B or multivariate tests, and multivariate testing is not about discovering new ideas. In fact, testing is the last step in the CRO process

Before you test anything, first come up with data-driven hypotheses about how to build a better experience and increase conversions. That means answering questions like:

  • Where do visitors leave your website?
  • Which elements do visitors engage with?
  • Which elements do visitors ignore?
  • What do visitors want to achieve and how can your website help them accomplish those goals?

6 steps to finding variables to test

  1. Conduct informal research: study product reviews and look at Customer Support feedback to see what people are saying about you and your products. Speak with your designers, Sales, and Support staff.

  2. Figure out where people leave: use traditional analytics tools (such as Google Analytics) and our conversion funnels tool to see where visitors leave your website.

  3. Find out which page elements people engage with: heatmaps show where the majority of users click, scroll, and hover their pointers (or tap their fingers on mobile devices). Spot trends that show how visitors interact with the most important pages, and zero-in on elements that might impact conversions if changed.

  4. Gather customer feedback: use on-site polls, feedback widgets, and surveys to collect open-ended feedback about how visitors see your website and what they really want from it.

  5. Look at session recordings: see individual users (anonymized) as they work their way through your website. Note what they do right before leaving.

  6. Explore usability testing: usability testing tools provide insight into how people use your site. Gather direct feedback about issues that visitors encounter and solutions they’d like to see.


Once you collect some solid data, it’s time to formulate a hypothesis and start thinking about testable elements. Focus your energy on the items that are likely to offer the biggest return, and balance your desire to test many variables with the traffic and resources at your disposal.

Understand your visitors

Get profound insights into visitor drives and behaviors—so you can study the elements that have the biggest impact on conversion.

Free forever. Get started!