Hotjar + Contentsquare 🎉 Hotjar + Contentsquare: we’re joining forces to build better experiences for all. 🎉 Learn more

Last updated Jan 28 2021

Leader in Spring 2021

Multivariate testing: a definition

What is multivariate testing?

Multivariate testing, or MVT, is the act of combining and testing multiple variables on a website as part of a controlled experiment, to determine which combination produces the most conversions. Multivariate testing enables companies to validate hypotheses before implementing them at scale, reducing risk and improving upside.

While an A/B test compares a standard ‘A’ version of a landing page with a modified ‘B’ one, multivariate testing changes more than one variable, testing all resulting combinations against each other at the same time. For example, a multivariate test that studies two images and four calls-to-action (CTAs) would test eight versions simultaneously, letting. you determine which one is most highly optimized.

How many versions are required for a given test?

Use the following formula to calculate how many websites versions you need for a given multivariate test:

[# of variations for first element] x [# of variations for second element] = total number of versions to test. 

Here’s a practical example: you're building a clickable button that requires an image and a line of text. Based on research, you have narrowed your options down to:  

  • 2 different images (a star and a smiley face)
  • 3 different lines ('try now', 'buy now' and 'order now')  

How many versions do you need? 

[2 images] x [3 CTAs] = 6 versions

How many versions are required for a given test?

Why is multivariate testing important?

Just like A/B testing, multivariate testing removes the guesswork from Conversion Rate Optimization (CRO) and lets you test your optimization theories. Users let you know which website version is likely to produce the most conversions, and you can make changes to your website accordingly.

Two advantages of MVT over A/B testing

Multivariate testing has the following advantages:

  • There’s no need to run a lengthy series of sequential A/B tests. Instead, the tests all run simultaneously: this enables rapid knowledge-gathering on a greater set of possible combinations.

  • You get to quickly see how a series of variables interact with one another. For example, one version of a CTA may work better than another CTA as a standalone change, but its performance could drop when combined with a different image. Multivariate testing shows you how a set of page variations combine to increase or decrease conversions.

The disadvantage of multivariate testing

Most A/B split testing software allows you to run multivariate tests and easily calculate the results, but there is one potential disadvantage: multivariate tests require a higher amount of traffic to achieve statistical significance than A/B tests simply because there are more pages to test. 

Before running a multivariate test, use a sample size calculator to estimate how much traffic you’ll need per variation to reach statistically significant results. If the sample size you need makes your test impractical to run, reduce the number of variables and/or test only the most important changes.

Multivariate testing is NOT the way to stumble across new ideas

CRO is not just a series of A/B or multivariate tests, and multivariate testing is not about discovering new ideas. In fact, testing is the last step in the CRO process

Before you test anything (or even start brainstorming different versions of landing pages), first come up with data-driven hypotheses about how to build a better experience and increase conversions. That means answering questions like:

  • Where do visitors leave your website?
  • Which elements do visitors engage with?
  • Which elements do visitors ignore?
  • What do visitors want to achieve and how can your website help them accomplish those goals?

6 steps to finding variables to test

Before you commit to this testing method, it pays to do your due diligence and understand the prerequisites for a successful test that gets meaningful results:

  1. Conduct informal research: study product reviews and look at Customer Support feedback to see what people are saying about you and your products. Speak with your designers, Sales, and Support staff.

  2. Figure out where people leave: use traditional analytics tools (such as Google Analytics) and our conversion funnels tool to see where visitors leave your website.

  3. Find out which page elements people engage with: heatmaps show where the majority of users click, scroll, and hover their pointers (or tap their fingers on mobile devices). Spot trends that show how visitors interact with the most important pages, and zero-in on elements that might impact conversions if changed.

  4. Gather customer feedback: use on-site polls, feedback widgets, and surveys to collect open-ended feedback about how visitors see your website and what they really want from it.

  5. Look at session recordings: see individual users (anonymized) as they work their way through your website. Note what they do right before leaving.

  6. Explore usability testing: usability testing tools provide insight into how people use your site. Gather direct feedback about issues that visitors encounter and solutions they’d like to see.

Once you collect some solid data, it’s time to formulate a hypothesis and start thinking about testable elements. Focus your energy on the items that are likely to offer the biggest return, and balance your desire to test many variables with the traffic and resources at your disposal.

Understand your visitors

Get profound insights into visitor drives and behaviors—so you can study the elements that have the biggest impact on conversion.

Free forever. Get started!

There are two types of multivariate test:

  • Full factorial testing → this type of MVT test will test every single combination of the options available. In the example given above, [2 images] x [3 CTAs] = 6 versions. To achieve statistical significance, this type of test requires a lot of traffic to each variation.
  • Fractional (or partial) factorial → this type of MVT test will test a smaller sampling or subset of all the available options. This requires less traffic, but—because it only tests a subset of all available options—is not exhaustive; it's mostly useful to get a general sense of whether a variation is directionally better or worse than the others.

Unlike multivariate or MVT testing, where different page element combinations are tested against each other all at once, an A/B/N test is used to test different versions of a page against each other. When the versions are two, the test is an A/B test (where A: control and B: variation); when there are three versions, the test is an A/B/C test, and so on.
This type of test can be useful when:

  • You want to compare more 'drastically' different versions of page layout and design (unlike an MVT test, where all variations are more subtly different from one another)
  • You don't have as much traffic as an MVT test would require to reach statistically significant conclusions.

Any company can use MVT testing—ecommerce, marketplaces, SaaS, etc.—provided they have sufficient traffic to reach statistically significant results.

We don't have a recommendation—the choice depends on your budget, setup, needs, etc.—but here are some of the most commonly used MVT tools:

  • Optimizely
  • AB Tasty
  • Adobe Target
  • Webtrekk
  • Oracle Maxymiser
  • Monetate
  • Omniconvert
  • VWO