How to conduct effective, user-centered A/B testing in 5 steps

This guide shows you how to conduct effective A/B testing to consistently ship website changes your users love.

It gets better! Our detailed walkthrough also explains how to know which version is winning users' hearts and uncover why that’s the case. Spoiler alert: the key to unlocking the why lies in using complementary tools like Hotjar (yes, that's us!).

Excited to see how? Let's begin.

How to do A/B testing the right way: 5 steps for success

A/B testing, sometimes called split testing, is your best bet in setting the direction for your teams—whether that's devs, user experience (UX), product, marketing, or design. While relatively simple to set up, impactful A/B testing involves more than you think.

Combining test results with recordings, heatmaps, and user feedback tools allows you to evaluate performance, spark new ideas, and encourage user-led innovation like never before. It also reduces the time needed to run tests that yield valid, actionable results—but more on that later.

First, let’s look at how to run A/B testing the right way in five easy steps.

Step 1: formulate an evidence-based hypothesis

It all starts with an idea. For example:

• A streaming platform wants to update its personalization system to help users find more content, with the goal of boosting customer satisfaction and retention

• An ecommerce store wants to streamline the sign-up flow for new users, leading to a potential sales uplift

But an A/B test must run on more than assumptions. It requires some informal research—digging into available data to generate ideas worth testing. Commonly, teams formulate a hypothesis based on blended quantitative and qualitative insights—like top exit page lists combined with exit-intent survey responses. 

🧪 What makes a good hypothesis?

 A good hypothesis is an educated and testable statement that:

• Proposes a solution or explanation to a problem

• Predicts an outcome for the experiment

• Provides reasoning for the expected result 

Here's a sample hypothesis we grabbed from Netflix:

When the user experience (UX) wizards at Netflix were planning to test the 'Top 10' lists we see on the web user interface (UI), the hypothesis went:

Step 2: select your A/B testing tool

Next, find a suitable A/B testing tool to run your experiment. Here’s a quick rundown of some of the top options:

Note: steps 2 and 3 are interchangeable. You can decide on the nitty-gritty of your test before settling on a tool with all the features you need to run your experiment effectively. 

Step 3: set up your experiment

Remember, your hypothesis clarifies what you're trying to change and the primary metric you're measuring. When you finalize your hypothesis, proceed to:

• Create two versions of…something. Anything!—an email subject line, a call-to-action (CTA) button, or a landing page layout. The baseline or 'control' version A displays the incumbent element, design, or page. At the same time, ‘variation’ B deploys with the change or group of changes you want to study. Both versions should be identical in all other ways.

• Take a subset of your users or members to act as your sample. Then, split your sample evenly into two groups using random assignment. One group sees the control version, and the other encounters the variation.

• Decide what your confidence level should be. Confidence level determines how sure you can be of your results. In market research, it's common to use a 95% confidence level. This means if you ran the experiment 20 times, you'd get the same results (with a margin of error) about 19 times.

• Determine the sample size and timeframe for your split tests. Depending on your A/B testing tool, you can set the sample size and test duration. If you need help deciding what to choose and there's no analyst to ask, try this sample size calculator.

• Set up and run your experiment on your chosen A/B testing tool. A/B test landing pages, CTAs, headers, and other variables one at a time for the same marketing campaign to ensure the causal chain will be linked back to a specific change or set of changes instead of a jumble of factors.

Step 4: analyze your A/B test results

When analyzing test results in your A/B testing tool, paying attention to statistical significance is super important. You don’t want to push forward with a version that only appears superior because of a sampling error or random chance. Generally, a statistical significance of 95% indicates you have a clear winner to give the go-ahead. 

Celebrate if you've got a statistically significant outcome, disable the losing variation in your A/B testing tool, and move forward with the winning version! 🎉

But what if neither variation is statistically better? In that case, you'll have to mark the test as inconclusive, indicating that the variable you tested didn't significantly impact the results. 

Don't worry, you can still learn from the experience: stick with the original version or use the failed data to help you create a fresh iteration for a new test.

For instance, imagine your ecommerce marketing team just finished A/B testing two different product page layouts. You've collected data on click-through rates, add-to-cart rates, and sales, but the results show no clear winner. 

Now what?

You could take a closer look at the data to see if there are any user subgroups that respond differently to the variations. Or, you could try a new test with a different variable, such as one involving alternative product descriptions or pricing strategies.

Step 5: view session recordings of your experiments

With the help of complementary A/B testing tools offering deeper behavior analytics insights, look at session recordings to see exactly how users interact with your experiment variant—where they move, what they click, and what they ignore. 

This way, you won’t just know which version is the winner—you’ll also know exactly why.

Here are some ways to analyze your recordings and add qualitative insights to your product experimentation method:

• Keep an eye out for engagement and expected behavior, but also note any unexpected signs of friction, like rage clicks

• Follow the visitor's path throughout the recording to see where the tested element fits into the user journey and if it's making the experience easier or changing the trajectory

A/B testing best practices with Hotjar

As we mentioned, you can also set up Hotjar's tools outside of the integrations to help monitor A/B tests and add depth to your A/B testing with the qualitative data these tools provide. You'll know what to do (or not do) next, ensuring no resources are wasted on ineffective changes.

Show a feedback widget or survey on a variant

Get user feedback on a specific page variant with Events in Hotjar. In this case, add the relevant event code to trigger the feedback widget or on-site survey. 

Keep in mind that events take priority over URL-based targeting rules, so they'll show up regardless of any URL-specific rules you've set. 

Need help setting up events? Check out our Events API documentation.

Compare heatmaps for your control and variant

Track your A/B test using heatmaps, depending on the following conditions:

1. When your variations are on the same URL: this means your A/B test is set up to randomly load any of your two variations for each user visiting the same page. In this case, use Events in Hotjar. Set up unique event names for your control and variant and apply session filters to view their corresponding heatmap.

2. When your variations are on different URLs: use the heatmap URL filter to track data for each variation. Select a heatmap URL filter and enter your control URL and/or variant URL, then view the corresponding heatmap. Here are the guidelines on how to choose the correct heatmap URL filter for your control and variant.

Note: implementing Events in Hotjar may involve adding JavaScript to your site or using Google Tag Manager or Segment. Read more of the technical stuff here before getting started.

Watch recordings of users exposed to the variant

Recordings show you how users behave on your site or page variations firsthand. Hotjar's Events API passes an event to Hotjar whenever users land on a specific version of an A/B test page. Use the event filter to isolate sessions where a particular variant was visited and see how people interacted with it.

Deliver conversion-driven changes consistently

Now there's nothing left to do but to take those steps. Start A/B testing even the tiniest tweak before shipping it to your entire user base. And combine multiple sources of qualitative data, like heatmaps, session recordings, and user surveys, to uncover even more opportunities that aren’t obvious when looking at test results alone.

Ensure you're not making changes that could hurt your conversion rates or user engagement—and waste precious company resources—when you enhance your experiments with Hotjar.