10 A/B testing mistakes that ruin your experiments (and how to avoid them)

Here are 10 mistakes to avoid if you want to run the kind of A/B test that produces squeaky-clean data—and leads you to make informed product decisions.

They’re divided into three categories:

• Pre-testing mistakes

• Mid-testing mistakes

• Post-testing mistakes

Pre-testing mistakes 

In A/B testing, also known as split testing, you present two variations of a web page to users at random. One of them is the standard version (control), and the other is the new version (variant) of that page you hope might encourage more users to take the action you want them to.

A successful, error-free A/B test begins in the planning phase, with a thoughtful consideration of your hypothesis and test conditions. Here are some common A/B testing pitfalls teams make before they’ve even hit ‘run’ on their experiments—and how to avoid them. 

1. Not having a clear hypothesis

It’s easy to start an experiment because you’ve just got a vibe that a specific change might be a good idea in some way. However, an A/B test can only answer a closed-ended question, so you need to pin your hypothesis down. 

If you run a test without a clear hypothesis statement, you might get distracted by interesting but inconsequential metrics. For example, perhaps you end up focusing on the fact that users spent more time on the page instead of the metric you actually care about.

How to fix it: to produce useful split testing results, you need to specify which metric(s) you hope to increase and why the change you’re proposing could increase it—i.e. you need to create a working hypothesis.

When formulating a hypothesis, do these three things: 

• First, dig into web analytics data to observe what your users are doing on the path to conversion. For example: “Google Analytics shows that 95% of our customers who start checking out bounce before they enter their card details.”

• Then, speculate on what prevents users from converting—your best guess, based on the information available 

“I speculate that users aren’t converting because they don’t trust our creaky old site with their card details.”

• Next, formulate a hypothesis on a change that might encourage users to convert. This should be a statement that clearly sets out your A/B test. You can use the 'if-then' format. 

“I hypothesize that if we add a ‘secure payment’ icon to the checkout page, then more users will convert.”

2. Failing to consider the customer journey

Make sure you’re A/B testing something that has the potential to move your metrics: 

• Test important pages, like your product, checkout, or registration pages, that are high in traffic and connected to your sales funnel

• Don’t run A/B tests on inconsequential pages without much traffic—like your about us page, for example, or else you won’t be able to make impactful changes with the results

One way to check you’ve chosen an important page is to review your customer journey map—the steps that lead up to a customer doing business with you, such as users:

• Clicking an advert

• Reviewing a product page

• Reading shipping information

This stops you from investing too much effort into optimizing a page that customers only see once they’ve already made a conversion decision—or when they’re not even close to making one.

How to fix it: formulate hypotheses with your customer journey map in mind. Whichever page you’re planning to A/B test, ask yourself: 

• What decisions has the user already taken at this point? 

• Is there a chance they’re not even on the path to conversion yet? 

3. Running a test before you’ve got enough users

A/B testing gets so much hype from product management and marketing media that it’s tempting to consider it the best-in-class solution for all digital businesses. In reality, that’s not the case.

Unless your page has significant weekly traffic, your A/B test will struggle to achieve statistically significant results. It’s industry standard to consider 95% probability ‘statistically significant’, which means that if you ran the experiment 20 times, you’d likely get the same results 19 times. 

Many A/B testing tools will automatically send a notification when your test reaches this threshold. 

How to fix it: investigate whether you have enough traffic and conversions to run a statistically significant test by using a sample size calculator like this one from Neil Patel or this one from CXL. 

If you don’t have enough traffic, remember: A/B testing isn’t the be-all and end-all of conversion rate optimization (CRO). 

Directly ask your customers what they want to see with a survey instead, or try using a scroll heatmap to see how far down your page customers typically get.

4. Not considering mobile traffic

It’s easy to prioritize the desktop version of your site. But this is a massive oversight because mobile users are responsible for over 60% of all web traffic in 2023. 

The mobile blindspot is behind two big A/B testing mistakes:

1. Some teams don’t dig into analytics data for the mobile version of their sites when coming up with hypotheses. So, even if they run dozens of A/B tests, they’re only optimizing 40% of the user experience. 

2. Some teams will A/B test a design tweak or new feature idea and—in the rush to put something out—not check how it appears on mobile. This leads to a situation where a user-centric, well-considered hypothesis for the desktop version of your site produces inexplicably poor results from an A/B test. It seems like the idea wasn’t a hit when, really, it might have just looked bad on iOS.

How to fix it: remind your team to spend time with the mobile version of your site. Here’s how:

• When you set up an A/B test, investigate how it looks and feels on mobile before you hit ‘run’. Many testing platforms offer simple tools for this. For example, when you’re about to launch an A/B test in Google Optimize, there’s a ‘Preview’ tool that allows you to see how your changes will appear on other devices. 

• When you’re looking at site data to form hypotheses, filter your data by device. Many user insights platforms like Google Analytics allow you to segment your data in this way. Similarly, you can filter Hotjar recordings and heatmaps to watch how smartphone users navigate the mobile version of your site. 

For example, in the heatmap below, you see readers trying to click on the ‘Homepage design’ and ‘Responsive web design’ bullet points. This could mean you need to incorporate jump links more into your content strategy to give readers what they want faster.

Mid-testing mistakes

Getting your test off to a strong, error-free start is important, but there are still several common pitfalls to avoid whilst it’s up and running. 

5. Testing too many hypotheses at once

Testing multiple page elements at once may seem like an obvious way to save time and achieve a high-converting page ASAP—but it can be more trouble than it’s worth.

Here’s what ‘too many hypotheses’ looks like: say you’re aiming to increase sales to your online store. You decide to run an A/B test on a product page, where the ‘A’ version is the option you currently use, and the ‘B’ variant includes:

• An offer banner

• A bigger ‘Buy now’ button

• Extra product information

You’ll never know which tweak was responsible for the results, so you waste time and resources implementing changes without evidence they’re going to increase conversions. 

How to fix it: take your A/B tests one at a time, so you can understand key learnings from each one, and move forward methodically. But A/B testing isn’t the only way to validate that users will like your changes—and it’s certainly not the fastest. 

Here’s what else you can do:

• If you’ve got plenty of traffic and you’re certain you want to test more than one element at a time—perhaps you’ve got a big design relaunch and need to make sure everything’s in order—then try multivariate testing. This is when your website traffic is split into smaller segments: so instead of A/B testing, it might be A/B/C/D testing. 

• If you’ve got less traffic, ask customers to rate your radical changes using a good old-fashioned user survey

6. Running the test for too short a time 

Just like running an A/B test on too few users, cutting your test duration short can lead to unrepresentative results. Once again, you need to achieve statistical significance—at least a 95% chance that these results weren’t random. 

It’s often tempting to stop an A/B test that shows initial signs of success, especially if you were very confident about your initial hypothesis. But remember: without statistical significance, you’re gambling. The more data you collect, the more accurate your results will be.

How to fix it: stay disciplined and don’t switch off an A/B test before it’s reached that magic number: 95% significance. Most A/B testing applications will declare a winner at this point automatically. 

However, an even surer threshold is to wait until you’ve hit your pre-calculated minimum sample size, as mentioned in the third point.

7. Not noticing your testing tool slows down site speed

Some A/B testing tools might slow your site down by up to a second. After all, splitting your user base between two different versions of your site creates an extra step in the page-loading process. What’s more, 40% of users abandon a site that takes more than three seconds to load, so this can seriously hurt your metrics. 

If your site slows down, your conversion rate will probably go down, and your bounce rate will go up. 

The good news: this is a short-term problem since your site will speed up again once you finish your A/B test and switch the tool off. The danger lies in not realizing what’s going on. 

How to fix it: do an A/A test before your A/B test. Run your testing software in the background of your page without an experiment. Many common testing tools, including VWO and Optimizely, support this practice. 

An A/A test lets you:

• See if your A/B testing tool slows down your page or has any negative effects on the metric you’re hoping to test

• Verify the tool has been set up correctly since the results of your A/A test should show a tie

• Factor a reduction in site speed into your results if you need to, or—if you’re planning on A/B testing often—find another tool

Post-testing mistakes 

Even once you’ve carefully planned and conducted your A/B tests, there’s still a chance of making mistakes. Here are some errors to avoid when turning your freshly acquired data into action. 

8. Leaving too little documentation

Between waiting for statistical significance and only being allowed to change one thing at a time, A/B tests are pretty high maintenance—so it’s important to squeeze as much learning out of each one as possible.

Thorough documentation helps you:

• Learn which ideas work

• Maximize resources

• Inform future product strategies

How to fix it: create a template for how you document internal A/B tests and ensure everyone sticks to it. It should include:

• The analytics data that inspired your hypothesis

• Your assumptions about why this data looks the way it does

• Your hypothesis, formed as a clear statement 

• The metrics you decided to measure

• Your test results, including a discussion and a list of further actions

Your A/B test document should be a simple one-page report. Once you’ve written up your test, share the report with the rest of your company for feedback. 

9. Not iterating on the test

When an A/B test proves your hypothesis wrong, it’s easy to write your report, shrug it off, and move on to a completely different idea—especially if you’ve waited a couple of weeks for this hypothesis to fail and are just over it. However, this approach leaves no space to fully digest your learnings. 

How to fix it: 

• If you have a hypothesis grounded in data about a customer pain point and an A/B test proves it false, try coming up with a different option. The issue you identified still exists—you just didn’t hit the solution the first time around.

• Consider an iteration if your hypothesis proves successful. Now that you’ve found something that works, can you double down on your win? Say, for example, you tested adding a carousel of pictures to your product page, and it led to a statistically significant uptick in conversions. Why stop there? Your next move could be A/B testing a product video.

10. Making too many changes based on your results 

If you’re trying to win your team over to an idea for a new feature or a design change, convincing A/B test results are highly persuasive. However, people feel so much more confident in their product decisions when there’s data involved that it can be tempting to overestimate your experiment’s implications. 

For example, say you’re trying to grow your mailing list sign-ups. In an A/B test, you learn that adding a sign-up pop-up to your 'about us' page increases this metric. So, based on that data, you add a pop-up to every page of your site. Perhaps the trend continues, and you get more sign-ups—or, perhaps your bounce rate goes through the roof because users casually browsing your blog are annoyed by seeing a pop-up after every article they click on.

How to fix it: go slow and steady when making changes based on your data. Remember that an A/B test only answers a narrow question and that site-wide changes require care and caution.

Accurate test results lead to successful growth strategies 

A/B tests are worth the patience—the results help ensure that every change you make is likely to boost your bottom line. However, A/B testing isn’t the only tool in the trunk.

Digital experience insights tools—like heatmaps, session recordings, and surveys—are powerful for identifying which changes encourage your users to convert, and where your hypotheses went wrong (or right!). Watch a handful of recordings of users clicking around your checkout page, and you’ll soon notice which areas need improvement. 

The most efficient way to optimize your site is to use these strategies hand-in-hand: enrich your understanding of A/B test results with digital experience insights, and use these insights to come up with great hypotheses for A/B tests.