Learn / Guides / A/B testing guide
An essential 6-step guide to mobile A/B testing
Mobile browsing isn’t just for on-the-go users anymore. People often pick up their phones or tablets to shop for mattresses or subscribe to accounting software, even if their desktop device is nearby.
To provide a smooth user experience (UX), mobile A/B testing is a must. By testing two different versions of your product or site, you learn which features work—and which need work.
Navigating the mobile A/B testing process can be challenging, as it’s easy to get overwhelmed with the sheer number of elements you could change or the various ways to set up your test.
This guide walks you through a six-step process to conduct effective mobile A/B tests. Learn how to optimize your site or product to create a seamless UX, increase user engagement, and boost conversion rates.
TL;DR
In an increasingly mobile-centric world, designing a smooth UX is essential. Mobile A/B testing gives you the data you need to build empathy with your users and improve your design.
Follow these six steps to run mobile testing with confidence:
Review your data: look at mobile-specific performance metrics and qualitative data to find ways to improve your UX
Create your hypothesis: craft a single sentence with the problem, potential solution, and predicted outcome to frame your experiment
Design your variant and test: gather or design any required creative elements for the alternate version, and set up a test in your A/B testing platform of choice
Segment your audience: apply mobile filters to look specifically at data for users on phones or tablets, and then layer in additional filters relevant to your hypothesis
Launch—and complete—the test: launch your test, running it until you’ve gathered the necessary sample size and achieved an appropriate test duration
Analyze your results: evaluate how each design performed and return to your hypothesis. Then, consider the next steps for improving your website or product.
Conduct mobile A/B testing with ease
Use Hotjar’s tools to see how users experience different versions of your product on mobile.
6 steps to effective mobile A/B testing
Mobile usage has exploded in recent years, dominating other traffic sources. So much so that, in June 2023, the global mobile market share of web traffic was more than double that of desktop.
Mobile A/B testing is no longer a nice-to-have—it’s a necessity.
Desktop and mobile aren’t just devices with different screen sizes. Users approach each with a distinct mindset, and their experience needs to reflect that. They might pop onto an ecommerce site while waiting for their kid’s piano lesson to end, or check their email while waiting in line for their coffee. To avoid frustrating users, you need an easy and intuitive mobile experience.
Increasingly, optimization is moving ‘mobile only’. It’s been ‘mobile-first’ for a while. That means doing usability testing on mobile, ideating and designing for mobile, and reviewing test previews on mobile.
That means you need to design and test specifically for mobile. Not sure where to start?
Read on for our six-step process.
1. Review your data
Mobile A/B testing is a way to seek out the rough edges and spot problems in your design to figure out how to make it better.
Do a deep dive into your
Analytics to check key metrics like average session duration and bounce rate
Survey data like Net Promoter Score® (NPS) to gauge customer satisfaction, or an exit-intent survey to see why users bounce
Rage click maps to locate areas where users repeatedly click due to confusion and frustration
Funnels to see where mobile users drop off, and how that compares to desktop users
Feedback to learn how mobile visitors feel about aspects of your site or product
Trends to see charts of key performance indicators (KPIs) for mobile users over time
Access all of this data from one centralized location—your Hotjar Dashboard—to see what is happening (through quantitative or numerical data) and why (through qualitative or text-based data).
Make sure to apply the ‘mobile’ filter to home in on device-specific issues. If you spot one, Hotjar lets you see relevant heatmaps, visual representations of how the average user views and interacts with your site, and recordings, which show individual visitors’ taps, swipes, and scrolls.
💡 Pro tip: filter your recordings by traffic channel to dig deeper into user behavior.
Hotjar’s traffic channel filter lets you see how mobile users from different channels behave on your site. For example, a visitor who arrived from paid social ads may experience your site differently than one who clicked a call-to-action (CTA) button in a promotional email. If you find an issue that hinders conversions for a specific group, it may be a worthy contender for an upcoming experiment.
The traffic channel filter in Hotjar Recordings shows you how users from different channels experience your site
2. Create your hypothesis
Once you’ve identified problem spots on mobile, choose one to prioritize and create a hypothesis for your experiment. A strong hypothesis outlines the problem, solution, and predicted result—typically in that order.
Let’s say you review data for your product’s landing page on mobile. You see the following:
Average time on page is up, but conversions are down
Clusters of angry face icons on your rage click maps appear on non-clickable descriptions of product features
Your Engagement Zones heatmap—which combines aggregate click, move, and scroll data to show you the most engaging parts of your site—reveals that the FAQ section at the bottom of your page is getting tons of attention
Your takeaway could be that users need more information before they buy. Then, you’d create a problem-solution-result hypothesis like:
If users need more information before they buy…then we should move the FAQ section higher up the page…so that at least 10% more visitors convert.
Hotjar saves us and our clients a lot of time. It’s such a convenient and functional tool—it gives us the reasonable, user-backed foundations for our experiment hypotheses, and lets us demonstrate how effective we’ve been through our redesigns at producing KPI improvements.
3. Design your variant and test
The more time-intensive part of this process is to gather all the assets you need. Depending on your hypothesis, you may need
A wireframe from your UX design team
Code from your dev team
Copy from copywriters
Custom illustrations from your graphic design team
If you’re only testing button sizes or rearranging existing page elements, you’re ready to set up your experiment.
The good news is that the same A/B testing tools that offer desktop testing also let you design mobile tests. (If you don’t have one yet, consider options like Omniconvert, Kameleoon, or Unbounce—all of which integrate easily with Hotjar. 🙌)
When building your test, consider how to distribute traffic to the two variants. If you run two brand-new homepage options side by side, you may opt to send an equal percentage of traffic to each. If you test an existing page against a new variant, consider diverting a smaller percentage of traffic, such as 25% or 30%, to the new option.
4. Segment your audience
The next step is deciding who sees each variant.
You already know you need to segment by mobile. For best results, you also want to target other specific segments of users relevant to the problem you called out in your hypothesis.
For example, say the issue is that new users abandon your mobile checkout flow much faster than existing customers. For your mobile A/B test, you’d need to target this particular segment. (An A/B testing platform makes this task simple.)
Also consider segmenting by
Demographics
Average order value (AOV), i.e. how much a customer typically spends on a transaction
Subscriber plan or tier type
Length of subscription
User operating system, like Android or iOS
Length of time since last visit
Different user groups have their own unique needs on your mobile site. Segmenting lets you collect the precise data you need to improve the experience for each group.
Define a segment and even context. Don’t expect all users to have the same needs and behavior.
5. Launch—and complete—the test
Once you hit launch on your mobile A/B test, you’re on your way to getting answers. But you still have two more aspects to consider: sample size and test duration. In other words, when do you end your experiment?
Without a background in statistics, calculating a statistically significant sample size—the number of visitors you need to run a test—is tricky. At the same time, you don’t want to simply pull a number out of thin air. That’s why most A/B testing tools include a test length calculator like this one.
Test duration—the length of time you let the test run—varies, too. If you can get enough visitors to test your variations on day one, should you? Nope. For a well-rounded, representative sample of insights, most experts recommend running your test for two to four weeks.
💡 Pro tip: you don’t need to wait until the test is over to peek at your results. Monitor your test for bugs or issues, and check out how a new design performs while the test is still in action.
By integrating your A/B testing platform with Hotjar—either through our dozens of native integrations or countless others through Zapier—you get a window into how users experience your test variants.
Say you’re running a checkout flow A/B test in Optimizely. Head to the Recordings tool in Hotjar, and filter sessions based on your Optimizely experiment. Then, add additional filters, like the specific variant or user actions you want to see.
Spot something interesting? With our Slack integration, send specific session recordings to your team members—and start an early conversation about your findings.
Hotjar’s Slack integration makes it simple to communicate with team members about A/B test results
6. Analyze your results
You’ve come full circle—from data analysis to, well, data analysis.
Once your test has run its course, look at the performance metrics. Ask yourself questions like
How did each of your variants perform?
How do these results compare to each other?
How do these results compare to your predicted outcome in your hypothesis?
Which version ‘won’?
But don’t stop there. Take a closer look at user behavior through relevant heatmaps and recordings, and explore user opinions through feedback and surveys. These valuable qualitative insights show you why users prefer one page over another and help inform future iterations.
A maxim in CRO goes that quantitative data tells you what is happening but not why. It’s easy to forget that all those metrics and graphs are just a reflection of human behavior.
👀 How The Good used heatmaps and recordings as part of their A/B testing process to optimize mobile UX
While reviewing sessions in Hotjar Recordings, ecommerce conversion rate optimization (CRO) company The Good noticed its client Swiss Gear’s mobile users struggling with the site’s icons and language. So the team developed an alternate mobile design and began cycling through A/B testing.
Maggie Paveza, CRO Strategist at The Good, noted, “After each change, we ran heatmaps and session recordings on that area again to further understand how the results of the A/B test impacted user behavior.”
The team’s design iterations resulted in a simple menu-driven user interface (UI) design for the mobile homepage that promoted top filters. With a streamlined UX on category pages, they reduced the mobile user bounce rate on Swiss Gear’s site by 8% and increased time on site by 84%.
Create an effortless, user-centric mobile experience
Following these six A/B testing steps gives you a glimpse into how users experience your mobile site—so you can take the next steps to improve it.
With results in hand, you’re ready to iterate and redesign to deliver a seamless mobile experience that delights your users and keeps them coming back for more.
Conduct mobile A/B testing with ease
Use Hotjar’s tools to see how users experience different versions of your product on mobile.
