Learn / Blog / Article
How to evaluate the impact and reach of website bugs
Those pesky bugs. Even with the most meticulous checks possible, one or two usually make it through to production.
It’s vital to evaluate website bugs as they come up, so you know how urgently you need to fix them. Without taking a moment to assess impact, you’ll be shooting in the dark.
You might invest your precious resources in unnecessary work, or worse—underestimate the problem and cause frustrated users to take their business elsewhere.
This article outlines a three-step framework to systematically gather all the information you need to understand the impact of a bug, so you can take action fast—and catch any critical errors before they hurt your bottom line and business goals.
TL;DR
Sizing the impact of a bug is a matter of compiling all the information you can about it, and then looking for patterns. Here’s how:
Gather bug reports, including those in your error logs and any user reports that come through customer support channels
Collect analytics data to understand how users behave on your pages, and how many of them encounter the bug
Apply deductive reasoning skills to estimate the size of the bug by comparing what you know about the error to your user base statistics
Learn how bugs affect your users
Hotjar gives you the behavior insights you need to tell how a bug impacts your site and users.
A 3-step framework to evaluate the impact of a bug
It’s crucial to evaluate how many users are affected by a bug, and to what extent it’s preventing them from achieving their goals on your site.
However, when you take time to stand back and make a proper assessment, make sure to:
Here’s how to size the reach of a bug in three simple steps.
1. Gather bug reports 🐞
Step number one to fixing a bug is realizing it’s there. First, gather the logs and reports that flagged this problem to you. Here’s how.
Look through error logs
In many cases, when a bug appears, you’ll catch wind via the error log in your web server. An error log gives you a text-based account of anything that’s gone noticeably wrong in your code.
The relevant entry in your error log will tell you
The page your user requested
The page your user was on before encountering the error
The time and date the bug occurred
The software version it occurred on
Whilst looking at error logs alone won’t be enough to judge the impact of your bug, you might be able to get an idea of its reach by counting the number of times it occurred.
💡 Pro tip: remember that error logs can’t capture everything. Problems that slip through the net include
UX-related bugs (e.g. if nothing’s wrong with the code, but your CTA looks too small to be clickable on mobile)
Silent errors (which happen without triggering an error log)
Heisenbugs (bugs that are hard to reproduce consistently)
Check out user reports
Next up: the word on the street.
Head over to your customer support channels and see what your users are saying. How often is the bug reported? Who is reporting it? This helps you spot trends, such as the operating system or device the bug seems to appear on, so you can estimate the scale of the problem.
User reports of software errors are like cockroaches: if you see one report, there are probably a lot more bug occurrences out of sight. It’s quite rare for users to actually contact you over bugs.
More often than not, if a user encounters an error, they either live with it or leave your site—that’s why it's crucial to be proactive about cross-referencing error reports with other data.
💡 Pro tip: encourage your users to report bugs without the friction of emailing your customer support team by adding a Hotjar Feedback widget to your important pages. This allows users to quickly and easily comment on their experience while they’re on your website. If there’s a particular element or section of a page they want to talk about, there’s also a highlighting feature (as seen below).
Hotjar’s Feedback widget lets users highlight parts of the page, so you know exactly what they’re talking about
2. Collect analytics data 📈
Once you have an idea of what the error looks like from the perspective of your developers and support team, it’s time to take the magnifying glass to your analytics software.
For a big-picture perspective on how this bug impacts user behavior, behavior analytics and digital experience insights tools are your best friend.
Examine rage click maps
Rage clicks—repeated clicks to the same spot, that typically indicate frustration—are a solid hint that users are encountering a bug. Tracking rage clicks helps you uncover the kind of UX issues that an error log just can’t see.
Use the Hotjar Heatmaps tool to create rage click maps, which show aggregate data on where users clicked angrily on a particular page. If a rage click map shows that many—or even most—users are experiencing frustration with a particular element, there’s a good chance you’re dealing with a high-reach bug.
💡 Pro tip: if you’re using Hotjar, you can also ‘zoom in’ on your rage click maps to watch session recordings of frustrated users.
To view these, select a particular rage click, and hit ‘View recording’. Watching a recording (more about this below) will help you discover which user flows the bug usually appears in, providing another valuable clue as to the bug’s impact.
Zoom in to view session recordings of users rage clicking on an element
Watch session recordings
Capture and watch some session recordings of the pages your bug affects. Session recordings are playbacks of users’ mouse movements as they click, scroll, and browse through your site, and they’re an invaluable way to see how an error appears through your users’ eyes.
Create your session recordings using Hotjar, and you’ll be able to filter them by behaviors and variables that could flag an error, helping you understand how many users are dealing with your bug. Filtering options include
Rage clicks: repeated clicks to the same spot which usually indicate something’s not working as it should
U-turns: when a user repeats a step of their journey on your site, presumably because it didn’t go as expected the first time
Frustration score: a mix of behaviors that indicate a user is having a hard time finding what they’re looking for on your site, including things like rage clicks and u-turns
JavaScript console errors: appearances of an error in the code of a page
💡 Pro tip: if you use Jira to manage your response to errors, connect your account to Hotjar. Once you spot a bug in a session recording, you can easily create a Jira ticket with the recording attached. Your team will not only get an error report, but also video evidence of what the bug looks like in context.
3. Implement deductive reasoning 🕵️
Now that you’ve gathered information on where the bug is, what it looks like, and the percentage of your users who experience it, it’s time to dig deeper by putting your detective hat on and inferring patterns. Here’s how.
Consider different user bases
To make educated guesses about how many users your bug will impact, consider the different versions of your site visible to different user bases. An error might only affect users who access your site via a particular device, browser, or geographic location.
Some of this information may be available in your error log, and some of it you’ll be able to see within Hotjar, or whichever analytics tool you’re using. For example, you might see from Hotjar that your checkout page error only shows up in session recordings of users browsing your site on mobile.
Identifying which segment of your user base encounters an error helps you calculate the percentage of users it may affect—though you won’t be able to calculate the exact amount until you’ve also identified the scenarios it affects them in.
Analyze key user paths
From what you know of your customer journey and key flows, consider which action your users are trying to complete when this error appears.
It may be that this error doesn’t interfere with any important processes, and only appears when users behave unexpectedly. Or it may be that this bug appears slap-bang in the middle of your conversion flow. Figuring out which processes are affected helps you assess how often this error could possibly come up.
💡 Pro tip: use Hotjar Funnels to visualize the percentage of users who pass through each stage of your important flows, and which stages see the most drop-offs.
Best of all, you can use your funnels chart to filter session recordings by different funnel stages. For example, if your bug is leading to user drop-off at a particular stage, you can watch recordings of the moment these users abandoned the process.
The Hotjar Funnels tool lets you track events leading up to important actions on your site and watch session recordings from users who dropped off
Run tests
Finally, try to replicate the error yourself, so you can get a clearer idea of when it appears to users. Try using the problem page under various conditions, including
Different devices (laptops, mobile phones, tablets)
Varying degrees of user authentication (whether they’re logged in or not)
Using the page after having navigated away from it, then returned to it via the back button
As different user types (admin, moderator, or end-user, for example)
Remember to document all your attempts at replication, so you and your team members don’t need to repeat work.
By accessing the problem page or element in different conditions, you can shed light on which scenarios and user paths your bug affects, and, ultimately, estimate the percentage of users who are impacted.
Size your bug → tackle it appropriately
By their nature, software errors are unpredictable. You never know where one will pop up or who it will impact—so you need to be proactive. Even if your site seems spotless right now, get familiar with your error logs, check in with support regularly, and set up a digital experience insights software.
A platform like Hotjar gives you a range of tools to understand the impact of a bug—whether it’s the kind that’s most visible in a session recording, a rage click map, or user responses to a survey.
Get started with Hotjar now, and when a bug does strike, you’ll quickly get a sense of whether it’s a flea or a jungle caterpillar—and dedicate resources accordingly. 🐛
FAQs about evaluating website bugs
Related articles
Solving common problems
Create an effective landing page using these free tools (video tutorial)
Your website is still one of your most important business tools, yet only 15% of websites average over 100,000 unique monthly visitors. In order to create successful websites that convert, consumers have access to an array of tools and features that promise to simplify their lives.
However, the overwhelming abundance of options can sometimes lead to more complications. Business owners often find themselves wondering, "How can I easily create a landing page that converts without design resources, IT support, or expensive optimization experts?"
Marc Hans
Solving common problems
10 ways Hotjar helps you understand your users better
Your sales are down; the well's running dry. What do you do to encourage people to buy from you: add a new product or service or multiply an existing product's features?
Increasing choices and features may be a business's automatic response to boost interest and, later on, conversion and sales. But more is needed to justify using company resources to create a solution that may or may not resonate with customers.
To achieve real, needle-moving change, you need to dig deeper and understand users better—what motivates or frustrates them and what hinders or helps them satisfy their 'jobs to be done'.
Hotjar team
Solving common problems
10 ways Hotjar helps you optimize your website UX
To build a great website, you need to help users complete their goals effortlessly at each step of their journey. But how do you determine what users really want, and need, from the pages you’re working on?
Hotjar team