How to monitor and spot issues after releasing changes to your site/app

Understanding why people leave

When you launch changes or updates to a website, app, page design, product and/or feature, there is always a risk of something unpredictable happening: people don’t stay as long as they once did, don’t spend as much money as they used to, responsive issues across different devices appear, errors pop up that won’t let visitors complete an action.   

Because it’s difficult to know in advance what impact the launch will have, it can be useful to have an overall plan of action to catch urgent issues (so you can deploy solutions in record time) and discover any additional problems (so you can limit the overall negative impact when things are not working perfectly).

Below is our 3-step framework to help you stay on top of any issues in the first minutes/hours/days of launch. We wrote it based on our experience and that of other Hotjar users to help improve your chances of being efficient and responsive:

 

1.Watch recordings to find and prioritize what needs fixing

The fastest way to figure out if something is going wrong is by watching people using your website or app. This is especially helpful if you don’t have large volumes of traffic or have no resources to do in-person testing.

In general, visitor recordings give everybody involved in developing (or fixing) the product a chance to see, and therefore understand, what people are really going through. Without this step, you will not be aware of all the bugs and glitches that come up, because people don't always report them: they just get frustrated and leave. 

 

Set up Recordings

Set up recordings as soon as possible after launching or releasing changes. Hotjar will then start recording samples of visitor sessions when new people visit the website/app, so you are able to watch videos fairly quickly.

From the Hotjar sidebar menu, select Recordings under the Analysis section and set them up as you need (find more information about settings on this technical page).

hotjar-menu-recordings

 

Start with 30-50 recordings

Depending on how much traffic you get to your website, you might find yourself with tens or even hundreds of videos in the hour(s) after you set up recordings. To get started, get every person on your team to watch 20-30 recordings in one go, or watch 30-50 if you are on your own. Based on our experience and that of our users, it takes only about 10-15 recordings for you to get into a good watching flow.

 

Sort recordings and watch a mix

We recommend sorting your videos by length and/or number of pages viewed and watching a mix: shorter videos might be caused by visitors encountering an issue and leaving immediately after, while longer ones help you paint a more accurate picture of the journey and potential stumbling blocks.

recordings_filters_explained4

(filtering options in recordings) 

Some other useful tips:

  • if you are dividing up the watching task amongst your colleagues, consider doing it on a page-by-page basis (you can use the filter function in recordings to select the URLs you are interested in);
  • use URL filters to look for recordings that include a 404 page (in our case, we would filter by https://www.hotjar.com/404): if you find any, following the journey backward will help you see how people got there;
  • skip pauses and/or watch at increased speed to save some time (see the two buttons below, highlighted in red).  
 
This is what an actual Hotjar recording looks like — 
the navigation bar at the bottom is where you can speed up and skip pauses

  

Look out for bugs

As you go through recordings, look out for the following bugs:

  • compatibility bugs, where pages load incorrectly on different devices/browsers;
  • functionality bugs, where the website/app doesn’t do something it should (example: users try to login but cannot, clickable buttons or elements don’t work);
  • layout and design bugs, where elements of a page don’t render correctly;
  • unusual mouse activity, such as wild scrolling or repeated clicking, which can indicate visitors are not finding what they need.

 

Use tags and notes 

Whenever you want to make note of a specific action (e.g. a click on a broken element, a drop-off on a specific page, etc.), tag recordings with that information; you and your team members can later filter by individual tags and re-watch relevant recordings.

You can agree on certain tags to use ahead of time, or as the need arises. Some of the ones we’ve been using at Hotjar include Broken link to contact us, Error, Filters used, Unsuccessful, To investigate, Didn’t seem to find pricing, Verification issue—maybe these can spark ideas for tags for your team.

Note: if you are on a PLUS plan or above, you can tunnel your team’s focus by automatically tagging recordings. 

 

Share recordings and findings with your product/dev team

You can generate a unique URL for each recording directly from the screen (for step-by-step instructions, check out this technical guide) and share it with anyone who needs to watch it. 

recordings_how_to_share_in_video

If you and your product/dev team are using a project management system like Jira or Trello, create and 
add recording links to individual tickets or requests for fixes: this gives whoever is in charge of dealing with the problem the opportunity to see it happen before going in and working on a fix. 

2. Review heatmaps to identify larger issues

While recordings let you dive deep into individual visitor journeys, heatmaps aggregate data from multiple visitors to give you an overall understanding of what people have viewed on your website or app. They are particularly useful if you have made substantial changes to your page layout, for example by moving elements around or restructuring the page hierarchy. 

Depending on what kind of insight you are looking for, you can use three types of heatmap:

  • Click and tap heatmaps show where visitors click their cursor on desktop devices and where visitors tap on mobile and tablet devices:

    click-map-example

  • Move Heatmaps (desktop only) show where visitors move their mouse on the screen:

    move-map-heat

  • Scroll Heatmaps show you to what position your visitors are scrolling on a page:

    scrollmap-average-fold

 

All three heatmaps use a color scale to show elements that get the least (blue) to the most (red) interaction: after a few dozen visits, you’ll start to see which element(s) your visitors are clicking on, scrolling past, and/or ignoring.

 

Set up Heatmaps

Set up heatmaps as soon as possible after launching or releasing changes: it only takes one person to visit the page for a heatmap to start working, but data based on just one  visitor  will not be very useful—so it’s important to give the tool some time to collect a good amount of data (100-200 visits are a good starting point).

From the Hotjar sidebar menu, select Heatmaps under the Analysis section and set them up as you need (you can find more information about settings on this technical page).

hotjar-menu-heatmaps


The number of  heatmaps  you want to set up depends on how many pages are affected by your changes:

  • if you have launched a new page, you will need just one  heatmap ;
  • if you have redesigned your entire website, you’d benefit from setting up multiple heatmaps to capture data from key pages.

 

Review click heatmaps to see what gets the most interaction

The red elements on a click heatmap are the ones that get the most interaction(s). If the reddest elements on your heatmap are not links, you might be creating confusion in visitors who expect to be taken elsewhere when clicking or tapping on them.

Red elements can also help you find out if your visitors’ attention is where you want it to be. If people are ignoring or missing something you’d rather they focused on, you can investigate the situation further and eventually change (or test changing) page elements accordingly.

See it in action: 

Our current menu raises a question: (why) are people not seeing the blog?


hotjar-menu-mapA Hotjar heatmap from our homepage — "pricing" is clearly the most clicked menu item. 

 

Review scroll heatmaps to see how much of the page visitors are viewing  

Scroll heatmaps show you how far down the page your visitors scroll. As with click heatmaps, red indicates the most viewed page portion, and blue the least. Views are also displayed as a percentage of page visits, showing how many visitors reached a particular point on the page.

If your heatmaps reveal that the majority of visitors don’t scroll far enough to see important information, you can consider moving elements around to make them more visible. If the change in color is sharp rather than gradual, there might be an issue with page design/layout that makes people think they reached the end of the page and there is no content below.

 

Check for differences across devices

While recordings might make you aware of individual glitches or problems on specific devices,  heatmaps  help you get a more comprehensive understanding of their impact and show you obvious inconsistencies that need fixing. For example, valuable content that is high on a desktop page could be far lower on a phone, and a mobile scroll heatmap would help you understand just how many people actually see it (or don’t) so you can make decisions accordingly.

 

Share heatmaps and findings with your team and/or clients

Because of their visual impact, heatmaps are usually very powerful in helping product/development teams and clients develop a rapid understanding of the overall performance of the change(s) you made. You can download heatmaps and/or generate unique URLs to share them as a link (for step-by-step instructions, check out this technical guide). 

3. Investigate why something doesn’t work with feedback polls


In addition to uncovering bugs and broken elements that need fixing, you might find that something does not work as intended, but you don’t know why. For example, individual recordings might show that people reach a certain page then do not progress to the next one; a page-specific heatmap might then confirm that these are not isolated  incidents,  because the “next page” button (or equivalent) does not get clicked as much as the “previous page” one.

 

So you know what is going on, but to know the why you have to interact with your visitors: in addition to watching their actions, you can solicit their feedback directly by setting up polls.

 

NOTE: polls works better if you have around 1000+ users a day. When you have lower traffic volumes, you will likely benefit from ‘guerilla research’ instead — where you just show the website to a few people and ask them questions directly.

 

Set up a Poll

From the Hotjar sidebar menu, select Polls under the Feedback section and choose the settings you need (find more information about creating your first poll on this technical page).

hotjar-menu-polls

 

Hotjar will start showing polls to visitors as soon as you launch them, but the speed at which you get answers will depend on people’s willingness to submit them.

If you haven’t received many answers after a few days and you are sure the page has been visited plenty of times, consider changing the position or color of the poll to make it stand out more, and/or re-formulating the question you are asking.

 

Ask people why they are leaving

If you’ve seen people drop-off at a specific page, set up a poll on that page to investigate what is going on: ask questions that allow you to understand the context of their decision to abandon the page. This is usually known as an exit poll.

You can use very direct questions:

  • Why are you leaving this page?
  • What is not clear about this step?
  • What is missing from this page?

Or you can try a less direct approach:

  • What would be the next logical step for you after this page?
  • Is there something you think should be added to this page?
  • If there was one thing you could add to this page, what would it be?

See it in action:

Conversion Rate Experts use exit surveys to ask questions as close to the “moment of truth” (the decision point on whether to continue or abandon) as possible, following this flow:

CRE-polls-flow

For more details, read the full  Conversion Rate Experts case study .

 

Ask people for guidance

If you have spotted a recurring problem or behavior but are not sure what the best solution to it is, ask your visitors for guidance:

  • What would you expect to happen when you click [insert description of  clicked  element]?
  • What would you expect to happen after this page?
  • How could this page/section/form/[etc.] be improved?
  • What would make you more likely to [insert desired action]?
  

See it in action: 

To figure out what was stopping people from continuing down a registration path, Hubspot asked a very simple, open-ended question that allowed visitors to express their concerns, doubt, or objections. The results gave the team at Hubspot the actionable insight that key information was missing from the page: this allowed them to start working on a content fix, which in turn would help ensure more people continued towards registering.    

Hotjar-Hubspot-9


For more details, read the full  Hubspot case study .
 

You don’t need to stick with open-ended questions; you can use a YES/NO approach to give people options and pair them with a follow-up question:

  • Is there anything on this page that doesn't work the way you expected it to? Yes/No (if yes: what?)
  • Would you prefer to see [X] on this page? Yes/No (if no: why?)

 

Trigger polls at specific points [Plus/Business Plan]

When you are on a Plus or Business plan, you can set up JavaScript triggers to initiate a poll only when visitors perform a specific action—for example, when they click on any elements that you want to investigate. Use this option when you want to be immediately responsive and have more control over when and where a question is asked.

Final tip: follow these instructions before making changes 

These steps will help you after a product launch, but we recommend that you use them as you prepare for the next one. Before you start building or planning to build anything,

  • sit and watch a visitor’s journey  on  your current site
  • set up and check heatmaps on existing important pages
  • ask questions to understand visitors more.  

 

Are you ready? Log into your account and get started!