Last updated Oct 27 2020

⚡ Heat map testing

Rich heatmapping through experiment tracking - Nick Disabato

Nick Disabato, a designer and writer, walks you through the process to create a per-variant heatmap in Hotjar.

What Nick covers:

  • How to use Javascript to create per-variant heatmaps in Hotjar
  • Why it’s essential to analyze all metrics and understand the bigger picture
  • The importance of archiving your heatmaps

Click below to read the transcript.

[Transcript]

Hi there. I'm Nick Disabato. I'm a designer and writer from Chicago. I run a consultancy called Draft. You can take a look at draft.nu if you're curious. Today, we're going to be talking a little bit about how to create per-variant heatmaps in Hotjar.

We have Google Optimize open right here. We're gonna be integrating Google Optimize and Hotjar so that any hits that actually go to the variant get sent over to Hotjar. And this is super useful because it gives you a sense of who is hitting your variant, what they're doing, and how their behavior is changing over time.

We presume you're probably running heatmaps already on your homepage in some capacity, on your product detail pages, but are you running them for your experiments? And are you trying to understand how behavior is changing over time?

Super, super important in our consultancy, and it's something that we've incorporated into our design practice.

So, if I go in here, I just set up an example AB test, and I'm going to go through, and this is the page for a book that I wrote recently, and if I want to just change the headline to something, I can go and edit the text and update it over here: “become more respected as a designer, period.” Right? If I wanted to change it from a question to a sentence, whatever, to a, you know, declarative, whatever, that's totally fine.

But that's not enough, right? If I want to integrate it with Hotjar, I have to inject some JavaScript, so I'm going to go through and add some JavaScript. I usually just do it kind of somewhere in the header by editing the logo. And I go to “Edit Element”, “Run JavaScript” (down here), and then it gives me a little brick of JavaScript that I can paste in here.

I've written this JavaScript that you see up top; you paste this in. This variable—experiment variant—you can change it to anything that you want. I just have it here. I usually change it so it specifies the actual experiment that I'm putting together. I hit “Apply”, I hit “Save”, and then I can go and launch the test.

Next, we're gonna tab over to Hotjar and actually set up the heat map. I go through here, Hotjar Lightning Talk. If you're on a Plus plan or above, you can do this. Go to 2000 page views, and then you go over here, “users who trigger a JavaScript snippet”, and then you have JavaScript trigger here. And this experiment variant here, you set up here and then you hit “Next” and you create your heatmap that way. That way, anybody who hits this particular variant ends up going to this particular heatmap.

You might need to capture a screenshot ahead of time and there you go. You've created your heatmap. You wait, and you run it for the length of your test.

So, with that in mind, you want to be able to compare heatmaps. I use a tool called Kaleidoscope. It's an app for the Mac, super, super useful. And that gives me a sense of what's happening on the control versus what's happening on the variant. I try and run a heat map on the control right before I'm running the experiment. Usually, I know what key pages I'm doing that on because I usually work in eCommerce and there's a pretty clear sense of how that maps.

And then you want to take those insights, what those differences look like, and incorporate them into your final analysis. This matters tremendously because you might have knock-on effects in customer behavior. You might see that the conversion rate goes up, but overall engagement goes down. People might not be interested or they might not be browsing other products. If the conversion rate goes up, and average order value goes down, you might want to know why, and heatmaps might be able to tell you that. Super, super useful.

It's never enough to just say, hey, we made this one number go up. You can maximize that, but it could kind of be a Pyrrhic victory because you end up hurting a lot of other lagging metrics. And that could really, really hurt you.

And, you know, obviously, keep an archive of your heatmaps. Not just on Hotjar, download them locally, make sure you understand what's been changing, when. Map it to annotations in Google Analytics as you're going through, and make sure that you're trying to understand exactly what you're doing with customers' behavior over time.

What are the blind spots with this that I've personally found? I haven't found a way to run a heatmap on just the control when an experiment is running. You can have a heatmap fire for a given control variable, but then it would run for everybody in the AB test, and I don't think that's so great. So, that's one blind spot that you would end up seeing here.

You probably want to run a control heatmap when there are no active tests going on on the page. But that might change over time as Hotjar's functionality changes or Google Optimize's functionality changes. I know Google Optimize, comparatively speaking, is relatively new. It's like a two-year-old platform, so that's something you want to be keeping in mind.

And that is it. Thank you so much. You can take a look at my consultancy at draft.nu, and I'm really grateful for your time today.