Rich heatmapping through experiment tracking - Nick Disabato
What Nick covers:
- Why it’s essential to analyze all metrics and understand the bigger picture
- The importance of archiving your heatmaps
Click below to read the transcript.
Rich heatmapping through experiment tracking - transcript
Hi there. I'm Nick Disabato. I'm a designer and writer from Chicago. I run a consultancy called Draft. You can take a look at draft.nu if you're curious. Today, we're going to be talking a little bit about how to create per-variant heatmaps in Hotjar.
We have Google Optimize open right here. We're gonna be integrating Google Optimize and Hotjar so that any hits that actually go to the variant get sent over to Hotjar. And this is super useful because it gives you a sense of who is hitting your variant, what they're doing, and how their behavior is changing over time.
We presume you're probably running heatmaps already on your homepage in some capacity, on your product detail pages, but are you running them for your experiments? And are you trying to understand how behavior is changing over time?
Super, super important in our consultancy, and it's something that we've incorporated into our design practice.
So, if I go in here, I just set up an example AB test, and I'm going to go through, and this is the page for a book that I wrote recently, and if I want to just change the headline to something, I can go and edit the text and update it over here: “become more respected as a designer, period.” Right? If I wanted to change it from a question to a sentence, whatever, to a, you know, declarative, whatever, that's totally fine.
You might need to capture a screenshot ahead of time and there you go. You've created your heatmap. You wait, and you run it for the length of your test.
So, with that in mind, you want to be able to compare heatmaps. I use a tool called Kaleidoscope. It's an app for the Mac, super, super useful. And that gives me a sense of what's happening on the control versus what's happening on the variant. I try and run a heat map on the control right before I'm running the experiment. Usually, I know what key pages I'm doing that on because I usually work in eCommerce and there's a pretty clear sense of how that maps.
And then you want to take those insights, what those differences look like, and incorporate them into your final analysis. This matters tremendously because you might have knock-on effects in customer behavior. You might see that the conversion rate goes up, but overall engagement goes down. People might not be interested or they might not be browsing other products. If the conversion rate goes up, and average order value goes down, you might want to know why, and heatmaps might be able to tell you that. Super, super useful.
It's never enough to just say, hey, we made this one number go up. You can maximize that, but it could kind of be a Pyrrhic victory because you end up hurting a lot of other lagging metrics. And that could really, really hurt you.
And, you know, obviously, keep an archive of your heatmaps. Not just on Hotjar, download them locally, make sure you understand what's been changing, when. Map it to annotations in Google Analytics as you're going through, and make sure that you're trying to understand exactly what you're doing with customers' behavior over time.
What are the blind spots with this that I've personally found? I haven't found a way to run a heatmap on just the control when an experiment is running. You can have a heatmap fire for a given control variable, but then it would run for everybody in the AB test, and I don't think that's so great. So, that's one blind spot that you would end up seeing here.
You probably want to run a control heatmap when there are no active tests going on on the page. But that might change over time as Hotjar's functionality changes or Google Optimize's functionality changes. I know Google Optimize, comparatively speaking, is relatively new. It's like a two-year-old platform, so that's something you want to be keeping in mind.
And that is it. Thank you so much. You can take a look at my consultancy at draft.nu, and I'm really grateful for your time today.