Last updated Oct 27 2020

⚡ Ecommerce testing

You don't know if you don't try - Christina Moritz

Christina Moritz, UX Strategy and Research Manager at Crate and Barrel, shares how she used surveys to collect data for her team.

What Christina covers:

  • The importance of using polls and surveys
  • How you can generate new insight from a simple survey

Click below to read the transcript.

[Transcript]

Hello, good morning, good evening, good afternoon. My name is Christina Moritz and I am the UX Strategy and Research manager at Crate and Barrel. I'm so excited to talk to you about experimentation today.

So much so, that we're still in the midst of COVID, I invited you into my kitchen 'cause what better place to talk about experimentation? That's not what this talks about today, and I'm not sure I got the right toolkit in the kitchen anyway, so let's get into the story.

The key takeaway, and I'm hitting it right off the bat, is you don't know if you don't try. I have been stressing this with the teams that I'm working with, all the pivoting that all of us have had to do in our organizations. And we don't know if we don't try and continue to learn and just see what we can get after.

So, all of this started a little less than a year ago. I began at Crate and Barrel and they said, "we have this platform called Hotjar. Can you dig in there and see what we might be able to do with it?"

So, I discovered the Hotjar polling tool. It's not something that we had been utilizing in the past. Start listening in meetings and start to really get curious, because let's be honest, this is UX and we've got nothing but questions and curiosity behind us.

It's a particular landing page, doesn't have a ton of visibility on it, and I thought it was a safe place to go ahead and test out the polling tool. And push it up, I'm so excited, we're live. I'm gonna start learning something. The first response comes in and how do we think that went? Let's try again.

So once it was discovered that this thing popped up, I had to take it down. But I was so excited. I did not get upset by this because I had ten insights that started to shed light on what was happening on that page.

Fast forward a few weeks, we're leading into the holiday season, you're in ecommerce, you know it really begins in hot October. And we really wanted to begin learning at Crate and Barrel more about our customers and gifting.

It was a perfect opportunity to align stakeholders, have a bit more deeper discussion and figure out our strategy to push and launch our first poll. So we did that as a collective team and as the results started coming in, we wanted to try some more. So we pushed another poll up, and that was very quickly discovered and not on a page that we were comfortable with, and so, in less than 24 hours, that needed to be taken down.

So again, try again. But what was really exciting is we went from 10 responses in the first, and then that 24 hours that we had the second poll up, we had over 200 responses. So collectively now, I've got over 400 customer insights that we didn't have before.

And so what we decided to do with that was compile a collective presentation and share that out with a cross-functional group. And this is really the key is, is what you learn you have to share, you might have to share it more than once, several times to hit all the key audiences and the more cross-functional and the more of a lens you can provide to those customers coming to your site the better.

Our VP was even in attendance for ecommerce and she was thrilled and it was new insight that we didn't have it all originated, because we tried this poll, and we tried that second one, which brought even more insight in.

What that gained was traction. And when you get that momentum, you've really just gotta run with it and stay proactive. That is on you. You need to think ahead of where else, what do we wanna learn, and how can I apply what we've already learned. And it's an experiment, even with your customers to find that right mix, how many questions can I ask? What will they tolerate? What will they abandon, and how do we take one poll and build upon it?

So that's exactly what we did. We kept repeating all the way up until COVID. And I wouldn't say that COVID really paused any of the trying, we've just had to try in different ways and be sensitive to our customer base and what's appropriate and what's not.

So, we have the backing of various insights for various reasons, not just gifting, that has allowed us to dip into our toolbox of other research tools and triangulate and build upon that.

And what's really cool is once you deliver a set of results, and then you deliver more, and now others have more questions, and you continue to partner, and you continue to get after it and stay proactive and try new methods, you can ask to do more and more. So congratulations, the lesson here is that you now have more work for yourself, but that's all exactly what we want. It is our job to bring customer insights after.

As we all know, one research effort doesn't yield over results, and the more that we can compile, based on those initial discovery and learnings from the polls, the better.

And so I'll leave you with try, try again. Even if something doesn't work out, try again, share, share your insights, make sure that the ones that you choose to share cross-functionally

are really going to have an impact and create a lens that can be shared across the organization and continue to repeat that process.

Build that cadence in for yourself, your team and your cross-functional peers. Learn, learn as you go and experiment. Expect to go faster because when you provide those valuable insights, they're only going to want more. And as that keeps going, all you have to do is keep trying again.

Go fast, but not too fast - Niccolò Gloazzo

Niccolò Gloazzo with Kettle and Fire shares a valuable lesson in customer research and tells us why it’s essential to understand your customer before making changes on your website.

What Niccolò covers:

  • How a small change to Kettle and Fire’s subscription made their churn rate increase
  • Why it’s important to analyze secondary KPIs
  • Why your first step should be to listen to and understand your customers

Click below to read the transcript.

[Transcript]

Hi there, my name is Niccolò and I'm part of the growth team for Kettle and Fire. I do growth and commercial rate optimization and Kettle and Fire is a fast-growing DTC company in United States, and we sell bone broth and soups. We started online and then move slightly to retail as well.

Today’s presentation is about actually “go fast, but not too fast”. That's the key learning I would like you guys to get from today’s presentation. Hear the voice of customer is actually the first step. What we did was, we had an idea, we tested, the result were incredible, and after three months we had to roll back to basic here actually.

So what happened? We, I think was around six, nine month ago, we actually decided to roll out and test and experiment that we had in mind, based from the idea that customer might be willing to get a subscription, most likely if they actually create their own box with their own product.

So online, we sell one-time purchase or as well as a subscription so that you get your product recurrently every month to your door. So, we NDP an idea. We create a test where basically the customer could choose his plan, for example, I want one time or I want a subscription. And then they find the box size, so six cartons, nine cartons, 12 or 18 plus, fill up your box with the product that you want, and then make the purchase.

What we found was that actually here the results were incredible, so we have a lot of subscriber. The number of subscription opt-in increase really really a lot compared to what we had before during this time while we had these tests running.

But there is another side of the coin. So what happened next? Next, after a month or two, the churn rate increase exponentially. We saw a lot of customer canceling their order and not their order specifically, but their upcoming orders, so their subscription.

What does it mean? And we try to understand because we run a poll on the website while we were testing these features where we ask customer, if there is anything that is not working as they expect, if it’s missing any kind of information so we run multiple poll, and after a month or two, the results were incredible, we decided to go straight and publish this as a baseline. We moved too fast in this case, we didn't really analyze all KPIs and customers said, “I don't want to buy a minimum six box, I would like to choose whatever I want. I don't want, how can I just add to cart one specific product?” So the concept was a little bit too much for them, too advance maybe. And didn't really match all customer needs.

I know that you can not match every single customer needs, but in this case, there are a lot of customer that still maybe want to purchase one or two carton as they need and not really be forced to go inside the specific funnel let's say.

So, the key insight for today that I would like to share with you is first of all, make sure to spend enough time analyzing secondary KPIs. This is very very important because during our work test, we focus mainly into the subscription opt-in part, but we totally, we were a little bit not confused, I would say, but maybe too happy to analyze also the others the secondary KPI. So, we didn't really pay much attention to, for example, the one-time orders that we miss and as well as the churn rate on day zero as well as days 27. Day 27 because it's the day that the customer actually receive the reminder email about their upcoming renewal.

And after two or three months that we actually implement this as a baseline is where we face a reality. And we noticed that there was some issues with the current baseline.

Second, key insight is a one size does not fit all. So, not because we had really really good subscription opt-in means that we can roll this out to everyone. There are still customer, and even current customer, that they just want to purchase a la carte as they go.

And third is, listen to the voice of your customer because they are really the one that then will make the purchase and that you should listen to.

So, what we did was, we rolled back to basic and decided to use this specific funnel only for a specific target customer and only for specific type of traffic.

5 one-liners to help with experimentation - Denise Visser

Denise Visser, Product Manager of Team Experimentation at bol.com, discusses experimentation and how you can convince other members of your team that experiments are important for growth.

What Denise covers:

  • The importance of experimentation
  • What the goal of every experiment should be
  • Five different conversation styles to help you convey the importance of testing

Click below to read the transcript.

[Transcript]

Hi, I am Denise Visser, and I'm Product Manager of Team Experimentation at bol.com. The goal of this team is to make it ridiculously easy for every colleague to run experiments. Bol.com is a Dutch ecommerce company with about 2,000 employees and 200 scrum teams or product teams.

Not everybody is aware of the great opportunities of experimentation. When you're already hooked up like me, it's sometimes difficult to hit right core to convince others that this is the way to go. And although you might work in a data-driven company, not everybody has the same level of enthusiasm regarding experimenting time.

Some teams at bol.com are completely self-supporting and running several tests simultaneously, but others did not even start yet. In my evangelism to make them also enthusiastic about running experiments to get a better understanding of the uses of the product, I use several stories linked to one-liners which I picked up during talks and books about experimentation in the previous years.

Each person is sensitive to different arguments. There is no one-size-fits-all approach. And in this talk, I give you five conversation styles.

The first is, what does your success look like? Of course, we are used to measure stuff, but do you really have the right metrics in place that represent your success? And what if you have access to those numbers, do you really know how successful you are? What was the results six months ago? What is it today, and what should it be next year? And what can you do to adjust those numbers? Experiments can bring you how you can affect the results. And if you don't know how well you are performing, how can you possibly get better?

But what if you compare yourself with others? Are you already number one in time to open a bottle of champagne? Or is there still an improvement to gain? The smarter a team gets, the more effective the team gets. The ultimate goal of a product team is to build the best possible experiences for the users.

Fortunately, the building resources are not unlimited and therefore, you have to make choices about what to do. If you learn from previous innovations and improvements but the impact was, you could have learned about what changes made the greatest impact.

Experimentation helps us learn about causality in a way that is grounded in evidence that is not anecdotal, and that may be statistically significant. Thus, we can develop informed opinions, which makes it easier to make the right choices and to decide which actions you have to put effort in to get the desired outcome. It makes your team more effective.

Without experimentation, you are testing without knowing. If you put a change, like without an experiment, you cannot learn what the direct impact is of this change because there might be a lot of other factors that influence the outcome.

In experiments, you make some change and measure that against the control to observe whether the change had an impact. The only difference between these groups is the change that you deliberately made. Therefore, if you observe a significant difference between these groups, in well-designed and well-controlled experiments, you can conclude that the difference was most likely caused by the change that you made.

If you put your change, like without an experiment, you don't have any proof about causality.

And for your more supportive colleagues, learning before earning, especially when you have a discussion about the success rate of your experiments, keep in mind that you have to practice to become better.

It's not realistic to run only successful experiments. Famous football players didn't enter the fields in Camp Nou or Anfield without practicing, they practiced a lot. And there is a saying that you have to put 20,000 hours of efforts to gain a scale.

And luckily to run an experiment and draw links from it, doesn't take that much of time. It's easier to become a successful experimental than a famous football player. And the goal of every experiment should be to learn something. And if you manage to do so, then every experiment is successful, no matter whether it caused additional revenue or reduced costs.

This is my favorite one, work smarter, not harder. Everybody wants to be smarter and being smarter to work, that makes you very successful. Just don't put more effort in your tasks, but pick the right tasks. You can be way more effective at work if you choose the tasks that have effects.

And besides that, it makes your work even more fun because you know the actual effect of all the effort you're putting. Don't do just the things you're used to, but pick the ones that benefit your goals. Measure your results, not your time.

Hopefully, you got some input by this presentation to inspire yourself and others to do experiments. And it would be great if you could give me some feedback or which argument works best for you and your company. Have a nice day and enjoy some other talks provided by Hotjar.