Episode 009

Rescuing a sinking ship: How customer feedback saved CXL Institute with Peep Laja

"I was ready to pull the plug in 30 days and tell everybody, 'Sorry, I have to let you go because I'm out of money.'"

Here's how Peep and his team at CXL Institute averted disaster and turned a profit by listening to their customers.

In this episode, we cover:

What do you do when you take a massive gamble on an initiative you think is a sure bet, and then it almost tanks your company?

In today’s episode, Peep, the founder of CXL, one of the most popular conversion optimization blogs on the internet, shares:

How the launch of the CXL Institute training program completely bombed, left the company with only 30 days worth of cash, and got Peep to the point where he was almost ready to pull the plug and let his whole team go.

BUT, he also shares how thanks to his people-first approach, Peep and his team were able to turn things around and start generating a profit.

We talk about the exact steps Peep and his team took to understand the needs of their customers as the ship was going down, including:

  • The exact questions they asked when their customers started AND left the course, and how that led them to give away a key part of their course for free
  • How they broke down the limited data they had to better understand who was staying in the course, who was leaving, and why
  • As well as the fundamental change they made to the course’s format that encouraged a positive change in their customers’ behavior and led to an increase in attendance

So if you’re interested in hearing how one of the top minds in CRO managed to rescue a sinking ship, you’ll definitely want to listen to this episode.

Show notes
  • [00:02:36] How the CXL Institute got started
  • [00:05:04] The size of the email list CXL had at the time of the launch of CXL Institute
  • [00:05:58] What Peep decided to do when it looked like CXL Institute was about to run out of money
  • [00:7:35] The things that Peep learned from the qualitative feedback they got from people who were using or canceling their service
  • [00:11:00] How CXL Institute conducted interviews and surveys
  • [00:12:53] The types of questions CXL asked in surveys
  • [00:14:17] How the surveys helped CXL decide what to stop doing and what to keep doing
  • [00:14:54] The process that CXL used for analyzing the data from interviews and surveys
  • [00:15:39] How changing the cancellation process helped CXL get necessary feedback
  • [00:17:02] The top reasons why people were cancelling their CXL Institute service
  • [00:18:29] How CXL used churn analysis to help them understand their cancellation data
  • [00:24:58] Resources that Peep recommends 

[00:00:03] Louis: Hey, this is Louis Grenier here and welcome to The Humans Strike Back by Hotjar, a weekly podcast designed to help you succeed by putting people first. I have the pleasure to co-host this podcast with my fellow human and colleague, David Peralta.

[00:00:18] David: Each week we're going to learn the stories of other humans who are making a difference and thriving by putting their users, customers and team members first, so you can learn from them, take action, and grow. On to the podcast.

[00:00:32] Louis: Hey, it's Louis Grenier here and today I am with Peep Laja, founder of ConversionXL, one of the most popular conversion optimization blogs on the internet. CXL is also a successful agency. In 2016, Peep and his team launched a CXL Institute to help teams improve their optimization, analytics, and UX skills. CXL Institute is doing extremely well today having trained teams from Google, Cisco, and Zalando but that wasn’t always the case.

In fact, in today's episode, Peep is going to share how the launch of the CXL Institute almost sank the entire company and left them with only 30 days worth of cash and got into the point where he was almost ready to pull the plug and let his whole team go. But he also shares how thanks to his first approach, Peep and his team were able to turn things around and start generating a profit.

We talk about the exact steps Peep and his team took to understand the needs of their customers as their ship was going down, including the exact question they asked when their customers started and left the course, and how that led them to give away the key part of the course for free, how they broke down the limited data they had to better understand who was staying in the course, who was leaving and why, as well as the fundamental change they made to the course's format that encouraged a positive change in their customers’ behavior and led to an increase in attendance.

If you're interested in hearing how one of the top minds in CRO managed risk your sinking ship, you'll definitely want to listen to this episode. Peep, welcome aboard.

[00:02:24] Peep: Thank you for having me, you're very kind to invite me.

[00:02:27] Louis: Let's dig a little bit more into the situation of the ConversionXL Institute that was born around two years ago, right?

[00:02:36] Peep: I started the company ConversionXL or now we call it CXL in 2011 and it was just an agency. We surveyed our email list people asking them, "What else would you want to buy if we were smart enough to buy?" People said courses. About two years ago, I thought, "Hey, let us start a spinoff business," and we called it CXL Institute.

In 2016, January, I hired a team to work with me on this—product people, researchers, and so on and so forth. Five months later in May of the same year, the product was launch ready.

We launched CXL Institute the first iteration of it, which was a subscription product. It cost $65 a month and it gave you access to a conversion optimization course. And also we were doing weekly UX research studies, so you would also get access to those studies.

We had a launch campaign, which basically was a bunch of emails and blog announcements. So we were marketing to our existing email list. But the launch didn't go as I had planned in my imagination. We only made about $25,000 in the first month, which was not even enough to cover our cost.

I think my cost, office, salaries, and everything else was about $40,000 or $50,000 a month. Okay, I have been losing money preparing for this launch and still losing money. But June will be better, but in June we made even less money. In July we made even less money. In August even less, and in September, we made just $10,000 and we had depleted the cash reserves. We had only enough money left for one month to pay salaries for the team.

I was ready to pull the plug in 30 days and tell everybody, "Sorry, I have to let you go because I'm out of money." Yet the next month we turned it around. We made $75,000 in revenue. It means we turned a small profit and we lived happily ever after.

[00:02:36] Louis: Let me ask you some detailed questions, because this is a very interesting snapshot. How many email subscribers did you have in your list when you emailed people about CXL?

[00:05:11] Peep: I think at that time we probably had around 50,000 or so.

[00:05:16] Louis: Okay. So 50,000 people on your list and $25,000–was it $25,000 in sales when you launched the first month?

[00:05:24] Peep: It's $25,000 in revenue, so 50,000 in the email list, open rate–I don't remember it by heart, but you know, maybe 25%, something like this, click-through rate probably, even worse, much worse compared to open rate.

I'm not sure how many people we actually had consuming the offer or the multiple emails that we sent but definitely the launch campaign could have been better as well. Yeah, it didn't go so well.

[00:05:58] Louis: You had one month of cash reserve left and you're about to pull the plug. What did you decide to do?

[00:06:06] Peep: Of course, it wasn't now that we are all out of money, now we started thinking, "Maybe we should change something." We were thinking what to change all along because we knew the ship was sinking because every single month we had less and less money.

Throughout these months of losing money, we were doing multiple things. We were very frantic about customer development meaning that everybody who canceled, let's say it was a subscription product, everybody who canceled, we sent them an email, tried to get them on the phone asking, "Why did you cancel? What did you not like? How could we improve?" Also, onboarding, when people joined, we called them up, said, "Hey, thank you for joining. What do you think?"

We started to get qualitative feedback from people who were canceling and people who were using. That was one thing that we did. Another thing we did was we were doing quantitative churn analysis. Essentially, everybody who canceled, and on average, people were canceling, some after one month, some after two months, three months, and our history wasn't huge right at that time.

We wanted to see what's the difference between people who churned right away—maybe after one month—and who are still users; what's the difference between those two groups?

What we learned was that there was only one thing that we could see, which was–company size. Bigger companies stayed on as subscribers, smaller companies churned.

Our hypothesis based on that was that while smaller companies just have less money to spend, especially on education. Also in our interviews with the small company people, we learned that they are very tactics oriented. They wanted something that I can implement right now and make money right now versus bigger companies thinking long term, thinking strategy, because they have the luxury to think long term.

One of the things we did based on this information, we decided to change who we're after. We thought we were targeting small businesses, people who can't afford our agency. But actually, the target audience turned out is the same audience that uses our agency services. So our ideal customer is somebody who makes at least $10 million a year online and agencies, agencies is the one rare exception where small business that thinks long term, invests in education, and is used to paying for it. So that was one thing.

Another thing we learned from our qualitative research was that some people just completed the CRO course, even though every single month we added new content. But some people were eager. They just completed–"Oh yes, I finished it, so why keep on paying?" I said, "Well, you have a point there." So what do we do then?"

Well, we need other stuff for these people to buy. We needed more products to sell, essentially. We started to think what those could be. Could we do more courses and what those could be?

Another thing we saw in the data was that a lot of people who were churning were actually not consuming the content. While some people finished everything and that was good, that was not the case for most. Most bought the course, logged in, and never logged back in again or maybe they logged in twice. Then we learned that this is not just–at first, we thought it's us and then we learned this is the learning industry. It's the same problem for Udemy, Udacity, Coursera, edX, the people who invested all this money with Coursera and Udacity, I don't think they're ever getting their money back because their completion rates are horrible. It's like 10%, 12% of people who enrolled in a course.

I thought, "Well, I've been running this conversion coaching programs for many years now," for a high ticket, $2,000, and these were very high engagement rates. The only difference was that I was delivering the classes live, and then we thought why don't we add on another product that will be a live course, live online sessions like webinar type of situations? And I don't scale so how about we recruit another expert to teach this course?

So we did. First, we added on a new product, which is another training, using a third-party trainer and that made a huge difference. Ever since then, now, we are producing two new live courses every single month because the format works. People learn better, we make money too, and then the completion rates for these live courses are way, way higher.

[00:10:59] Louis: That's great delivery of the story. Let's break it down practically for people who are interested in doing the same–in their own business and perhaps they struggle with the same struggles that you had when you started at the institute.

Let's go back to customer development, because you said a few very interesting things. So first of all, you did interviews and surveys?

[00:11:21] Peep: Right.

[00:11:22] Louis: How did you go about that?

[00:11:23] Peep: We agreed on a protocol that every single time somebody signs up, that we basically sent them a welcome email, "Thank you for joining. By the way, would you have time to hop on a 10-minute phone call with us?" Most did not but enough people did. So we started to get people on Skype calls and online meetings with us, or just phone calls, and then we developed an interview set of questions and we had two people basically as part of the daily routine, call up maybe 5, 10, 15 people a day, so that was one. The cancellation, why people...

[00:12:08] Louis: Let me cut you right there because I think we need to dig a bit deeper again into this. You had two people, part of their routine every day was to call those people. What is the rough amount of people who agreed to be on a call with you from those emails? Do you remember?

[00:12:28] Peep: Not very many, I would say there were less than 10% for sure. Maybe 8% will be the right ballpark. So most people don't–they did reply, "Oh yeah, thanks for the offer but no thanks." People are busy; they have their own lives–it's okay. We didn't let ourselves be discouraged by the low response rate.

[00:12:54] Louis: What questions did you ask them?

[00:12:56] Peep: We wanted to understand who they are, what made them sign up, which problem they are solving for themselves, how are they going to use this knowledge. Basically, really trying to understand who these people are and what are they trying to get out of it.

We would also do a small walkthrough of the product, look in here for this thing and look in here for that thing that you might need. It was mostly about figuring out who they are and what they want and then informing our product. That, we coupled with actual analytics because we had analytics on what are people doing inside our product.

For instance, when we launched we also had weekly UX research studies and people were not consuming those based on analytics. When people were on the onboarding course, they did not mention that they signed up because of the studies, so we killed it. We stopped doing those studies. I actually made them all free on our blog and that also basically helped us cut costs so we stopped producing content that nobody wanted, or at least they didn't want to pay for it.

[00:14:10] Louis: Which is a big difference and it helped your top of the funnel, like your very top of the funnel making this content available for anyone.

[00:14:19] Peep: Sure, the content on the blog performed on average a little bit better than our average blog posts. Not hugely better but also I mean–I like to study because nobody else does as the original content great for attracting backlinks. But most of all it helped us gain focus for our paid product development. Stop doing this, do more of this other thing. Change the messaging on the website. Let's stop talking about the UX research studies and focus on the CRO aspect.

[00:14:55] Louis: You did those interviews and surveys. How did you make sense out of them? What is the typical process that you used for this one?

[00:15:04] Peep: Every call was documented. We used Google spreadsheets. During our weekly meetings, basically we read through them and the number of responses–it was not like we had to read through thousands and thousands of responses, we were just getting started.

Basically, every week we had 10, 20, more. It was a manageable quantity and the things that people say started to repeat a lot. So we immediately started to see patterns.

[00:15:40] Louis: Then you did cancellation on interviews and surveys. That's different from the ones where you're actually onboarding them and asking questions about their intention.

[00:15:48] Peep: Exactly. Here, as soon as they canceled, we sent them an email asking, "What made you cancel?" Even worse, response rate, compared to onboarding–but then we flipped something around. We flipped it around by when people are canceling, they couldn't just cancel. They had to choose cancellation reason in our product itself.

Before they could hit "cancel", they have to choose from five top reasons plus "other" where they could just write in the reason like, "I don't have time. I don't have money. I don't have X, Y, Z"–typical reasons that we saw. That was a really good move because every single cancellation, we are seeing the reason why people are canceling.

[00:16:35] Louis: You used the multiple choice question–I mean, the single choice but the questions–the categories that people could answer with in this cancellation survey were filled from cancellation interviews you did in the past and you can now have a rough idea of the reasons?

[00:16:52] Peep: Exactly. Then we just wanted to, A, get more responses and B, quantify what are the top reasons out of those.

[00:17:03] Louis: Remind us, if you had to pick the top three reasons why people were canceling, what were they?

[00:17:10] Peep: Number one by far, "I don't have time," because when is learning a priority? People have their to-do list–a million of things in them. How often is learning? Do you have learning in your daily to-do list?

[00:17:23] Louis: Nope.

[00:17:24] Peep: Yeah, most people don't. That's again why live classes helps us combat that because now the classes happen in a specific time, like Tuesday 2:00 PM, so people actually put it on their calendar. They make time for learning. Whereas self-study, difficult, very difficult.

The second reason was it's too expensive. We can't justify it.

[00:17:49] Louis: This is probably one of the toughest thing to do, isn't it? To change people's behavior, to make them do something that they genuinely don't do. Instead of maybe pushing people to do something that you know will never do what you've done is change what you are offering in order for them to change their behavior.

[00:18:06] Peep: Exactly. Change their mindset around what online learning is like, because if I have all the time in the world to consume a course, chances are, I never will.

[00:18:15] Louis: Exactly. Those are the very qualitative and human-based things you've done to understand what was going on. This is really based on people, isn't it? This is just individuals like you and me that you talk to. So that's interesting.

[00:18:28] Peep: Exactly.

[00:18:29] Louis: But then you did something a little bit more, I would say, scientific, even though it was also based on people. You did churn analysis. So tell us a little bit more about this particular thing?

[00:18:38] Peep: What we did was–we exported out our subscription data into Excel and in Excel we split the subscription listing to two people who were members only, maybe one-two month and people who were still members or who canceled after four months, who had a longer retention period. What's the difference between those?

Some were Gmails and Yahoos, so we couldn't really use those. We filtered out all the free email servers and were left with actual business domains. Then, for those domain names, we did SEO analysis, meaning that we looked at their domain authority, whether they're big sites or small sites, and then also with Clearbit, we enriched the list.

Clearbit gives you data on revenue and who these people are, employment seniority level, and things like this.

[00:19:46] Louis: Let me break it down right now because once again this is very critical. SEO analysis–SEO stands for Search Engine Optimization, and Domain Authority is a measure that was invented by Moz to understand how popular a website is and how influential it is. It's ranked from 0 to 100.

This is interesting. First of all, you looked at that, you had Excel, you use Google spreadsheet. Do you use an extension to plug the domain authority–how did you go about it? Do you remember?

[00:20:15] Peep: To be honest, I don't remember. I think we extracted the domain names from the spreadsheet using a filter in Excel and then we put them in a bulk–basically exported bulk into an SEO tool. Maybe we used Ahrefs. I don't remember which one, actually. Basically, we got the data for each domain fairly quickly.

[00:20:40] Louis: You have this column where you had this number of domain authority.

[00:20:43] Peep: Exactly.

[00:20:45] Louis: You did something quite interesting. You used data enrichment, which is a way from one email address to actually get a lot more information about people, their role, the company they're in, their industry, the size of the company–there are so many things, right? You used Clearbit for that. What information were you looking at in particular?

[00:21:06] Peep: Clearbit will give you like 50 data points. We were mostly interested in the company. We didn't know what they were looking for. But quite soon, we started to see that maybe company size is one of those things. We looked at annual revenue.

Also, it gives you Alexa Rank, so we looked at that and job titles, like who these people are. Are they specialists, are they managers? Of course, with Clearbit, the job title is like One-Man Show says that I'm a CEO. Both the CEO of Walmart and the One-Man Show is both the CEO, so it gets muddy. But the revenue stuff combined with the domain authority made it pretty clear that more established, bigger companies were the ones that retained longer.

[00:22:01] Louis: This is crazy. This is very a good insight from an Excel spreadsheet. It's further away from human-to-human type of relationship, even though you use that to really understand people.

Clearbit is a great tool from our own experience in Hotjar when we used it. You don't expect to get 100% of the data on 100% of the people. What happens most of the time is you get some information on certain things and not on all others.

[00:22:31] Peep: Exactly.

[00:22:32] Louis: You need to have a certain amount of data to start making sense out of it like you've done.

[00:22:37] Peep: Right. That is the reality of the world, that we are always operating with imperfect data. Nobody ever has perfect data–even in our lives. Who are you going to marry? Well, you don't have perfect data how it's going to work out. You don't have perfect data about taking a job offer. We are always making decisions based on imperfect data. It was a race against the time, right? We needed to make a decision.

Especially once we got closer to the point where we had one month left. Like in A/B test, if you need a big uplift, you can't change just one small thing. You need to change a whole bunch of things. In this one month leading up to our running out of money, we changed a lot of things and it worked.

[00:23:24] Louis: To summarize your story and especially the key points that you said that are critical. You did customer development, which is basically a way to understand people and what they're trying to achieve. You did customer development interviews during onboarding, to understand what people were looking for and who they were. You did cancellation interviews and surveys to understand why people were leaving, and you added this customer survey at the end of each when somebody was trying to cancel to understand why [editor's note: if you would like to send surveys like Peep but don't know how to make sense of the answers, here's the method we developed at Hotjar to analyze open-ended questions.]

[00:23:55] Peep: Product analytics to see what people are using and what are they not using.

[00:24:00] Louis: Yeah. Product analytics as well, as part of customer development, then you did churn analysis to understand at scale the profiles of people depending on their behavior. You used two things. You used SEO analysis to understand the influence of the domain and then you used data arrangement, to understand those email address, who they are like behind this email address.

As you said at the end, which I really like, "You will never be able to take a decision based on 100% perfect data." Most of the time you will have 20% of the data that usually gives you 80% of the insights. It's easy for me to say that from now, but from experience it seems like this is usually the case.

[00:24:43] Peep: Absolutely. I mean, that is the daily life of anybody who's doing any optimization online. You build your hypothesis based on some data points you have available—sometimes it works, sometimes it doesn't.

[00:24:58] Louis: Outside of the resources you mentioned during this call, do you have anything that you would recommend to listeners or the viewers to be more human, to be closer to the customers? It could be podcasts, blog, videos–anything that you think could be helpful for them.

[00:25:15] Peep: For customer development, I've been in the past really influenced by Steve Blank, Startup Owner's Manual, some steps to epiphany. I don't remember how many.

[00:25:27] Louis: Four.

[00:25:29] Peep: The Lean Startup.

[00:25:32] Louis: I would say that you should also check out ConversionXL, and CXL the blog, and CXL Institute. As you mentioned, new content every two weeks is really helpful with practitioners that are experts in what they do and very niched. Definitely check that out.

Peep, once again, thank you very much for your time.

[00:25:53] Peep: Thank you.

[00:26:02] Louis: Thanks for listening, my fellow human. We know how fast paced life is. If you're listening to this on your daily commute, while running, or even cooking, you can always go to Hotjar.com/humans and look for today's episode. That's where you'll find access to all the resources and humans we talked about, the full transcript of the conversation, and even links to really see the episodes.

[00:26:25] David: If you like today's episode, please help us out by leaving your honest rating and review on iTunes or wherever you get your podcast. The more honest feedback we get, the more we can improve the show for you, and the more this podcast will be discovered by other humans. It's a win-win situation. Until next time, take care and be human.