Lean UX: definition, process, & a detailed case study [2018]

October 31, 2018 by Louis Grenier

Scott Adams, creator of the Dilbert comic strip, has spent the past 30 years making fun of inefficiency in business—including our efforts to reduce inefficiency.

Lean UX Dilbert Comic

The truth is, it’s hard to get everyone on the same page in any organization, big or small. That’s why Doodle, a Software as a Service (SaaS) company with just over 50 employees and 250 million active users, turned to Lean UX to improve its products in an efficient, evidence-based, customer-centric way. In other words, Lean UX helps Doodle’s product teams make the changes their customers want in a smart, efficient way, so they can grow faster and boost their bottom line.

I recently sat down with Jack Berglund, Chief Product Officer at Doodle, to talk about his product development strategy. He was very transparent, so if you’re looking for a real-life example of how to use Lean UX for digital product design, you’ve come to the right place.

Table of Contents

What is Doodle and who is Jack Berglund?

Doodle is an online tool that helps people schedule meetings more efficiently. It syncs with all the major calendaring systems (Google, Outlook, Apple’s iCal, etc.), and it lets users poll meeting invitees to find a time that works for everyone.

Founded in 2007, Doodle is headquartered in Zurich with offices in Berlin, Tel Aviv, Israel, and Belgrade. Today, more than 250 million people use Doodle each year, and it’s available in more than 25 languages.

As Doodle’s Chief Product Officer, Jack Berglund uses Lean UX to guide agile, cross-functional teams to develop products that customers love.

Does that sound a little jargon-y? Don’t worry, I’ll unpack it all for you… and I promise it won’t sound so much like a Dilbert cartoon by the time we’re done.

Lean UX Dilbert comic2

What is Lean UX?

Lean User Experience (UX) Design is a process that embraces Lean and Agile methods to reduce waste and build products centred around the users. Lean UX grew out of earlier process management systems like Lean Manufacturing, which has been used by major companies (such as Intel, Nike, Toyota, and Ford) to eliminate waste in production. Lean UX took principles that were originally designed for physical products and adapted them for software development.

Lean UX production plant image

Lean UX takes its roots in Lean Manufacturing,
which is used by major companies across the World.

What makes software production different? For one thing, you don’t have to follow the old “measure twice, cut once” idea. Instead, your Agile product teams (which are composed of experts from different departments) can come up with ideas, test them out quickly, and revise them based on customer response.

As Jeff Gothelf puts it in Lean UX: Designing Great Products with Agile Teams, “You can measure once, cut once, measure once, cut once, measure once, cut once, forever.”  That’s the luxury that SaaS and e-commerce product teams have. Since they work in a virtual space, they just don’t have the same physical restrictions as Toyota or Nike.

How so? If you’re Toyota, you can’t try out five different steering wheels in one month to see which one improves the customer experience. But if you’re Doodle, you can add a new checkbox to the “Make a Poll” product, see if it gets any use, and tweak it accordingly.

 

Lean UX in action: Doodle saw a 54% increase in the number of free trial signups by changing the language on their homepage to focus on business use-cases, which was a greater increase than expected.

3 Lean UX phases: Think, Make, Check

In the Lean UX model, product teams are usually made up of members from different departments (e.g., engineers, product managers, designers). Each team works to improve a specific product by focusing on improvement processes that cycle through three phases: Think, Make, and Check.

Lean UX process

The Lean UX model follows a three-phase cycle: Think, Make, and Check.

Think

Teams brainstorm possible areas for improvement based on customer feedback, customer research, competitor comparisons, and observing their product in use. Then they decide which areas they want to improve.

Make

Designers and developers build a new feature that will (hopefully) solve a problem and/or improve the product.

Check

Product teams test the new feature, using tools like surveys and A/B testing, to figure out whether their hypothesis was correct. If customers respond well to the new feature, it becomes part of the new design. If it doesn’t improve the customer experience, they return to the Think phase and try something new.

How Doodle uses Lean UX

Doodle uses Lean UX to anticipate their users’ needs. This allows them to build cutting-edge products their customers will gladly pay for, and keeps them one step ahead of the competition.

They have five product teams, and each team is usually composed of at least one product manager, one product designer, and a collection of engineers—possibly including data scientists or data engineers, as needed.

Lean UX Product Team

A typical product team at Doodle includes a Product Manager, a Product Designer, a Data Scientist, and multiple Engineers.

The idea is to create a cross-functional team (meaning, teams from different departments) who come together to work on specific product improvements (such as Doodle’s “MeetMe” product, Polls, etc.). Each team has a slightly different makeup depending on the product they manage.

 

Lean UX in action: Doodle tested out a new calendar invite feature that put the agreed-upon meeting time in all attendees’ calendars, and 40% of their customers used the first version of this feature. This demonstrated how strong the demand was for the feature, and gave the team a clear sense of where to go next because there was already plenty of room for improvement on that initial version.

Here’s how Doodle approaches their Think, Make, Check phases:

1. The “Think” Phase at Doodle

Doodle uses Airtable to maintain their “Idea bank,” which is a collection of all ideas submitted for improvement and, at the time we talked, contained 175 ideas.

Lean UX Idea bank in Airtable

Doodle uses Airtable to collect ideas and prioritize them.

“We use the idea bank to capture all ideas, wherever they come from. Even if we think they're terrible ideas, we still write them down, and that serves two purposes. One, we want the person who made the suggestion to know that it's been captured and it's up for consideration. Two, we do this because what might not be a great idea now, for whatever reason, might become a great idea in the future.”

 

Lean UX Jack Berglund

 

 

Jack Berglund - Chief Product Officer, Doodle

Of course, ideas don’t come out of thin air! The product teams at Doodle use the following tools to gather qualitative research from their customers and capture ideas for product improvement.

3 Lean UX tools Doodle uses to find areas for improvement

  1. Google Analytics: “Google Analytics is a good source for us in terms of user behavior within the product,” says Jack.

    Doodle uses Google Analytics for cohort analysis, which is all about breaking users down into groups (cohorts) and studying the behavior of each group. For example, Doodle can tell whether a user who connected through their Calendar Connect product is retained better than those who have not. They have their own data warehouse to complement this information.

  2. Desk.com: Desk.com, a Salesforce product, allows Customer Success (CS) to tag different types of issues that arise.

    “For obvious bugs,” Jack says, “it’s a bit easier: if something isn’t working the way it’s supposed to, it’s pretty obvious that you need to fix it. Where it becomes more complicated is when you receive input, either from research or CS, that speaks to something that isn’t so black and white, like a feature we could add down the line. At that point, it goes from an engineering/problem solving matter to a question of prioritizing and using your judgment,” he says.

    In the next section, I’ll talk about how Doodle prioritizes projects like these.

  3. Hotjar: Doodle uses Hotjar’s Incoming Feedback tool to have users rate their level of satisfaction and describe their experience. If the customers provide an email address, someone responds. “However,” Jack says, “even the anonymous feedback can be helpful.”

Lean UX Incoming feedback

Doodle collects feedback from different types of users, segmenting the data according to each group. In particular, they want to know which product the respondents are using (e.g., MeetMe, Polls), whether they are the ones who created the poll or simply participated in it, and whether they’re using the paid version of the product.

Lean UX Hotjar Feedback example

Doodle uses Incoming Feedback by Hotjar to collect feedback from users at scale and in context.

Product managers and product designers read the replies and look for patterns, but at the moment they don’t have a formal system in place to review this data.

Where Doodle did take a more organized approach to Voice-of-the-Customer feedback was when they redesigned their website in 2017. This was a complete redesign, going from a late-90s style to something much more modern, and they used Hotjar’s Incoming Feedback tool to figure out what their users liked, disliked, and needed.

Categorizing projects according to goals

Once they submit ideas, Doodle’s project teams categorize each one based on the goals they want to achieve, and those goals are grouped into Objectives and Key Results (OKRs). OKRs, in short, describe big-picture objectives, such as “convert more users after the trial ends.” To learn more about organizing by OKRs, take a look at this article by Jeff Gothelf (the author of Lean UX, mentioned above).

Lean UX Company Objectives

Doodle uses Objectives and Key Results (OKRs) to measure their projects.

Doodle uses OKRs to identify their larger goals. Then they break their OKRs down by Objectives and Key Results, which identify the smaller goals that will hopefully help them achieve their Objectives. For example, the “convert more users after the trial period ends” Objective might be achieved by encouraging users to create more polls, since Doodle’s research shows that users who create polls are more likely to convert.

Therefore, “get users to create a poll during the trial period” could become a Key Result, and that KR would fall under the “convert more users” Objective.

Lean UX Objective - Key Results

An example of an Objective (convert more users) and Key Results attached to it.

As the team captures more ideas, they tag them according to the appropriate OKR. Then, when they discuss plans for reaching those goals, they can use Airtable to filter the results, only showing those that relate to certain Objectives and Key Results.

Idea generation and problem-solving: a tale of two meetings

In the beginning, the product teams would brainstorm new ideas and discuss the progress they’d made on current projects at one meeting, but that got confusing.

These days, they have two separate meetings. Each team meets monthly to generate ideas, and they meet weekly to discuss their progress on current projects.

“In order to get a balance there, we actually have two different versions of the same meeting. One is weekly, where we go through the stuff we're working on, making sure that it's progressing. And then there’s a separate meeting where we generate ideas and talk about what we want to do next.”

 

Lean UX Jack Berglund

 

 

Jack Berglund - Chief Product Officer, Doodle

2. The “Make” phase at Doodle

How Doodle prioritizes projects

During the monthly meetings, whoever owns or champions an idea gives a 60-second talk about the idea, the data related to it, and how they might approach it as an improvement project. After the team reviews all new ideas they take a vote, pick the two or three most promising ideas, and really dive into them. The best ideas get added to the list of active projects.

Lean UX Idea meeting

An example of an Objective (convert more users) and Key Results attached to it.

The ICE framework

Doodle uses a system called ICE, designed by Sean Ellis. ICE stands for Impact, Certainty, and Effort, and potential projects are evaluated on these qualities. Here’s what the ideal project would look like in the ICE model:

Impact (on User Experience): High

Certainty (whether it will work): High

Effort (what it would take to make the change and test it): Low

Of course, most projects aren’t so straightforward.  Jack says, “There are ones where you say, ‘Look, the effort is significant. We think the impact is high, but we're not completely sure.’ In that case, before we spend that full effort, we see if there’s something simpler we can work on.”

Jack stressed that they don’t take anything for granted when they assess certainty. If the idea’s champion thinks it’s a great project to work on, that counts for .01 on a certainty scale of 0 to 10. If the stakeholders love the idea and think it’s brilliant, that might bump you up another 0.1, but you’re still a long way from a 1, much less a 10.

Lean UX Doodle team meeting

The Doodle team prioritizes ideas using the ICE framework.

“What I really like about this approach,” he concludes, “is that it pushes us to do the necessary things to validate our ideas.” Whether it’s asking very specific questions to customers, bringing out specific data, or doing more A/B tests (see below), this framework allows Jack and his teams to save a lot of time down the line and make sure that all the work they are doing counts. That said, the ICE model isn’t a rigid scoring system. It’s a simple tool that gives you a rough idea of what to prioritize.

 

Curious to see how Hotjar prioritizes projects? Check out Mastering the art of prioritization: how Hotjar decides what to work on next.

3. “Check” phase at Doodle

Lean UX doesn’t rely on gut feel to decide whether a change has been successful, and neither does Jack Berglund. The teams at Doodle test their changes to see how users respond and later refine their approach accordingly.

A/B tests

If you have a lot of traffic to your website, A/B tests (or split tests) are a great way to validate your hypotheses scientifically and determine if the changes you made are working. Once you’ve got that result, you can go ahead and implement the changes confidently.

“We have a good amount of traffic, so we can do a lot of A/B testing. And if we're smart about the tests we run, we can save ourselves a lot of time by not implementing things that are basically low confidence,” says Jack.

Example #1: An 800% increase in the use of a feature

Sometimes the results can be surprising—like, “an eight-fold increase in feature use” surprising. When users create a poll, which they send to meeting invitees to determine which date works best for everyone, they can add a “Yes, if need be” option. This allows invitees to select their preferred date (the green check mark) along with any dates that are less-than-ideal but would still work (the yellow check mark).

Lean UX screencast-doodle.com-2018.10.12-10-13-00

Originally, this was an advanced option, meaning that it didn’t pop up in the “Make a Poll” wizard that the average user sees. The product team assumed most users wouldn’t be interested in that feature, but they questioned their assumptions and tested it out by adding it to the wizard as a checkbox.

The result? They saw an 800% increase in the use of that feature. Who knew?

Example #2: The fake door that got them nowhere

Jack shared this with me as a cautionary tale—the story of a time they set out to study a problem with A/B testing but ended up studying the wrong thing. They didn’t get the information they needed to move to the next steps, but they learned a valuable lesson.

Doodle wanted to see whether having a “Poll Deadline” button, which allowed poll creators to set a deadline for respondents to reply, would get any traction. They set up a fake door, which is a button that offers a feature that isn’t actually available, so the UX team can see whether people will use it before spending the time and money to develop it. When users click on a fake door, they receive a message telling them: “Sorry! This feature is not yet available.”

Doodle showed the fake door to a portion of their users, and people clicked, but the team soon realized they had no way of knowing how useful the feature would be to their clients. Sure, they got plenty of clicks… but was the button valuable enough to influence buying behavior? Would it increase conversions? They couldn't tell from this A/B test.

To answer that question, Doodle went back to the drawing board. They’re currently planning to run a test that measures how likely this feature is to influence a user’s decision to start a free trial, which is a more complicated and business-focused question than whether or not a user will click.

“The core lesson is that we need to measure behavior change that demonstrates real value to the user.”

 

Lean UX Jack Berglund

 

 

Jack Berglund - Chief Product Officer, Doodle

By designing smart, strategic A/B tests that answer basic questions about buying behavior, the teams at Doodle can prioritize their tasks and focus on those that offer the greater return.

Starbucks tests

It’s not always practical to do A/B testing, even when you have a ton of traffic, because sometimes you need to move quickly to see if an idea is worth looking into at all. You can always do more structured testing later if an idea shows promise after ‘guerilla-testing’ it (an idea that Jeff Gothelf introduces in Lean UX).

Guerilla tests are ‘quick and dirty’ user tests that give you a sense of how your ideal users might respond. Much like focus groups, which advertising agencies have been using for more than 70 years, guerilla testing gives you quick feedback when you need it.

Since Doodle has 250 million users (that’s the population of Indonesia, by the way), and since many of these users are similar to people the Doodle team members rub shoulders with at their local Starbucks, they like to use Starbucks for guerilla testing new features. They go down to the Starbucks near their office and offer people a voucher for a free cup of coffee to test out new features.

“If you ever want to try to get a Starbucks voucher for a free cup of coffee, just hang out at the Starbucks near the Doodle office in Zurich. You never know!”

 

Lean UX Jack Berglund

 

 

Jack Berglund - Chief Product Officer, Doodle

Starbucks testing is just one form of anecdotal feedback they use. Doodle also goes to customers, employees (including marketers), and product management stakeholders to get their input.

The guerrilla testing approach of Lean UX works better for SaaS and eCommerce than it does for companies that build physical products because, again, you can make changes quickly and reverse them if they don’t work. And in many ways, that points to the basic difference between Lean Manufacturing and Lean UX.

Lean UX is all about agility, experimentation, testing, and reassessing in a continuous cycle. It has allowed companies like Doodle to scale to 250 million active users—and it’s helped them thrive in a competitive market.

 

Lean UX in Action: Doodle had the “Settings” page automatically appear as the third step in their poll creation wizard, which increased the number of free trial signups by 25%, boosted the use of their “Ask Attendees for Contact Information” option by 300%, and increased the use of the “Hidden Polls” feature (which allows the creator to keep poll results hidden) by 400%.

Want to give Lean UX a try? A simple roadmap to get started

Think Phase

  1. Gather customer feedback using tools like Incoming Feedback from Hotjar and reviewing insights from your Customer Success or sales team.
  2. Process this info in an idea bank using a simple spreadsheet or a tool like Airtable.

Make Phase

  1. Set up a meeting to prioritize projects according to their Impact, Certainty, and Effort (keeping in mind that certainty is determined by facts, not stakeholder opinions).

Check Phase

  1. Use guerilla testing to get quick feedback where appropriate.
  2. Use A/B testing, if your traffic is large enough, to get scientifically valid results.

Join 6,600+ marketers and designers who receive our blog posts in their inbox

Louis Grenier

Content Lead at Hotjar - Louis is a major marketing geek who believes that good marketing starts with understanding people (and not tricking them).

comments powered by Disqus