Learn / Guides / Product experimentation guide
How product validation can help you run better experiments
Building a product from the ground up can be quite the journey—full of unknowns and assumptions. Having reliable evidence before you make any big product changes or investments can be a lifesaver in terms of reducing risk and saving valuable time, money, and resources.
After all, who wants to put a product into the market only to find out months later that it isn’t commercially viable?
That’s where product validation comes in.
Like experimentation, validation is a critical part of building new products. It proves that customers want your concept, that you can build it, and that it’s profitable. This guide shows you how the process of product validation can tie into product experiments—and how the two areas are aligned.
Test and prove your business ideas, fast
Hotjar’s product experience insights help you reduce assumptions and build products that users want and love—before making heavy investments in time, money, or resources.
What is product validation?
Product validation is the process of testing an idea with potential users to get feedback on a product’s viability.
It checks whether a product or feature addresses the needs and pain points of current and potential customers. Engaging in product validation early in the development process means you’re strategically directing your time and funds to create the best product possible.
Product validation comes in different shapes and sizes, but the most common things to assess are:
Desirability: is this something users want?
Usability: is it intuitive enough for users?
Feasibility: can we actually build it?
Validation comes in the form of data that demonstrates that the current product has addressed key risks in the business. These risks will initially show up as assumptions.
Defining risky assumptions—and implementing a product experimentation framework to confirm or refute them—is the way to validate desirability, usability, and feasibility at various stages of product development.
How is product validation different from product experimentation?
Validation is most useful to assess a new feature or product. It often happens at the earliest stages of product development and can be essential in determining if a product is on track to success.
Experimentation is a form of product validation. It’s most useful to assess iterations and improvements to a feature or product once it’s already been built. Validation can help inform the direction of product experiments and help to reduce overall assumptions.
Later in this guide, we’ll explain the nuances between product experimentation and product validation so you know when to use each one to your advantage.
How to recognize true product validation
Product validation is not a one-time event—it’s an ongoing process that will last throughout the product management lifecycle.
Products can find sources of validation along a variety of metrics, like high engagement and viral coefficient or long-term retention. What’s important is that the data tells a compelling product narrative to demonstrate that the business is on solid ground.
Identifying your riskiest assumptions and creating a plan to validate them ultimately provides you with a data-backed way to stay ahead of any bottlenecks. Your goal is to reduce overall assumptions and focus on what matters for your users. This way, you can lower the risk of wasting resources on development.
A good product validation process will help you generate assumptions to answer the important questions about your product:
Can it be built?
Should it be built?
Will people be able to use it?
Once you’ve established these questions, the next step is to answer them as sufficiently and cost-effectively as possible.
4 common product validation methods
Validating your business ideas before investing too much too soon will set your product up for success. To do that, you need to go through the four types, or levels, of idea validation methods:
Assessment
Fact-Finding
Tests
Experiments
Each category contains a variety of validation methods, and more are being created as you read this. Product validation is really only limited by creativity and a willingness to step outside your comfort zone.
Some methods are better suited to test desirability, while others work best for usability, and feasibility. Whichever method you use, make sure you validate all aspects across the entire assumption validation effort.
1. Assessment
This step is all about determining quickly, with no external research, if an idea is worth moving forward with at this time. These methods will act as important filters to help you evaluate your idea objectively and in a structured way, considering:
Goals alignment
Business modeling
Stakeholder reviews
Assumption mapping
2. Fact-Finding
Next, you want to look for available facts and data that support your idea—or refute it.
These key fact-finding techniques help you collect feedback on use cases, user expectations, and ideas for improvements, and include:
Data analysis
User interviews
Surveys
Field research
Competitor analysis
3. Tests
It’s time to put a version of your product in front of users and measure their reactions. The structured process of testing your assumptions will save you time and confirm that each part of your work creates value for users. Test types include:
Usability tests
Human-operated tests
Smoke tests
Dogfood tests
Early-adopter programs
Large scale tests
4. Experiments
If you’ve already read the first chapter of this product experimentation guide, this step should be familiar territory. Experiments are tests that include a control element to guard against false results caused by chance. For example, an A/B test is an experiment, but a usability test is not.
As a form of validation, product experimentation helps you gauge the effect of a change before actually implementing it. Experimentation tests include:
Percent experiments
Holdback experiments
Use Hotjar to collect data and generate ideas for experiments
Take advantage of Hotjar’s flexible settings for heatmaps, surveys, and session recordings, which is especially useful when you’re A/B testing and want to see how users experience each version of your design:
Determine which page elements visitors engage with using heatmaps. Look for trends in the way people interact with your most important pages so you can zero in on elements that need preserving vs changing.
Study session recordings to observe individual users as they make their way through your product, so you can see the experience from their perspective.
Collect customer feedback using on-page surveys and feedback tools to get open-ended feedback about what customers think about your site.
Why is validation important in product development?
As product people, you often have many ideas bubbling in your heads. Before getting too excited, you need a way to decide which ones are the most viable to pursue.
There are two ways to figure out if there will be demand for a particular product:
You can build and validate the idea at the final release. With this type of waterfall approach, you risk building the wrong thing, resulting in poor adoption or complete rejection of your product.
You can validate individual assumptions through tests and experiments at various points along the development process. This de-risk strategy will increase your chance of a successful launch. By testing your business idea and gathering evidence, you can determine whether people are willing to buy your product before you build it.
Validation works like a flashlight in the dark for teams building new products. In a fast-moving product development landscape, it enables you to move forward with proof that:
Customers will buy your product
The concept can be profitable
It’s feasible to build and bring to the market
Why teams skip product validation
One of the biggest and most common mistakes product teams make is rushing or skipping the validation process. Here’s one example of what that can look like:
They start with an idea for a product they think people want.
They spend months, sometimes years, perfecting that product without ever speaking with, listening to, or showing prospective customers the product, even in its most basic form.
They keep making mistakes, including testing the wrong aspect of the business, asking the wrong questions, and neglecting to define a criterion for success.
When customers ultimately communicate—mostly through indifference—that they don't care about the idea, the product fails.
Teams that don’t validate their ideas often have far more confidence in their product specifications than they should. Most often, they:
Don’t know how to validate or don’t want their idea to fail at the validation stage.
Think they already know what their customers want. They move forward and plan to adjust the product—if necessary—once they get beta feedback, far past the time for major changes.
Rush through the process because they're anxious to launch the product as quickly as possible.
💡 As a product manager, it's your responsibility to ensure that your team doesn't skip the product validation process.
The key to doing this is to reduce assumptions and increase confidence in your product. Validate ideas for future initiatives with reliable data based on unbiased experience—and empathize with how people use your product—to prioritize brilliantly.
You can do this, and it costs far less than you probably think.
Product validation strengthens your commitment to your business while keeping you laser-focused on the actions that will get you to market faster.
4 effects of skipping product validation
New business ideas will always have unpredictable elements. Without evidence-based foresight, there can be serious consequences.
Skipping product validation can lead to:
1. Out-of-touch product improvement decisions
For early-stage companies, feedback can become a bit of an echo chamber. You need more data points to truly get a sense of whether you're tapping into product-market fit.
Without a detailed understanding of user habits, behaviors, frustrations, needs, and wants, it’s impossible to make smart decisions about how, when, and where to improve a product.
2. Losing focus
The post-launch period can be a chaotic time, and it’s easy to lose focus and chase after shiny objects. In reality, you’ve probably only tapped into a very small fraction of the people you originally set out to reach.
When you focus on figuring out the right thing to build—what customers want and will pay for—you won’t need to spend months waiting for a beta launch to adjust the product's direction. Instead, you can adapt your plans incrementally, inch by inch, minute by minute.
3. Neglecting your team
Without product validation, the development team is often trying to build a product to an ever-changing spec, based on intuitions from the founders about what might be able to sell itself.
Without access to customer data, the product development team keeps building feature after feature based on what they think might be useful. Their efforts often don’t add up to much.
4. Talking to the wrong people
Assuming that others have the same problem you've experienced is a classic product development mistake. Many teams realize too late that the problems they’re trying to solve are not painful enough to sustain a business.
Instead of scratching your own itch, focus on turning ideas into products, measuring how customers respond, and learning whether to pivot or persevere. All successful processes should accelerate that feedback loop.
How does product validation relate to product experimentation?
Validation ties into product experiments, and the two are closely aligned in their goal of helping teams build better products. According to Andrei Beno, one of Hotjar’s Senior Product Managers, they’re also quite different.
Besides the fact that experimentation is, itself, a form of product validation, the core difference between the two, he says, is the stage at which each process takes place:
Product validation implies assessing if the product actually provides enough user value and makes sense to build (i.e. is there a viable business model to support it), and it’s often most useful for the earliest stages of a product’s development.
[...] experimentation is often done at a later stage, because you need to compare a new variation against the control, so you already need to have the control experience live (that’s your baseline).
Having validation for a product’s desirability, usability, and feasibility helps product teams conduct better experiments and feel more confident with their decisions.
Sometimes we test a new idea just to learn if it works. Other times we test a hypothesis and set clear success or failure criteria in advance. Here are the best times for product experimentation and product validation and when to use each one to your advantage.
When you should use product validation
Experiments may be the gold standard of validation, but there are other, more immediate ways to test an idea in the early stages of product development. By fixating on experiments too soon, many companies set the bar too high, miss out on easier opportunities, and often give themselves an excuse to keep doing things the old way.
Use product validation methods at the earliest stages of product development
Validation methods help ensure the solution will provide value by selling the concept to customers before building the product.
Even if the product isn't in its best shape, early validation will establish a baseline against which the product development team can try and improve. By the time it’s ready to be widely distributed, it will have already gained established customers, solved real problems, and offered detailed specifications for what needs to be built.
Set specific parameters by creating a clear plan for your research
Be as detailed as you can—it will keep you focused on gathering relevant, actionable data for your product development team.
When this process of measuring and learning is done correctly, it will be clear that your product is moving the drivers of the business model.
If it's successful, you can get started with your campaign: enlist early adopters, add employees to each further experiment or iteration, and eventually start to build a better, more successful product.
If it's not successful, it's a sign that it's time to pivot or make a structural course correction to test new fundamental assumptions about the product, strategy, and engine of growth.
Reap the benefits of early product validation
Companies that validate early know their customers extremely well, and are free to implement a consistent product experimentation framework.
Continuously experiment with new product features and test product marketing efforts to increase the conversion rate of each new batch of customers you bring in.
Over time, you'll find a formula for acquiring, qualifying, and selling to customers in the market segments you've targeted. In other words, you learn to grow renewable audiences.
Given the data you've collected about these early customers, you can also estimate with better precision how big the market is for your product in its current form and engage in rapid growth.
Pro tip: the earlier you start validating assumptions, the higher your chance of success. You can analyze log data, clickstreams, funnel analytics, heatmaps, session replays, feedback, CRM data, and anything else that comes from your users.
Piriya Kantong, a Senior Online Marketing Analyst at Zenprint, used Hotjar to reveal what numbers don’t:
“Funnels helped me identify where in the customer journey people drop off. Recorded user sessions let me understand what people see when they arrive on our website—what they click and what they don’t. Heatmaps helped me identify where they spend most of their time and assess if they should be spending time there or not.”
When you should use product experimentation
Companies don’t exist just to make stuff and earn money. They exist to learn how to build a sustainable business by consistently meeting customer needs. You can validate these learnings scientifically, by running experiments that allow you to test each element of your vision.
By the time you reach the experimentation stage, product validation will have helped you and your team feel more confident with your decisions. True validation should inform the direction of product experiments by helping define and reduce overall assumptions.
Then, you can use product experimentation to validate a hypothesis and test if a specific change in the product will have the type of impact you’re expecting. That change can be as small as a copy or button revision, or as large as a feature or product release.
How to validate ideas and create a product that users love
It can be tempting to build your product secretly. 🤫 But it's better to validate your ideas and what you’re building along the way, get input from your potential customers, and use experiments to keep working towards a product they’ll love.
To do that, you need first to map out assumptions, and then design experiments to get the evidence you need to take your product forward. Ask yourself:
What type of assumption are you testing?
How much evidence do you already have for that specific assumption?
What type of data is best to validate the assumption?
The answers will help you identify your assumptions, prioritize them, and think about how you can validate them.
Going through the assessment process before planning your experiments will save you lots of time and money. In the end, you want to end up in a virtuous cycle situation, where you:
Identify and map out assumptions
Validate them
Experiment
Analyze results
Identify new assumptions and start over
Here’s what that can look like:
Identify and validate your target market
Knowing your target audience is the first step to validating whether your product is viable. This is where you would determine if there’s an actual market with consumers who need and will pay for your product. Their feedback checks your company’s assumptions about your product—and this data is at the heart of product validation:
Set up interviews: start with a narrow audience and prioritize your roadmap by asking customers about their needs.
Conduct surveys: use surveys to validate the problem space and gain confidence in your hypothesis before building.
Pro tip: consider the purpose of your pre-launch surveys and craft your questions (and audience) accordingly:
Design a set of questions to validate that you empathize with your customer profile, their pain points, current solutions they use, etc.
Keep your survey as short as possible and follow up with an interview or feedback tool to gather more nuanced data.
It's helpful to both validate your understanding of your customer profile and learn new information about what outcomes they are responsible for, so you can better empathize with them.
A 5-point scale will give you a good sense of whether there's a market opportunity.
Research the market
At this stage, you’re looking to validate your production needs and analyze your competition.
Define important market dynamics like price value or range, possible suppliers, costs of production, and other barriers you might face in building the product.
Develop the long-term product vision and a strategy to execute it.
Create a strong story of what you need to build and sell the idea to different stakeholders across the business.
Conduct experiments
Build a prototype or minimum viable product (MVP) and identify problem areas and opportunities for optimization.
Confirm a hypothesis by watching users in the wild with session recordings.
Release new features with confidence by collecting ongoing user feedback from beta groups.
Generate leads with a 'smoke-test' landing page.
Use Hotjar to analyze a Smoke test
Smoke tests, also known as Fake Door tests, help you estimate the demand for a non-existent product or feature.
Create a convincing opportunity for customers to check out the new product and 'opt in' by creating a landing page and analyzing session recordings as customers go through the prototype.
Conversion rates at each step of the test will give you a sense of how desirable the product or feature is. When users choose to opt in, inform them that the product or feature isn’t ready yet, and offer the option to join a waitlist as a way to generate leads.
Review the effort to impact ratio
Stop assuming new launches will work. Build confidence in your product by collecting a reliable amount of qualitative feedback and a constant stream of quantitative data to validate your ideas.
Use recordings to check if the user’s behavior has improved.
Build Hotjar into your routine to regularly spot improvement opportunities.
Find new solutions to problems that resonate with users.
💡 True product validation leads to the best version of your product
Remember: product teams should feel confident with their decisions—and at the end of the day, product validation helps build confidence in your ideas.
Test and prove your business ideas, fast
Hotjar’s product experience insights help you reduce assumptions and build products that users want and love—before making heavy investments in time, money, or resources.
