Learn / Product Forge / Article
Different types of product testing and why they matter for product teams
Successfully launching a product requires a lot of testing—then more testing, and then testing again.
Often, product testing is seen as a separate activity to product development. But when testing isn’t integrated into the product development process, product teams can miss the insights they need to ensure their products work smoothly and solve user problems.
Last updated11 Jan 2022
This article will help you understand why product testing is so important across the whole product team, and which types of testing will work for you.
Hotjar helps product teams test their assumptions
Hotjar’s product experience insights tools support concept testing, user testing, and product tests for a product that delivers.
Why product testing is key for successful product management
As the old saying goes, you can’t manage what you don’t measure. Product testing is crucial to measuring the success of your product ideas, product functionality, and product changes.
Strategic product testing helps PM teams to:
Unearth valuable insights to better understand your product and, more importantly, your customers
Better meet user needs
Remove blockers to convert, attract, and retain users
Save time and resources by eliminating non-viable product ideas or prototypes early on
Gather clear data, and use it to make a case for stakeholder buy-in
Ensure your product stays relevant in a changing marketplace
Regular product testing is the key to satisfying user needs, meeting business targets, and keeping the product team focused on the right initiatives.
But remember: things change! Even if you’ve already done stellar research on your customers, competition, and current technologies, the product landscape is always evolving. To stay on top of customers’ shifting needs, you need a culture of product testing to test your assumptions.
Pro tip: test your successes. Don’t only test when you’re feeling uncertain or things are going wrong. Test products and features that are working well to understand why they’re succeeding at meeting user needs, and replicate that success.
What to keep in mind for successful product testing
When to test in the product development process
The best product managers create a culture of product testing that keeps the whole team connected with product functionality and user needs at every stage.
But there are different schools of thought:
Product teams using a waterfall approach typically run initial testing of the market and product concept, then don’t test again until product development is complete.
Agile product teams engage in continuous testing to gather information at every stage of the product development process. They test early ideas and markets, prototypes and MVPs, and keep testing after release in a process of continuous discovery. Instead of waiting for a fully developed product, agile product management teams test every functional piece for quality within the development phase, checking new product iterations as they go.
The lean startup methodology promotes testing cycles built on 'build, measure, and learn' principles. Lean teams are engaged in a continuous feedback loop, testing everything they build and using the data to improve the next iteration.
Whatever your preferred method, keep in mind that testing should be part of the full product lifecycle.
If you only test right before launch, you’ll find it’s often too late to make substantial changes to your product timeline, leaving you with a tough choice: go back to the drawing board and delay release, or push through with a product that may not resonate with your users.
It is a complete waste of resources to get to the final testing stage only to realize that a major component has to be changed.
To avoid such a scenario, product managers should be involved in the testing process from the beginning. They should gather and analyze feedback as it comes and attempt to identify areas for improvement. It is much better to make smaller changes along the way and perhaps prolong the testing phase than to leave everything for the very end when a product is all done.
How you’ll design your test
To test effectively, you need to be clear on what you’re testing.
Excellent test design starts with a strong research question—and your question should emerge from real user pain points and be as specific as possible.
For example, you might ask, “why are 40% of users not clicking on the new profile feature?” or “will the code for this upgrade allow users to more easily export their data?”
You can then develop a hypothesis to test, which could be something like: “if we integrate the new profile feature into the user dashboard, more users will click.”
The best option for preparing questions for testing is to involve a business analyst who knows the requirements of the market and client along with a product manager who understands the vision of the product. This way, you can create really great questions that will help make testing as useful as possible for development.
You’ll also need to establish how you’ll track and measure results and determine the test parameters:
Which users your test will target
How long you’ll test for
Which metrics you’ll use to test quality, value, and user interest or purchase intent
Which resources and operational costs will be deployed
Pro tip: don’t design test questions assuming you already know the answers. This can create confirmation bias and influence users. For accurate results, it’s important to develop fail conditions and go into product testing with an open mind.
Which team members should be involved
Traditional test methods often create silos between product development, product testing, and product management professionals, who are all involved at different stages of the process.
With a substantial team devoted only to product testing, the development and management teams may hardly be involved in larger companies.
It's absolutely critical for product managers who want to help build quality products to get involved with product testing as early as possible. A seemingly amazing idea can fall down immediately when it's put into practice, and when you're involved in product testing regularly you begin to see common pitfalls early.
In an agile culture of continuous discovery, developers, software testers, and PMs work together to design and run effective product testing. DevOps teams combine development and operations efforts throughout the product lifecycle.
Cross-functional collaboration is vital for product testing success. Aligning different team members on testing brings different perspectives and priorities to the testing design.
For example, if you work with the development team to turn product requirements into tests, you’ll end up with highly specific, actionable, technical test criteria.
Product testing as a team brings in vital perspectives that create a satisfying product with a wide reception. Bringing in stakeholder participation, rather than relying on idle communication, prevents a situation where a key stakeholder is out of the loop and ends up stopping the development or release of a product.
Leading a team culture of product testing also keeps everyone connected with the product’s impact and user experience—which cultivates user empathy, improves motivation, and unearths new insights that enable iterative improvements to more effectively meet user needs with your product.
When all product team members are involved in the testing process, responding to test data is smoother and faster.
Software becomes more complex every day: there are too many features for one person to have in their head at once. By having the entire team care about testing, people can cover each other’s blind spots.
How you’ll use the data
Gathering test data is one thing; using it is another.
Product managers are responsible for turning insights into action by using test results to guide development priorities, develop new solutions, and get buy-in from stakeholders.
When you test different options—as with A/B or multivariate testing—it’s important not just to pick the winning option and forget about the test data: every experiment you run is an opportunity to learn something about your product and users. Decide how to preserve that knowledge and incorporate it into your product workflow and future product research.
10 product testing methods: benefits and limitations
Deciding which type of product test to run depends on:
The kind of questions you need answers to
The user base at your disposal
The resources you have available
How far along you are in the development process
Make sure you consider the benefits and drawbacks of each. Here’s our breakdown of 10 key product testing methods:
1. Concept testing
Concept testing explores the viability of your initial product ideas. It’s a way of 'taking the temperature' on specific product concepts or directions, using written or oral presentations, surveys, paper prototypes, or wireframes.
It’s a great way to decide whether to proceed with an idea before you’ve invested significant resources.
Positive input can be used to get buy-in on your ideas.
It often generates new insights, helping you better understand what your users need.
Users aren’t interacting with your actual product, so the results can be a little vague.
Concept tests can generate false positives—users who seem enthusiastic about your product ideas but may not be willing to invest their time or money in it. You can ask specific questions about purchase intent to try to limit this.
2. QA testing
The quality of your product determines your users’ experience.
Quality assurance testing is crucial throughout the product development process, usually performed in production or staging environments. It can involve creating and running test cases (like running through what happens if a user tries to use a feature but forgets to upload a necessary file); full-scale, in-depth regression testing for changes and features; performance testing; and sanity testing.
The staging environment is crucial to QA testing because it’s the first stage where everything should work as designed and developed. This environment is not a version that the end customer can use, but it is totally stable so it’s ideal for polishing possible bugs before launching the product.
Learning whether the product operates as expected
Detecting bugs and defects quickly
Insights into product functionality
Avoiding user frustration by detecting blockers before launch
It’s functional testing rather than user testing, which means it won’t give you insights into real user experience or obstacles real users may encounter.
3. A/B split testing
A/B testing is a common test type that involves splitting your user base into two groups and giving them two versions of a product, site, or feature. There should only be a single variable, but it can be relatively minor (like using a different color button) or more significant (like a different feature hierarchy). The best A/B testing tools streamline or automate the process.
Gives you clear answers to very specific product design questions
Quick way of lowering the risk factor of new ideas
Can be run with current product users
It only works for specific, single-variable testing goals.
It can be risky to try a potentially unpopular idea with half of your user base.
It can be resource-intensive to set up, and deciding on the variables is tricky.
4. Multivariate testing
The principles of multivariate tests are similar to A/B tests, but while A/B tests only change one variable at a time, multivariate tests experiment with many variables or multiple segments of the user base. Multivariate tests try out different combinations of variables to determine which is the most effective.
Saves time compared to running several A/B tests one after the other
Gives you an overall sense of how much each variable contributes to the product experience
Since there are more variables to test than with A/B testing, each combination will be assigned to a smaller percentage of users, making the test process slower.
More complex results mean it may be difficult to make a confident decision based on the test process alone.
5. Tree testing
In tree testing methods, users are shown a streamlined sitemap with tree-like hierarchies. Then they're asked to complete specific tasks that test how easy it is to access the product functionality they need and offer insights into its navigability.
Can give quick confirmation of whether the user journey and product navigability are clear
Shows what needs to be restructured or relabelled for optimal user experience
Quick tests that are easy to recruit for and offer clear answers to usability questions
Since users are shown a stripped-down product map, you don’t learn how they would navigate the full product with visual elements that could help or hinder their use.
Offers only quantitative data— you won’t get a sense of why users make certain choices or struggle to navigate.
6. Card sorting
Card sorting methods ask a group of test users to sort product or website navigation items into the categories they think make the most sense.
In closed card sorting tests, users are given fixed categories and decide where to place each navigation item. In open card sorting tests, users create categories that seem logical to them.
Gives you insight into how your users think, which can help you understand how to design better UX navigation but also to better understand their key needs
Quick, easy, low-resource exercise
You can’t always translate the results directly into your product’s navigation hierarchy since users aren’t asked to do tasks with the navigation items, just to sort them.
You’ll need more information to understand the context behind your test group’s decisions.
7. Time-lapse testing
Time-lapse testing compares user experience metrics and product KPIs before and after a major product change has been released, to test whether the change has positively or negatively influenced user experience.
Gives you critical insights into how your real users are responding to product changes
Allows you to show stakeholders the success of product changes
Negative data can alert you to possible issues with product changes
There’s no clear causal relationship between user metrics and the product change as you’re testing in an uncontrolled environment with several variables.
Again, you won’t know why your users’ product experience has changed without collecting more qualitative data.
8. Eye-tracking and mouse-tracking
Eye-tracking tests use advanced technology to monitor users’ eye movements while interacting with your product, typically using a headset. Results are represented on heatmaps that show you areas where users looked longest and the places where they lost interest.
Click maps and scroll maps track the user’s mouse movements, showing where they click and scroll and how they engage with your product overall.
Gives you a quick, highly visual overview of how users are behaving
Gives an objective account of user behavior
Alerts you to potential issues by showing you where users are dropping off
Gives quantitative data that can help you justify where to prioritize resources
Raises questions that can be used to design more detailed surveys
Eye-tracking tools are costly, though mouse-tracking tools (like Hotjar's Heatmaps) are much more accessible and can be used to test your current users’ product experience.
The tests don't show you why users are behaving a certain way.
You’ll need to combine these tools with surveys or user interviews to get deep into the specifics of the user experience.
Eye-tracking tools can generate useful insights into what attracts the eye and improve visual design. But just because something attracts the eye doesn’t mean it stays in the mind. The most impactful elements are those that users understand, retain, and are able to recount five minutes later.
9. Canary deployments
In canary deployments, product teams release a new feature to a small group of users at the start and monitor the product experience over a short period to collect data on how users are engaging. The user groups can be filtered according to different criteria—they can be early adopters, users from a specific industry or region, or a random subset.
Offer real-world insights into how specific user groups respond to features
Show performance issues that might have been missed in a staging or production environment
It’s easy to roll back to the previous version if there’s a strong negative response or if a major issue emerges
The first 'canary' group may get frustrated by bugs and put off the new feature. This can be partially addressed by creating an opt-in program for early adopters who are aware they’re being used as testers.
Deploying two versions of the product side-by-side requires additional infrastructure investments and can take up the product team’s time and budget.
10. User observation
There’s no substitute for watching real people use your product or website. As well as gathering key data from their mistakes and frustrations, user observation tests get you closer to understanding what your real users experience.
Tests product experience (PX) in a holistic way instead of just testing one specific element
You see key blockers and bugs in action
Allows you to identify what’s working well by watching users who smoothly navigate the product
Can build empathy with users
In-person user observation tests require focus groups and can be time-consuming to arrange, not to mention you don’t see how users engage with the product in their home environment. Watching session recordings of real users can help you get around these limitations.
Without asking further questions, you still won’t understand the context or emotions behind your users’ product experience.
Pro tip: use Hotjar's Session Recordings to see how real users interact with your product—and whether they’re moving smoothly through the user journey or getting frustrated and ‘rage clicking’. Deploy specific filters to compare the product experiences of satisfied vs dissatisfied users and quickly see what the major blockers and successful product elements are.
Combining controlled test methods with open-ended research
Product testing methods can offer you clear answers to specific questions to help you improve your users’ overall product experience.
Controlled tests are critical to making key product decisions and feeling confident at product launch.
But you should also gather open-ended data to put all of these different types of product testing into context. VoC data that tells you what your users are thinking and feeling helps you stay connected to users and learn why they struggle with particular features or iterations.
Hotjar helps product teams test their assumptions
Use Hotjar’s product experience insights tools for concept testing, user testing, and product tests that will help your product to deliver.