Learn / Guides / User sentiment survey analysis

Back to guides

Survey sentiment analysis: how to bring the voice of the customer to your business

When you understand your customers, you can improve your products and the customer experience. But it’s not enough to know what your customers do on your website. You also need to know what they’re thinking and feeling at all points of the customer journey.  

So, go beyond the ‘what’ and dive deep into the ‘why.’

Last updated

14 Sep 2022

Surveys are a great way to collect feedback from your customers. But they yield a lot of subjective information that isn’t that easy to turn into objective, actionable insights. 

To do this, you need to use survey sentiment analysis techniques. 

We’ve put together a guide to how and why you should analyze survey sentiment. And give you practical tips on how you can apply insights to your business to create customer delight.

Want to know how customers really feel about your brand?

Hotjar Surveys let you gather valuable feedback in customers’ own words—so you can go deep into the why of user behavior.

Why should you analyze survey sentiment? 

Survey sentiment analysis involves analyzing the sentiment behind answers to open-ended survey questions. This is key to measuring a user’s emotional connection to your brand and product. The more connected they feel, the happier they are with it. And the happier they are, the more likely they are to buy, use, or recommend it.  

Ask a question like “How do you feel about our product” and customers are likely to answer using words like ‘love,’ ‘hate,’ etc. So the sentiment will probably be fairly clear. However, if you ask “What’s stopping you from recommending our product to a friend?” it can be harder—but just as important—to decipher the sentiment behind their words.  

Survey sentiment analysis lets you uncover hidden insights behind your customers’ words. This lets you achieve empathy with them—and make changes to your branding, product, and processes to boost customer satisfaction and retention.  

For example, an ecommerce shopper might abandon their cart if they can’t find information about your refund policy. To address this, you need to know what they’re thinking and feeling at that point. Analyzing the sentiment of exit-intent surveys will tell you what doubts and worries they have. Then, you can create materials—like an FAQ section at the point of sale—to provide the answers they need at the right time to build trust and boost conversions.  

Or, when SaaS customers cancel their subscriptions, you can identify the emotional factors involved in their decision by analyzing churn surveys. For example, you might discover that end-users hate your user interface (UI). So user sentiment analysis from churn surveys tells you what you need to change to keep your remaining customers happy. Like all product experience (PX) insights, survey sentiment analysis helps get cross-functional teams on the same page by eliminating disagreements about ‘what the customer wants.’ It also helps you get stakeholder buy-in for changes. 

Survey sentiment analysis means no more second guessing what customers think and feel. Instead, you can make user-centric decisions based on solid qualitative data. 

Open-ended vs closed-ended survey questions  

Asking open-ended questions is key to survey sentiment analysis. Let’s take a look at the benefits and challenges involved. 

What are open-ended vs closed-ended questions? 

Closed-ended questions can be answered with a simple ‘Yes/No.’ For example, “Are you satisfied with our product?” Or with a numerical score or rating: “Please rate your level of satisfaction on a scale of 1-10.” 

By contrast, open-ended questions require a more detailed answer. For example, an open-ended follow-up to the above questions would be “Why did you give us that rating?” 

Open-ended questions provide valuable contextual qualitative data that complements your quantitative data. So you get the what and the why. They provide authentic feedback in the customer’s own words that helps you understand things from their perspective. It also provides context for their actions.  

You can also change the wording of open-ended questions for different customer segments. For example, you’ll want to ask slightly different questions of new and returning customers. Or of occasional vs regular users of a SaaS tool. This lets you personalize interactions and go deeper into sentiment for certain groups.  

Open-ended questions also maximize the insights you can gain from a relatively small sample size. For example, a SaaS product team planning a launch can survey beta users to find out which features need improvement. These early adopters represent only a small fraction of the eventual target market, but their input is invaluable for achieving product-market fit before rolling out to a larger audience.

#Open-ended questions provide valuable insights that you can use to improve your website and products 
Hotjar.com
Open-ended questions provide valuable insights that you can use to improve your website and products Hotjar.com

Challenges of analyzing open-ended survey questions 

While there are many benefits, it’s harder to analyze open-ended questions than closed-ended ones. Here’s why: 

  • Open-ended questions give you subjective rather than objective answers. But to be able to act on them, you need to turn them into objective insights. And it can be very time-consuming to code and analyze hundreds of long-form answers manually.  

  • It’s more challenging to identify patterns and themes in text than extract a numerical score from ratings.  

  • Your analysis can suffer from observation bias. It’s tempting to ask leading questions like “Tell us why you love our product,” or “What's the best thing about our new feature?” as opposed to neutral questions like “How do you feel about our new feature?” These leading questions can cause customers to give a more positive response than they otherwise would. 

It’s also tempting to interpret results according to what you’d like to see. So all survey questions need to be carefully written to avoid bias. 

#This table makes it easy to see how open-ended questions let you gather much more contextual information about user motivations. 
Hotjar.com
This table makes it easy to see how open-ended questions let you gather much more contextual information about user motivations. Hotjar.com

 

  • You usually get a smaller sample size with open-ended surveys. This is because it’s  more time-consuming for people to answer them, as opposed to just giving a rating. This is why many surveys use open-ended questions following on from a closed-ended one to avoid putting customers off. 

  • It can be expensive to perform qualitative research with hundreds of customers. This plus a smaller sample can mean you never get a truly representative sample of your customer base. 

Survey sentiment analysis helps overcome these challenges by making it easier to uncover patterns, themes, and trends in qualitative data. 

Let’s take a look at how it works.

How to do survey sentiment analysis

To perform survey sentiment analysis, start by gathering the right data. Then, organize and analyze it and act on the insights. 

1. Choose your qualitative data analysis method 

There are four basic ways to analyze qualitative survey data

  1. Content analysis examines and groups responses based on expressed emotion. It’s particularly useful to understand customer behavior and measure brand reputation. 

  2. Thematic analysis identifies and interprets themes and patterns in qualitative data.  For example, if you spot that users universally hate your new releases, your product teams can make changes to the UI and features.   

  3. Grounded theory analysis involves creating theories and hypotheses by examining real-world data. For example, if churn rates are high, you can come up with a hypothesis about why you’re losing customers and look for data to back it up. 

  4. Discourse analysis tells you what people think of a topic, and why. It works best for longer text but is also useful for marketers to understand customer motivations expressed in surveys. 

Which method you choose will depend on what and why you want to learn about your customers and how you'll use the insights.  

2. Gather the right insights

The next step in the sentiment analysis process is information gathering. 

Choose your survey type 

There are several types of surveys you can use to ask open-ended questions: 

  • Net promoter score (NPS) surveys measure customer loyalty and satisfaction. They’re usually closed-ended, but you can ask open-ended follow-up questions after a rating.  

  • Customer satisfaction surveys (CSAT) tell you what customers feel about your product, brand, and customer support.

  • Customer effort score (CES) surveys are a type of customer experience survey. They measure the effort involved for customers to use your product or resolve issues with your help desk.

  • Point of conversion (POS) surveys launch when customers reach specific milestones, like completing a purchase on your site.

  • Retention surveys tell you why customers cancel or downgrade a subscription or return an item.

#Launching surveys when users churn can tell you how you can keep your remaining customers happy. 
Hotjar.com
Launching surveys when users churn can tell you how you can keep your remaining customers happy. Hotjar.com

If you’re using Hotjar, you can take advantage of our large survey template library, as well as getting inspiration for questions.

Run your survey

Create and run an on-site survey (that appears on a website page) or an online survey (that exists on a separate URL). Place them at the right place on your website to capture the data you want. For example, a SaaS provider looking to gather point-of-conversion insights can launch them when customers sign up for a demo or trial. 

Pro tip: use Hotjar Observe tools—Heatmaps and Session Recordings to map the user journey and identify where to place your surveys. 

Heatmaps show you which areas of your site get the most attention. Session Recordings reveal where users get frustrated and drop off, so they’re a good place for exit surveys.

#If you know most of your site visitors don’t make it below the fold, you need to place your surveys on the top half of the page.

#If you know most of your site visitors don’t make it below the fold, you need to place your surveys on the top half of the page.

Finally, use a sample size calculator to determine how many data points you need to achieve statistical significance—and get a representative sample of your customer base. This tells you how many survey responses you need before you can start your analysis. 

Once you’ve collected your data, the next step is to organize it. 

2. Organize your data 

Once your surveys are in, you need to organize the data. You can organize it by type, topic, theme, etc. One way to do this is to upload it to a research repository. This unifies it with other data and makes it easily accessible. 

If you're using Hotjar to view your survey data, you should also go to the responses tab and use filters to remove incomplete answers. These could affect the accuracy of your data. 

3. Analyze your survey data

There are several ways to analyze survey sentiment. You can do it manually or use dedicated sentiment analysis software.  

How to analyze survey sentiment manually  

If you don’t have the budget to invest in software, you can analyze survey data manually. Here are two ways to do it: 

  1. Use a spreadsheet like Hotjar’s open-ended question analysis template. This is a quick and affordable way to structure your data and extract quantitative insights from qualitative data. 

In a nutshell, this is how it works: 

  • Export data from your surveys to the spreadsheet 

  • Use a text analyzer like Textalyser to identify broad categories of responses. Or you can sort responses alphabetically to see what patterns emerge. 

  • Assign a value to each category. Then, go through the spreadsheet and assign each response to a category. So, if ‘satisfied customers’ are Category 1, label each response that indicates satisfaction with a 1. 

  • Group categories together if you notice respondents use different words to describe the same concept.  

  • Represent the data visually by creating a chart in the spreadsheet

#Using a spreadsheet takes a lot of the manual work out of classifying your survey sentiment data.
Hotjar.com
Using a spreadsheet takes a lot of the manual work out of classifying your survey sentiment data. Hotjar.com
  1. Look at the Hotjar Dashboard to compare metrics over time and filter by month to see how sentiment changes. This makes it easy to spot trends. You can also visualize responses per question as tables, charts, or word clouds. 

#Word clouds are a great way to spot patterns, like repeated use of the same words.
Hotjar.com
Word clouds are a great way to spot patterns, like repeated use of the same words. Hotjar.com

Automated survey sentiment analysis

If you don’t have time for manual data analysis, you can automate it. This saves a ton of time and eliminates bias in the interpretation. 

You can use qualitative data analysis tools like Quirkos, ATLAS.ti, or MonkeyLearn, which use machine learning to classify data through text and keyword analysis. This helps you find themes in customer feedback and anticipate what people are going to do next. 

Many of these tools use natural language processing (NLP), a component of AI that helps computers understand and interpret human language. For example, an NLP tool like Thematic lets you sort survey insights by theme and matches them to historical behavior data. This way, you can predict users’ future actions such as whether ecommerce shoppers are likely to make a purchase.

#Thematic dashboard showing survey sentiment analysis results and visualizations
Hotjar.com
Thematic dashboard showing survey sentiment analysis results and visualizations Hotjar.com

4. Act on insights 

Once you’ve analyzed survey sentiment, it’s time to act on those insights. First, prepare some charts and graphs to visualize the data. This will be especially helpful if you need stakeholder buy-in to make changes. 

Here are just a few ways you and your customers can benefit from survey sentiment analysis insights: 

  • Address customer complaints and make changes to improve the customer experience. For example, ask open-ended questions to identify user challenges at different touchpoints. Then, based on these surveys, identify common customer complaints—like timed-out searches, a poor onboarding experience, or unclear policies.

  • Improve your UX to boost conversions. For example, launching a Hotjar exit survey when visitors are about to leave your website will reveal friction points. Like unclear language that creates doubt in customers’ minds. Once you know this, you can tweak your copy to make it clearer.

  • Improve the purchase process. For example, running session recordings on an ecommerce site will reveal when visitors rage-click or drop off. Then, you can launch surveys at those points to understand what’s stopping them from buying. It could be as simple as confusing a non-clickable element for a clickable button, which is easy to fix.

  • Create compelling marketing and website content based on the language your satisfied customers use. This brings the voice of the customer to your copy and helps them identify with your brand. 

  • Develop or improve features and onboarding flows to help users get even more value from your product.  

  • Improve customer support and success to boost customer satisfaction and retention rates. If you’re using Hotjar, you can easily share PX insights via the Slack integration to keep all your teams in the loop.

#Hotjar integrations let you easily collect and share PX insights across popular collaboration tools like Slack
Hotjar integrations let you easily collect and share PX insights across popular collaboration tools like Slack

Survey sentiment analysis: the key to achieving customer empathy  

Survey sentiment analysis puts the voice of the customer at the heart of your decision-making. Combining quantitative insights—what people do—with qualitative data—why they do it—from surveys lets you experience the customer journey through their eyes.  

To use survey sentiment analysis, first decide what you want to know, why you want to know it, and how you’ll act on the insights. Then, gather your data, and organize and analyze it. You can do this manually or with software tools. Then, visualize your data and decide how to act on insights. Use this guide to work out how and where you can apply survey sentiment analysis to your business. 

Once you’ve done that, you’ll be well on the way to achieving true empathy with your customers. Which puts you in a better position to give them exactly what they want.

Want to know how customers really feel about your brand?

Hotjar Surveys let you gather valuable feedback in customers’ own words—so you can go deep into the why of user behavior.

Frequently asked questions about survey sentiment analysis