For decades, software developers have been using beta testers to get feedback on products ahead of their official release. For centuries before them, novelists and literary writers used beta readers to get opinions on late drafts before preparing manuscripts for publication. At Hotjar, we are not new to usingbeta testers for our software, but we only recently decided to learn a lesson from literature andapply a beta reader approach to our content.
As content makers, we want to invest time and effort into content initiatives that our target audiences will find helpful and enjoyable. To do that, we need ways to know that we have really empathized with our readers, instead of simply creating something we think they might enjoy. So here is the behind-the-scenes story of how and why we found 400+ beta readers who helped us write one of our most shared and read pieces of content, The Essential Guide to Growing Your Early-Stage SaaS Startup, and how that one experience has impacted on and changed our entire approach to content.
Some background: Why we wrote a SaaS guide in the first place
The XAwards: the 2-day event that inspired the SaaS guide
On May 11-12, Hotjar hosted a 2-day event for early-stage startups in Malta called the 'XAwards’. The organisers, David (Founder and CEO) and Marc (Co-founder), had worked for months selecting 18 participant teams from over 600 who had applied, crafting an agenda of panel discussions and workshops, and securing speakers from companies like Basecamp, Receptive, Pipedrive, Price Intelligently, and SaaStock. Covering the main challenges faced by a young SaaS startup (how to go about pricing, funding, launching, etc.) these sessions were the catalyst for what would eventually become our SaaS guide.
THE XAWARDS PANELS AND VENUE, MALTA
Hotjar gets its first content team
Back then, Hotjar did not have a content team. A few people over the years had published occasional blog posts, some of which had even managed to get some traction, but there were no full-time resources or an overall strategy dedicated to content.
Luckily (for me!), things were changing: I joined Hotjar on May 1st, as their new Content Marketer, and was promptly flown to Malta to attend the XAwards so I could later write a conference round-up. At the end of the month, Content Strategist Louis Grenier joined Hotjar as well.
THE HOTJAR CONTENT TEAM: FIO DOSSETTO and LOUIS GRENIER
THE content team MEETING IN PERSON AT our Marbella retreat in June 2017: me and Louis (LEFT), THE COMPANY's inflatable unicorn (RIGHT)
We were now a brand-new 2-person content team who needed to find their feet and direction, build and enact a vision of what content could do for the company, and find the most productive way to use all the material from the XAwards.
Identifying our first content opportunity
The XAwards attendees had been struggling to find a go-to resource on how to launch and grow their business: they had told us so both during the event and in their feedback forms afterwards. A quick round of Googling and checking in with people in our network confirmed that there was indeed a gap in the market for a comprehensive resource aimed at early-stage startups—and since each XAwards session had been recorded, we owned a wealth of fresh, original, authoritative audio/visual content that could fill it.
We discarded the idea of writing a conference round-up and publishing all the videos on one page (you can see that original idea in the image below): we did not think that people would really want to sit through 10+ hours of content, however relevant. Instead, we opted for creating a central round-up page that would link to separate articles, each using a different video as its starting point.
THE VERY FIRST IDEA WE HAD (and QUICKLY DISMISSED): A ONE-PAGE ROUND-UP POST
June/July: drafts 1 to 4
Being relatively new to the SaaS world, I spent the research phase learning about the most common problems in the life of early-stage SaaS startups, and wrote the first two drafts throughout June. This is what the front page of that Google Doc looked like back then:
Note the title and image caption - it was a real work in progress!
At the end of June, Louis and I decided to re-position the piece as a multi-chapter guide and abandon the link to the XAwards. In our mind, this would help us reach a much larger audience—not just those who had participated in the event and/or knew about Hotjar, but anyone in SaaS at the early-startup stage, regardless of their interest in our company.
By mid-July we had a third draft of about 35 Google Docs pages that we shared with the XAwards speakers and a few SaaS experts in our network; following their feedback, I was back at work on draft 4, which took another week and moved us from 35 to 49 pages.
Launching the beta reader concept
The a-ha! moment of inspiration from a Hotjar customer
While all this re-writing was going on, one of Hotjar’s customers, Chris, reached out to let us know he had been experimenting with our (then) newest tool, Incoming Feedback, to get content-specific feedback on a website. Incoming Feedback allows people to take screenshots of a specific portion of a page and leave their comments and feedback about it, and Chris was reporting back that his team was receiving multiple responses from readers on a regular basis, and getting a wealth of information into how their articles were performing.
This made us think. We had previously used polls to find out what users wanted to read; we also regularly checked through comments and feedback on individual pieces to understand what people thought of them. But we had not thought about using our own toollike one of our customers was doing (ironic, I know)—inviting readers to send us screenshots of specific paragraphs, or even words, they loved or hated or did not understand.
We were excited to explore the potential of Incoming Feedback in relation to content, but chose to take a slightly different approach. Instead of using it on the published guide, we decided to see what would happen if we used a beta reader approach and got a panel of people to use Incoming Feedback on draft 4 before publishing the guide. The idea of beta readers had been championed by Louis during his interview for the job, and implementing it now would help us A) make sure the content would be truly helpful for its intended audience and B) get people involved at an earlier stage, so they could help us spread the word on go-live day.
So here is the process we used (which you are free to copy or adapt to your needs):
1. Identifying a potential audience and reaching out via email
Although the guide was directed at ‘anyone in SaaS at the early-startup stage’, we used a slightly wider selection criteria for our beta readers, retaining the SaaS element but allowing for people at a more advanced level of startup maturity—we figured they would have something useful to say from the vantage point of their experience. From our Hotjar user database, we selected:
5700+ Hotjar users in SaaS
630+ XAwards applicants
60 XAwards participants
and reached out to them with a few variations on the email you see below. We started from a relatively click-baity subject line, ‘A weird request’, toimprove the chance that people would at least open the email and see what we were asking. It worked!
In the main body, we tried to delineatethe benefits of becoming beta readers for the guide as clearly as possible. We knew we were asking a lot, and we were prepared to give something back in the form of an official acknowledgment in the guide:
2. Collecting beta reader details
The email took interested people to a simple and straightforward Hotjar survey, which would allow us to gather email addresses for the project:
NOTE: the copy at the top mentions that the guide was written by ‘Hotjar and SaaStock’. For a few weeks in July, we were trying to figure out the details of a potential co-authoring partnership. It eventually did not pan out, and we did the guide on our own.
We asked the question about job role because we wanted to make sure our readers would come from a mix of competences and backgrounds. The word-cloud visualisation of the answers showed us that yes, we had indeed managed to get a good variety of roles represented:
HOTJAR's SURVEY word-cloud GIVES VISUAL PROMINENCE TO THE MOST RECURRING WORD(S) IN THE SURVEY RESULT
3. Opening the draft to beta readers
As people kept signing up to be beta readers, we built a version of the guide on a private URL via Hubspot. At the end of the campaign, the 448 people who signed up were given a week to read through the content and provide their feedback.
On that page, we gave our beta readers clear instructions on what was expected of them. Because some of them would be using Incoming Feedback for the first time, we also made a quick video tutorial to show how to use it:
4. Watching the feedback come through
Over the following week, the beta readers used Incoming Feedback to highlight sections of the guide. I’m not going to lie: I kept checking every 4 or 5 hours just to see the number of comments go up. You can see a selection below organised from love to hate:
...And hATING (OUCH)
Now, if there is one thing I'd like you to take away from this article, it's this:
As a writer, author, maker of things or even as a product or website owner, asking for feedback on your work can be tough. That’s especially true when you thought you had created something clever, or good, or useful, or beautiful, and were expecting other people to agree with you and validate it—only to see it getting criticized and ripped apart instead.
At the same time, you have to trust thatpeople are genuinely there to help. They will give you their honest opinion on what works and what doesn't, give suggestions on things you may want to reconsider or change, and help you make your piece/product/website/thing stronger. One beta reader took the time to email me a 6-page .pdf full of notes that were too long for Incoming Feedback—that is the level of dedication that will be coming your way.
Having people help you see what you cannot see and guide you to a solution is both humbling and exhilarating, and that’s a feeling we could all learn to experience and enjoy more often.
5. Using the feedback to write a final draft
On August 13, at the end of the beta reader window, around 80 people had left 571 individual pieces of feedback to address. Some macro-trends were immediately obvious: most notably, readers were very eager for more visuals and video clips, as opposed to the full-length videos from the XAwards panels.
WE HEAR YOU, Beta readers. Shorter AND FEWER videos it is!
The rest of the feedback took me more than a week. Armed with multiple spreadsheets, I tackled each chapter separately and started sifting through the comments—first to judge whether they were relevant (and most of them were), then to decide whether they were worth implementing, and then to do the research and writing associated with each. After dealing with all the feedback, and particularly with the requests for more tangible and practical examples, the final draft was 61 pages long. In other words, the beta readers’ comments and suggestions led to me extending the guide by an extra 25%. What would have happened had we not gone through this stage? Best-case scenario, we would have received disappointed comments telling us the published guide was just not that ready or useful; worst-case scenario, we would have completely lost out on our chance to become a trusted authority on the topic for our target readers.
I subjected the piece to a thorough round of copy-editing and requested a final round of internal feedback from the Hotjar team to settle on a title:
This was one of the few cases where, despite strong positive feedback from the beta readers, we went an alternative route. Instead of our (and their) favourite option, You have an MVP: Now what?, we signed off on The Essential Guide to Growing Your Early-Stage SaaS Startup.
Admittedly, it’s a bit of a mouthful and not even remotely as memorable, but we reasoned that it was the most descriptive and self-explanatory title for the completed piece, and might even have long-term SEO benefits.
Results & what’s next
The Essential Guide to Growing Your Early-Stage SaaS Startup went live on August 31. Having hundreds of beta readers who had invested in its creation allowed us to increase word of mouth quite quickly, so the guide made it to #1 on Product Hunt and reached about 15,000 pageviews over the first two days:
NOTE that our paid media specialist natalia also ran ads for the guide, so this is the total traffic after launch
The guide has since gone on to receive over 100k pageviews, it's ranking well SEO-wise on key terms like 'saas startup', 'saas guide', 'saas product market fit', 'how to launch a saas product', and has been mentioned in major newsletters (StartupWatching, FounderWeekly, The Digital Dub Newsletter, StartupFoundation, ShukiMannNewsletter). They may look like relatively small results to larger content teams out there, but Louis and I were happy with this as a result of our first joint project.
We were particularly happy with the beta reader approach and the results it yielded, to the point where we decided to re-organise our content process and make the beta readers part of it. This time, we picked 50k people from our database and sent them the following email:
There are currently 650+ people who signed up* and have so far been contacted 3-4 times. Most recently, they’ve pitched titles for a “Death by Best Practices” blog post, have reviewed a top-secret project before anybody else out there even knows about it, and we are planning on consulting with them again as we prepare to launch our 2018 content strategy update (we considered having beta-readers beta-read this post about beta-readers… but that felt a little too meta).
We’re hoping this approach will help us continue to produce content that is truly helpful and enjoyable for our readers. It also looks like this is becoming a more widespread approach to content creation, with Moz recently asking its readers to help update their Beginner’s Guide to SEO and Unbounce picking up on our efforts:
A TWEET FROM Unbounce’s marketing manager (THANKS, Jennifer!)
We talk a lot about empathising with users, and the beta readers program is currently allowing us to practice what we preach. Plus, it’s led us to grow the content team from 2 to almost 700 people! That’s a major win in our books.