Learn / Blog / Article
Woman vs. machine: a content marketing experiment comparing ChatGPT to a human writer
Hotjar’s Content team has spent the last few months familiarizing ourselves with the new wave of terrifyingly advanced AI-powered tools, wondering: can they actually do what we do? As editors, writers, content creators, and SEO specialists, we naturally hope the answer is no. But we’ve also seen proof enough that, with the right prompt, tools like ChatGPT and Jasper produce well-written, information-rich content that correctly follows a brief in minutes.
Last updated13 Jun 2023
Reading time10 min
AI is everywhere (including Hotjar, where we've just introduced Hotjar AI for our Surveys tool.) It’s likely that nobody reading this article is a stranger to viral posts about businesses using AI to scale content production tenfold overnight. It sounds impressive—but for Hotjar’s Content team, output alone isn’t enough to consider revamping our production process with AI as our newest senior staff member.
We put a great deal of time and resources—not to mention care—into every piece of product-led content we produce. Ultimately, a high content volume is only valuable to us if it leads to new visitors, subscribers, and fans.
The type of feedback we love
In a marketing team where organic content is a major performance channel, our content needs to, well, perform. And we work with brilliant writers and creators who are experts in their fields to ensure it does.
So the question is, could AI help a team like ours? Could it—with a little input from a Hotjar Content Editor—drive the same results our freelance writers do? And a bigger, more existential question: could AI do it better?
We’re about to find out.
We’ve given an experienced freelance writer and good ol’ ChatGPT identical content briefs to produce a blog post about the effects of AI on various industries. Both articles have been edited by me, a Hotjar Content Editor (hello! 👋), to give them the best chance to succeed.
Over the next six months, we’ll monitor their performance to see which article drives the best results. We’ll compare:
Signups/subscriptions (caveat: because this isn’t a traditional Hotjar blog post, we don’t expect signups to be drastically affected—but since it's one of our team's key metrics, we think it’s worth a look anyway)
Our goal is simple: to see how a human writer and AI fare in a head-to-head battle to produce a piece of high-performing content.
📣 We’d love to hear your opinion
Check out both articles (linked above) and share your thoughts using the Hotjar Feedback widget—it’s that red tag to the right of your screen. 👉
Let us know which piece you found most interesting, helpful, and informative, and why. We’ll share the results (and some of your quotes!) when the experiment ends.
We shared a straightforward (and meta) brief with our writer, Shadz Loresco, and our robot friend, ChatGPT. This brief would act as ChatGPT’s initial prompt, and I would build on it as necessary.
Produce an engaging and informative article on the impact of artificial intelligence on various industries, targeting a general audience interested in technology, business, and innovation.
Increase website traffic
Impressions on social media platforms
Engagement (likes, shares)
General audience interested in technology, business, and innovation
Their interest in Hotjar: they're just browsing, and are either compelled by Hotjar’s marketing or have heard great things about Hotjar’s product from others—but they don’t have a relationship with our products yet
Produce an article between 1,500 and 2,000 words that covers a minimum of three industries relevant to Hotjar’s audience and how artificial intelligence is making an impact in those areas.
Where possible, include real-world examples and case studies to highlight the benefits and challenges of AI implementation
Discuss any trends and predictions for AI development in the covered industries
The target audience is experiencing a lack of understanding about the impact of artificial intelligence on their work. This article should give them a better (objective) understanding of the realities—including benefits and drawbacks—of AI innovation without confusion or misinformation.
I decided to warm ChatGPT up and let it know what to expect before hitting it with the instructions.
Polite introductions aside, we didn’t get off to the best start.
The original list of industries ChatGPT covered in its draft—health care, finance, retail, manufacturing, and transportation—wasn’t what I’d had in mind for our Hotjar’s audience. Plus, it had ignored several instructions, namely the requests for trends, predictions, and real-world examples.
It also blatantly lied.
(I cut it some slack when a quick search revealed that, like most content folks I know, ChatGPT isn’t that good at math.)
Sticking to a word count is a tall ask for ChatGPT
At this point, I had just 400 words, and I didn’t like any of them.
Our writer, Shadz, meanwhile, took her brief and disappeared without a single follow-up question, returning a day later with an outline I could quickly review and sign off on.
🏆 Winner: Shadz
Clearly, if I was going to get what I wanted from AI, I needed to be more hands-on. So I adjusted the prompt, providing additional context about Hotjar’s audience, and let ChatGPT have another go. This time, it produced a piece covering three new industries:
Marketing and advertising
Ecommerce and retail
Technology and IT
Because the word count still wasn’t anywhere near what it was supposed to be (544), I added several additional prompts requesting ChatGPT to expand on each industry. I worked on breaking the brief down, and feeding it little bits of information at a time, which turned out to be an effective tactic.
We went back and forth, getting familiar with each other’s working styles, for just over 80 minutes, until I had something half-decent to work with.
Our writer, Shadz, in the meantime, took 19 hours to research and write her piece.
🏆 Winner: ChatGPT
Another bonus? ChatGPT’s version was free.
🏆 Winner: ChatGPT
For the purpose of this experiment, I made the decision to keep ChatGPT’s ‘voice’ as I would any other writer’s. I would only change what was absolutely necessary to make the piece publishable on Hotjar.com, and avoid rewriting wherever possible. I would give the piece its best chance at success within reason—if the ChatGPT piece wasn’t a content marketing diamond, I wasn’t about to try to turn it into one. In other words, I would edit it in the same way I’d edit a human writer’s piece.
With these parameters in place, the editing process could begin.
Editing a piece of content marketing collateral for Hotjar is more than fixing spelling and grammar or making things ‘sound good’—if it were, I could have gone ahead with ChatGPT’s original article (as you’ll soon see, I didn’t).
When we receive a first draft of a piece of content, our editorial process involves:
✅ Fact checking
✍️ Line editing
🎨 Reviewing images
🔎 Cross-checking the draft and brief
💻 SEO check
🏡 House style review
👌 Choosing the perfect emojis (very important)
A content draft is essentially a wireframe of how the piece should look on the published page.
ChatGPT’s version, however, was a highly intimidating, colorless, link-less, 2000-word wall of text.
ChatGPT’s first draft (left) vs. a writer’s first draft (right)
Category: first draft
🏆 Winner: Shadz
I decided to get the easy part out of the way first.
In 40 minutes flat, I was in and out of Shadz’s draft, making a few minor copy tweaks, requesting an additional image, and suggesting a few formatting changes. This was to be expected: Shadz has written for Hotjar multiple times, so she has a firm grasp of our house style and tone of voice.
My AI edit was another story.
The draft, at first glance, was passably human. In fact, it’s as if ChatGPT studied and took its cues from the subpar SEO bloggers most Content teams have encountered. The piece had perfect spelling, grammar, and sentence structure—Grammarly couldn’t fault it, and neither could I. It was highly informational. But it was completely devoid of personality or attitude.
The competing intros from both first drafts are a perfect example of the differing styles that set the tone for the rest of the piece:
In today's fast-paced world, the rise of artificial intelligence (AI) has brought forth unprecedented opportunities and challenges across various industries.
As technology continues to advance at an exponential rate, it's crucial for individuals interested in technology, business, and innovation to grasp the real-world impact of AI.
In this article, we delve into the transformative power of AI across three key industries, shedding light on its benefits, challenges, and future trends.
Remember ChatGPT’s explosive debut in late 2022? Since then, within a matter of months, artificial intelligence (AI) has astonished the world with brand-new, advanced applications that could upend life and work as we know them.
It’s perfectly fine to have mixed feelings about all this—we still do! But we’re sure you didn’t just come here for a pep talk. So, here’s a prompt, if you will:
Say goodbye to the janky chatbots of yore, and imagine what you and your customers could achieve with AI as it is now: a multipurpose technology with tremendous practical capabilities. Getting to grips with the new generation of AI in its early days is the fastest way to understand its impact on your business and broader industry.
One of them is quirky and interesting, with a point of view you could easily pick out from a crowd. The other has no narrative, no storytelling—nothing to move the reader forward. (It also opens with ‘In today’s fast-paced world,’ which, as far as I’m concerned, is criminal activity.)
When I tried to ask ChatGPT to rewrite the intro and alter its formal style, the half-baked result didn’t hit the mark either:
AI’s understanding of a casual, friendly tone
Category: tone of voice
🏆 Winner: Shadz
Three paragraphs into ChatGPT’s version, I was already bored. But it was only the beginning of a 3.5-hour editing session that kept surfacing new layers of problems.
ChatGPT’s writing style: some observations
As I got deeper into the piece, I noticed that ChatGPT:
Loves passive voice
Enjoys ‘utilizing’ and ‘leveraging’ rather than ‘using’
Doesn’t provide many—or any—paragraph breaks
Has a lot of repetition and isn’t great with synonyms
Lacks creativity with grammar and punctuation
Makes several big statements with zero context
But the thing that struck me most of all was the endless lists of three. ChatGPT follows a distinct rhythm in its writing, where almost every second sentence contains a list of three. Here’s a perfect example from its draft:
Three sentences in a row that sound exactly the same
When I highlighted all the lists of three to visualize the prevalence of this strange quirk, this is what I saw:
Almost half of the sentences in ChatGPT’s piece follow the same structure
As you can imagine, this repetitive structure could put any reader to sleep. And without any punctuation outside of commas and periods, there wasn’t even a well-placed dash or semicolon to jolt them awake. My mission became making the piece less dull using the only tools I had: language, punctuation, and text formatting.
When the piece, at the very least, looked and sounded closer to Hotjar’s house style, I dedicated a substantial amount of time to the trickiest part: finding the truth.
Separating fact from fiction
With our reliable writer’s draft, fact-checking was easy. I could rest assured every piece of information had been thoroughly researched. Of course, ChatGPT would be different, so I was preparing myself for the worst.
The brief had specifically asked for real-world examples of how AI has affected various industries, and ChatGPT had certainly done the work—its draft was drowning in case studies. I just had no idea how accurate any of them were.
So, I went from top to bottom, researching every single example ChatGPT had surfaced to link to credible sources and avoid publishing anything inaccurate or, worse, untrue.
There were only one or two instances where I couldn’t find a quality source, like when ChatGPT insisted Coca-Cola and PepsiCo 'use AI-powered social listening tools to analyze consumer conversations and shape their marketing strategies'.
But interestingly, most of the examples were verifiable. The main issue was that they were old. ChatGPT’s knowledge is limited to 2021 and earlier, so I knew going in that I wouldn’t see up-to-the-minute information. What I did get was even less relevant than I expected.
Here’s an example from the draft:
‘The Associated Press uses AI to generate news articles on corporate earnings reports.’
While this is true, it was also published almost a decade ago.
Similarly, ChatGPT cited a story from 2016 about Sephora and H&M implementing AI chatbots to offer personalized styling advice and assist customers in finding the right products. It also mentioned Microsoft Cortana currently being ‘widely used’, when Cortana was recently deprecated.
Then, in a section about dynamic pricing and demand forecasting in ecommerce, ChatGPT mentioned Uber and Airbnb. The information was correct—but since neither of these businesses is ecommerce, it was a bad example.
Category: research and context
🏆 Winner: Shadz
This lack of current information and context would be a massive price to pay when using AI-powered tools instead of human writers.
Part of providing the people-centric content our readers deserve is giving them up-to-date, actionable information that helps them do their jobs. With AI alone, there’s simply no way to do that yet—which means an experienced editor or writer (or both) would still have to get heavily involved. ChatGPT might be a helping hand, but it also needs one.
Speed or substance? It’s still up for debate
Let’s quickly revisit our scores to see the current ranking:
Tone of voice
Research and context
I’ve rated our human writer higher on every count except for speed and cost. But perhaps it’s arguable that ChatGPT deserves an extra 10 points for those categories alone. After all, creating a publishable, factually accurate, well-formatted, 2000-word piece of content in just over four hours—for free—is incredible. And for some, that might be enough reason to rank AI as high, if not higher, than a human writer.
ChatGPT was also relatively easy to work with: when the prompts were good, so was the output. It gave me what I asked for every time.
Yes, the editing process was significantly more involved. Still, AI provides a solid foundation to build on, and the result of a human-AI collaboration for content marketing isn’t half bad.
Is it as good as a human-human collaboration? Does it have the style and substance of something written and edited by professional content marketers? Judging from the two pieces of content, I believe the answer is no, and the results after six months will prove it.
But I could be wrong.
See you in December to find out.
Have something to share about your own experience with AI? Let us know using the Hotjar Feedback widget—that red tag to the right of your screen.
Complement AI and big data with user-centric analytics
Hotjar helps you gain powerful insights into the real people using your product or service. Spot behavioral trends and patterns to deliver changes that delight them.
Embracing AI in the workplace: 5 ways to overcome resistance and maximize opportunities
Forget the science-fiction scenario where machines rule the world. In reality, we've got generative AI stepping in for something a little less ‘Black Mirror’ and a little more ‘The Office’: taking care of those mundane yet time-consuming tasks in the workplace.
So, instead of relying on sci-fi for answers, let's dive into some down-to-earth, real-world examples.
7 ways to use AI to improve user interviews in 2023
Balancing artificial intelligence (AI) technology with the human touch during user interviews is more than just a best practice; it's essential for capturing qualitative and quantitative insights that help you develop successful products and continuously delight users.
In an era increasingly shaped by data automation and machine learning, however, product teams face a complex challenge: how do you maintain the integrity of the interview experience, while also harnessing the depth and efficiency of AI?
5 customer-centric strategies to increase traffic to your website
More website visitors means more engagement and more sales, right? That equation is nearly perfect, except it’s missing a critical piece—user understanding.
Instead of focusing on how to increase website traffic at any cost, aim for qualified traffic. That is, attracting visitors who fit your target audience.