Learn / Blog / Article

Back to blog

Woman vs. machine: a content marketing experiment comparing ChatGPT to a human writer

Hotjar’s Content team has spent the last few months familiarizing ourselves with the new wave of terrifyingly advanced AI-powered tools, wondering: can they actually do what we do? As editors, writers, content creators, and SEO specialists, we naturally hope the answer is no. But we’ve also seen proof enough that, with the right prompt, tools like ChatGPT and Jasper produce well-written, information-rich content that correctly follows a brief in minutes. 

Last updated

13 Jun 2023

Reading time

10 min

Share

AI is everywhere (including Hotjar, where we've just introduced Hotjar AI for our Surveys tool.) It’s likely that nobody reading this article is a stranger to viral posts about businesses using AI to scale content production tenfold overnight. It sounds impressive—but for Hotjar’s Content team, output alone isn’t enough to consider revamping our production process with AI as our newest senior staff member.

We put a great deal of time and resources—not to mention care—into every piece of product-led content we produce. Ultimately, a high content volume is only valuable to us if it leads to new visitors, subscribers, and fans.

The type of feedback we love

In a marketing team where organic content is a major performance channel, our content needs to, well, perform. And we work with brilliant writers and creators who are experts in their fields to ensure it does.

So the question is, could AI help a team like ours? Could it—with a little input from a Hotjar Content Editor—drive the same results our freelance writers do? And a bigger, more existential question: could AI do it better?

We’re about to find out.

The experiment

We’ve given an experienced freelance writer and good ol’ ChatGPT identical content briefs to produce a blog post about the effects of AI on various industries. Both articles have been edited by me, a Hotjar Content Editor (hello! 👋), to give them the best chance to succeed. 

  1. Beyond the prompt: the impact of artificial intelligence on 5 major industries” by Shadz Loresco 

  2. Exploring AI's impact on retail, technology, and marketing: from disruption to innovation” by ChatGPT

Over the next six months, we’ll monitor their performance to see which article drives the best results. We’ll compare:

  • Total traffic

  • User sentiment

  • Signups/subscriptions (caveat: because this isn’t a traditional Hotjar blog post, we don’t expect signups to be drastically affected—but since it's one of our team's key metrics, we think it’s worth a look anyway)

Our goal is simple: to see how a human writer and AI fare in a head-to-head battle to produce a piece of high-performing content.

📣 We’d love to hear your opinion

Check out both articles (linked above) and share your thoughts using the Hotjar Feedback widget—it’s that red tag to the right of your screen. 👉

Let us know which piece you found most interesting, helpful, and informative, and why. We’ll share the results (and some of your quotes!) when the experiment ends.

The prompt

We shared a straightforward (and meta) brief with our writer, Shadz Loresco, and our robot friend, ChatGPT. This brief would act as ChatGPT’s initial prompt, and I would build on it as necessary.

Project overview

Produce an engaging and informative article on the impact of artificial intelligence on various industries, targeting a general audience interested in technology, business, and innovation.

(Performance) objectives

  • Increase website traffic

  • Impressions on social media platforms

  • Engagement (likes, shares)

Target audience

  • General audience interested in technology, business, and innovation

  • Their interest in Hotjar: they're just browsing, and are either compelled by Hotjar’s marketing or have heard great things about Hotjar’s product from others—but they don’t have a relationship with our products yet

Deliverables

Produce an article between 1,500 and 2,000 words that covers a minimum of three industries relevant to Hotjar’s audience and how artificial intelligence is making an impact in those areas.

  • Where possible, include real-world examples and case studies to highlight the benefits and challenges of AI implementation

  • Discuss any trends and predictions for AI development in the covered industries

Key message

The target audience is experiencing a lack of understanding about the impact of artificial intelligence on their work. This article should give them a better (objective) understanding of the realities—including benefits and drawbacks—of AI innovation without confusion or misinformation.

I decided to warm ChatGPT up and let it know what to expect before hitting it with the instructions.

Polite introductions aside, we didn’t get off to the best start.

The original list of industries ChatGPT covered in its draft—health care, finance, retail, manufacturing, and transportation—wasn’t what I’d had in mind for our Hotjar’s audience. Plus, it had ignored several instructions, namely the requests for trends, predictions, and real-world examples.

It also blatantly lied.

(I cut it some slack when a quick search revealed that, like most content folks I know, ChatGPT isn’t that good at math.)

Sticking to a word count is a tall ask for ChatGPT

At this point, I had just 400 words, and I didn’t like any of them. 

Our writer, Shadz, meanwhile, took her brief and disappeared without a single follow-up question, returning a day later with an outline I could quickly review and sign off on.

Category: outline

🏆 Winner: Shadz

Clearly, if I was going to get what I wanted from AI, I needed to be more hands-on. So I adjusted the prompt, providing additional context about Hotjar’s audience, and let ChatGPT have another go. This time, it produced a piece covering three new industries:

  • Marketing and advertising

  • Ecommerce and retail

  • Technology and IT

Better! 

Because the word count still wasn’t anywhere near what it was supposed to be (544), I added several additional prompts requesting ChatGPT to expand on each industry. I worked on breaking the brief down, and feeding it little bits of information at a time, which turned out to be an effective tactic.

We went back and forth, getting familiar with each other’s working styles, for just over 80 minutes, until I had something half-decent to work with. 

Our writer, Shadz, in the meantime, took 19 hours to research and write her piece.

Category: time

🏆 Winner: ChatGPT

Another bonus? ChatGPT’s version was free.

Category: cost

🏆 Winner: ChatGPT

The edit

For the purpose of this experiment, I made the decision to keep ChatGPT’s ‘voice’ as I would any other writer’s. I would only change what was absolutely necessary to make the piece publishable on Hotjar.com, and avoid rewriting wherever possible. I would give the piece its best chance at success within reason—if the ChatGPT piece wasn’t a content marketing diamond, I wasn’t about to try to turn it into one. In other words, I would edit it in the same way I’d edit a human writer’s piece. 

With these parameters in place, the editing process could begin.

Editing a piece of content marketing collateral for Hotjar is more than fixing spelling and grammar or making things ‘sound good’—if it were, I could have gone ahead with ChatGPT’s original article (as you’ll soon see, I didn’t). 

When we receive a first draft of a piece of content, our editorial process involves:

  1. 🖼 Formatting

  2. ✅ Fact checking

  3. ✍️ Line editing

  4. 🎨 Reviewing images

  5. 🔎 Cross-checking the draft and brief

  6. 💻 SEO check

  7. 🏡 House style review

  8. 👌 Choosing the perfect emojis (very important)

A content draft is essentially a wireframe of how the piece should look on the published page.

ChatGPT’s version, however, was a highly intimidating, colorless, link-less, 2000-word wall of text.

ChatGPT’s first draft (left) vs. a writer’s first draft (right)

Category: first draft

🏆 Winner: Shadz

I decided to get the easy part out of the way first.

In 40 minutes flat, I was in and out of Shadz’s draft, making a few minor copy tweaks, requesting an additional image, and suggesting a few formatting changes. This was to be expected: Shadz has written for Hotjar multiple times, so she has a firm grasp of our house style and tone of voice.

My AI edit was another story.

The draft, at first glance, was passably human. In fact, it’s as if ChatGPT studied and took its cues from the subpar SEO bloggers most Content teams have encountered. The piece had perfect spelling, grammar, and sentence structure—Grammarly couldn’t fault it, and neither could I. It was highly informational. But it was completely devoid of personality or attitude. 

The competing intros from both first drafts are a perfect example of the differing styles that set the tone for the rest of the piece:

AI

Human writer

In today's fast-paced world, the rise of artificial intelligence (AI) has brought forth unprecedented opportunities and challenges across various industries.

As technology continues to advance at an exponential rate, it's crucial for individuals interested in technology, business, and innovation to grasp the real-world impact of AI.

In this article, we delve into the transformative power of AI across three key industries, shedding light on its benefits, challenges, and future trends.

Remember ChatGPT’s explosive debut in late 2022? Since then, within a matter of months, artificial intelligence (AI) has astonished the world with brand-new, advanced applications that could upend life and work as we know them.

It’s perfectly fine to have mixed feelings about all this—we still do! But we’re sure you didn’t just come here for a pep talk. So, here’s a prompt, if you will:

Say goodbye to the janky chatbots of yore, and imagine what you and your customers could achieve with AI as it is now: a multipurpose technology with tremendous practical capabilities. Getting to grips with the new generation of AI in its early days is the fastest way to understand its impact on your business and broader industry.

One of them is quirky and interesting, with a point of view you could easily pick out from a crowd. The other has no narrative, no storytelling—nothing to move the reader forward. (It also opens with ‘In today’s fast-paced world,’ which, as far as I’m concerned, is criminal activity.) 

When I tried to ask ChatGPT to rewrite the intro and alter its formal style, the half-baked result didn’t hit the mark either:

AI’s understanding of a casual, friendly tone

Category: tone of voice

🏆 Winner: Shadz

Three paragraphs into ChatGPT’s version, I was already bored. But it was only the beginning of a 3.5-hour editing session that kept surfacing new layers of problems.

ChatGPT’s writing style: some observations

As I got deeper into the piece, I noticed that ChatGPT:

  • Loves passive voice

  • Enjoys ‘utilizing’ and ‘leveraging’ rather than ‘using’

  • Doesn’t provide many—or any—paragraph breaks

  • Avoids contractions

  • Has a lot of repetition and isn’t great with synonyms

  • Lacks creativity with grammar and punctuation

  • Makes several big statements with zero context

But the thing that struck me most of all was the endless lists of three. ChatGPT follows a distinct rhythm in its writing, where almost every second sentence contains a list of three. Here’s a perfect example from its draft:

Three sentences in a row that sound exactly the same

When I highlighted all the lists of three to visualize the prevalence of this strange quirk, this is what I saw:

Almost half of the sentences in ChatGPT’s piece follow the same structure

As you can imagine, this repetitive structure could put any reader to sleep. And without any punctuation outside of commas and periods, there wasn’t even a well-placed dash or semicolon to jolt them awake. My mission became making the piece less dull using the only tools I had: language, punctuation, and text formatting.

When the piece, at the very least, looked and sounded closer to Hotjar’s house style, I dedicated a substantial amount of time to the trickiest part: finding the truth.

Separating fact from fiction

With our reliable writer’s draft, fact-checking was easy. I could rest assured every piece of information had been thoroughly researched. Of course, ChatGPT would be different, so I was preparing myself for the worst. 

The brief had specifically asked for real-world examples of how AI has affected various industries, and ChatGPT had certainly done the work—its draft was drowning in case studies. I just had no idea how accurate any of them were. 

So, I went from top to bottom, researching every single example ChatGPT had surfaced to link to credible sources and avoid publishing anything inaccurate or, worse, untrue.

There were only one or two instances where I couldn’t find a quality source, like when ChatGPT insisted Coca-Cola and PepsiCo 'use AI-powered social listening tools to analyze consumer conversations and shape their marketing strategies'.

But interestingly, most of the examples were verifiable. The main issue was that they were old. ChatGPT’s knowledge is limited to 2021 and earlier, so I knew going in that I wouldn’t see up-to-the-minute information. What I did get was even less relevant than I expected.

Here’s an example from the draft:

‘The Associated Press uses AI to generate news articles on corporate earnings reports.’

While this is true, it was also published almost a decade ago.

Similarly, ChatGPT cited a story from 2016 about Sephora and H&M implementing AI chatbots to offer personalized styling advice and assist customers in finding the right products. It also mentioned Microsoft Cortana currently being ‘widely used’, when Cortana was recently deprecated.

Then, in a section about dynamic pricing and demand forecasting in ecommerce, ChatGPT mentioned Uber and Airbnb. The information was correct—but since neither of these businesses is ecommerce, it was a bad example.

Category: research and context

🏆 Winner: Shadz

This lack of current information and context would be a massive price to pay when using AI-powered tools instead of human writers. 

Part of providing the people-centric content our readers deserve is giving them up-to-date, actionable information that helps them do their jobs. With AI alone, there’s simply no way to do that yet—which means an experienced editor or writer (or both) would still have to get heavily involved. ChatGPT might be a helping hand, but it also needs one.

Speed or substance? It’s still up for debate

Let’s quickly revisit our scores to see the current ranking:

AI

Human writer

Outline

🥈

🥇

Time

🥇

🥈

Cost

🥇

🥈

First draft

🥈

🥇

Tone of voice

🥈

🥇

Research and context

🥈

🥇

I’ve rated our human writer higher on every count except for speed and cost. But perhaps it’s arguable that ChatGPT deserves an extra 10 points for those categories alone. After all, creating a publishable, factually accurate, well-formatted, 2000-word piece of content in just over four hours—for free—is incredible. And for some, that might be enough reason to rank AI as high, if not higher, than a human writer. 

ChatGPT was also relatively easy to work with: when the prompts were good, so was the output. It gave me what I asked for every time.

Yes, the editing process was significantly more involved. Still, AI provides a solid foundation to build on, and the result of a human-AI collaboration for content marketing isn’t half bad.

Is it as good as a human-human collaboration? Does it have the style and substance of something written and edited by professional content marketers? Judging from the two pieces of content, I believe the answer is no, and the results after six months will prove it. 

But I could be wrong. 

See you in December to find out.

Have something to share about your own experience with AI? Let us know using the Hotjar Feedback widget—that red tag to the right of your screen.

Complement AI and big data with user-centric analytics

Hotjar helps you gain powerful insights into the real people using your product or service. Spot behavioral trends and patterns to deliver changes that delight them.