Study finds students without ChatGPT produce more diverse ideas than AI-generated essays

A new study has found that students writing without the help of AI tools such as ChatGPT contribute significantly more diverse ideas than essays generated by large language models, raising fresh questions about the impact of artificial intelligence on education and creativity.

The research, published in Computers in Human Behavior: Artificial Humans, analysed 2,200 college admissions essays across three studies. Researchers compared essays written by real university applicants between 2018 and 2022, before ChatGPT became publicly available, with essays generated by GPT-4 using the same admissions prompt.

While GPT-4 often produced essays that appeared highly creative on their own, researchers found a major difference when they looked at creativity across a larger group of essays rather than individual pieces of writing.

‘Diversity growth rate’

To measure this, the researchers created a new metric called the “diversity growth rate.” Instead of examining whether a single essay is creative, the metric tracks how much each new essay contributes to the overall pool of ideas.

The study found that every additional human-written essay introduced fresh experiences, perspectives and combinations of ideas. As more essays were added, the collective pool of ideas continued to expand.

GPT-4 essays behaved differently. Although many individual essays scored well on creativity measures, new AI-generated essays added much less novelty to the group. Researchers found that the model repeatedly drew from similar themes, patterns and ways of expressing ideas.

Across the three studies, human-written essays increased collective diversity between two and eight times more than GPT-4 essays. The gap became larger as the number of essays increased.

The researchers described this as a “homogenising effect”, a tendency for AI-generated writing to converge around similar ideas rather than continuously introducing new ones.

GPT-4 can be creative, but .

The researchers stressed that the findings should not be interpreted as proof that AI cannot be creative.

In fact, earlier studies have shown that GPT models can perform as well as, and sometimes better than, humans on several creativity tests. The new research reached a similar conclusion in some cases. When assessed individually, GPT-4 essays often matched human-written essays and sometimes exceeded them on measures of semantic diversity.

To evaluate creativity, the researchers used a technique known as semantic distance, which measures how many different concepts and ideas are connected within a piece of writing. A greater semantic distance suggests a broader and more original range of ideas.

However, the researchers argued that creativity is not only about how impressive a single essay appears. It is also about whether many different people bring different perspectives to a discussion.

Their findings showed that while GPT-4 could generate creative essays, large numbers of GPT-4 essays tended to resemble one another more than large numbers of human-written essays.

Attempts to make AI more diverse

The research team also tested whether the homogenising effect could be reduced.

In one experiment, they instructed GPT-4 to be as creative as possible. In another, they adjusted model settings designed to encourage more novel language and less repetition. They also tested chain-of-thought prompting, a technique that encourages AI systems to reason through a task step by step.

These interventions improved the creativity of individual essays. In some cases, modified GPT-4 outputs even surpassed human essays on individual diversity scores. Yet the broader pattern remained unchanged.

Even after prompt changes, parameter adjustments and chain-of-thought prompting, GPT-4 essays continued to contribute fewer new ideas to the collective pool than human-written essays.

The study found that the newer GPT-4 model tested in one experiment showed an even stronger tendency towards homogenisation than an earlier version.

Why the findings matter

According to the researchers, the main concern is not that students will become worse writers by using AI.

Instead, they warn that widespread reliance on the same AI systems could gradually reduce the diversity of perspectives appearing in classrooms, universities and other creative environments.

The paper links this concern to the idea of “algorithmic monoculture”, where large numbers of people rely on the same technology and therefore produce increasingly similar outputs.

“If organisations, educational institutions, or creative industries rely too much on a certain AI model, the collective pool of ideas may become more uniform over time,” the researchers wrote.

The authors said the findings highlight the need for AI literacy and policies that encourage originality when students use AI tools. They also called for further research into how AI affects creativity in areas such as academic writing, journalism, literature and social media.

The study concludes that while AI can support individual creativity, human writers still contribute far more diversity of thought when viewed collectively, a difference that may become increasingly important as AI tools become a routine part of education.

Ready to navigate global policies? Secure your overseas future. Get expert guidance now!