One of the inevitable side effects of the current AI boom is a certain level of distrust in communicating with artificial intelligences. This is certainly not unfounded paranoia either; AI has repeatedly demonstrated that it is prone to hallucinations and inaccuracies, can be easily fooled, and can even perform better if you are naughty, according to Live Science. Now a new paper has shown that it may be capable of impersonating humans.
A preliminary study (via arxiv) by researchers at Stony Brook University and UC San Diego showed that, under certain conditions, AI can outperform humans on the Turing test, a classic (if somewhat outdated) measure of a computer system’s ability to mimic human behavior. However, it is not necessarily time to panic about the inevitability of a Replicator/Cylon takeover: these results were produced under a very specific set of circumstances and have limited applicability to real-world scenarios.
Methodology and results
Artistic creations from the summit/Shutterstock
Proposed by Alan Turing in 1950, the Turing test is theoretically a way to assess a machine’s ability to impersonate a human. In the classic three-way test, replicated in the study, a human judge is asked to converse with a computer and another human via text message. The judge then attempts to determine which of the participants is human and which is the machine.
The study authors conducted two randomized, controlled, pre-registered Turing test experiments (meaning the parameters were set before the test was performed, so that they could not be changed later to adapt to the results). Sessions lasted five minutes and involved a human judge speaking with an LLM and a UCSD student or volunteer, then selecting which one was human. During the study, hundreds of human participants participated and four AI models were tested.
The main result came from OpenAI’s GPT-4.5 model, which fooled judges into thinking it was a human 73% of the time. LLaMa-3.1-405B performed just above chance at 56%, while the other two models were much less successful, with ELIZA at 23% and GPT-4o at 21%.
Some important caveats
Evgeniyshkolenko/Getty Images
Before you join Generation Z in their hatred of AI and rush your local data center with a sledgehammer in your hand, there are some important caveats to keep in mind. First, the Turing test is not the most scientific. It is extremely subjective and relies on both the judgment of the human evaluator and the responses of the other human participant. It also only reports a single binary metric, indicating whether or not a machine was able to fool the sole human judge.
It’s also worth noting that the best results (73% for GPT-4.5 and 56% for LLaMa-3.1-405B) came when models were encouraged to adopt a young, introverted, chronically online personality. This suggests that the model’s ability to succeed may rely on the model’s orientation toward a specific conversation style rather than a model’s general intelligence or innate adaptability. The only, very specific, takeaway is that in short text conversations, within a specific experimental design, it is possible that an LLM is more often mistaken for a person than for the person they are associated with.
That said, as the authors point out, this raises concerns about the growing risks of deception, trust, and the limits of human detection in casual conversation with AI (which has also been shown to affect your brain). It’s also worth mentioning that an AI had already passed the Turing test as early as early last year and the models have only become more sophisticated since then.
