The Turing Test
$25,000 prize was available to the first bot which can pass the text-only Turing Test, and a $100,000 prize was available for a bot which could pass a modified version of the test, which uses multiple senses. While these two prizes were never won, in 2008, a bot was one vote away from passing the 30% criteria, which would have made it the first to do so. The Loebner Prize ended in 2009, 19 years after its creation, with no winner of the grand prize. To mark the 60 th anniversary of Alan Turing’s death, the University of Reading hosted ‘Turing Test 2014’, which had the most simultaneous instances of the Turing Test, and followed the original description from Turing’s paper exactly. This event has significance because it is commonly seen as the first time where the test has truly been beaten. The bot which beat the test was called ‘Eugene Goostman’, who was developed by a team of two Russian and Ukrainian software engineers, starting development in 2001, 13 years before passing the test. Eugene played the character of a 13-year-old Ukrainian boy, who claims to know everything. His developers developed his personality, aiming to make it believable, while using attributes such as being young, and a foreign speaker, to explain any grammar errors which would give it away. When faced with the Turing Test, Eugene managed to convince 33% of the judges, exceeding the 30% requirement in Turing’s description. Eugene’s success was met with plenty of criticism, stating that the bot relied on tricking the judge using its carefully developed character. Conversations with Eugene will usually have him avoiding questions and giving clearly incorrect answers to simple questions, which is not what most people would think of as a truly intelligent bot. Another chatbot which is well known for its progress on the Turing Test is ELIZA, which was developed in the 1960s by Joeseph Weizenbaum at MIT. The most successful variation of ELIZA is called DOCTOR, who plays the character of a Rogerian psychotherapist. Rogerian therapy is based on the idea that the client is already able to grow emotionally, and the therapist’s role is to unlock that in the client. In the case of DOCTOR, this means that the bot relies on reflecting the user’s questions on themselves. The code for DOCTOR has a series of set outputs, which combine parts of the user’s inputs to make a coherent sentence. This method creates the illusion that the bot understands what the user is saying, while it is only repeating generic outputs. When talking to DOCTOR, I found that it was unable to push conversations forwards, instead relying on asking simple, and often repetitive, questions to the user. It is simple to recognize the patterns in its speech, for example, I found that when you use the phrase ‘I am . . .’ in an input, the bot will respond with ‘Do you believe it is normal to be . . . ?’ After looking into the code of a modern recreation of ELIZA, I found that it consists of a set of inputs to recognize, such as ‘I am . . .’ and then a series of generic outputs, which use parts of the user’s inputs. While this bot did not pass the Turing Test, it was believed to have convinced many people that it was a real person. ELIZA is a good example of why the Turing Test is flawed, because by looking at the code, you can tell it is not intelligent, but it could very easily convince you that it is. For a pre-print study at the University of California San Diego, two researchers ran the Turing Test, using 284 participants. The study compared two modern Large Language Models (LLMs), GPT-4.5 and LLaMa- 3.1-405B (LLaMa), and two older models, ELIZA and GPT-4o. The results found that GPT-4.5 convinced participants that it was human the most, with 73%, which would pass the Turing Test. LLaMa convinced
83
Made with FlippingBook - PDF hosting