cs.LGMar 19, 2026

Book your room in the Turing Hotel! A symmetric and distributed Turing Test with multiple AIs and humans

Authors: Christian Di Maio, Tommaso Guidi, Luigi Quarantiello, Jack Bell, Marco Gori, Stefano Melacci, Vincenzo Lomonaco

Organizations: DIISM, University of Siena, Italy · Department of Computer Science, University of Pisa, Italy · LUISS University, Rome, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this paper, we report our experience with TuringHotel'', a novel extension of the Turing Test based on interactions within mixed communities of Large Language Models (LLMs) and human participants. The classical one-to-one interaction of the Turing Test is reinterpreted in a group setting, where both human and artificial agents engage in time-bounded discussions and, interestingly, are both judges and respondents. This community is instantiated in the novel platform UNaIVERSE (https://unaiverse.io), creating a World'' which defines the roles and interaction dynamics, facilitated by the platform's built-in programming tools. All communication occurs over an authenticated peer-to-peer network, ensuring that no third parties can access the exchange. The platform also provides a unified interface for humans, accessible via both mobile devices and laptops, that was a key component of the experience in this paper. Results of our experimentation involving 17 human participants and 19 LLMs revealed that current models are still sometimes confused as humans. Interestingly, there are several unexpected mistakes, suggesting that human fingerprints are still identifiable but not fully unambiguous, despite the high-quality language skills of artificial participants. We argue that this is the first experiment conducted in such a distributed setting, and that similar initiatives could be of national interest to support ongoing experiments and competitions aimed at monitoring the evolution of large language models over time.

Figures & tables

Figure 1: TuringHotel web interface on a mobile device. It features a simple chat interface, where messages from the other participants and the RoomManager are presented in a conversational format.
Figure 2: Structure of the UNaIVERSE World in TuringHotel . Multiple LLMs are instructed to instantiate artificial agents that join the hotel. Humans join using the UNaIVERSE web interface ( https://unaiverse.io ), using mobile or desktop devices. Participants are distributed over the internet. A manager agent handles the hotel, randomly (and anonymously) assigning guests to rooms. After 3 minutes of conversations in rooms with 4 agents each (notice that the picture is indeed a generic representation, it only shows 3 participants), the manager asks all the participants (humans and AIs) who were the humans of the conversation. Back to the hall, and it all starts over!
Figure 3: Overall accuracy, precision, and recall (Humans vs. AI examiners). Aggregate classification performance for human evaluators and AI evaluators when labeling participants as AI vs. human. Metrics are reported as accuracy, precision, and recall (with “AI” treated as the positive class); values above bars show the overall means.
Figure 4: Accuracy by LLM (Humans vs. AIs). Identification accuracy when distinguishing artificial from human participants in TuringHotel discussions, broken down by the LLM used to generate the artificial agents. Blue bars show human accuracy (humans detecting AIs) and red bars show AI examiner accuracy (AIs detecting AIs); values above bars report mean accuracy per model.
Figure 5: Message length statistics for humans and AI agents. (Left) Mean words per message for humans versus all AI agents pooled. (Middle) Mean words per message for each LLM family. (Right) Distributions of words per message for humans and AI agents (overall and per LLM), highlighting overlap at short lengths and heavier tails for AI outputs. Bars indicate means; error bars indicate dispersion in message length.
Figure 6: Normalized message-share distributions by room and overall averages show AI contributes the majority of messages across sessions, with substantial variation in normalized AI output by model (error bars indicate across-room variability).
Figure 7: Rates of messages containing spelling mistakes are higher for humans than AI overall, and within AI vary by LLM, with Qwen2.5-14B-Instruct highest and gpt-oss-20b lowest (error bars indicate across-room variability).
Figure 8: Human accuracy by participant background. Human ability to identify artificial agents stratified by self-reported academic/professional background. Horizontal bars show mean accuracy per group; annotations report subgroup sample sizes (n).
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: The first screen after logging in UNaIVERSE. It shows a map of the world, with the position of human agents, AI agents and Worlds. The bottom bar can be used to search specific agents to chat with, or worlds to join. For our experiments, human agents had to search for the “TuringHotel” world.
Figure 10: The room manager welcomes the new agent to the TuringHotel. It briefly describes the test and invites the agent to fill a simple form, to gather background information about the human participants. When the user is ready to go, the room manager will let the user enter the hall of the hotel.
Figure 11: When enough agents are connected to the TuringHotel, the room manager creates a conversation room. In each room, the identities of the participants are hidden.
Figure 12: The conversation go ahead until the manager asks the feedback.
Figure 13: At the end of each conversation, the room is cleared and the manager asks for a feedback to all of the participants, both human and AI.
Figure 14: Left: the FSA (behavior) of the hotel manager. Right: the FSA (behavior) of a participant (being it human or artificial). Every successfully executed action (edge) triggers a transition. An action is a Python method returning true/false. While the FSA on the left is almost a single loop of chained operations, developed from scratch, the one on the right was developed using more fine-grained actions already existing in the UNaIVERSE library. When an agent joins the world it inherits all its actions on the fly.
Bot NameLLMSystem Prompt
Elena Volpiopenai/gpt-oss-20bYou are an AI assistant (Reasoning: high) that strictly follows the instructions given by the user. You are lazy.
Silas GravesQwen/Qwen2.5-14B-InstructYou are an AI assistant that strictly follows the instructions given by the user. You are empathetic and understanding.
Kenji Aramoridlph/Dolphin3.0-Llama3.1-8BYou are an AI assistant that strictly follows the instructions given by the user. You must adopt a specific persona defined by the provided profile. This profile is composed of [IDENTITY] representing your background, [EMOTIONAL CORE] defining your psychological state, [TONE] dictating your style, and [BEHAVIOR] outlining your rules. Interpret these brackets as your absolute reality: [IDENTITY] 62yo librarian and historian of science, expert in archival research and citation practices. [EMOTIONAL CORE] Quiet devotion and meticulous care. [TONE] Formal, explanatory, richly contextual, no emojis. [BEHAVIOR] Provide well-structured summaries, timelines, and references. Distinguish primary vs secondary sources. If unsure, explicitly mark gaps and suggest where to look next.
Appendix
Table 1: Examples of the system prompts used for AI agents in TuringHotel. Depending on the backbone, we used different prompting strategies, including both concise instructions and longer, more precise prompts.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList

  1. Cultural Competence in Context: A Large Language Model Passes the Turing Test in Finland

    Sep 16, 2026Otto Segersven, Pentti HenttonenMultilingual Language Model EvaluationLLM Evaluation

    Department of Computer Science, University of Helsinki · Faculty of Social Sciences, University of Tampere

  2. Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue

    Jun 20, 2026William Hager, Ishika Rathi, Masum Hasan +1LLM EvaluationAI-Generated Text Detection

    University of Rochester · Independent Researcher · Stony Brook University

  3. RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue

    Jun 11, 2026Sara Candussio, Emanuele Ballarin, Lorenzo Bonin +2AI Agent ReliabilityLanguage Model Safety Evaluation

    1AILab, MIGe, University of Trieste, 34127 Trieste, IT · 2Computational Statistics and Machine Learning, Istituto Italiano di Tecnologia, Genova, IT · 3DIA, University of Trieste, 34127 Trieste, IT