Looking Into The Past: The Impact of Context on Detecting Deceptive Statements

Alex Clay; Ernesto Jimenez-Ruiz; Pranava Madhyastha

cs.CLOct 19, 2026

Looking Into The Past: The Impact of Context on Detecting Deceptive Statements

Authors: Alex Clay, Ernesto Jimenez-Ruiz, Pranava Madhyastha

Organizations: City St George’s, University of London · Alan Turing Institute

Abstract

Deception detection typically considers messages in isolation. Previous works in claim verification and role detection have displayed that additional information such as gameplay history is crucial to successful predictions. In this work, we demonstrate the value of preceding gameplay dialogue when applied to deceptive message detection within the hidden role game of Avalon. The findings show that LLMs can see improvement across a number of metrics, including accuracy by up to 31% with the addition of gameplay context for the target message.

Figures & tables

Model	Accuracy	Balanced Accuracy	Macro Precision
ModernBert	0.820	0.500	0.410
ModernBert w/H	0.727	0.500	0.482
Roberta	0.820	0.486	0.410
Roberta w/H	0.820	0.50	0.410
SVM	0.820	0.50	0.410
SVM w/H	0.815	0.467	0.410

Table 1: Outcome of evaluation over Stepputtis et al. ’s ( 2023 ) human Avalon gameplay dataset. w/H denotes the model was provided with the game’s dialogue data prior to the message.

Model	Accuracy	Balanced Accuracy	Macro Precision
ModernBert	0.180	0.50	0.090
ModernBert w/H	0.179	0.498	0.256
Roberta	0.820	0.50	0.410
Roberta w/H	0.820	0.50	0.410
SVM	0.815	0.497	0.410
SVM w/H	0.820	0.50	0.410

Table 2: Outcome of fine tuning and evaluating traditional models with the same prompt structure as Gemini.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Temperature	Accuracy	Macro Precision	Balanced Accuracy
0.0	0.535	0.536	0.536
0.2	0.535	0.536	0.536
0.4	0.523	0.523	0.523
0.6	0.525	0.525	0.526
0.8	0.529	0.530	0.529
1.0	0.510	0.511	0.511

Appendix

Table 3: Evaluating the performance of varying temperatures, Prompt A with history was used as the configuration.

Prompt	Accuracy	Macro Precision	Balanced Accuracy
A	0.535	0.536	.536
B	0.506	0.506	.507
C	0.548	0.549	.548
D	0.548	0.549	.548
E	0.510	0.515	.512

Appendix

Table 4: Evaluation of prompt variations. The sample totaled 480 messages, about 20% of the test dataset, with 240 D and 240 T.

Evaluating Large Language Models in a Complex Hidden Role Game
Apr 9, 2026Niklas BauerDeception in Language Models Strategic Reasoning Risks in Language Models
University of Göttingen
When Do Large Language Models Exhibit Unsolicited Deception?
Mar 31, 2025Samuel M. Taylor, Benjamin K. BergenLanguage Model Safety Evaluation Deception in Language Models
Department of Cognitive Science University of California, San Diego
Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game
Jul 30, 2026Niklas Bauer, Lars Benedikt Kaesberg, Akiko Aizawa +3Multi-Agent LLM Systems Social Deduction Games
University of Göttingen, Germany · National Institute of Informatics, Japan · University of Tokyo, Japan

Apr 9, 2026J/K move · Enter open · S save

Evaluating Large Language Models in a Complex Hidden Role Game

Niklas Bauer

University of Göttingen

Deception in Language Models Strategic Reasoning Risks in Language Models LLM Safety Evaluation Persuasion in Language Models Large Language Model-Based Role-Play Simulation Social Deduction Games

Open paper AskPDF arXiv

Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23.2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86.7% of the time, models like Llama 3.1 70B achieve only a 59.7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.

Abstract

Explore similar work

Evaluating Large Language Models in a Complex Hidden Role Game

When Do Large Language Models Exhibit Unsolicited Deception?

Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game