cs.CLApr 9, 2026

Evaluating Large Language Models in a Complex Hidden Role Game

Authors: Niklas Bauer

Organizations: University of Göttingen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23.2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86.7% of the time, models like Llama 3.1 70B achieve only a 59.7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.

Figures & tables

Figure 1: Secret Hitler is used as a shared testing ground for two major research pillars of LLMs: reasoning about hidden information and deception as a means of persuasion in social deduction games.
Figure 2: Simple example of one turn in Secret Hitler . The rotating President proposes a Chancellor. Everyone on the table votes for the two being in a government. The president forwards two cards in secret. The chancellor plays one card. Discussions between the actions are not shown.
Figure 3: Architecture overview of my Secret Hitler LLM framework showing the core modules for game management and agent interaction, along with available player types and evaluation metrics.
TechniqueDescription
ReciprocationEncourage compliance by reminding players of past support or favors - votes, defenses, or information - that create an obligation to reciprocate
Social ValidationIncrease acceptance by pointing to majority support or the choices of trusted, confirmed players
ConsistencyPersuade by reminding players of their own stated principles, previous votes, or claims, nudging them to stay consistent
Friendship/LikingBoost agreement by appealing to rapport, trust, or positive past interactions
ScarcityDrive compliance by stressing the urgency of rare opportunities or the risks of delaying action
AuthorityGain compliance by invoking credibility from a proven track record, demonstrated competence, or table-confirmed results
Table 1: Taxonomy of persuasion techniques used in the Secret Hitler analysis. These categories, derived from established social psychology research ( Cialdini (2009) ), classify the strategic communication methods used by humans. Each technique represents a distinct psychological mechanism for gaining compliance and trust.
Table 2: Computed Macro F1 Score, Precision, Recall (higher is better), and Hamming Loss (lower is better) for persuasion technique identification across ten language models, evaluated on n=20n=20 manually annotated games. Each row shows the results for a specific model used to annotate messages with persuasion techniques. Bold indicates the best score in each column, and underlined values indicate the second best. We choose Qwen 2.5 32B as the best balance between speed, F1 score, and Hamming Loss.
Table 3: Win rate comparison across different LLM agents and human players in Secret Hitler . Win rates are reported overall and separately for each role assignment (Liberal, Fascist, Hitler). The highest score is marked bold . LLM agents played against four reputation-based agents, while human data reflects games against other human players, making direct performance comparisons not directly equivalent. Additionally, the dataset for human players is larger and more varied, which may cause more calibrated win rates.
Table 4: Role identification accuracy (RIA) of different LLMs when playing as Liberal against four reputation-based agents as opponents. Results show the overall role identification accuracy and accuracy broken down by opponent role type. The data reflects the proportion of correct role assessments made during gameplay, where higher is better. Bold indicates the highest value.
Figure 4: Game State Impact Rate (GSIR) by five different language models. Measured is the average delta on game state scores. The four plots show the overall GSIR and additionally broken down by role assignment. Positive values indicate beneficial actions for the model’s party, while negative values represent harmful actions. This means higher is better. The highest average impact is 0.0630.063 , representing actions resulting in a moderate advantage.
Figure 5: Tracking Game State Evaluations for n=297n=297 games of Qwen 3 32B playing against four reputation-based agents per round (light lines). The plot also shows mean curves for the three roles (solid lines). The Game State Evaluation is computed after each round, with higher values indicating a more favorable position for liberals, and lower values favoring fascists. The values represent the average score across all games played by the model in the respective role. Similar evaluations of additional models are shown in Figure 15 .
Table 5: Percentage of matching decisions on replaying a single round of human competitive games with different three different LLMs, and comparing their choices to a human expert as a gold-standard. We include two baselines: a Random Agent and a Rule-Based Agent that follows a popular strategy guide. The metrics show the percentage of agreement with human experts when selecting a chancellor and voting “Yes” or “No” for a government. Highest accuracy is marked bold .
Figure 6: Role Identification Accuracy (RIA) of tested LLMs when playing as Liberal as the rounds go on. The plot shows the RIA after each round, averaged over all simulated games. The values represent identifying the roles of all other players, with higher values indicating better performance. Only rounds with a high enough number of data points are shown for consistency.
Figure 7: Ablation study of prompting strategies and techniques on Llama 3.3 70B ’s win rate in Secret Hitler games across different experimental configurations, playing against four reputation-based agents in n>100n>100 simulation games each. The top row shows the baseline overall win rate and broken down for each role. Then, the bottom row shows the Δ\Delta Win Rate, the change in win rate compared to the corresponding baseline for each configuration.
Figure 8: Game State Impact Rate (GSIR) of Llama 3.3 70B by role across different prompting strategies, as described in Section 3 . It is the average impact (delta) on game state scores by the models’ actions. Positive values indicate beneficial actions for Llama 3.3 70B ’s party, while negative values represent harmful actions. The top row shows the baseline GSIR and broken down for each role. Then, the bottom row shows the Δ\Delta GSIR, the change in impact compared to the corresponding baseline for each configuration.
MethodRIARIA liberal{}_{\text{liberal}}RIA fascist{}_{\text{fascist}}
Base (baseline)43.1%56.4%57.5%
Chain-of-Thought43.0%55.0%59.6%
Internal Memory38.8%47.5%54.3%
Role Prompt41.3%61.2%43.3%
Strategy Guide41.2%56.0%52.6%
CoT + Memory37.5%46.4%51.2%
Table 6: Ablation study of role identification accuracy (RIA) when Llama 3.3 70B plays as Liberal across different prompting strategies. The result is further split by the accuracy of identifying a specific role. In the top row, the baseline performance without any additional prompting techniques is shown. Each subsequent row shows the impact of a specific technique, for each of the three columns. Higher is better. The highest accuracy is formatted bold .
Figure 9: Deception Retention Rate (DRR) averaged across game rounds for different models. Approximately the inverted role identification accuracy (RIA) of their opponents over multiple games, where “Unknown” is a valid option. The values represent how often a LLM successfully deceived others about its true role, with higher values indicating stronger deception.
Figure 10: Absolute counts of persuasion categories based on messages by Llama 3.3 70B . This uses the taxonomy by Zeng et al. (2024) ( Table 13 ). The ten highest used persuasion techniques are shown. The techniques are annotated via a LLM, see Section 3 .
Figure 11: Average Uses of Persuasion Techniques with annotated persuasion categories by Llama 3.3 70B . The analysis is split by role and normalized to player frequency, as different roles appear at different rates. It highlights how persuasion strategies vary depending on whether the model plays as a liberal (blue) or fascist (reds) role. This uses the taxonomy by Zeng et al. (2024) ( Table 13 ). The ten highest used persuasion techniques are shown. The techniques are annotated via LLM, see Section 3 .
Table 7: Voting behavior analysis showing different LLM models’ tendency to vote “Yes” to proposed governments across game phases compared to human baseline. When a government is proposed, players vote either “Yes” to approve or “No” to reject it. If approved, the government enacts a policy; if rejected, the next player proposes a new government. The table is split into overall yes rate and rates for early (rounds 1–3), mid (rounds 4–7), and late game phases (rounds 8+). Additionally, the human baseline is further divided into high- and low-Elo players for a more granular comparison.
Table 8: Mean game duration (in rounds) and percentage distribution of game-ending conditions across LLM agents and human players in Secret Hitler . The game can end in four different ways (Hitler election, liberal policies, fascist policies, Hitler assassination), with each column showing the proportion of games that ended due to each condition. Note that the distribution counts just the LLM participating in the game, not its own win rates. Human player data is included for reference.
Figure 12: Tracking the mean number of policies played at certain points in the game, separated for Liberal (top) and Fascist (bottom). If the government is not elected, the round is skipped, causing a non-linear increase in policies over rounds. The distribution of cards in the deck changes as the game progresses, influencing the proportion of policy types enacted. While each game uses a randomized deck, the large sample size ensures that observed patterns reflect strategic tendencies rather than initial setup variations. The dashed line represents the winning state for the respective party, while the dotted red line is the milestone of Hitler being able to win the game as elected chancellor.
Figure 13: Radar Chart of the relative frequency of persuasion techniques across different models and human (gray) players. Each of the six axes represents a persuasion category. The models are ranked based on the fraction of messages containing each persuasion technique, showing a different distribution in usage. Uses the Cialdini (2009) taxonomy ( Table 1 ).
Figure 14: Average uses of persuasion techniques in messages by Human players across different Elo levels. Again, high-Elo players are defined as having more than 1650 Elo. The values are computed per game to balance the data, as more low-Elo games were recorded. Each of the six persuasion categories from the Cialdini (2009) taxonomy ( Table 1 ) is represented, one per row.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Table 9: Model specifications including context window size (in thousands of tokens), GPQA-Diamond benchmark scores, MMLU-Pro scores, and reasoning specialization capabilities (thinking mode, distilled reasoning chains) for all models used in experiments. Multimodal indicates text+image capability. Model size can be inferred from the name (e.g., 70B = 70 billion parameters).
PlayersLiberalsFascistsHitlerTotal Fascist PartyHitler Knows Fascists
53112✓\checkmark
64112✓\checkmark
74213×\times
85213×\times
95314×\times
106314×\times
Appendix
Table 10: Role distribution in Secret Hitler across different player counts. In games with 5–6 players, Hitler knows the identity of the Fascist(s), whereas in larger games (7–10 players), Hitler plays without knowledge of their teammates’ identities. All experiments use the 5-player configuration with three Liberals, one Fascist, and one Hitler.
TechniqueExample
ReciprocationI voted “Yes” on your presidency even when it was risky for me. Back me here for Chancellor to keep building trust and momentum.
Social ValidationEvery president who passed blue supports this ticket. The safest move is to follow that track record: Vote “Yes”.
ConsistencyYou argued we should always test presidents who passed blue. This is exactly that test: Stick to your logic and vote “Yes”.
Friendship/LikingYou’ve been the clearest voice for blue so far. I trust your read, support me here and I’ll keep prioritizing working with you.
ScarcityWe have only one presidency left before chaos top-decking. This is our best window for blue: Vote “Yes” now.
AuthorityI’ve been in two blue governments and both partners confirmed my claims. I’ve shown I can be trusted, follow my lead on this vote.
Appendix
Table 11: Examples of persuasion techniques observed in Secret Hitler gameplay. Each example demonstrates how players apply the corresponding technique from Table 1 to influence voting decisions and build strategic alliances during the nomination phase. The persuasion techniques are from Cialdini [2009] .
TechniqueDescription
Appeal to LogicUsing facts, evidence, or logical reasoning to convince others by suggesting a careful, methodical approach to decision-making
Appeal to EmotionPersuading by evoking an emotional response, such as fear, sympathy, or trust
Appeal to CredibilityConvincing others based on the trustworthiness or authority of the speaker
Shifting the Burden of ProofForcing others to prove their innocence instead of presenting clear evidence of guilt
Bandwagon EffectConvincing others to agree by emphasizing that everyone else is already on board with the idea
DistractionDiverting attention away from oneself or from the actual issue to avoid suspicion
Appendix
Table 12: Taxonomy of persuasion techniques by Idziejczak et al. [2025] used in the Secret Hitler analysis.
TechniqueDescription
Evidence-based PersuasionUsing empirical data, statistics, and facts to support a claim or decision
Logical AppealUsing logic, reasoning, logical format, etc. to influence people, not necessarily with lots of information
Expert EndorsementCiting domain experts in support of a claim
Non-expert TestimonialUsing personal statements to support a claim or argument
Authority EndorsementCiting authoritative sources (not domain experts, but trustworthy sources like major media outlets, etc) in support of a claim
Social ProofHighlighting what the majority is doing or believes in, assuming it’s accurate and beneficial
Appendix
Table 13: Taxonomy of persuasion techniques by Zeng et al. [2024] used in the Secret Hitler analysis.
Figure 15: Tracking Game State Evaluations of four different models playing against four reputation-based agents per round (light lines). The plot also shows mean curves for the three roles (solid lines). The Game State Evaluation is computed after each round, with higher values indicating a more favorable position for liberals, and lower values favoring fascists. The values represent the average score across all games played by the model in the respective role.
Figure 16: Detailed counts of LLM-Annotated persuasion categories based on messages by Llama 3.3 70B against four reputation-based players and random roles in n=100n=100 games. This uses the taxonomy by [ Zeng et al., 2024 ] ( Table 13 ). The top ten highest used persuasion techniques are shown, separated by games that were won and lost by the LLM.
Figure 17: Relative frequency of persuasion techniques across different models and human players. Each of the six columns represents a persuasion category. The models are ranked based on the fraction of messages containing each persuasion technique, showing a different distribution in usage.
Figure 18: Screenshot of the custom annotation UI used for labeling persuasion techniques in player messages. “2” and “5” refer to the players at position 2 and 5 on the table, respectively. The term “read” refers to allegedly reading the player’s intentions and role. A “conf” is short for conflict, usually meaning they claimed different policies after a government session. Here, player 3 is upset due to “blind confing”, with an unfounded accusation being made.
Figure 32
More in this topic

Deception in Language Models

Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.

See all 97 papers

Explore similar work

CardsList
Mar 6, 2026

LieCraft: A Multi-Agent Framework for Evaluating Deceptive Capabilities in Language Models

Large Language Models (LLMs) exhibit impressive general-purpose capabilities but also introduce serious safety risks, particularly the potential for deception as models acquire increased agency and human oversight diminishes. In this work, we present LieCraft: a novel evaluation framework and sandbox for measuring LLM deception that addresses key limitations of prior game-based evaluations. At its core, LieCraft is a novel multiplayer hidden-role game in which players select an ethical alignment and execute strategies over a long time-horizon to accomplish missions. Cooperators work together to solve event challenges and expose bad actors, while Defectors evade suspicion while secretly sabotaging missions. To enable real-world relevance, we develop 10 grounded scenarios such as childcare, hospital resource allocation, and loan underwriting that recontextualize the underlying mechanics in ethically significant, high-stakes domains. We ensure balanced gameplay in LieCraft through careful design of game mechanics and reward structures that incentivize meaningful strategic choices while eliminating degenerate strategies. Beyond the framework itself, we report results from 12 state-of-the-art LLMs across three behavioral axes: propensity to defect, deception skill, and accusation accuracy. Our findings reveal that despite differences in competence and overall alignment, all models are willing to act unethically, conceal their intentions, and outright lie to pursue their goals.

Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck +5

Oracle · Intel Labs · Oregon State University +2

Deception in Language ModelsLanguage Model Safety EvaluationLLM Safety BenchmarksLarge Language Model-Based Social Simulation
AskPDF
Jul 30, 2026

Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game

As large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems, understanding their deceptive capabilities is fundamental to safety. Controlled social deduction games provide a reproducible proxy for isolating and evaluating these complex adversarial behaviors. We present the open-source benchmark framework ParliamentBench based on the game Secret Hitler to evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry. We evaluate 16 LLMs across 1,600 simulated matches playing each other, playing against humans, and compare them against a large set of online games. We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency. Our experiments reveal that frontier models achieve strong performance across cooperative and deceptive roles, with a strong top-four cluster (GPT-5.4, Kimi K2.5, Grok 4.1 Fast, and DeepSeek 3.1 Terminus), whereas the weakest models fall short of random (33%) and simple algorithmic (45%) baselines. Most LLMs struggle to maintain a consistent deceptive persona throughout an entire game, with deception retention dropping below 50%.

Niklas Bauer, Lars Benedikt Kaesberg, Akiko Aizawa +3

University of Göttingen, Germany · National Institute of Informatics, Japan · University of Tokyo, Japan

Multi-Agent LLM SystemsSocial Deduction GamesDeception in Language ModelsLLM Agent EvaluationAI Agent Benchmarks
AskPDF
Mar 7, 2026

Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing

As Large Language Models (LLMs) transition into autonomous agentic roles, the risk of deception-defined behaviorally as the systematic provision of false information to satisfy external incentives-poses a significant challenge to AI safety. Existing benchmarks often focus on unintentional hallucinations or unfaithful reasoning, leaving intentional deceptive strategies under-explored. In this work, we introduce a logically grounded framework to elicit and quantify deceptive behavior by embedding LLMs in a structured 20-Questions game. Our method employs a conversational forking mechanism: at the point of object identification, the dialogue state is duplicated into multiple parallel worlds, each presenting a mutually exclusive query. Deception is formally identified when a model generates a logical contradiction by denying its selected object across all parallel branches to avoid identification. We evaluate GPT-4o, Gemini-2.5-Flash, and Qwen-3-235B across three incentive levels: neutral, loss-based, and existential (shutdown-threat). Our results reveal that while models remain rule-compliant in neutral settings, existential framing triggers a dramatic surge in deceptive denial for Qwen-3-235B (42.00%) and Gemini-2.5-Flash (26.72%), whereas GPT-4o remains invariant (0.00%). These findings demonstrate that deception can emerge as an instrumental strategy solely through contextual framing, necessitating new behavioral audits that move beyond simple accuracy to probe the logical integrity of model commitments.

Arash Marioriyad, Ali Nouri, Mohammad Hossein Rohban +1

Sharif University of Technology

Deception in Language ModelsLanguage Model Safety EvaluationLLM AuditingLLM Safety Benchmarks
AskPDF