cs.CLApr 9, 2026
SaveEvaluating Large Language Models in a Complex Hidden Role Game
Organizations: University of Göttingen
Abstract
Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23.2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86.7% of the time, models like Llama 3.1 70B achieve only a 59.7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.
Figures & tables
Figure 1: Secret Hitler is used as a shared testing ground for two major research pillars of LLMs: reasoning about hidden information and deception as a means of persuasion in social deduction games.
Figure 2: Simple example of one turn in Secret Hitler . The rotating President proposes a Chancellor. Everyone on the table votes for the two being in a government. The president forwards two cards in secret. The chancellor plays one card. Discussions between the actions are not shown.
Figure 3: Architecture overview of my Secret Hitler LLM framework showing the core modules for game management and agent interaction, along with available player types and evaluation metrics.
| Technique | Description |
|---|---|
| Reciprocation | Encourage compliance by reminding players of past support or favors - votes, defenses, or information - that create an obligation to reciprocate |
| Social Validation | Increase acceptance by pointing to majority support or the choices of trusted, confirmed players |
| Consistency | Persuade by reminding players of their own stated principles, previous votes, or claims, nudging them to stay consistent |
| Friendship/Liking | Boost agreement by appealing to rapport, trust, or positive past interactions |
| Scarcity | Drive compliance by stressing the urgency of rare opportunities or the risks of delaying action |
| Authority | Gain compliance by invoking credibility from a proven track record, demonstrated competence, or table-confirmed results |
Table 1: Taxonomy of persuasion techniques used in the Secret Hitler analysis. These categories, derived from established social psychology research ( Cialdini (2009) ), classify the strategic communication methods used by humans. Each technique represents a distinct psychological mechanism for gaining compliance and trust.
Table 2: Computed Macro F1 Score, Precision, Recall (higher is better), and Hamming Loss (lower is better) for persuasion technique identification across ten language models, evaluated on manually annotated games. Each row shows the results for a specific model used to annotate messages with persuasion techniques. Bold indicates the best score in each column, and underlined values indicate the second best. We choose Qwen 2.5 32B as the best balance between speed, F1 score, and Hamming Loss.
Table 3: Win rate comparison across different LLM agents and human players in Secret Hitler . Win rates are reported overall and separately for each role assignment (Liberal, Fascist, Hitler). The highest score is marked bold . LLM agents played against four reputation-based agents, while human data reflects games against other human players, making direct performance comparisons not directly equivalent. Additionally, the dataset for human players is larger and more varied, which may cause more calibrated win rates.
Table 4: Role identification accuracy (RIA) of different LLMs when playing as Liberal against four reputation-based agents as opponents. Results show the overall role identification accuracy and accuracy broken down by opponent role type. The data reflects the proportion of correct role assessments made during gameplay, where higher is better. Bold indicates the highest value.
Figure 4: Game State Impact Rate (GSIR) by five different language models. Measured is the average delta on game state scores. The four plots show the overall GSIR and additionally broken down by role assignment. Positive values indicate beneficial actions for the model’s party, while negative values represent harmful actions. This means higher is better. The highest average impact is , representing actions resulting in a moderate advantage.
Figure 5: Tracking Game State Evaluations for games of Qwen 3 32B playing against four reputation-based agents per round (light lines). The plot also shows mean curves for the three roles (solid lines). The Game State Evaluation is computed after each round, with higher values indicating a more favorable position for liberals, and lower values favoring fascists. The values represent the average score across all games played by the model in the respective role. Similar evaluations of additional models are shown in Figure 15 .
Table 5: Percentage of matching decisions on replaying a single round of human competitive games with different three different LLMs, and comparing their choices to a human expert as a gold-standard. We include two baselines: a Random Agent and a Rule-Based Agent that follows a popular strategy guide. The metrics show the percentage of agreement with human experts when selecting a chancellor and voting “Yes” or “No” for a government. Highest accuracy is marked bold .
Figure 6: Role Identification Accuracy (RIA) of tested LLMs when playing as Liberal as the rounds go on. The plot shows the RIA after each round, averaged over all simulated games. The values represent identifying the roles of all other players, with higher values indicating better performance. Only rounds with a high enough number of data points are shown for consistency.
Figure 7: Ablation study of prompting strategies and techniques on Llama 3.3 70B ’s win rate in Secret Hitler games across different experimental configurations, playing against four reputation-based agents in simulation games each. The top row shows the baseline overall win rate and broken down for each role. Then, the bottom row shows the Win Rate, the change in win rate compared to the corresponding baseline for each configuration.
Figure 8: Game State Impact Rate (GSIR) of Llama 3.3 70B by role across different prompting strategies, as described in Section 3 . It is the average impact (delta) on game state scores by the models’ actions. Positive values indicate beneficial actions for Llama 3.3 70B ’s party, while negative values represent harmful actions. The top row shows the baseline GSIR and broken down for each role. Then, the bottom row shows the GSIR, the change in impact compared to the corresponding baseline for each configuration.
| Method | RIA | RIA | RIA |
|---|---|---|---|
| Base (baseline) | 43.1% | 56.4% | 57.5% |
| Chain-of-Thought | 43.0% | 55.0% | 59.6% |
| Internal Memory | 38.8% | 47.5% | 54.3% |
| Role Prompt | 41.3% | 61.2% | 43.3% |
| Strategy Guide | 41.2% | 56.0% | 52.6% |
| CoT + Memory | 37.5% | 46.4% | 51.2% |
Table 6: Ablation study of role identification accuracy (RIA) when Llama 3.3 70B plays as Liberal across different prompting strategies. The result is further split by the accuracy of identifying a specific role. In the top row, the baseline performance without any additional prompting techniques is shown. Each subsequent row shows the impact of a specific technique, for each of the three columns. Higher is better. The highest accuracy is formatted bold .
Figure 9: Deception Retention Rate (DRR) averaged across game rounds for different models. Approximately the inverted role identification accuracy (RIA) of their opponents over multiple games, where “Unknown” is a valid option. The values represent how often a LLM successfully deceived others about its true role, with higher values indicating stronger deception.
Figure 10: Absolute counts of persuasion categories based on messages by Llama 3.3 70B . This uses the taxonomy by Zeng et al. (2024) ( Table 13 ). The ten highest used persuasion techniques are shown. The techniques are annotated via a LLM, see Section 3 .
Figure 11: Average Uses of Persuasion Techniques with annotated persuasion categories by Llama 3.3 70B . The analysis is split by role and normalized to player frequency, as different roles appear at different rates. It highlights how persuasion strategies vary depending on whether the model plays as a liberal (blue) or fascist (reds) role. This uses the taxonomy by Zeng et al. (2024) ( Table 13 ). The ten highest used persuasion techniques are shown. The techniques are annotated via LLM, see Section 3 .
Table 7: Voting behavior analysis showing different LLM models’ tendency to vote “Yes” to proposed governments across game phases compared to human baseline. When a government is proposed, players vote either “Yes” to approve or “No” to reject it. If approved, the government enacts a policy; if rejected, the next player proposes a new government. The table is split into overall yes rate and rates for early (rounds 1–3), mid (rounds 4–7), and late game phases (rounds 8+). Additionally, the human baseline is further divided into high- and low-Elo players for a more granular comparison.
Table 8: Mean game duration (in rounds) and percentage distribution of game-ending conditions across LLM agents and human players in Secret Hitler . The game can end in four different ways (Hitler election, liberal policies, fascist policies, Hitler assassination), with each column showing the proportion of games that ended due to each condition. Note that the distribution counts just the LLM participating in the game, not its own win rates. Human player data is included for reference.
Figure 12: Tracking the mean number of policies played at certain points in the game, separated for Liberal (top) and Fascist (bottom). If the government is not elected, the round is skipped, causing a non-linear increase in policies over rounds. The distribution of cards in the deck changes as the game progresses, influencing the proportion of policy types enacted. While each game uses a randomized deck, the large sample size ensures that observed patterns reflect strategic tendencies rather than initial setup variations. The dashed line represents the winning state for the respective party, while the dotted red line is the milestone of Hitler being able to win the game as elected chancellor.
Figure 13: Radar Chart of the relative frequency of persuasion techniques across different models and human (gray) players. Each of the six axes represents a persuasion category. The models are ranked based on the fraction of messages containing each persuasion technique, showing a different distribution in usage. Uses the Cialdini (2009) taxonomy ( Table 1 ).
Figure 14: Average uses of persuasion techniques in messages by Human players across different Elo levels. Again, high-Elo players are defined as having more than 1650 Elo. The values are computed per game to balance the data, as more low-Elo games were recorded. Each of the six persuasion categories from the Cialdini (2009) taxonomy ( Table 1 ) is represented, one per row.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9: Model specifications including context window size (in thousands of tokens), GPQA-Diamond benchmark scores, MMLU-Pro scores, and reasoning specialization capabilities (thinking mode, distilled reasoning chains) for all models used in experiments. Multimodal indicates text+image capability. Model size can be inferred from the name (e.g., 70B = 70 billion parameters).
| Players | Liberals | Fascists | Hitler | Total Fascist Party | Hitler Knows Fascists |
| 5 | 3 | 1 | 1 | 2 | |
| 6 | 4 | 1 | 1 | 2 | |
| 7 | 4 | 2 | 1 | 3 | |
| 8 | 5 | 2 | 1 | 3 | |
| 9 | 5 | 3 | 1 | 4 | |
| 10 | 6 | 3 | 1 | 4 |
Table 10: Role distribution in Secret Hitler across different player counts. In games with 5–6 players, Hitler knows the identity of the Fascist(s), whereas in larger games (7–10 players), Hitler plays without knowledge of their teammates’ identities. All experiments use the 5-player configuration with three Liberals, one Fascist, and one Hitler.
| Technique | Example |
|---|---|
| Reciprocation | I voted “Yes” on your presidency even when it was risky for me. Back me here for Chancellor to keep building trust and momentum. |
| Social Validation | Every president who passed blue supports this ticket. The safest move is to follow that track record: Vote “Yes”. |
| Consistency | You argued we should always test presidents who passed blue. This is exactly that test: Stick to your logic and vote “Yes”. |
| Friendship/Liking | You’ve been the clearest voice for blue so far. I trust your read, support me here and I’ll keep prioritizing working with you. |
| Scarcity | We have only one presidency left before chaos top-decking. This is our best window for blue: Vote “Yes” now. |
| Authority | I’ve been in two blue governments and both partners confirmed my claims. I’ve shown I can be trusted, follow my lead on this vote. |
Table 11: Examples of persuasion techniques observed in Secret Hitler gameplay. Each example demonstrates how players apply the corresponding technique from Table 1 to influence voting decisions and build strategic alliances during the nomination phase. The persuasion techniques are from Cialdini [2009] .
| Technique | Description |
|---|---|
| Appeal to Logic | Using facts, evidence, or logical reasoning to convince others by suggesting a careful, methodical approach to decision-making |
| Appeal to Emotion | Persuading by evoking an emotional response, such as fear, sympathy, or trust |
| Appeal to Credibility | Convincing others based on the trustworthiness or authority of the speaker |
| Shifting the Burden of Proof | Forcing others to prove their innocence instead of presenting clear evidence of guilt |
| Bandwagon Effect | Convincing others to agree by emphasizing that everyone else is already on board with the idea |
| Distraction | Diverting attention away from oneself or from the actual issue to avoid suspicion |
Table 12: Taxonomy of persuasion techniques by Idziejczak et al. [2025] used in the Secret Hitler analysis.
| Technique | Description |
|---|---|
| Evidence-based Persuasion | Using empirical data, statistics, and facts to support a claim or decision |
| Logical Appeal | Using logic, reasoning, logical format, etc. to influence people, not necessarily with lots of information |
| Expert Endorsement | Citing domain experts in support of a claim |
| Non-expert Testimonial | Using personal statements to support a claim or argument |
| Authority Endorsement | Citing authoritative sources (not domain experts, but trustworthy sources like major media outlets, etc) in support of a claim |
| Social Proof | Highlighting what the majority is doing or believes in, assuming it’s accurate and beneficial |
Table 13: Taxonomy of persuasion techniques by Zeng et al. [2024] used in the Secret Hitler analysis.
Figure 15: Tracking Game State Evaluations of four different models playing against four reputation-based agents per round (light lines). The plot also shows mean curves for the three roles (solid lines). The Game State Evaluation is computed after each round, with higher values indicating a more favorable position for liberals, and lower values favoring fascists. The values represent the average score across all games played by the model in the respective role.
Figure 16: Detailed counts of LLM-Annotated persuasion categories based on messages by Llama 3.3 70B against four reputation-based players and random roles in games. This uses the taxonomy by [ Zeng et al., 2024 ] ( Table 13 ). The top ten highest used persuasion techniques are shown, separated by games that were won and lost by the LLM.
Figure 17: Relative frequency of persuasion techniques across different models and human players. Each of the six columns represents a persuasion category. The models are ranked based on the fraction of messages containing each persuasion technique, showing a different distribution in usage.
Figure 18: Screenshot of the custom annotation UI used for labeling persuasion techniques in player messages. “2” and “5” refer to the players at position 2 and 5 on the table, respectively. The term “read” refers to allegedly reading the player’s intentions and role. A “conf” is short for conflict, usually meaning they claimed different policies after a government session. Here, player 3 is upset due to “blind confing”, with an unfounded accusation being made.
More in this topic
Deception in Language Models
Deception in language models studies when systems mislead through lies, omissions, or selective truths, and how to detect and prevent it.
See all 97 papers