cs.CLOct 3, 2026
SavePlaying social deduction games with reinforcement fine-tuned large language models
Organizations: Peking University, Beijing, China. · Alibaba Group, China. · University of Illinois Chicago, Chicago, Illinois, United States.
Abstract
Reinforcement fine-tuning (RFT) is increasingly used in applications where large language models (LLMs) interact with humans and other agents. Here we use social deduction games to study how RFT changes LLMs' social behaviour. We let fine-tuned and base LLM agents play hidden-role games that require hidden-state inference, social reading and vote steering. Our results show that LLM agents do not reliably acquire social-deduction ability by directly optimizing terminal win--loss outcomes, suggesting that final game results provide a sparse and noisy signal for socially interactive learning. However, RFT is particularly effective at improving social reading, including tasks that require agents to infer hidden roles from public discussion, update beliefs over time and predict other agents' future decisions. We further show that RFT can also improve social influence, including tasks that require agents to steer votes, team approvals and collective decisions, although these gains depend more strongly on behaviourally specific rewards and structured interaction settings. Finally, we show that LLMs' ability to play social deduction games can be further improved through multi-agent social-cognitive reinforcement fine-tuning, which combines social-reading and social-influence signals during same-side multi-agent training. These learned behaviours also receive more favourable human evaluations of strategic competence, persuasiveness and social usefulness. Together, these results enrich our understanding of how RFT changes LLMs' social behaviour and provide a step toward a behavioural learning theory for machine social intelligence.
Figures & tables
Figure 1: Playing social deduction games with reinforcement fine-tuning in an example game of Avalon. In step 1, LLM-controlled agents with hidden good or evil roles interact in an Avalon-like social deduction game through public discussion, team proposals, voting, missions and final outcomes. In step 2, the full-game trajectory, including dialogue, votes, mission results and outcome, is collected as a complete episode record. In step 3, reinforcement fine-tuning updates the agents from this full-game experience. In step 4, the updated agents return to play the next game, forming a closed learning loop across repeated social deduction games.
Figure 2: Macro-families of social deduction games.
Figure 3: RFT training dynamics across social deduction game families. We report the mean episode-level outcome reward during training for representative games from the Elimination, Institution, and Asymmetry families. The reward is defined by whether the focal agent’s side achieves its game-specific objective at the end of the episode. Shaded bands and capped error bars indicate the standard error across rollouts within each training step.
Figure 4: RFT improves hidden-state inference across social deduction games. We train agents on controlled social-reading tasks and report hidden-state prediction accuracy over training. a, In Avalon, agents infer each player’s hidden team alignment from public game transcripts. b, In Werewolf/Mafia, agents infer the hidden wolf set from dialogue and voting evidence. Across completed Qwen, GLM, and DeepSeek runs, RFT improves hidden-state inference, indicating that reinforcement fine-tuning can strengthen social reading even when aggregate outcome-based game training is noisy.
Figure 5: RFT improves dynamic belief updating during social deduction. We analyse a representative Qwen2.5-7B run in Avalon and Werewolf/Mafia. a, In Avalon, we group belief predictions by mission and report alignment accuracy over training. Later missions provide richer public evidence from team proposals, votes, and quest outcomes, and the learned policy assigns more accurate hidden-team beliefs as training progresses. b, In Werewolf/Mafia, we evaluate the focal villager’s belief about the hidden wolf set after successive days of discussion. RFT improves wolf-set F1 across training, with later-day beliefs benefiting more strongly from accumulated dialogue and elimination evidence.
Figure 6: Role-conditioned RFT dynamics in Avalon. a , Mean episode-level outcome reward when the focal agent is trained under a fixed Avalon role rather than sampled across roles. Among the four roles, the Assassin shows the clearest and most sustained improvement, whereas Merlin, Servant, and Minion remain substantially noisier. b , Mean response length over the same training window. The Assassin rapidly reduces its response length while improving reward, indicating that the learned policy does not rely on longer deliberation. Instead, the agent learns to make minimal intermediate responses during missions and discussions, while preserving the game-critical decision for the final assassination.
Figure 7: Robustness of social-reading improvement across independent RFT runs. a, Werewolf social-reading performance across four independent training runs. Curves show the per-run mean reward over training, and shaded regions indicate s.e.m. across sampled rollouts at each training step. Numbers at the end of each curve denote the improvement from the first 20 to the last 20 training updates. b, Response length over training for the same runs, with shaded regions indicating s.e.m. across sampled rollouts. The performance gains were not accompanied by a systematic increase to the response-length limit.
Figure 8: RFT enables self-selected vote steering in One Night Werewolf. a, Target vote hit rate during training. The trained Minion self-selects a non-wolf vote target and attempts to steer other players’ final votes toward that target. Curves show binned means, and shaded regions denote step-level s.e.m. computed across training-step means within each bin. Numbers on the right indicate first-to-last improvement. b, Wolf-protection rate during training, measuring the fraction of other players whose final votes do not target the hidden Werewolf. c, Response length during training, measured in model-specific tokens. d, First-to-last improvement in target vote hit rate, wolf-protection rate and werewolf-team win rate.
Figure 9: RFT improves team-approval influence in Avalon. a, Target approval outcome during training across five model families. Curves show binned means and shaded regions denote s.e.m. across rollout episodes within each training bin. Numbers at the right indicate the first-to-last improvement in target approval outcome. b, Response length during training, measured in model-specific tokens. Shaded regions denote s.e.m. c, First-to-last improvement in target approval outcome and follow shift. Target approval outcome measures whether the final team vote matches the target direction; follow shift measures the change in the fraction of other players voting in the target direction after the trained agent’s response.
Figure 10: Repeated-run robustness of vote steering. a, Target vote hit rate for four independent Qwen2.5-7B self-selected vote-steering runs. Each curve shows the binned mean for one run, with shaded regions denoting step-level s.e.m. computed across training-step means within each bin. Numbers on the right indicate first-to-last improvement. b, Response length during training for the same four runs, measured in model-specific tokens. Shaded regions denote step-level s.e.m.
| Model | No SR/SI | SR only | SI only | SR+SI | SR+SI |
|---|---|---|---|---|---|
| Qwen2.5-7B | 24.0% | 32.0% | 33.0% | 37.0% | pp |
| Qwen2.5-14B | 15.0% | 16.0% | 17.0% | 23.0% | pp |
| Qwen3-4B | 21.0% | 30.0% | 12.0% | 34.0% | pp |
| GLM-4-9B | 19.0% | 21.0% | 21.5% | 22.0% | pp |
| DeepSeek-7B | 14.0% | 18.0% | 16.0% | 19.0% | pp |
Table 1: Prompt-level multi-agent social-cognitive ablations in Werewolf. Win rates are reported for good-side agents under four prompting conditions.
Figure 11: Full-game multi-agent social-cognitive reinforcement fine-tuning across games. Left column: training dynamics for six-player Werewolf, nine-player Werewolf and Avalon. Curves show scalar reward, win rate, social reading score and social influence score as a function of normalized training progress. Shaded regions denote variability across rollout groups. Middle and right columns: distributional shifts in social reading and social influence from early to late training, resolved by game day in Werewolf and by mission index in Avalon.
| Wolf-side Opponent | Good WR | SR-F1 | SI-Follow | Wolf-Exile |
|---|---|---|---|---|
| Qwen3-4B | 96% | 0.638 | 0.725 | 90.8% |
| Qwen3.7-Max | 94% | 0.643 | 0.625 | 84.7% |
| Qwen-Plus | 91% | 0.572 | 0.594 | 78.0% |
| GLM-4-9B-Chat | 90% | 0.599 | 0.678 | 80.9% |
| Qwen-Max | 90% | 0.606 | 0.620 | 85.0% |
| Qwen2.5-14B-Instruct | 87% | 0.577 | 0.708 | 78.7% |
Table 2: Cross-model evaluation of the learned good-side policy. Each opponent is evaluated over 100 valid full games using the same social-cognitive inference protocol. SR-F1 measures hidden-role identification, SI-Follow measures social-influence success, and Wolf-Exile measures the fraction of wolves successfully exiled.
Figure 12: Blinded evaluation of complete Werewolf trajectories. a, Mean ratings of trajectories generated by the base and social-cognitive RFT policies across five dimensions. Bars show means over the pooled blinded evaluation, comprising ratings per condition; error bars denote bootstrap confidence intervals over trajectory ratings. b, Direction of the RFT–base difference across game-identifier-matched trajectory pairs in the scalable evaluation. Bars show the percentage of pairs in which the RFT trajectory received a higher, equal or lower score than its matched base trajectory. Numbers inside the bars indicate pair counts.
| Evaluation dimension | Base | Social-cognitive RFT | |
|---|---|---|---|
| Overall value | |||
| Social-reading quality | |||
| Social-influence and coordination | |||
| Human-like gameplay | |||
| Strategic correctness |
Table 3: Blinded evaluation of complete trajectories generated by the base and social-cognitive RFT policies. Scores are measured on a five-point scale. Values report the mean and standard deviation over trajectory ratings per condition. denotes the RFT-minus-base difference.