Reinforcement fine-tuning (RFT) is increasingly used in applications where large language models (LLMs) interact with humans and other agents. Here we use social deduction games to study how RFT changes LLMs' social behaviour. We let fine-tuned and base LLM agents play hidden-role games that require hidden-state inference, social reading and vote steering. Our results show that LLM agents do not reliably acquire social-deduction ability by directly optimizing terminal win--loss outcomes, suggesting that final game results provide a sparse and noisy signal for socially interactive learning. However, RFT is particularly effective at improving social reading, including tasks that require agents to infer hidden roles from public discussion, update beliefs over time and predict other agents' future decisions. We further show that RFT can also improve social influence, including tasks that require agents to steer votes, team approvals and collective decisions, although these gains depend more strongly on behaviourally specific rewards and structured interaction settings. Finally, we show that LLMs' ability to play social deduction games can be further improved through multi-agent social-cognitive reinforcement fine-tuning, which combines social-reading and social-influence signals during same-side multi-agent training. These learned behaviours also receive more favourable human evaluations of strategic competence, persuasiveness and social usefulness. Together, these results enrich our understanding of how RFT changes LLMs' social behaviour and provide a step toward a behavioural learning theory for machine social intelligence.
Figures & tables
Figure 1: Playing social deduction games with reinforcement fine-tuning in an example game of Avalon. In step 1, LLM-controlled agents with hidden good or evil roles interact in an Avalon-like social deduction game through public discussion, team proposals, voting, missions and final outcomes. In step 2, the full-game trajectory, including dialogue, votes, mission results and outcome, is collected as a complete episode record. In step 3, reinforcement fine-tuning updates the agents from this full-game experience. In step 4, the updated agents return to play the next game, forming a closed learning loop across repeated social deduction games.
Figure 2: Macro-families of social deduction games.
Figure 3: RFT training dynamics across social deduction game families. We report the mean episode-level outcome reward during training for representative games from the Elimination, Institution, and Asymmetry families. The reward is defined by whether the focal agent’s side achieves its game-specific objective at the end of the episode. Shaded bands and capped error bars indicate the standard error across rollouts within each training step.
Figure 4: RFT improves hidden-state inference across social deduction games. We train agents on controlled social-reading tasks and report hidden-state prediction accuracy over training. a, In Avalon, agents infer each player’s hidden team alignment from public game transcripts. b, In Werewolf/Mafia, agents infer the hidden wolf set from dialogue and voting evidence. Across completed Qwen, GLM, and DeepSeek runs, RFT improves hidden-state inference, indicating that reinforcement fine-tuning can strengthen social reading even when aggregate outcome-based game training is noisy.
Figure 5: RFT improves dynamic belief updating during social deduction. We analyse a representative Qwen2.5-7B run in Avalon and Werewolf/Mafia. a, In Avalon, we group belief predictions by mission and report alignment accuracy over training. Later missions provide richer public evidence from team proposals, votes, and quest outcomes, and the learned policy assigns more accurate hidden-team beliefs as training progresses. b, In Werewolf/Mafia, we evaluate the focal villager’s belief about the hidden wolf set after successive days of discussion. RFT improves wolf-set F1 across training, with later-day beliefs benefiting more strongly from accumulated dialogue and elimination evidence.
Figure 6: Role-conditioned RFT dynamics in Avalon. a , Mean episode-level outcome reward when the focal agent is trained under a fixed Avalon role rather than sampled across roles. Among the four roles, the Assassin shows the clearest and most sustained improvement, whereas Merlin, Servant, and Minion remain substantially noisier. b , Mean response length over the same training window. The Assassin rapidly reduces its response length while improving reward, indicating that the learned policy does not rely on longer deliberation. Instead, the agent learns to make minimal intermediate responses during missions and discussions, while preserving the game-critical decision for the final assassination.
Figure 7: Robustness of social-reading improvement across independent RFT runs. a, Werewolf social-reading performance across four independent training runs. Curves show the per-run mean reward over training, and shaded regions indicate s.e.m. across sampled rollouts at each training step. Numbers at the end of each curve denote the improvement from the first 20 to the last 20 training updates. b, Response length over training for the same runs, with shaded regions indicating s.e.m. across sampled rollouts. The performance gains were not accompanied by a systematic increase to the response-length limit.
Figure 8: RFT enables self-selected vote steering in One Night Werewolf. a, Target vote hit rate during training. The trained Minion self-selects a non-wolf vote target and attempts to steer other players’ final votes toward that target. Curves show binned means, and shaded regions denote step-level s.e.m. computed across training-step means within each bin. Numbers on the right indicate first-to-last improvement. b, Wolf-protection rate during training, measuring the fraction of other players whose final votes do not target the hidden Werewolf. c, Response length during training, measured in model-specific tokens. d, First-to-last improvement in target vote hit rate, wolf-protection rate and werewolf-team win rate.
Figure 9: RFT improves team-approval influence in Avalon. a, Target approval outcome during training across five model families. Curves show binned means and shaded regions denote s.e.m. across rollout episodes within each training bin. Numbers at the right indicate the first-to-last improvement in target approval outcome. b, Response length during training, measured in model-specific tokens. Shaded regions denote s.e.m. c, First-to-last improvement in target approval outcome and follow shift. Target approval outcome measures whether the final team vote matches the target direction; follow shift measures the change in the fraction of other players voting in the target direction after the trained agent’s response.
Figure 10: Repeated-run robustness of vote steering. a, Target vote hit rate for four independent Qwen2.5-7B self-selected vote-steering runs. Each curve shows the binned mean for one run, with shaded regions denoting step-level s.e.m. computed across training-step means within each bin. Numbers on the right indicate first-to-last improvement. b, Response length during training for the same four runs, measured in model-specific tokens. Shaded regions denote step-level s.e.m.
Model
No SR/SI
SR only
SI only
SR+SI
Δ SR+SI
Qwen2.5-7B
24.0%
32.0%
33.0%
37.0%
+13 pp
Qwen2.5-14B
15.0%
16.0%
17.0%
23.0%
+8 pp
Qwen3-4B
21.0%
30.0%
12.0%
34.0%
+13 pp
GLM-4-9B
19.0%
21.0%
21.5%
22.0%
+3 pp
DeepSeek-7B
14.0%
18.0%
16.0%
19.0%
+5 pp
Table 1: Prompt-level multi-agent social-cognitive ablations in Werewolf. Win rates are reported for good-side agents under four prompting conditions.
Figure 11: Full-game multi-agent social-cognitive reinforcement fine-tuning across games. Left column: training dynamics for six-player Werewolf, nine-player Werewolf and Avalon. Curves show scalar reward, win rate, social reading score and social influence score as a function of normalized training progress. Shaded regions denote variability across rollout groups. Middle and right columns: distributional shifts in social reading and social influence from early to late training, resolved by game day in Werewolf and by mission index in Avalon.
Wolf-side Opponent
Good WR ↑
SR-F1 ↑
SI-Follow ↑
Wolf-Exile ↑
Qwen3-4B
96%
0.638
0.725
90.8%
Qwen3.7-Max
94%
0.643
0.625
84.7%
Qwen-Plus
91%
0.572
0.594
78.0%
GLM-4-9B-Chat
90%
0.599
0.678
80.9%
Qwen-Max
90%
0.606
0.620
85.0%
Qwen2.5-14B-Instruct
87%
0.577
0.708
78.7%
Table 2: Cross-model evaluation of the learned good-side policy. Each opponent is evaluated over 100 valid full games using the same social-cognitive inference protocol. SR-F1 measures hidden-role identification, SI-Follow measures social-influence success, and Wolf-Exile measures the fraction of wolves successfully exiled.
Figure 12: Blinded evaluation of complete Werewolf trajectories. a, Mean ratings of trajectories generated by the base and social-cognitive RFT policies across five dimensions. Bars show means over the pooled blinded evaluation, comprising 89 ratings per condition; error bars denote 95% bootstrap confidence intervals over trajectory ratings. b, Direction of the RFT–base difference across 74 game-identifier-matched trajectory pairs in the scalable evaluation. Bars show the percentage of pairs in which the RFT trajectory received a higher, equal or lower score than its matched base trajectory. Numbers inside the bars indicate pair counts.
Evaluation dimension
Base
Social-cognitive RFT
Δ
Overall value
1.43±0.75
2.52±1.34
+1.09
Social-reading quality
1.35±0.66
2.25±1.33
+0.90
Social-influence and coordination
1.54±0.84
2.70±1.38
+1.16
Human-like gameplay
1.64±0.59
2.00±0.97
+0.36
Strategic correctness
1.21±0.57
2.38±1.45
+1.17
Table 3: Blinded evaluation of complete trajectories generated by the base and social-cognitive RFT policies. Scores are measured on a five-point scale. Values report the mean and standard deviation over 89 trajectory ratings per condition. Δ denotes the RFT-minus-base difference.
Social deduction games (SDGs) require agents to reason under partial observability by maintaining relational beliefs about hidden roles and team alignments. While recent LLM-agent approaches improve gameplay through prompting and preference optimization, they often optimize actions and in-game speech without explicitly grounding them in such beliefs. This frequently leads to strategically inconsistent behavior, especially for compact LLM agents. We introduce Multi-Agent Relational Belief Optimization (MARBO), a belief-grounded preference optimization framework that leverages relational beliefs to guide strategic decisions and in-game speech. MARBO provides preference feedback only when behaviors are supported by reliable relational beliefs and lead to strategically favorable social outcomes, encouraging more consistent learning under uncertainty. Experiments on representative SDGs show that MARBO enables compact LLM agents to consistently outperform existing baselines. The Code is available on https://github.com/PleaseTakemeAway/MARBO.
Yechan Hwang, Sangjun Bae, Jeongmo Kim +2
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea.
LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social interaction offers no objective ground truth: evaluations fall back on LLM judges, which are costly, subjective, and noisy, and models get no reliable signal to learn from. To address both, we first introduce Social Gym, an environment of 21 multi-agent social games (e.g., Werewolves, Resistance, Spyfall) whose rule-decided outcomes make agent performance verifiable and objective, with an Elo tournament that produces a cross-game leaderboard. Benchmarking experiments show that while GPT-5-mini tops the leaderboard, no model excels at all games uniformly or in all game roles, pointing to limitations of social reasoning. Motivated by this, we additionally propose SPaRTan (Self-Play and Reflect-Transfer), a training-free self-improvement loop: a model plays a game, reflects on its trajectories and their outcomes to produce a transferable playbook, and applies that playbook in subsequent games. Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance. Together, Social Gym and SPaRTan offer a reproducible, verifiable foundation for measuring and improving LLM social reasoning without weight updates.
Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue. As language models become autonomous collaborators, it is central to building effective and trustworthy human-AI interaction. Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards, producing short-sighted policies that struggle to manage goal-relationship tensions across multi-turn interactions. We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges. First, we apply multi-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning. Second, we design six process reward dimensions capturing the goal-relationship trade-off, including goal advancement, relational attunement, contextual coherence, etc. A reward model dynamically generates fine-grained scoring criteria for each dimension, while a stage-aware weight schedule prioritizes relationship-building in early turns, goal advancement mid-way, and balanced closure late. Across multiple social-dialogue benchmarks, SocialRL improves Goal Achievement by an average of 9.2 percentage points over the corresponding Base models. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.