Developing socially intelligent AI remains heavily dependent on human-annotated data, limiting the scale and breadth of social understanding models can acquire. Methods that derive training signals from unlabeled data offer a path beyond this dependence, but social predictions lack the verification oracles available in mathematics and coding. Moreover, core social targets such as affect, intent, preference, and pragmatic meaning are often ambiguous. The same behavior can support multiple plausible interpretations, making it difficult to verify which is best supported. To address this challenge, we introduce Reinforcement Learning with Comparative Evidence (RLCE), a reinforcement learning method that learns social understanding from unlabeled training data without constructing rewards from ground-truth annotations. Given distinct answers in a rollout group, RLCE constructs evidence tests that identify observable evidence favoring an answer over another, validates these tests against the input sample, and aggregates test outcomes to determine the best-supported interpretation. Tests are regenerated as the policy produces new answers, enabling them to evolve with the policy. Across four benchmarks spanning affect, pragmatics, communicative intent, and preference, RLCE attains the strongest performance among seven methods that use no ground-truth training labels for rewards, including consensus, policy LLM-judge verification, multimodal co-evolution, and rubric-based rewards. Gains over the strongest baseline reach up to +18.93 points. Analyses further show that RLCE exhibits a larger share of reward variation between correct and incorrect predictions than compared rubric methods, can overturn erroneous policy-derived preferences, and benefits from pairwise test construction, compositional test aggregation, and on-policy test evolution.
Figures & tables
Figure 1: Method Overview. Reinforcement Learning with Comparative Evidence ( RLCE ) verifies candidate answers by comparing them pairwise with multiple tests that check the sample for support favoring an answer over another, and aggregating the test outcomes into a verdict. RLCE instantiates this through three steps: (i) Evidence-test construction: constructing pairwise evidence tests that check for observable evidence favoring an answer over another, (ii) Compositional aggregation: validating and aggregating these tests across answers to identify the best-supported answer for reward assignment, and (iii) On-policy evolution: regenerating evidence tests as the policy is updated and produces new answers during training.
Method
IntentQA ( ↑ )
Δ
MOSEI ( ↑ )
Δ
UR-FUNNY ( ↑ )
Δ
PrefEval † ( ↑ )
Δ
Avg. Rank ( ↓ )
Qwen3.5-9B (base model)
55.25
–
64.86
–
63.11
–
44.72
–
–
Universal Self-Consistency ( Chen et al., 2023 )
56.77
+1.52
66.84
+1.98
59.86
−3.25
41.37
−3.35
5.75
Policy LLM-Judge ( Yuan et al., 2024 )
57.33
+2.08
67.35
+2.49
60.44
−2.67
46.73
+2.01
4.00
Video-Zero ( Zhang et al., 2026a )
55.06
−0.19
64.80
−0.06
62.91
−0.20
43.30
−1.42
6.75
VisPlay ( He et al., 2025 )
55.34
+0.09
65.00
+0.14
64.96
+1.85
42.55
−2.17
5.75
Rubrics as Rewards ( Gunjal et al., 2026 )
55.60
+0.35
65.03
+0.17
64.72
+1.61
56.45
+11.73
4.25
Table 1: Held-out test performance after training on unlabeled training data. All methods start from the same Qwen3.5-9B base model, and we report the mean over three training runs with different random seeds. For IntentQA, MOSEI, and UR-FUNNY, which are obtained from Human Behavior Atlas ( Ong et al., 2026b ) , we follow its evaluation setup and report LLM-Judge accuracy, binary weighted F1 (F1w2), and weighted F1, respectively. For PrefEval, we follow its standard evaluation protocol and report LLM-Judge accuracy, macro-averaged over its three preference tasks. Δ denotes change from the base model (green/red: improvement/regression), and ↑/↓ denote higher/lower is better. Overall, RLCE achieves the most consistent performance across the baseline methods.
Method
IntentQA ( ↑ )
Δ
MOSEI ( ↑ )
Δ
UR-FUNNY ( ↑ )
Δ
PrefEval † ( ↑ )
Δ
Qwen3.5-9B (base)
55.25
–
64.86
–
63.11
–
44.72
–
TTRL ( G=4 )
56.00
+0.75
63.47
−1.39
65.81
+2.70
36.52
−8.20
TTRL ( G=16 )
53.84
−1.41
68.81
+3.95
60.13
−2.98
18.26
−26.46
RLCE ( G=4 ) (ours)
58.34
+3.09
70.65
+5.79
70.33
+7.22
73.70
+28.98
Table 2: Test-time training performance. All methods are initialized from the same Qwen3.5-9B base model. Δ is relative to the base model (green/red denote improvement/regression). Following TTRL ( Zuo et al., 2026 ) , we scale test-time adaptation with test-set size, and report performance at the end of the allocated adaptation budget: three passes for IntentQA, UR-FUNNY, and PrefEval, and two for the larger MOSEI test set, using the same dataset-specific budget across methods. G denotes rollout group size; we use G=4 for RLCE and report TTRL with G=4 and an expanded-rollout G=16 setting. † PrefEval is the macro-average over its three task settings (Choice, Persona, Explicit). Overall, RLCE improves over the base model on all four benchmarks, while TTRL exhibits more variable performance changes with both G=4 and G=16 .
Figure 2: Analysis of reward variation and correction of erroneous selections. (Left) On MOSEI, among rollout groups with reward variation, the fraction of total reward variation that distinguishes correct from incorrect predictions, comparing RLCE, Rubrics as Rewards (RaR), and RLER. (Middle, Right) For RLCE, RaR, and RLER methods, among the cases where Universal Self-Consistency (USC) or Policy LLM-Judge selects an incorrect rollout over a correct one, fraction in which each method reverses this ordering by assigning the correct rollout higher reward, on MOSEI and UR-FUNNY.
Reward
MOSEI
† PrefEval
Static evidence tests
69.40
46.57
Evolving evidence tests
71.73
77.77
Table 3: Ablation comparing evolving and static evidence tests across a single training run. Evolving evidence tests yield stronger performance than static tests, with a larger performance difference on free-form PrefEval compared to closed-label MOSEI. † PrefEval reports the macro average across three task settings (Choice, Persona, Explicit).
Figure 3: Analysis of compositional aggregation and evolving evidence tests. (Left) top panel: On training examples, RLCE ’s aggregated verdict on the best-supported answer is correct on 86-100% of cases when 0-40% of individual evidence tests are incorrect; bottom panel: verdict accuracy also increases as more evidence tests are composed on MOSEI and UR-FUNNY. For verdict accuracy, MOSEI uses positive and negative polarity, while UR-FUNNY uses binary humor labels, consistent with Ong et al. (2026b) . (Right) Regenerating evidence tests as the policy evolves (i.e. evolving tests) yields higher validation performance than keeping them fixed (i.e. static tests), with larger performance gains on the free-form PrefEval tasks than on closed-set MOSEI.
Reward
IntentQA
PrefEval †
Pointwise evidence tests
56.61
68.51
Pairwise evidence tests
59.61
77.77
Table 4: Ablation comparing pairwise and pointwise evidence-test construction across a single training run. Pairwise evidence test construction yields a stronger performance than pointwise construction on IntentQA and PrefEval. † PrefEval reports the macro average across its three task settings (Choice, Persona, Explicit).
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Task
Inputs
Metric
Val n
Test n
MOSEI-Sentiment
Sentiment, 7 phrases
F, T
Binary-weighted F1
1,871
4,659
UR-FUNNY
Humour, 2 labels
F, T
Weighted F1
192
224
IntentQA
Intent, free text
F
Accuracy
2,044
2,134
PrefEval, Overall
Personalized preferences
–
Macro-Average
—
597
PrefEval, Explicit
Preferences, free text
T
Accuracy
100
199
PrefEval, Choice
Preferences, free text
T
Accuracy
100
199
Appendix
Table 5: Overview for the social intelligence tasks in this work. "Inputs" is what the model receives — "F" indicates frames and "T" indicates text. n is the number of rows in each split. To ensure consistency and robustness, we follow the previous work’s (HBA ( Ong et al., 2026b ) and PrefEval ( Zhao et al., 2025 ) ) evaluation procedure and metric. The overall PrefEval score is computed as the macro-average of the three subtasks: explicit, choice, and persona.
Figure 4: Evaluation prompt template. The task content is identical across methods, while the final response-format instruction follows the output schema used during each method’s training. The user turn is the split’s own question text; the <video> line inside it is expanded into frame tokens.
Figure 5: Real user turns from the three HBA test splits, exactly as the model receives them. Each dataset has a single question template; only the transcript or question changes between rows. The IntentQA instruction mentions audio, which the system prompt overrides.
Figure 6: The IntentQA judge prompt. Message 1 is HBA’s grading instruction, copied verbatim from its grader; message 2 carries the question, candidate and reference. The verdict is requested through a JSON schema with the two fields shown.
Figure 7: PrefEval acknowledge judge check, reproduced from the benchmark’s released check prompt. Each check is sent as a single user message.
Figure 8: PrefEval hallucination judge check, reproduced from the benchmark’s released check prompt. The <preference> quote returned by acknowledge fills the restatement slot of hallucination.
Figure 9: PrefEval violation judge check, reproduced from the benchmark’s released check prompt.
Figure 10: PrefEval helpful judge check, reproduced from the benchmark’s released check prompt. Helpful is the only check whose “Yes” is the good outcome.
Method
IntentQA ( ↑ )
MOSEI (S) ( ↑ )
UR-FUNNY ( ↑ )
PrefEval † ( ↑ )
Qwen3.5-27B (zero-shot)
56.44 ± 0.03
65.14 ± 0.04
70.28 ± 0.06
76.63 ± 0.36
LLaVA-OneVision-2-8B (zero-shot)
52.60 ± 0.43
36.17 ± 0.77
62.61 ± 0.55
14.82 ± 0.59
Qwen3.5-9B (base model)
55.25
64.86
63.11
44.72
Universal Self-Consistency ( Chen et al., 2023 )
56.77 ± 0.89
66.84 ± 1.28
59.86 ± 1.96
41.37 ± 6.16
Policy LLM-Judge ( Yuan et al., 2024 )
57.33 ± 0.50
67.35 ± 1.77
60.44 ± 2.20
46.73 ± 5.21
Video-Zero ( Zhang et al., 2026a )
55.06 ± 0.13
64.80 ± 0.21
62.91 ± 0.92
43.30 ± 0.83
Appendix
Table 6: Performance of methods over three runs with different seeds. Qwen3.5-9B is the common initialization from which all methods are trained; Qwen3.5-27B and LLaVA-OneVision-2-8B are included as reference models run with zero-shot inference without training. We report the mean result of each task with the standard deviation ± over three independent seed runs. ↑ indicates higher is better. Across the four benchmarks, RLCE achieves the highest mean performance among these methods. The margin is modest on IntentQA (+1.15 points over Policy LLM-Judge), but larger on MOSEI (+3.58), UR-FUNNY (+4.50), and PrefEval (+18.93). Among the RLCE and seven baseline methods that construct rewards from unlabeled training data (excluding zero-shot reference models), bold denotes the best result, second best results are underlined.
Method
Explicit
Choice
Persona
Avg.
Qwen3.5-9B (base model)
55.28
39.20
39.70
44.72
[0.5pt/1.5pt] Universal Self-Consistency ( Chen et al., 2023 )
62.31 ± 9.95
33.92 ± 1.78
27.89 ± 6.75
41.37 ± 6.16
Policy LLM-Judge ( Yuan et al., 2024 )
57.03 ± 5.33
39.95 ± 4.62
43.22 ± 5.69
46.73 ± 5.21
Video-Zero ( Zhang et al., 2026a )
51.26 ± 2.84
40.70 ± 1.42
37.94 ± 1.07
43.30 ± 0.83
Visplay ( He et al., 2025 )
50.50 ± 3.20
41.71 ± 1.42
35.43 ± 3.20
42.55 ± 1.66
Rubrics as Rewards ( Gunjal et al., 2026 )
65.33 ± 0.71
51.26 ± 0.13
52.74 ± 3.55
56.45 ± 0.95
Appendix
Table 7: Per-form PrefEval results. Accuracy is judge accuracy on the test split. We report the mean with standard deviation ± over three independent training runs. Avg. is the macro-average over the three forms and is the value reported in Table 6 and Table 1 . Bold denotes the best result, second best results are underlined.
Method
IntentQA ( ↑ )
MOSEI (S) ( ↑ )
UR-FUNNY ( ↑ )
PrefEval † ( ↑ )
Qwen3.5-9B (base)
55.25
64.86
63.11
44.72
[0.5pt/1.5pt] TTRL ( G=4 ) ( Zuo et al., 2026 )
56.00
63.47
65.81
36.52
TTRL ( G=16 ) ( Zuo et al., 2026 )
53.84
68.81
60.13
18.26
RLCE ( G=4 ) (ours)
58.34
70.65
70.33
73.70
Appendix
Table 8: Test-time training performance. All methods are initialized from the same Qwen3.5-9B base model. Following TTRL ( Zuo et al., 2026 ) , we scale test-time adaptation with test-set size: three passes for IntentQA, UR-FUNNY, and PrefEval, and two for the larger MOSEI test set, using the same dataset-specific budget across methods. G denotes rollout group size; we use G=4 for RLCE and report TTRL with G=4 and an expanded-budget G=16 setting. † PrefEval is the macro-average over its three preference tasks. Bold denotes the best result, second best results are underlined.
Method
Explicit
Choice
Persona
Avg.
Qwen3.5-9B (base model)
55.28
39.20
39.70
44.72
[0.5pt/1.5pt] TTRL ( G=4 )
52.26
34.17
23.12
36.52
TTRL ( G=16 )
21.61
19.60
13.57
18.26
RLCE ( G=4 ) (ours)
49.75
81.41
89.95
73.70
Appendix
Table 9: Test-time training performance on the three PrefEval task settings. All methods start from the same Qwen3.5-9B base model. We report the final score after three passes for RLCE and TTRL with G=4 , and the third-pass score for TTRL with G=16 . Avg. is the macro-average across Explicit, Choice, and Persona. Bold denotes the best result, second best results are underlined.
Figure 11: Example prompt used to construct evidence tests.
Figure 12: Example of the prompt used to validate evidence tests against the observed sample.
Figure 13: System prompt used by the RLCE trainable policy during training and evaluation.
Figure 14: Example of the prompt used to canonicalize free-text rollout answers.
Figure 15: Universal Self-Consistency prompt used for free-form tasks. Following Chen et al. (2023) , the selector receives the question and sampled responses and selects the response most consistent with the majority.
Figure 16: Pointwise Policy LLM-Judge prompt used to evaluate each candidate answer independently. The judge receives the question, available input evidence, and a single proposed answer, and returns a binary correctness judgment.
Figure 26Figure 27Figure 28Figure 29
Setting
Policy
Optimization algorithm
GRPO
Learning rate
5×10−7
KL coefficient
0.001
Epoch cap
4
Unique clips per rollout batch
32
Rollouts per clip for GRPO
8
Appendix
Table 10: Optimization settings for Evolving and Static Rubrics. We follow the original work’s settings wherever applicable.
Figure 17: An example of conflicting criteria in RLER. UR-FUNNY clip train_00088_00122 (label: not humour ). At visit 2 the generator wrote a4 and a5 : word-for-word copying the two active criteria a1 and a2 with the sign reversed. a1 and a2 both reward arguing for “not humour” and agreeing with the label; even worse, the three seeded persistent rubrics ( p1 , p2 , p3 ) incorrectly assume the clip is humorous. Note that the pruning mechanism for RLER only checks for the standard deviation of past rewards for each criterion; thus, despite this conflict, these criteria may not get pruned. a6 , a7 and a8 (weight +1 , not shown) reword the three seeds; they scored 0 on every rollout and were culled at visit 2, but they still count in P .
Figure 18: Criterion Collapse: conflicting or near-redundant criteria neutralize relative advantages, inhibiting the effectiveness of GRPO. Following the example in Figure 17 , each conflicting pair contributes wisi+wjsj=0 to the numerator; yet, it contributes ( +1 ) to P . This affects normalization; and in turn, dilutes the relative advantages of individual rollouts. Without the negative (conflicting) copies, the relative advantages (second column) may become more discriminative; thus, carrying a stronger training signal for GRPO.
PrefEval EXP
PrefEval CHO
PrefEval PER
UR-FUNNY
MOSEI SEN
IntentQA
Checkpoint
Val
Test
Val
Test
Val
Test
Val
Test
Val
Test
Val
Test
Iteration 1
38.00
–
43.00
40.70
36.00
37.69
59.15
–
62.70
–
52.94
–
Iteration 2
43.00
52.76
36.00
–
36.00
–
61.39
64.49
63.58
64.60
52.35
–
Iteration 3
43.00
52.76
39.00
43.22
35.00
36.68
61.03
63.84
62.49
64.75
53.13
54.64
Appendix
Table 11: VisPlay Reasoner checkpoints across iterations for one training run. Validation is the full validation split; test is the full test split. The tasks are reported in their corresponding metrics defined in Appendix A . Bold marks the iteration with the best validation score, which is the reported checkpoint; ties go to the earlier iteration. Only that checkpoint and the final one are test-evaluated. The results in Table 1 , 6 and 7 uses the mean values across three runs.
Figure 34Figure 35
Setting
Questioner
Reasoner
Optimization algorithm
GRPO
GRPO
Learning rate
1×10−6
1×10−6
KL coefficient
0.01
0.01
Iterations
3
3
Optimizer steps per iteration
20
20
Prompts per rollout batch
30
30
Appendix
Table 12: Optimization settings for VisPlay. We follow the original work’s released code, except the max response lengths, trimmed from 4096 to fit classification-length answers, and the video front end, which follows the evaluation configuration.
PrefEval EXP
PrefEval CHO
PrefEval PER
UR-FUNNY
MOSEI SEN
IntentQA
Checkpoint
Val
Test
Val
Test
Val
Test
Val
Test
Val
Test
Val
Test
Iteration 1
40.00
–
43.00
–
37.00
–
56.39
62.26
63.01
–
52.69
–
Iteration 2
42.00
–
45.00
39.70
38.00
–
54.73
–
63.17
–
52.30
–
Iteration 3
43.00
53.27
43.00
–
37.00
–
54.65
–
63.29
64.65
52.40
–
Iteration 4
41.00
53.27
44.00
44.22
39.00
38.69
53.92
62.79
63.16
64.93
52.98
54.83
Iteration 5
43.00
49.75
–
–
39.00
36.68
–
–
63.19
64.97
53.13
54.97
Appendix
Table 13: Video-Zero Solver checkpoints across iterations for one full training run. Validation is the full validation split; test is the full test split. The tasks are reported in their corresponding metrics defined in Appendix A . Bold marks the iteration with the best validation score, which is the reported checkpoint; ties go to the earlier iteration. Only that checkpoint, the final one, and iteration 4 (the fixed iteration count of the original work) are test-evaluated. Rows after an early stop (patience 2) are left empty. The results in Table 1 , 6 and 7 uses the mean values across three runs
Figure 38Figure 39Figure 40
Setting
Questioner
Solver
Optimization algorithm
GRPO
GRPO
Learning rate
1×10−6
1×10−6
Epochs per iteration
1
1
KL coefficient
0.01
0.01
Rollouts per prompt for GRPO
4
5
Rollouts for pseudo-labeling
–
10
Appendix
Table 14: We used the same optimization settings in Video-Zero.
Figure 42Figure 43Figure 44
Setting
Generator
Critic
Optimization algorithm
GRPO
GRPO
Learning rate
1×10−6
1×10−6
KL coefficient
0.01
0.01
Adversarial rounds
5
5
Optimizer steps per round
18
18
Prompts per rollout batch
32 clips
32 pairs
Appendix
Table 15: Optimization settings for RLAC. The original work trains both players with DPO but states that the method is agnostic to the policy optimization algorithm, e.g., GRPO. We use GRPO with the same batch size, group size, learning rate and KL coefficient as the other baselines. The reward includes a format term, which DPO does not need because an unparseable response is left unpaired. Generator and Critic are trained in alternating phases within each round from the same Qwen 3.5 9B backbone, which also serves as the frozen Validator.
While large language models demonstrate remarkable capabilities across numerous domains, social intelligence - the capacity to perceive social cues, infer mental states, and generate appropriate responses - remains a critical challenge, particularly for enabling effective human-AI collaboration and developing AI that truly serves human needs. Current models often rely on superficial patterns rather than genuine social reasoning. We argue that cultivating human-like social intelligence requires training with challenging cases that resist shortcut solutions. To this end, we introduce ToMBench-Hard, an adversarial benchmark designed to provide hard training examples for social reasoning. Building on this, we propose Social-R1, a reinforcement learning framework that aligns model reasoning with human cognition through multi-dimensional rewards. Unlike outcome-based RL, Social-R1 supervises the entire reasoning process, enforcing structural alignment, logical integrity, and information density. Results show that our approach enables a 4B parameter model to surpass much larger counterparts and generalize robustly across eight diverse benchmarks. These findings demonstrate that challenging training cases with trajectory-level alignment offer a path toward efficient and reliable social intelligence.
Jincenzi Wu, Yuxuan Lei, Jianxun Lian +5
The Chinese University of Hong Kong · Microsoft Research Asia · Princeton University
Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue. As language models become autonomous collaborators, it is central to building effective and trustworthy human-AI interaction. Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards, producing short-sighted policies that struggle to manage goal-relationship tensions across multi-turn interactions. We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges. First, we apply multi-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning. Second, we design six process reward dimensions capturing the goal-relationship trade-off, including goal advancement, relational attunement, contextual coherence, etc. A reward model dynamically generates fine-grained scoring criteria for each dimension, while a stage-aware weight schedule prioritizes relationship-building in early turns, goal advancement mid-way, and balanced closure late. Across multiple social-dialogue benchmarks, SocialRL improves Goal Achievement by an average of 9.2 percentage points over the corresponding Base models. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.
Social intelligence, the ability to navigate complex interpersonal interactions, presents a fundamental challenge for language agents. Training such agents via reinforcement learning requires solving the credit assignment problem: determining how individual utterances contribute to multi-turn dialogue outcomes. Existing approaches directly employ language models to distribute episode-level rewards, yielding attributions that are retrospective and lack theoretical grounding. We propose SAVOIR (ShApley Value fOr SocIal RL), a novel principled framework grounded in cooperative game theory. Our approach combines two complementary principles: expected utility shifts evaluation from retrospective attribution to prospective valuation, capturing an utterance's strategic potential for enabling favorable future trajectories; Shapley values ensure fair credit distribution with axiomatic guarantees of efficiency, symmetry, and marginality. Experiments on the SOTOPIA benchmark demonstrate that SAVOIR achieves new state-of-the-art performance across all evaluation settings, with our 7B model matching or exceeding proprietary models including GPT-4o and Claude-3.5-Sonnet. Notably, even large reasoning models consistently underperform, suggesting social intelligence requires qualitatively different capabilities than analytical reasoning.
Xiachong Feng, Yi Jiang, Xiaocheng Feng +9
The University of Hong Kong · Harbin Institute of Technology · Harbin Institute of Technology, Shenzhen