Reinforcement Learning with Comparative Evidence for Social Intelligence

Keane Ong; Yuriel Ryan; Sabri Boughorbel; Vladimir Necula; Jack Wei Lun Shi; Rui Mao; Roy Ka-Wei Lee; Adriel Kuek

cs.AIOct 2, 2026

Reinforcement Learning with Comparative Evidence for Social Intelligence

Authors: Keane Ong, Yuriel Ryan, Sabri Boughorbel, Vladimir Necula, Jack Wei Lun Shi, Rui Mao, Roy Ka-Wei Lee, Adriel Kuek, +4 more

Organizations: MIT · NUS · Prince Sattam bin Abdulaziz University · University of Cambridge · NTU · UBC · DSO National Laboratories · A*STAR

Abstract

Developing socially intelligent AI remains heavily dependent on human-annotated data, limiting the scale and breadth of social understanding models can acquire. Methods that derive training signals from unlabeled data offer a path beyond this dependence, but social predictions lack the verification oracles available in mathematics and coding. Moreover, core social targets such as affect, intent, preference, and pragmatic meaning are often ambiguous. The same behavior can support multiple plausible interpretations, making it difficult to verify which is best supported. To address this challenge, we introduce Reinforcement Learning with Comparative Evidence (RLCE), a reinforcement learning method that learns social understanding from unlabeled training data without constructing rewards from ground-truth annotations. Given distinct answers in a rollout group, RLCE constructs evidence tests that identify observable evidence favoring an answer over another, validates these tests against the input sample, and aggregates test outcomes to determine the best-supported interpretation. Tests are regenerated as the policy produces new answers, enabling them to evolve with the policy. Across four benchmarks spanning affect, pragmatics, communicative intent, and preference, RLCE attains the strongest performance among seven methods that use no ground-truth training labels for rewards, including consensus, policy LLM-judge verification, multimodal co-evolution, and rubric-based rewards. Gains over the strongest baseline reach up to +18.93 points. Analyses further show that RLCE exhibits a larger share of reward variation between correct and incorrect predictions than compared rubric methods, can overturn erroneous policy-derived preferences, and benefits from pairwise test construction, compositional test aggregation, and on-policy test evolution.

Figures & tables

Figure 1: Method Overview. Reinforcement Learning with Comparative Evidence ( RLCE ) verifies candidate answers by comparing them pairwise with multiple tests that check the sample for support favoring an answer over another, and aggregating the test outcomes into a verdict. RLCE instantiates this through three steps: (i) Evidence-test construction: constructing pairwise evidence tests that check for observable evidence favoring an answer over another, (ii) Compositional aggregation: validating and aggregating these tests across answers to identify the best-supported answer for reward assignment, and (iii) On-policy evolution: regenerating evidence tests as the policy is updated and produces new answers during training.

Method	IntentQA ( $\uparrow$ )	$\Delta$	MOSEI ( $\uparrow$ )	$\Delta$	UR-FUNNY ( $\uparrow$ )	$\Delta$	PrefEval † ( $\uparrow$ )	$\Delta$	Avg. Rank ( $\downarrow$ )
Qwen3.5-9B (base model)	55.25	–	64.86	–	63.11	–	44.72	–	–
Universal Self-Consistency ( Chen et al., 2023 )	56.77	$+1.52$	66.84	$+1.98$	59.86	$-3.25$	41.37	$-3.35$	5.75
Policy LLM-Judge ( Yuan et al., 2024 )	57.33	$+2.08$	67.35	$+2.49$	60.44	$-2.67$	46.73	$+2.01$	4.00
Video-Zero ( Zhang et al., 2026a )	55.06	$-0.19$	64.80	$-0.06$	62.91	$-0.20$	43.30	$-1.42$	6.75
VisPlay ( He et al., 2025 )	55.34	$+0.09$	65.00	$+0.14$	64.96	$+1.85$	42.55	$-2.17$	5.75
Rubrics as Rewards ( Gunjal et al., 2026 )	55.60	$+0.35$	65.03	$+0.17$	64.72	$+1.61$	56.45	$+11.73$	4.25

Table 1: Held-out test performance after training on unlabeled training data. All methods start from the same Qwen3.5-9B base model, and we report the mean over three training runs with different random seeds. For IntentQA, MOSEI, and UR-FUNNY, which are obtained from Human Behavior Atlas ( Ong et al., 2026b ) , we follow its evaluation setup and report LLM-Judge accuracy, binary weighted F1 (F1w2), and weighted F1, respectively. For PrefEval, we follow its standard evaluation protocol and report LLM-Judge accuracy, macro-averaged over its three preference tasks.

\Delta

denotes change from the base model (green/red: improvement/regression), and

\uparrow/\downarrow

denote higher/lower is better. Overall, RLCE achieves the most consistent performance across the baseline methods.

Method	IntentQA ( $\uparrow$ )	$\Delta$	MOSEI ( $\uparrow$ )	$\Delta$	UR-FUNNY ( $\uparrow$ )	$\Delta$	PrefEval † ( $\uparrow$ )	$\Delta$
Qwen3.5-9B (base)	55.25	–	64.86	–	63.11	–	44.72	–
TTRL ( $G{=}4$ )	56.00	$+0.75$	63.47	$-1.39$	65.81	$+2.70$	36.52	$-8.20$
TTRL ( $G{=}16$ )	53.84	$-1.41$	68.81	$+3.95$	60.13	$-2.98$	18.26	$-26.46$
RLCE ( $G{=}4$ ) (ours)	58.34	$+3.09$	70.65	$+5.79$	70.33	$+7.22$	73.70	$+28.98$

Table 2: Test-time training performance. All methods are initialized from the same Qwen3.5-9B base model.

\Delta

is relative to the base model (green/red denote improvement/regression). Following TTRL ( Zuo et al., 2026 ) , we scale test-time adaptation with test-set size, and report performance at the end of the allocated adaptation budget: three passes for IntentQA, UR-FUNNY, and PrefEval, and two for the larger MOSEI test set, using the same dataset-specific budget across methods.

G

denotes rollout group size; we use

G{=}4

for RLCE and report TTRL with

G{=}4

and an expanded-rollout

G{=}16

setting. † PrefEval is the macro-average over its three task settings (Choice, Persona, Explicit). Overall, RLCE improves over the base model on all four benchmarks, while TTRL exhibits more variable performance changes with both

G{=}4

and

G{=}16

Figure 2: Analysis of reward variation and correction of erroneous selections. (Left) On MOSEI, among rollout groups with reward variation, the fraction of total reward variation that distinguishes correct from incorrect predictions, comparing RLCE, Rubrics as Rewards (RaR), and RLER. (Middle, Right) For RLCE, RaR, and RLER methods, among the cases where Universal Self-Consistency (USC) or Policy LLM-Judge selects an incorrect rollout over a correct one, fraction in which each method reverses this ordering by assigning the correct rollout higher reward, on MOSEI and UR-FUNNY.

Reward	MOSEI	† PrefEval
Static evidence tests	69.40	46.57
Evolving evidence tests	71.73	77.77

Table 3: Ablation comparing evolving and static evidence tests across a single training run. Evolving evidence tests yield stronger performance than static tests, with a larger performance difference on free-form PrefEval compared to closed-label MOSEI. † PrefEval reports the macro average across three task settings (Choice, Persona, Explicit).

Panel of Figure 3 — Figure 3: Analysis of compositional aggregation and evolving evidence tests. (Left) top panel: On training examples, RLCE ’s aggregated verdict on the best-supported answer is correct on 86-100% of cases when 0-40% of individual evidence tests are incorrect; bottom panel: verdict accuracy also increases as more evidence tests are composed on MOSEI and UR-FUNNY. For verdict accuracy, MOSEI uses positive and negative polarity, while UR-FUNNY uses binary humor labels, consistent with Ong et al. (2026b) . (Right) Regenerating evidence tests as the policy evolves (i.e. evolving tests) yields higher validation performance than keeping them fixed (i.e. static tests), with larger performance gains on the free-form PrefEval tasks than on closed-set MOSEI.

Reward	IntentQA	PrefEval †
Pointwise evidence tests	56.61	68.51
Pairwise evidence tests	59.61	77.77

Table 4: Ablation comparing pairwise and pointwise evidence-test construction across a single training run. Pairwise evidence test construction yields a stronger performance than pointwise construction on IntentQA and PrefEval. † PrefEval reports the macro average across its three task settings (Choice, Persona, Explicit).

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Inputs	Metric	Val $n$	Test $n$
MOSEI-Sentiment	Sentiment, 7 phrases	F, T	Binary-weighted F1	1,871	4,659
UR-FUNNY	Humour, 2 labels	F, T	Weighted F1	192	224
IntentQA	Intent, free text	F	Accuracy	2,044	2,134
PrefEval, Overall	Personalized preferences	–	Macro-Average	—	597
PrefEval, Explicit	Preferences, free text	T	Accuracy	100	199
PrefEval, Choice	Preferences, free text	T	Accuracy	100	199

Appendix

Table 5: Overview for the social intelligence tasks in this work. "Inputs" is what the model receives — "F" indicates frames and "T" indicates text.

n

is the number of rows in each split. To ensure consistency and robustness, we follow the previous work’s (HBA ( Ong et al., 2026b ) and PrefEval ( Zhao et al., 2025 ) ) evaluation procedure and metric. The overall PrefEval score is computed as the macro-average of the three subtasks: explicit, choice, and persona.

Panel of Figure 4 — Figure 4: Evaluation prompt template. The task content is identical across methods, while the final response-format instruction follows the output schema used during each method’s training. The user turn is the split’s own question text; the <video> line inside it is expanded into frame tokens.

Panel of Figure 5 — Figure 5: Real user turns from the three HBA test splits, exactly as the model receives them. Each dataset has a single question template; only the transcript or question changes between rows. The IntentQA instruction mentions audio, which the system prompt overrides.

Panel of Figure 6 — Figure 6: The IntentQA judge prompt. Message 1 is HBA’s grading instruction, copied verbatim from its grader; message 2 carries the question, candidate and reference. The verdict is requested through a JSON schema with the two fields shown.

Panel of Figure 7 — Figure 7: PrefEval acknowledge judge check, reproduced from the benchmark’s released check prompt. Each check is sent as a single user message.

Panel of Figure 8 — Figure 8: PrefEval hallucination judge check, reproduced from the benchmark’s released check prompt. The <preference> quote returned by acknowledge fills the restatement slot of hallucination.

Panel of Figure 9 — Figure 9: PrefEval violation judge check, reproduced from the benchmark’s released check prompt.

Panel of Figure 10 — Figure 10: PrefEval helpful judge check, reproduced from the benchmark’s released check prompt. Helpful is the only check whose “Yes” is the good outcome.

Method	IntentQA ( $\uparrow$ )	MOSEI (S) ( $\uparrow$ )	UR-FUNNY ( $\uparrow$ )	PrefEval † ( $\uparrow$ )
Qwen3.5-27B (zero-shot)	56.44 $\pm$ 0.03	65.14 $\pm$ 0.04	70.28 $\pm$ 0.06	76.63 $\pm$ 0.36
LLaVA-OneVision-2-8B (zero-shot)	52.60 $\pm$ 0.43	36.17 $\pm$ 0.77	62.61 $\pm$ 0.55	14.82 $\pm$ 0.59
Qwen3.5-9B (base model)	55.25	64.86	63.11	44.72
Universal Self-Consistency ( Chen et al., 2023 )	56.77 $\pm$ 0.89	66.84 $\pm$ 1.28	59.86 $\pm$ 1.96	41.37 $\pm$ 6.16
Policy LLM-Judge ( Yuan et al., 2024 )	57.33 $\pm$ 0.50	67.35 $\pm$ 1.77	60.44 $\pm$ 2.20	46.73 $\pm$ 5.21
Video-Zero ( Zhang et al., 2026a )	55.06 $\pm$ 0.13	64.80 $\pm$ 0.21	62.91 $\pm$ 0.92	43.30 $\pm$ 0.83

Appendix

Table 6: Performance of methods over three runs with different seeds. Qwen3.5-9B is the common initialization from which all methods are trained; Qwen3.5-27B and LLaVA-OneVision-2-8B are included as reference models run with zero-shot inference without training. We report the mean result of each task with the standard deviation

\pm

over three independent seed runs.

\uparrow

indicates higher is better. Across the four benchmarks, RLCE achieves the highest mean performance among these methods. The margin is modest on IntentQA (+1.15 points over Policy LLM-Judge), but larger on MOSEI (+3.58), UR-FUNNY (+4.50), and PrefEval (+18.93). Among the RLCE and seven baseline methods that construct rewards from unlabeled training data (excluding zero-shot reference models), bold denotes the best result, second best results are underlined.

Method	Explicit	Choice	Persona	Avg.
Qwen3.5-9B (base model)	55.28	39.20	39.70	44.72
[0.5pt/1.5pt] Universal Self-Consistency ( Chen et al., 2023 )	62.31 $\pm$ 9.95	33.92 $\pm$ 1.78	27.89 $\pm$ 6.75	41.37 $\pm$ 6.16
Policy LLM-Judge ( Yuan et al., 2024 )	57.03 $\pm$ 5.33	39.95 $\pm$ 4.62	43.22 $\pm$ 5.69	46.73 $\pm$ 5.21
Video-Zero ( Zhang et al., 2026a )	51.26 $\pm$ 2.84	40.70 $\pm$ 1.42	37.94 $\pm$ 1.07	43.30 $\pm$ 0.83
Visplay ( He et al., 2025 )	50.50 $\pm$ 3.20	41.71 $\pm$ 1.42	35.43 $\pm$ 3.20	42.55 $\pm$ 1.66
Rubrics as Rewards ( Gunjal et al., 2026 )	65.33 $\pm$ 0.71	51.26 $\pm$ 0.13	52.74 $\pm$ 3.55	56.45 $\pm$ 0.95

Appendix

Table 7: Per-form PrefEval results. Accuracy is judge accuracy on the test split. We report the mean with standard deviation

\pm

over three independent training runs. Avg. is the macro-average over the three forms and is the value reported in Table 6 and Table 1 . Bold denotes the best result, second best results are underlined.

Method	IntentQA ( $\uparrow$ )	MOSEI (S) ( $\uparrow$ )	UR-FUNNY ( $\uparrow$ )	PrefEval † ( $\uparrow$ )
Qwen3.5-9B (base)	55.25	64.86	63.11	44.72
[0.5pt/1.5pt] TTRL ( $G{=}4$ ) ( Zuo et al., 2026 )	56.00	63.47	65.81	36.52
TTRL ( $G{=}16$ ) ( Zuo et al., 2026 )	53.84	68.81	60.13	18.26
RLCE ( $G{=}4$ ) (ours)	58.34	70.65	70.33	73.70

Appendix

Table 8: Test-time training performance. All methods are initialized from the same Qwen3.5-9B base model. Following TTRL ( Zuo et al., 2026 ) , we scale test-time adaptation with test-set size: three passes for IntentQA, UR-FUNNY, and PrefEval, and two for the larger MOSEI test set, using the same dataset-specific budget across methods.

G

denotes rollout group size; we use

G{=}4

for RLCE and report TTRL with

G{=}4

and an expanded-budget

G{=}16

setting. † PrefEval is the macro-average over its three preference tasks. Bold denotes the best result, second best results are underlined.

Method	Explicit	Choice	Persona	Avg.
Qwen3.5-9B (base model)	55.28	39.20	39.70	44.72
[0.5pt/1.5pt] TTRL ( $G{=}4$ )	52.26	34.17	23.12	36.52
TTRL ( $G{=}16$ )	21.61	19.60	13.57	18.26
RLCE ( $G{=}4$ ) (ours)	49.75	81.41	89.95	73.70

Appendix

Table 9: Test-time training performance on the three PrefEval task settings. All methods start from the same Qwen3.5-9B base model. We report the final score after three passes for RLCE and TTRL with

G{=}4

, and the third-pass score for TTRL with

G{=}16

. Avg. is the macro-average across Explicit, Choice, and Persona. Bold denotes the best result, second best results are underlined.

Panel of Figure 11 — Figure 11: Example prompt used to construct evidence tests.

Panel of Figure 12 — Figure 12: Example of the prompt used to validate evidence tests against the observed sample.

Panel of Figure 13 — Figure 13: System prompt used by the RLCE trainable policy during training and evaluation.

Panel of Figure 14 — Figure 14: Example of the prompt used to canonicalize free-text rollout answers.

Panel of Figure 15 — Figure 15: Universal Self-Consistency prompt used for free-form tasks. Following Chen et al. (2023) , the selector receives the question and sampled responses and selects the response most consistent with the majority.

Panel of Figure 16 — Figure 16: Pointwise Policy LLM-Judge prompt used to evaluate each candidate answer independently. The judge receives the question, available input evidence, and a single proposed answer, and returns a binary correctness judgment.

Setting	Policy
Optimization algorithm	GRPO
Learning rate	$5\times 10^{-7}$
KL coefficient	0.001
Epoch cap	4
Unique clips per rollout batch	32
Rollouts per clip for GRPO	8

Appendix

Table 10: Optimization settings for Evolving and Static Rubrics. We follow the original work’s settings wherever applicable.

Figure 17: An example of conflicting criteria in RLER. UR-FUNNY clip train_00088_00122 (label: not humour ). At visit 2 the generator wrote a4 and a5 : word-for-word copying the two active criteria a1 and a2 with the sign reversed. a1 and a2 both reward arguing for “not humour” and agreeing with the label; even worse, the three seeded persistent rubrics ( p1 , p2 , p3 ) incorrectly assume the clip is humorous. Note that the pruning mechanism for RLER only checks for the standard deviation of past rewards for each criterion; thus, despite this conflict, these criteria may not get pruned. a6 , a7 and a8 (weight $+1$ , not shown) reword the three seeds; they scored 0 on every rollout and were culled at visit 2, but they still count in $P$ .

Figure 18: Criterion Collapse: conflicting or near-redundant criteria neutralize relative advantages, inhibiting the effectiveness of GRPO. Following the example in Figure 17 , each conflicting pair contributes

w_{i}\,s_{i}+w_{j}\,s_{j}=0

to the numerator; yet, it contributes (

+1

) to

P

. This affects normalization; and in turn, dilutes the relative advantages of individual rollouts. Without the negative (conflicting) copies, the relative advantages (second column) may become more discriminative; thus, carrying a stronger training signal for GRPO.

	PrefEval EXP		PrefEval CHO		PrefEval PER		UR-FUNNY		MOSEI SEN		IntentQA
Checkpoint	Val	Test	Val	Test	Val	Test	Val	Test	Val	Test	Val	Test
Iteration 1	38.00	–	43.00	40.70	36.00	37.69	59.15	–	62.70	–	52.94	–
Iteration 2	43.00	52.76	36.00	–	36.00	–	61.39	64.49	63.58	64.60	52.35	–
Iteration 3	43.00	52.76	39.00	43.22	35.00	36.68	61.03	63.84	62.49	64.75	53.13	54.64

Appendix

Table 11: VisPlay Reasoner checkpoints across iterations for one training run. Validation is the full validation split; test is the full test split. The tasks are reported in their corresponding metrics defined in Appendix A . Bold marks the iteration with the best validation score, which is the reported checkpoint; ties go to the earlier iteration. Only that checkpoint and the final one are test-evaluated. The results in Table 1 , 6 and 7 uses the mean values across three runs.

Setting	Questioner	Reasoner
Optimization algorithm	GRPO	GRPO
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$
KL coefficient	0.01	0.01
Iterations	3	3
Optimizer steps per iteration	20	20
Prompts per rollout batch	30	30

Appendix

Table 12: Optimization settings for VisPlay. We follow the original work’s released code, except the max response lengths, trimmed from 4096 to fit classification-length answers, and the video front end, which follows the evaluation configuration.

	PrefEval EXP		PrefEval CHO		PrefEval PER		UR-FUNNY		MOSEI SEN		IntentQA
Checkpoint	Val	Test	Val	Test	Val	Test	Val	Test	Val	Test	Val	Test
Iteration 1	40.00	–	43.00	–	37.00	–	56.39	62.26	63.01	–	52.69	–
Iteration 2	42.00	–	45.00	39.70	38.00	–	54.73	–	63.17	–	52.30	–
Iteration 3	43.00	53.27	43.00	–	37.00	–	54.65	–	63.29	64.65	52.40	–
Iteration 4	41.00	53.27	44.00	44.22	39.00	38.69	53.92	62.79	63.16	64.93	52.98	54.83
Iteration 5	43.00	49.75	–	–	39.00	36.68	–	–	63.19	64.97	53.13	54.97

Appendix

Table 13: Video-Zero Solver checkpoints across iterations for one full training run. Validation is the full validation split; test is the full test split. The tasks are reported in their corresponding metrics defined in Appendix A . Bold marks the iteration with the best validation score, which is the reported checkpoint; ties go to the earlier iteration. Only that checkpoint, the final one, and iteration 4 (the fixed iteration count of the original work) are test-evaluated. Rows after an early stop (patience 2) are left empty. The results in Table 1 , 6 and 7 uses the mean values across three runs

Setting	Questioner	Solver
Optimization algorithm	GRPO	GRPO
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$
Epochs per iteration	1	1
KL coefficient	0.01	0.01
Rollouts per prompt for GRPO	4	5
Rollouts for pseudo-labeling	–	10

Appendix

Table 14: We used the same optimization settings in Video-Zero.

Setting	Generator	Critic
Optimization algorithm	GRPO	GRPO
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$
KL coefficient	0.01	0.01
Adversarial rounds	5	5
Optimizer steps per round	18	18
Prompts per rollout batch	32 clips	32 pairs

Appendix

Table 15: Optimization settings for RLAC. The original work trains both players with DPO but states that the method is agnostic to the policy optimization algorithm, e.g., GRPO. We use GRPO with the same batch size, group size, learning rate and KL coefficient as the other baselines. The reward includes a format term, which DPO does not need because an unparseable response is left unpaired. Generator and Critic are trained in alternating phases within each round from the same Qwen 3.5 9B backbone, which also serves as the frozen Validator.

Explore similar work

Social-R1: Towards Human-like Social Reasoning in LLMs
Mar 10, 2026Jincenzi Wu, Yuxuan Lei, Jianxun Lian +5Social Reasoning RL for Language Model Reasoning
The Chinese University of Hong Kong · Microsoft Research Asia · Princeton University
SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
Sep 9, 2026Jianing Wang, Xintao Wang, Aili Chen +7RL for Language Models Reinforcement Learning
Fudan University · Hello Group
SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution
Apr 21, 2026Xiachong Feng, Yi Jiang, Xiaocheng Feng +9Credit Assignment in RL Step-Level Credit Assignment in RL
The University of Hong Kong · Harbin Institute of Technology · Harbin Institute of Technology, Shenzhen

Mar 10, 2026J/K move · Enter open · S save

Social-R1: Towards Human-like Social Reasoning in LLMs

Jincenzi Wu, Yuxuan Lei, Jianxun Lian, Yitian Huang+4

The Chinese University of Hong Kong · Microsoft Research Asia · Princeton University

Social Reasoning RL for Language Model Reasoning Theory of Mind Process Supervision

Open paper AskPDF arXiv

While large language models demonstrate remarkable capabilities across numerous domains, social intelligence - the capacity to perceive social cues, infer mental states, and generate appropriate responses - remains a critical challenge, particularly for enabling effective human-AI collaboration and developing AI that truly serves human needs. Current models often rely on superficial patterns rather than genuine social reasoning. We argue that cultivating human-like social intelligence requires training with challenging cases that resist shortcut solutions. To this end, we introduce ToMBench-Hard, an adversarial benchmark designed to provide hard training examples for social reasoning. Building on this, we propose Social-R1, a reinforcement learning framework that aligns model reasoning with human cognition through multi-dimensional rewards. Unlike outcome-based RL, Social-R1 supervises the entire reasoning process, enforcing structural alignment, logical integrity, and information density. Results show that our approach enables a 4B parameter model to surpass much larger counterparts and generalize robustly across eight diverse benchmarks. These findings demonstrate that challenging training cases with trajectory-level alignment offer a path toward efficient and reliable social intelligence.

Abstract

Explore similar work

Social-R1: Towards Human-like Social Reasoning in LLMs

SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design

SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution