AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes

Yibo Wang; Rui Yang; Jisheng Dang; Bimei Wang; Yitao Wu; Pengfei Cao; Wencan Zhang; Hong Peng

cs.AIAug 24, 2026

AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes

Authors: Yibo Wang, Rui Yang, Jisheng Dang, Bimei Wang, Yitao Wu, Pengfei Cao, Wencan Zhang, Hong Peng, +2 more

Organizations: Lanzhou University, Lanzhou, China · Hainan University, Haikou, China · School of Computing, National University of Singapore, Singapore

Abstract

Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding, yet affective reasoning remains vulnerable to shortcut behavior. Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language, which weakens traceability and external verification. Prior reinforcement learning approaches mainly reward context or logical coherence without explicitly enforcing attention to human evidence. In addition, LLM as a Judge scoring often suffers from score clustering, which reduces reward discriminability. We propose AffectOmni, a GRPO trained framework for verifiable affective reasoning. AffectOmni introduces People Focus and Temporal Order rewards to encourage people-centric evidence selection and temporally structured reasoning, and it adopts within-group comparative scoring to produce more stable and discriminative reward signals. For verification, a Thinking Summarizer converts free form rationales into executable evidence instructions, which are grounded into pixel level evidence regions via SAM3 to provide an externally auditable interface outside the training loop. Experiments on IntentBench, Daily Omni, and WorldSense show consistent improvements over open source 7B scale baselines, including gains of 4.66% on emotion recognition and +14.29% on temporally sensitive tasks. Code is available at https://github.com/eliot127825-rgb/AffectOmni_nobody.

Figures & tables

Fig. 1 : Shortcut reasoning vs. human-centric reasoning. Left: without People Focus reward, evidence attribution drifts to background cues, leading to an incorrect answer. Right: with People Focus reward, evidence anchors on human cues such as hand placement and posture change, enabling evidence-grounded affective inference. The displayed frame is a representative snapshot from the full video, previous events provide context, while human cues provide discriminative evidence.

Fig. 2 : AffectOmni framework. The pipeline consists of three stages. (1) LOCATE applies GRPO training with human-centric rewards ( $R_{\mathrm{ppl}}$ , $R_{\mathrm{tmp}}$ ) and comparative scoring to guide evidence-grounded reasoning. (2) REASON generates structured output and compresses the reasoning chain into a Minimal Evidence Package (MEP) via a Thinking Summarizer. (3) VERIFY grounds the MEP to pixel-level masks using SAM3, enabling post-hoc auditing of whether the evidence claims can be localized in the visual input. Stars indicate our key contributions. We further evaluate the verification module in Section V .

Fig. 3 : Visualization of free-form reasoning transformed into structured evidence, i.e., key points, focus objects, and SAM3 instructions for SAM3 based pixel level grounding.

Proprietary MLLMs
Methods	LMM	Why	How	What	When	Who/Which	Other	Emotion	Deception	Avg
GPT-4o †	–	61.46	55.69	60.00	35.71	76.00	63.31	60.99	59.00	59.98
GPT-o1 † (think)	–	68.19	65.82	66.04	57.14	76.00	68.83	67.26	59.50	66.69
Gemini-2.5-Pro † (think)	–	68.57	67.41	65.12	57.14	64.00	70.03	68.23	60.00	67.15
Open-Source MLLMs
MiniCPM-o	8B	57.87	53.48	57.14	57.14	68.00	61.14	23.85	49.50	54.51

TABLE I : Comparison on IntentBench. We report category-wise intent understanding and social reasoning performance. Models marked with † are proprietary. The last three rows are ablation variants of AffectOmni built incrementally on HumanOmniV2 by adding People-Focus (

R_{\text{ppl}}

), Temporal-Order (

R_{\text{tmp}}

), and both rewards jointly (Full Model).

Method	Context	Infer.	Reason.	60s	Avg
Gemini 2.0 Flash †	63.73	76.62	75.43	56.57	67.84
Gemini 2.0 Flash Lite †	58.03	74.03	72.00	53.01	61.32
Unified-IO-2 (8B)	26.42	35.06	29.71	30.00	28.24
VideoLLaMA2 (7B)	35.75	40.91	34.29	31.82	35.17
Qwen2.5-Omni (7B)	38.86	57.79	61.71	38.36	47.45
Ola (7B)	39.89	61.03	66.28	48.72	49.87

TABLE II : Comparison with existing models on Daily-Omni. Models marked with

\dagger

are proprietary. We report accuracy (%) on representative sub-dimensions; full results are provided in the supplemental material.

Method	Tech	Film	Perform.	Music	Avg
Claude 3.5 Sonnet †	43.70	36.50	30.70	33.90	34.80
GPT-4o †	48.00	43.50	41.90	42.70	42.60
Gemini 1.5 Pro †	53.70	50.40	52.40	42.00	48.00
Unified-IO-2 XXL (7B)	27.10	23.70	25.50	27.30	25.90
VideoLLaMA2 (7B)	29.40	24.50	26.20	27.10	25.40
VITA-1.5 (7B)	38.20	39.80	41.20	39.90	36.90

TABLE III : Comparison with existing models on WorldSense. Models marked with

\dagger

are proprietary. We report accuracy (%) on representative domains; full results are provided in the supplemental material.

Configuration	Emotion	Deception	When	Why	Avg
Baseline (Acc-only)	80.78	66.50	57.14	66.05	69.23
+ People-Focus ( $R_{\text{ppl}}$ )	84.23	63.50	57.14	66.48	69.78
+ Temporal-Order ( $R_{\text{tmp}}$ )	83.67	60.50	57.14	64.33	69.62
Full Model	85.44	62.50	71.43	66.91	71.89

TABLE IV : We report accuracy (%) on IntentBench across key categories. All variants share the same base model and training setup, and differ only in the enabled reward terms, showing the marginal gain brought by each component.

Metric	Independent	Comparative
CV	0.153	0.325
Within-Group CV	0.049	0.138
IntentBench Avg (%)	69.78	71.89

TABLE V : Comparison of scoring strategies. “Ind.” denotes independent scoring. “Comp.” denotes our comparative scoring. CV is the coefficient of variation, computed as standard deviation over mean.

Fig. 4 : Reward Signal Discrimination Analysis. Each arc represents a score interval (1–10 scale), with arc length proportional to the percentage of samples falling in that interval. Left (Independent): LLM as a Judge scores each candidate in isolation, causing severe high score clustering, over 73% of scores fall in the 9–10 range. Right (Comparative): within-group comparative scoring forces relative ranking among candidates, yielding a more balanced distribution with meaningful differentiation across the full score spectrum.

Fig. 5 : Reasoning to evidence grounding framework. The framework connects structured reasoning chains to pixel-level grounding via SAM3: AffectOmni summarizes focus entities from the reasoning, converts them into segmentation instructions, and visualizes the exact regions to enable automatic consistency checks and human inspection.

Category	$n$	$\Delta$ (pp)	95% CI (pp)	$p$
Emotion ( $\leftarrow R_{\mathrm{ppl}}$ )	133	$+4.66^{**}$	$[+1.2,\ +8.3]$	$0.008$
How ( $\leftarrow R_{\mathrm{ppl}}$ )	632	$+3.64^{**}$	$[+1.3,\ +5.9]$	$0.002$
Why	698	$+0.86$	$[-1.4,\ +3.2]$	$0.484$
What	480	$+1.46$	$[-1.2,\ +4.2]$	$0.317$
Other	507	$+0.39$	$[-1.8,\ +2.6]$	$0.779$
Deception	200	$-4.00$	$[-9.0,\ +1.0]$	$1.000$

TABLE VI : One-sided paired bootstrap significance test (10,000 resamples) on IntentBench (

n=2{,}689

), testing whether AffectOmni

>

HumanOmniV2 per category.

\Delta

denotes the mean accuracy difference (AffectOmni

-

HumanOmniV2). Shaded rows are categories directly targeted by

R_{\mathrm{ppl}}

{}^{**}p{<}0.01

;

{}^{*}p{<}0.05

Method	Size	Overall Acc. (%)
LLaVA-Video [ 67 ]	7B	83.20
PLLaVA [ 54 ]	7B	81.00
Magma [ 56 ]	8B	80.90
AffectOmni (Ours)	7B	80.52
HumanOmniV2 [ 58 ]	7B	79.78
LLaVA-OneVision [ 27 ]	7B	79.40

TABLE VII : Results on NExT-QA multiple-choice test set. AffectOmni surpasses HumanOmniV2 without domain-specific fine-tuning, confirming that affective RL training does not cause catastrophic forgetting of general video understanding.

Method	Causal	Temporal	Descriptive
AffectOmni	79.28	79.44	86.48
HumanOmniV2	80.50	76.50	83.77

TABLE VIII : Fine-grained NExT-QA results by question type. AffectOmni shows gains over HumanOmniV2 on Temporal and Descriptive subsets, consistent with the transferable reasoning capabilities trained by People-Focus and Temporal-Order rewards.

Evidence Type	Count	Rate
Visual Evidence	84	84%
Temporal Evidence	72	72%
Behavioral Evidence	89	89%
$\geq$ 2 Types	93	93%

TABLE IX: Occurrence rates of three types of structured evidence (Visual/Temporal/Behavioral) and coverage rate of

\geq

2 evidence types from 100 randomly sampled think outputs.

Type	Samples	Visual	Temporal	Behavioral
Other	28	75%	75%	89%
Why	27	81%	70%	81%
How	19	95%	74%	89%
What	14	86%	64%	93%
Deception	8	88%	62%	100%
Emotion	4	100%	100%	100%

TABLE X: Evidence occurrence rates grouped by question type.

Indicator	Count	Rate
Mentions Person	98	98%
Has Appearance (color/clothing)	39	39%
Has Visual Feature (expression/posture)	91	91%
Has Locatable Action	94	94%
Basic Grounding	98	98%
High-Quality Grounding	86	86%

TABLE XI: Occurrence rates of grounding-relevant elements and two levels of grounding criteria from 100 randomly sampled reasoning chains.

Type	Samples	Basic	High-Quality
Other	28	96%	82%
Why	27	100%	85%
How	19	100%	89%
What	14	93%	86%
Deception	8	100%	88%
Emotion	4	100%	100%

TABLE XII: Grounding attainment rates grouped by question type.

	Seg. Relevant	Seg. Irrelevant	Total
Answer Correct	55	12	67
Answer Wrong	11	22	33
Conditional Acc.	83.3%	35.3%	67%

TABLE XIII : Annotators judge, based only on SAM3 segmentation, whether it is related to the question target, and we report the contingency table between this relevance judgment and answer correctness, together with conditional accuracy (n=100).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
Sep 29, 2026Cheng Ye, Weidong Chen, Zhaobo Qi +2Multimodal Large Language Models Multimodal Emotion Recognition
University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing
Jun 14, 2026Zebang Cheng, Shuimu Chen, Boxue Yang +7On-Policy Self-Distillation Multimodal Large Language Models
Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University +6
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
Jun 24, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +6Multimodal Reasoning Multimodal Hallucination
University of Science and Technology of China · SenseTime · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1

Sep 29, 2026J/K move · Enter open · S save

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

Cheng Ye, Weidong Chen, Zhaobo Qi, Beier Zhu+1

University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai

Multimodal Large Language Models Multimodal Emotion Recognition Visual Token Pruning

Open paper AskPDF arXiv

While multimodal large language models (MLLMs) have demonstrated exceptional capabilities in objective understanding tasks, their performance in affective reasoning still falls significantly short of human standards. We attribute it to a central capability gap: MLLMs are difficult to reliably distinguish semantically proximal emotions based on fine-grained visual evidence, which could be decoupled as two limitations: 1) Insufficient Attribution. The global reasoning paradigm of conventional MLLMs severely dilutes fine-grained emotion cues, where subtle emotional states are usually implicitly encoded, thereby generating emotional misjudgments in complex scenarios. 2) Insufficient Discrimination. Existing methods could only identify regions generally associated with emotions, which fails to distinguish discriminative regions between semantically similar emotions, leading to ambiguous emotion judgements. To overcome these limitations, we present a training-free inference-time optimization framework, named Decoding Affective Nuances (DAN). Specifically, we propose a Hierarchical Emotional Reasoning Chain (HERC) that enhances the insufficient attribution by harmonizing fine-grained scene/object-level cues and performing a soft-gated reasoning. Furthermore, to discriminate between semantically proximal emotions, we design a Contrastive Discriminative Visual Pruning (CDVP), which isolates discriminative visual tokens to reason the final emotion category by computing the absolute discrepancy between the attention distributions of similar emotions. Performances on several benchmarks demonstrate that DAN significantly improves discrimination for affective nuances without consuming additional training resources, especially achieving +10.47% improvements with Qwen3-VL-8B-Instruct on WebEmo25 dataset that contains 25 fine-grained emotion categories.

Abstract

Explore similar work

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing

Omni-Perception Policy Optimization for Multimodal Emotion Reasoning