Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition

Yu Liu; Lei Zhang; Haoxun Li; Hanlei Shi; Yuxuan Ding; Leyuan Qu; Taihao Li

cs.AIMar 17, 2026

Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition

Authors: Yu Liu, Lei Zhang, Haoxun Li, Hanlei Shi, Yuxuan Ding, Leyuan Qu, Taihao Li

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China.

Abstract

Open-Vocabulary Multimodal Emotion Recognition (OV-MER) is inherently challenging due to the ambiguity of equivocal multimodal cues, which often stem from distinct unobserved situational dynamics. While Multimodal Large Language Models (MLLMs) offer extensive semantic coverage, their performance is often bottlenecked by premature commitment to dominant data priors, resulting in suboptimal heuristics that overlook crucial, complementary affective cues across modalities. We argue that effective affective reasoning requires more than surface-level association; it necessitates reconstructing nuanced emotional states by synthesizing multiple evidence-grounded rationales that reconcile these observations from diverse latent perspectives. We introduce HyDRA, a Hybrid-evidential Deductive Reasoning Architecture that formalizes inference as a Propose-Verify-Decide protocol. To internalize this abductive process, we employ reinforcement learning with hierarchical reward shaping, aligning the reasoning trajectories with final task performance to ensure they best reconcile the observed multimodal cues. Systematic evaluations validate our design choices, with HyDRA consistently outperforming strong baselines--especially in ambiguous or conflicting scenarios--while providing interpretable, diagnostic evidence traces.

Figures & tables

Figure 1: HyDRA overview. ( A ) Propose–Verify–Decide inference protocol: given multimodal input, HyDRA proposes multiple latent-context hypotheses, adjudicates them via evidence-constrained comparison with explicit citations, and outputs the most plausible emotion set. ( B ) Learning HyDRA with GRPO and hierarchical reward shaping: for each input we sample a group of structured trajectories in the <hyp>--<think>--<ans> format, score them with six rewards, compute group-relative advantages, and update the policy to favor evidential closure and robust decisions under ambiguity. Textual rationales are grounded to human-verified textualized multimodal cues (ObsG) provided by the datasets, and enforced via semantic alignment to ground-truth cue descriptions.

Model	#Params	Basic		Sentiment		OV-FG		Avg
Model		MER2023	MER2024	SIMS	MOSI	$S_{1}$	$S_{2}$
Otter	7B	16.41	14.65	60.57	54.27	14.64	4.46	27.50
Video-LLaVA	7B	36.93	30.25	54.64	57.62	27.40	12.18	36.50
R1-Omni	0.5B	59.61	70.92	58.42	30.17	32.80	18.85	45.13
VideoChat2	7B	33.67	54.50	69.59	67.23	34.07	17.78	46.14
Video-ChatGPT	7B	44.86	46.80	64.43	57.77	35.29	19.77	44.82

Table 1: Main results on general and open-vocabulary emotion recognition. We compare HyDRA against several multimodal baselines (up to 7B).

S_{1}

and

S_{2}

denote coarse- and fine-grained F1-scores on the OV-FG task, respectively. Despite its smaller scale, HyDRA achieves the best average performance and significantly outperforms 7B models on open-vocabulary reasoning. Best results are bolded ; second-best are underlined . ∗ : AffectGPT trained on OV-MERD ( Lian et al., 2024a ) .

Model	HCS		LCS
Model	$S_{1}$	$S_{2}$	$S_{1}$	$S_{2}$
Baseline	30.85	17.24	31.33	18.50
Baseline $\ddagger$	33.05	17.80	34.56	19.17
PandaGPT	41.15	22.15	45.43	25.59
AffectGPT ∗	43.63	27.89	48.82	29.76
HyDRA	54.78	30.56	55.51	30.13

Table 2: Conflict robustness results on MER-FG. We evaluate HyDRA against HumanOmni (Baseline) and other models across High Conflict (HCS) and Low Conflict (LCS) subsets. HyDRA maintains superior performance even under significant modal conflict.

\ddagger

: HumanOmni (7B); ∗ : AffectGPT trained on OV-MERD. Best results are bolded , second-best underlined .

Figure 2: Left : The original input (visual/audio/text). Visual priors contradict subtle audio/textual cues. Middle (Success) : HyDRA resolves the conflict via the Propose–Verify–Decide protocol. Right (Failure) : R1-omni commits prematurely to salient visual signals. Due to the presence of real individuals in the original videos, personal identifiable information has been removed and processed via visualization to address copyright and privacy concerns.

For.	Acc.	Cit.	Sem.	OV-FG
For.	Acc.	Cit.	Sem.	$S_{1}$	$S_{2}$	Avg
•	◦	◦	◦	47.93	25.67	36.80
◦	•	◦	◦	49.27	25.73	37.50
◦	◦	•	◦	47.30	25.85	36.57
◦	◦	◦	•	49.79	26.00	37.90
•	◦	•	•	50.46	26.35	38.40

Table 3: Ablation study of GRPO reward components on MER-FG. Symbols

\bullet

and

\circ

indicate enabled and disabled rewards, respectively. For.: Format constraints; Acc.: Accuracy reward; Cit.: Hierarchical citation; Sem.: Semantic Grounding. Bold : Best; Underline : Second-best.

Model	OV-FG
Model	$S_{1}$	$S_{2}$	Avg
Baseline	30.98	17.58	24.28
No-hypotheses	52.14	26.88	39.51
1-hypothesis	49.24	25.50	37.37
3-hypotheses	54.48	26.05	40.26
4-hypotheses	53.51	26.28	39.89

Table 4: Impact of hypothesis cardinality (

K

) on MER-FG. ”No-hypotheses” denotes the standard linear reasoning (R1-style) without the Propose–Verify–Decide protocol.

K=2

achieves the optimal balance between analytical diversity and evidential focus. All variants use the same 0.5B backbone. Bold : Best; Underline : Second-best.

Model	OV-FG
Model	$S_{1}$	$S_{2}$	Avg
Prompt-only	23.26	12.34	17.80
Baseline	30.98	17.58	24.28
$\text{SFT}_{\text{cold}}$	45.91	24.22	35.07
PPO Training	51.09	26.93	39.01
$\text{SFT}_{\text{full}}$	49.60	25.31	37.46

Table 5: Comparison of training paradigms on MER-FG. Cold-start and GRPO RL Subsets are used for initial and RL stages.

\text{SFT}_{\text{full}}

, PPO, and HyDRA share the same data budget. Results show that RL (especially our GRPO-based HyDRA) significantly outperforms SFT scaling. Bold : Best; Underline : Second-best.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Component	Weight $\lambda_{k}$	Primary Optimization Goal
Performance	Accuracy ( $r_{\text{acc}}$ )	4.0	Maximize F1-score with length penalty $P_{\ell}$ .
Protocol	Consistency ( $r_{\text{fmt}}$ )	0.5	Ensure JSON validly and tag ordering.
	Reasoning ( $r_{\text{think}}$ )	0.5	Enforce comparative/differential thinking blocks.
Reliability	Citation ( $r_{\text{cite}}$ )	1.0	Encourage explicit hypothesis referencing.
	Evidence ( $r_{\text{evid}}$ )	2.0	Enforce intra-trace consistency via fuzzy matching.
	Grounding ( $r_{\text{sem}}$ )	2.0	Align reasoning with multimodal cue annotations.

Appendix

Table 6: Hyperparameters for Hierarchical Reward Shaping.

Figure 3: Overview of the Generation stages of ObsG.

Dataset Type	Raw Dataset	Selected Subset	# Samples
Sentiment Analysis	CMU-MOSI ( Zadeh et al., 2017 )	Test	686
Sentiment Analysis	CH-SIMS ( Yu et al., 2020 )	Test	457
Basic Emotion Recognition	MER2023 ( Lian et al., 2023a )	MER-MULTI (Test)	411
Basic Emotion Recognition	MER2024 ( Lian et al., 2024b )	MER-SEMI (Test)	1,169
Fine-grained Emotion Detection	MER-FG ( Lian et al., 2025b )	Test	1,200

Appendix

Table 7: Dataset card of the benchmarks used in our experiments. We group datasets by task type and report the official split adopted for evaluation.

Panel of Figure 4 — Figure 4: Emotion wheel. We use five emotion wheels, all of which are derived from previous research ( Lian et al., 2025a ) .

Figure 5: Evaluation scores across various epochs on the MER-FG.

Figure 6: Analysis of Emotional Label Cardinality Distribution

Explore similar work

Rationale-Guided Learning for Multimodal Emotion Recognition
Aug 11, 2026Sujung Oh, Jung Uk Kim, Sangmin LeeMultimodal Reasoning Emotion Recognition in Conversations
Pixel Lab, Sungkyunkwan University, South Korea · Visual AI Lab, Kyung Hee University, South Korea · Pixel Lab, Korea University, South Korea
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
Jun 24, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +6Multimodal Reasoning Multimodal Hallucination
University of Science and Technology of China · SenseTime · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition
Apr 14, 2026Zeheng Wang, Zitong Yu, Yijie Zhu +9Multimodal Emotion Recognition Multimodal RAG
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, Guangdong, China · Great Bay University, Dongguan, Guangdong, China · Guangdong Provincial Key Laboratory of Intelligent Information Processing & Shenzhen Key Laboratory of Media Security, Shenzhen University, Shenzhen, Guangdong, China

Aug 11, 2026J/K move · Enter open · S save

Rationale-Guided Learning for Multimodal Emotion Recognition

Sujung Oh, Jung Uk Kim, Sangmin Lee

Pixel Lab, Sungkyunkwan University, South Korea · Visual AI Lab, Kyung Hee University, South Korea · Pixel Lab, Korea University, South Korea

Multimodal Reasoning Emotion Recognition in Conversations Multimodal Emotion Recognition

Open paper AskPDF arXiv

Multimodal emotion recognition in conversation (MERC) requires understanding complex interactions between verbal and non-verbal cues. However, most existing approaches fundamentally treat this as a direct input-output (multimodal cues-emotion labels) mapping problem, overlooking the causal reasoning that humans use when interpreting emotions. We propose rationale-guided learning (RGL), a novel framework that transforms MERC into a cognitively-inspired reasoning task. Based on dual-process theory, we decompose emotional reasoning into three facets: Intuitive (immediate perception, System 1), Contextual (situational analysis, System 2), and Integrative (synthesis of both). We leverage an MLLM offline to generate structured rationales, which are encoded as memories to guide model training via aligning internal representations with human-like reasoning patterns. Our final model operates without any MLLM overheads at inference time. Experimental results show that RGL achieves state-of-the-art performance on the IEMOCAP and MELD benchmarks. Further, for interpretation, we demonstrate that the model's internal features effectively retrieve semantically correct rationales for unseen test samples, validating its rationale reasoning capabilities.

Abstract

Explore similar work

Rationale-Guided Learning for Multimodal Emotion Recognition

Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition