EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning

Zilong Huang; Kong Aik Lee; Zezhong Jin; Man-Wai Mak; Tatsuya Kawahara

cs.MMOct 9, 2026

EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning

Authors: Zilong Huang, Kong Aik Lee, Zezhong Jin, Man-Wai Mak, Tatsuya Kawahara

Organizations: The Hong Kong Polytechnic University, Hong Kong SAR, China · Kyoto University, Japan

Abstract

Multimodal emotion preference learning uses pairwise comparisons to identify which open-ended emotion descriptions align more closely with human judgments. However, existing methods primarily focus on the final preference decision and provide little explicit supervision for localizing and correcting erroneous content. To address this limitation, we propose EmoGavel, a Grounding-Aware reinforcement learning framework with Verifiable rewards for multimodal Emotion preference Learning. We first introduce grounded supervised fine-tuning (Grounded SFT), which exploits controlled-error construction information to jointly learn preference prediction, error localization, repair, and missing-evidence recovery. We then optimize these verifiable outcomes with grounding-aware group relative policy optimization (GA-GRPO), combining preference-gated grounding rewards with dynamic zero-variance rescue to prioritize preference correctness and recover learning signals from all-wrong groups. Experiments on EmoPrefer and controlled-error evaluation sets show that EmoGavel improves both natural preference prediction and controlled-error discrimination while maintaining strong error-grounding performance. Our code and model checkpoints will be released once the paper is accepted.

Figures & tables

Figure 1: Overview of the proposed EmoGavel framework for multimodal emotion preference learning. Grounded SFT learns preference prediction and error grounding from natural and controlled-error pairs, and it initializes the trainable and reference policies. GA-GRPO samples multiple outputs, computes verifiable rewards and group-relative advantages, applying zero-variance rescue to all-wrong binary preference groups and updating the policy with KL regularization. The reward function uses preference-only rewards for natural pairs and preference-gated grounding rewards for controlled-error pairs.

Judge Configuration				4 Controlled-Error Subsets
ID	Model	Setting / Method	Natural	Emotion Flip	Intensity Mismatch	Evidence Contradiction	Modality Omission	4-Error Avg	Swap Cons
1	Qwen3-Omni-30B-Thinking [ 25 ]	S2 Zero-shot	73.36	92.27	54.51	71.20	65.95	70.98	62.29
2	Qwen3-Omni-30B-Instruct [ 25 ]	S2 Zero-shot	73.43	87.06	52.29	63.49	77.75	70.15	54.13
3	Qwen3-Omni-30B-Instruct	S2 Error-Aug SFT	76.56	92.74	61.64	71.48	76.79	75.66	65.47
4	MiniCPM [ 14 ] + Qwen2.5 + Qwen3	EAPO [ 10 ]	80.31	96.21	77.01	75.46	92.72	85.35	76.38
5	Qwen2.5-Omni-7B-Instruct [ 24 ]	S2 Zero-shot	68.65	83.80	49.30	51.82	47.59	58.01	34.96

Table 1: Preference performance (%) on EmoPrefer-Data and its EAPO extension. The column “Natural” shows the WAF on natural pairs, while “4-Error Avg” and “Swap Cons” report the mean WAF and candidate-order consistency over the four controlled-error types, respectively. Each Judge ID identifies a model with its inference setting (S2) and optimization strategy. SFT uses standard preference supervision, whereas Error-Aug SFT denotes preference-only SFT using both natural and EAPO controlled-error pairs. [ 10 ] . Best and second-best results are shown in bold and underline , respectively.

Qwen2.5-Omni-7B-Instruct		Emotion Flip		Intensity Mismatch		Evidence Contradiction		Modality Omission	Overall
Variant	Parse	Loc.	Rep.	Loc.	Rep.	Loc.	Rep.	Evid. Recovery	Natural	4-Error Avg	Grounding
Zero-shot	87.00	27.09	25.45	48.33	39.66	49.24	46.30	18.70	68.65	58.01	34.18
SFT	90.06	31.19	30.20	60.42	53.31	52.76	51.68	26.78	77.08	70.21	41.64
Error-Aug SFT	91.51	35.12	34.50	62.66	57.56	56.81	55.04	29.61	77.15	77.13	45.11
Grounded SFT	89.73	78.43	78.21	78.29	75.42	72.29	72.24	63.91	78.13	79.18	72.84
EmoGavel	98.42	92.39	92.37	80.14	78.42	76.30	75.97	62.51	79.75	88.20	77.27

Table 2: Error-grounding performance (%) across training strategies. “Loc.” and “Rep.” denote token-level F1 scores for localization and repair, respectively. “Evid. Recovery” evaluates missing-evidence recovery for Modality Omission. “Parse” denotes the rate of correctly formatted outputs. “Natural” and “4-Error Avg” report preference WAF on natural pairs and the mean over four controlled-error types, respectively. “Grounding” denotes the macro grounding score. Best and second-best results are shown in bold and underline , respectively.

Variant	Natural	4-Error Avg	Grounding
Grounded SFT	78.13	79.18	72.84
Vanilla GRPO	78.53	81.13	72.73
+ Grounding Reward	78.38	81.53	76.32
+ Grounding Reward + Dynamic Rescue	79.03	82.11	78.12
+ Grounding Reward + Preference Gate	78.88	85.62	76.99
GA-GRPO (Full)	79.75	88.20	77.27

Table 3: Ablation results for GA-GRPO. Best and second-best results are shown in bold and underline , respectively.

Explore similar work

A Regret Minimization Framework on Preference Learning in Large Language Models
Jun 8, 2026Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim +4Regret Minimization in RL LLM Alignment
Seoul National University, Seoul, South Korea · LG AI Research, Seoul, South Korea · UNIST, Ulsan, South Korea +1
General Preference Reinforcement Learning
May 18, 2026Muhammad Umer, Muhammad Ahmed Mohsin, Ahsan Bilal +5Multi-Objective Language Model Alignment RL Fine-Tuning
Stanford University · The University of Oklahoma
PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
May 1, 2026Ziqin Yuan, Ruiqi Wang, Dezhong Zhao +2Reward Modeling Mixture of Experts
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.

Jun 8, 2026J/K move · Enter open · S save

A Regret Minimization Framework on Preference Learning in Large Language Models

Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim, Yu Jin Kim+3

Seoul National University, Seoul, South Korea · LG AI Research, Seoul, South Korea · UNIST, Ulsan, South Korea+1

Regret Minimization in RL LLM Alignment Preference-Based RL RL from Human Feedback

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization

(\textbf{RePO})

, which reframes RLHF through

\textit{regret minimization}

rather than reward maximization. Human preferences are often shaped by

\textit{prospective}

anticipation of outcomes and

\textit{counterfactual}

comparisons to alternative behaviors, rather than by immediate, outcome-independent utility.

\textbf{RePO}

captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Experiments on mathematical reasoning benchmarks and human preference datasets demonstrate consistent performance gains, indicating that

\textbf{RePO}

is an effective and human-aligned approach for training large language models.

Abstract

Explore similar work

A Regret Minimization Framework on Preference Learning in Large Language Models

General Preference Reinforcement Learning

PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning