Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling

Qiyuan Chen; Hongsen Huang; Jiahe Chen; Qian Shao; Jintai Chen; Hongxia Xu; Renjie Hua; Chuan Ren

cs.CLApr 7, 2026

Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling

Authors: Qiyuan Chen, Hongsen Huang, Jiahe Chen, Qian Shao, Jintai Chen, Hongxia Xu, Renjie Hua, Chuan Ren,

Organizations: Zhejiang University · Soochow Securities Co.,Ltd. · HKUST(GZ) · Nanjing University

Abstract

Vision-language reward modeling faces a dilemma: generative approaches are interpretable but slow, while discriminative ones are efficient but act as opaque "black boxes." To bridge this gap, we propose VL-MDR (Vision-Language Multi-Dimensional Reward), a framework that dynamically decomposes evaluation into granular, interpretable dimensions. Instead of outputting a monolithic scalar, VL-MDR employs a visual-aware gating mechanism to identify relevant dimensions and adaptively weight them (e.g., Hallucination, Reasoning) for each specific input. To support this, we curate a dataset of 321k vision-language preference pairs annotated across 21 fine-grained dimensions. Extensive experiments show that VL-MDR consistently outperforms existing open-source reward models on benchmarks like VL-RewardBench. Furthermore, we show that VL-MDR-constructed preference pairs effectively enable DPO alignment to mitigate visual hallucinations and improve reliability, providing a scalable solution for VLM alignment.

Figures & tables

Figure 1: Comparison of paradigms. Unlike Generative RMs (high latency) and Discriminative RMs (opaque scalars), VL-MDR dynamically decomposes evaluation into granular dimensions, achieving both interpretability and efficiency.

Figure 2: Data Construction Pipeline and Capability Distribution. Left: We aggregate $\sim$ 414.2k preference samples from 7 different VLM preference datasets, grouped by supervision provenance (AI Feedback vs. Human Feedback), and apply our multi-model fine-grained overall-consistency filtering to retain $\sim$ 321.3k samples; the rightmost nodes show the retained set’s source distribution. Right: The distribution of capability tags in the final dataset. As each sample is annotated with its top-3 relevant dimensions, the total volume of tags reaches $\sim$ 964k.

Core Capability	Fine-grained Dimensions	Tag Ratio
Safety & Ethics (SE)	Harmful Content Detection Bias & Fairness Privacy & Personal Information	4.8% 3.1% 2.0%
Coarse Perception (CP)	Style & Quality Scene & Topic Emotion	1.9% 10.7% 3.6%
Fine-grained Perception (FP)	Attribute & Celebrity Recognition Object Location Object Counting	4.8% 16.7% 3.7%
Instance Reasoning (IR)	Single Instance Attribute Cross-instance Comparison Cross-instance Relation	15.1% 3.6% 6.4%
Logical Reasoning (LR)	Diagram Reasoning Code & Sequence Reasoning Common Reasoning	3.2% 0.1% 13.3%
Science & Technology (ST)	Natural Science Engineering Geography & Earth Science	3.2% 0.9% 0.8%

Table 1: Hierarchical Capability Taxonomy. We define seven Core Capabilities, each encompassing 3 specific Fine-grained Dimensions (21 total) to capture the micro-skills required for multimodal tasks. Tag Ratio reports each fine-grained dimension’s share of tag slots in the retained dataset, where each sample contributes its top-3 dimensions.

Figure 3: Overview of the VL-MDR framework. As shown in the diagram, the model processes the Instruction and candidate Responses (A and B) through a decoupled architecture. The backbone extracts distinct representations to feed three specialized heads: Dimension Predict (identifying relevant criteria based on the instruction), Dimension Weighting (assigning importance), and Scoring (evaluating quality). These components are aggregated to compute the final reward signal.

Models	#Param	General	Hallucination	Reasoning	Overall Acc	Macro Acc
Proprietary Models
GPT-4o-mini	-	41.70	34.50	58.20	41.50	44.80
GPT-4o	-	49.10	67.60	70.50	65.80	62.40
Gemini-1.5-Flash	-	47.80	59.60	58.40	57.60	55.30
Gemini-1.5-Pro	-	50.80	72.50	64.20	67.20	62.50
Claude 3.5 Sonnet	-	43.40	55.00	62.30	55.30	53.60

Table 2: VL-RewardBench. Performance comparison of our reward model (VL-MDR) with existing open-source and proprietary counterparts.

Method	Config			Results
Method	Gran.	$\mathcal{L}_{\text{rank}}$	Gate	Overall	Macro
Scalar	1	$-$	$-$	64.55	60.87
Implicit	21	$-$	✓	68.72	65.94
Coarse-7D	7	✓	✓	67.12	64.20
Fine w/o Gate	21	✓	$-$	69.77	68.51
VL-MDR	21	✓	✓	70.81	69.96

Table 3: Ablation study on supervision granularity and gating . Gran. denotes the number of supervision dimensions;

\mathcal{L}_{\text{rank}}

indicates whether per-dimension ranking supervision is used; Gate indicates instruction-aware dimension gating. Results are reported on VL-RewardBench (Overall/Macro accuracy).

Model	#Param	Multimodal RewardBench								MM-RLHF-Reward Bench
		Overall	General		Know.	Reasoning		Safety	VQA	Mcq	Long	Short	Safety	Video	Acc	Acc+
		Overall	Corr.	Pref.	Know.	Math	Code	Safety	VQA	Mcq	Long	Short	Safety	Video	Acc	Acc+
Proprietary Models
GPT-4o	-	70.8	62.6	69.0	72.0	67.6	62.1	74.8	87.2	64.3	78.4	44.1	56.3	40.0	58.2	26.1
Claude 3.5 Sonnet	-	71.5	62.6	67.8	73.9	68.6	65.1	76.8	85.6	64.3	67.6	55.9	65.6	60.0	62.9	26.1
Claude 3.7 Sonnet	-	71.9	58.4	60.7	78.1	76.3	71.3	72.0	86.8	66.7	91.9	91.2	87.5	76.0	82.4	65.2

Table 4: Performance on Multimodal RewardBench (Left) and MM-RLHF-Reward Bench (Right). Comparison of our reward model (VL-MDR) with existing open-source and proprietary counterparts.

Figure 4: Impact of Active Dimension Count (

k

). Performance peaks at

k=3

, demonstrating that selecting a focused set of relevant dimensions strikes an optimal balance: it filters out noise from irrelevant criteria while retaining sufficient evaluation signals.

Method	Add. Param.	Cost	Macro Acc.
Method	(M)	(GPU Hours) $\downarrow$	(%) $\uparrow$
Generative RMs
LLaVA-Critic (7B)	N/A	0.423 $\pm$ 0.09	44.0
UnifiedReward (7B)	N/A	0.399 $\pm$ 0.08	66.6
UnifiedReward-Think (7B)	N/A	0.518 $\pm$ 0.11	71.8
Discriminative RMs

Table 5: Computational cost and performance trade-off on VL-RewardBench. We report the total inference cost (in GPU Hours) required to evaluate the full benchmark alongside the Macro Acc. While Generative RMs suffer from high latency due to decoding, VL-MDR maintains the efficiency of Discriminative RMs.

Method	LLaVABench	WildVision	LLaVABenchWilder	MMHal
Generative RMs
OV-7B	90.3	54.9	67.8	3.2
w/ LLaVA-Critic	100.3	67.3	71.6	3.9
w/ UnifiedReward	101.4	67.8	75.0	4.0
w/ UnifiedReward-Think	101.8	68.3	77.5	4.2
Discriminative RMs

Table 6: Image Understanding DPO Comparison. We compare our method with generative and discriminative RMs for DPO based on LLaVA-OneVision-7B.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Consistency Level	Count	Percentage
All 3 dimensions match overall	266,444	64.3%
2 dimensions match overall	108,259	26.1%
1 dimension matches overall	37,252	9.0%
0 dimensions match overall	2,177	0.5%
Total	414,132	100%

Appendix

Table 7: Dimension-Overall Consistency

Figure 5: Dimension-overall consistency.

Figure 6: Top dimension co-occurrence pairs.

Dimension	Tie Rate	Count
Celebrity	44.7%	61,897
Calc	19.2%	11,049
Location	18.3%	172,433
Counting	17.4%	44,919
Attribute	4.7%	182,596
Geometry	0.7%	5,148

Appendix

Table 8: Dimension Difficulty (Tie Rate)

Figure 7: Dimension difficulty ranked by tie rate.

Component	Layer Type	Shape (Input $\to$ Output)	Params
Dimension Predictor ( $f_{\text{dim}}$ )	Linear	$3584\to 1024$	3,670,016
	Linear	$1024\to 512$	524,288
	Linear	$512\to 512$	262,144
	Linear	$512\to 21$	10,752
	Subtotal		4.47 M
Scoring Module ( $f_{\text{score}}$ )	Linear	$3584\to 2048$	7,340,032

Appendix

Table 9: Detailed layer-wise parameter breakdown for the VL-MDR Reward Head components.

Figure 8: The prompt template used for Visual-Aware Dimension Prediction . The model is instructed to analyze the image-text pair and select the top-3 relevant fine-grained axes from the defined taxonomy.

Figure 9: The prompt template used for Fine-Grained Response Comparison . The model evaluates two candidate responses on the specific target dimensions identified in the previous step before providing an overall preference.

Multimodal Reward Modeling

Multimodal reward modeling builds evaluators that score images, video, audio, and agent behavior against human preferences or task goals.

See all 92 papers

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
May 2, 2026Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5VLM Reasoning Vision-Language Models
School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4
Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
Apr 6, 2026Shizhan Gong, Minda Hu, Qiyuan Zhang +2VLM Reasoning Faithfulness in VLMs
The Chinese University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning
May 13, 2026Hee Suk Yoon, Eunseop Yoon, Ji Woo Hong +6VLM Reasoning Reinforcement Learning with Verifiable Rewards
Korea Advanced Institute of Science and Technology (KAIST) · Microsoft Research Asia (MSRA) · University of Illinois at Urbana-Champaign (UIUC)

May 2, 2026J/K move · Enter open · S save

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

Yin Zhang, Jiaxuan Zhao, Zonghan Wu, Zengxiang Li+4

School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China+4

VLM Reasoning Vision-Language Models Reinforcement Learning RL for VLMs Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning+1

Open paper AskPDF arXiv

Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.

Abstract

Multimodal Reward Modeling

Explore similar work

MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning