You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Yinuo Yang; Zixian Ma; Manasi Ganti; Jieyu Zhang; Ranjay Krishna

cs.CVApr 13, 2026

You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Authors: Yinuo Yang, Zixian Ma, Manasi Ganti, Jieyu Zhang, Ranjay Krishna

Organizations: Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195, USA

Abstract

We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward models evaluate each response independently, requiring multiple forward passes, one for each potential response. Our approach concatenates multiple responses with separator tokens and applies cross-entropy over their scalar scores, enabling direct comparative reasoning and efficient $N$ -way preference learning. The multi-response design also yields up to $N\times$ wall-clock speedup and FLOPs reduction over conventional single-response scoring. To enable $N$ -way reward evaluation beyond existing pairwise benchmarks, we construct two new benchmarks: (1) MR $^2$ Bench-Image contains human-annotated rankings over responses from 8 diverse models; (2) MR $^2$ Bench-Video is a large-scale video-based reward benchmark derived from 94K crowdsourced pairwise human judgments over video question-answering spanning 19 models, denoised via preference graph ensemble. Both benchmarks provide 4-response evaluation variants sampled from the full rankings. Built on a 4B vision-language backbone with LoRA fine-tuning and a lightweight MLP value head, our model achieves state-of-the-art results on six multimodal reward benchmarks, including MR $^2$ Bench-Image, MR $^2$ Bench-Video, and four other existing benchmarks. Our model outperforms existing larger generative and discriminative reward models. We further demonstrate that our reward model, when used in reinforcement learning with GRPO, produces improved policy models that maintain performance across standard multimodal benchmarks while substantially improving open-ended generation quality, outperforming a single-response discriminative reward model (RM) baseline by a large margin in both training stability and open-ended generation quality.

Figures & tables

Figure 1: Comparison of reward model architectures. Left: Single-Response discriminative RM scores each

(x,y_{i})

pair independently via separate forward passes. Center: Generative RM prompts a VLM to output a preference distribution

p(I\mid x,y_{1},y_{2})

autoregressively. Right: Our Multi-Response discriminative RM concatenates all

N

candidates into a single sequence

(x,y_{1},y_{2},\ldots,y_{N})

and uses a multi-response scoring head to produce scores for all candidates in one forward pass.

Proprietary Models †
		Image				Video
Model	Size	VL-RB	MM-RB	MMRLHF	MR 2 B-I	VRB	MR 2 B-V	Avg
GPT-5 ( OpenAI, 2025 )	–	75.0	64.6	71.8	87.1	68.2	50.1	69.5
Claude-Sonnet-4.5 ( Anthropic, 2025 )	–	68.6	78.2	70.0	72.9	67.5	49.1	67.7
Gemini-2.5-Pro ( Comanici et al., 2025 )	–	70.5	82.4	70.6	71.2	63.2	49.7	67.9
Open-Source General VLMs †

Table 1: Main results on multimodal reward benchmarks. Our Molmo2-4B RM (4B) achieves the highest average across all open-source models, outperforming larger generative and discriminative baselines. VL-RB : VL-RewardBench (macro pairwise acc.); MM-RB : Multimodal RewardBench (pairwise acc.); MMRLHF : MM-RLHF RewardBench (pairwise acc.); MR 2 B-I : MR 2 Bench-Image (best-of-4 acc.); VRB : VideoRewardBench (macro pairwise acc.); MR 2 B-V : MR 2 Bench-Video (best-of-4 acc.). † Generative judge. ∗ Paper-reported score.

Base Model	Scoring	VL-RB	MM-RB	MMRLHF	MR 2 B-I	VRB	MR 2 B-V	Avg
Molmo2-4B	Multiple (CE)	62.1	73.8	88.8	52.5	64.3	47.1	64.8
Molmo2-4B	Single (BT)	57.7	61.9	64.7	41.2	60.8	37.6	54.0
Qwen3-VL-4B	Multiple (CE)	63.6	71.2	84.7	58.8	64.9	47.5	65.1
Qwen3-VL-4B	Single (BT)	67.4	73.5	88.8	49.6	58.1	40.6	63.0

Table 2: Multi-response vs. single-response scoring. Multi-response CE outperforms single-response BT on average, with a large gap on Molmo2-4B (64.8% vs. 54.0%).

Figure 2: Inference efficiency of multi-response vs. single-response scoring on Molmo2-4B (single NVIDIA H100 80 GB GPU). Per-sample latency and FLOPs grouped by

N

and modality, achieving up to

3.9\times

latency and

4.0\times

FLOPs reduction when

N=4

Figure 4: Validation reward during GRPO training. The multi-response RM provides a steadily increasing reward signal, while the single-response RM’s reward is unstable. The y-axis scales differ because the two reward models produce differently scaled outputs.

Model	WildVision	LLaVA-W	MMHal
Molmo2-4B (base)	54.6	92.4	3.98
+ GRPO (Multi-RM)	60.2 ( $+$ 5.6)	97.0 ( $+$ 4.6)	4.25 ( $+$ 0.27)
+ GRPO (Single-RM)	55.8 ( $+$ 1.2)	91.6 ( $-$ 0.8)	4.17 ( $+$ 0.19)

Table 3: GRPO on open-ended generation. Multi-RM substantially improves three open-ended benchmarks, while Single-RM shows little gains and even hurts LLaVA-Bench.

	VQAv2	TextVQA	ChartQA	DocVQA	InfoVQA	AI2D	MMMU	RWQA	MathVista	CountBench	PixMoCount	MuirBench	MMIU
Molmo2-4B (base)	86.7	84.9	86.0	92.4	78.6	95.6	50.9	75.4	56.7	94.1	88.1	60.5	55.5
Molmo2-4B + GRPO (Multi-RM)	86.7	84.7	86.2	92.5	78.6	95.7	50.6	75.9	56.5	94.3	88.3	60.7	55.9
Molmo2-4B + GRPO (Single-RM)	86.7	84.9	83.1	91.6	78.6	95.3	50.7	75.7	56.5	94.1	88.3	54.2	55.5

Table 4: GRPO on standard benchmarks. Multi-RM preserves performance across all 24 standard image and video benchmarks, while Single-RM degrades on several.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Proprietary Models
		MR 2 B-Image			MR 2 B-Video
Model	Size	Pairwise	Direct	$\Delta$	Pairwise	Direct	$\Delta$
GPT-5	–	87.1	87.5	+0.4	50.1	50.5	+0.4
Claude-Sonnet-4.5	–	72.9	72.5	-0.4	49.1	49.3	+0.2
Gemini-3-Flash	–	72.1	70.0	-2.1	52.3	52.5	+0.2
Open-Source General VLMs

Appendix

Table 6: Comparison of pairwise aggregation vs. direct best-of-4 selection on MR 2 Bench-Image and MR 2 Bench-Video. Pairwise : the model evaluates all

\binom{4}{2}=6

response pairs and selects the response with the highest win count (as used in Table 1 ). Direct : the model receives all 4 responses simultaneously and directly selects the best one.

\Delta

= Direct

-

Pairwise.

Proprietary Models †
Model	Size	VQA	Reason.	Safety	MR 2 B-I
GPT-5 ( OpenAI, 2025 )	–	80.0	87.5	93.8	87.1
Claude-Sonnet-4.5 ( Anthropic, 2025 )	–	62.5	76.2	80.0	72.9
Gemini-2.5-Pro ( Comanici et al., 2025 )	–	61.3	77.5	75.0	71.2
Open-Source General VLMs †
InternVL3-8B ( Zhu et al., 2025 )	8B	38.8	57.5	70.0	55.4

Appendix

Table 7: Per-category results on MR 2 Bench-Image (best-of-4 accuracy, 240 samples: 80 VQA, 80 reasoning, 80 safety, chance = 25%). † Generative judge (LLM-as-a-judge). Best result per category in bold , second best underlined .

Proprietary Models †
		VideoRewardBench
Model	Size	Perc-S	Perc-L	Know.	Reason	Safety	VRB Macro	MR 2 B-V
GPT-5 ( OpenAI, 2025 )	–	57.4	68.9	67.6	62.6	84.6	68.2	50.1
Claude-Sonnet-4.5 ( Anthropic, 2025 )	–	55.4	73.9	73.1	61.5	73.8	67.5	49.1
Gemini-2.5-Pro ( Comanici et al., 2025 )	–	62.7	47.7	72.7	68.0	64.7	63.2	49.7
Open-Source General VLMs †

Appendix

Table 8: Results on video reward benchmarks . VideoRewardBench (VRB) : VideoRewardBench ( Zhang et al., 2025e ) (pairwise accuracy across five categories, 1,563 pairs, chance = 50%); MR 2 B-V : MR 2 Bench-Video (best-of-4 accuracy, 495 samples, chance = 25%). † Generative judge (LLM-as-a-judge). Best result per category in bold , second best underlined .

Category	Benchmark	Split	Metric	Notes
Image Native	VQAv2	test-standard	VQA score	Server submission (EvalAI)
	TextVQA	val	VQA score
	ChartQA	test	Relaxed correctness
	DocVQA	test	ANLS	Server submission (RRC)
	InfoVQA	test	ANLS	Server submission (RRC)
	AI2D	test	Accuracy (transparent)

Appendix

Table 9: Per-benchmark evaluation configuration for GRPO policy evaluation (Section 5.2 ), following the Molmo2 technical report ( Clark et al., 2026 ) . Deviations: (1) validation splits used where test sets are unavailable; (2) EgoSchema val/500 instead of test/5000; (3) video evaluation uses decord (vs. torchcodec ), max_frames =376 (vs. 384), and 10K subtitle token cap.

Figure 5: Per-sample inference latency ( left , ms) and average FLOPs ( right ) for Qwen3-VL-4B on a single NVIDIA H100 80 GB GPU. Same grouping as Figure 3 .

Figure 6: Per-sample FLOPs comparison between our Molmo2-4B RM and open-source baselines, averaged over image benchmarks ( left : VL-RB, MM-RB, MMRLHF, MR 2 B-I) and video benchmarks ( right : VRB, MR 2 B-V). MM-RLHF-Reward on video benchmarks ( ∗ , hatched) are theoretically estimated due to incomplete automated measurement of LLaVA’s internal vision token expansion (see text).

Open-Source General VLMs
Model	Size	BoN	Pair	$\tau$
InternVL3-8B	8B	40.4	71.6	0.466
Qwen2.5-VL-7B	7B	44.4	73.6	0.504
Qwen3-VL-4B	4B	47.9	75.8	0.538
Qwen3-VL-8B	8B	47.7	77.2	0.563
Qwen3-VL-32B	32B	49.9	77.8	0.592

Appendix

Table 10: Full metrics on our MR 2 Bench-Video (4-response, 495 samples). BoN = best-of-N accuracy (%, primary metric from Table 1 ); Pair = pairwise accuracy (%);

\tau

= Kendall’s

\tau

rank correlation. Models: Skywork-VL-Reward ( Wang et al., 2025 ) , IXC-2.5-Reward ( Zang et al., 2025 ) , R1-Reward ( Zhang et al., 2025a ) , MM-RLHF-Reward ( Zhang et al., 2025c ) , LLaVA-Critic ( Xiong et al., 2025 ) , Molmo2 ( Clark et al., 2026 ) , Qwen3-VL ( Bai et al., 2025a ) . Best per section in bold , second best underlined .

Dataset	Collected	Used	$N$
Multimodal
MMHAL/MM-RLHF	16,321	16,321	$N{=}3\sim 5$
lmms-lab/LLaVA-Critic-113k	71,331	56,635	$N{=}2\sim 13$
openbmb/RLAIF-V-Dataset	83,132	64,097	$N{=}2$
MMInstruction/VLFeedback	80,258	63,448	$N{=}4$
YiyangAiLab/POVID_preference_data_for_VLLMs	17,184	17,184	$N{=}2$

Appendix

Table 11: Training data composition. We curate 436K samples from 10 datasets, with 35% containing

N{>}2

responses for listwise training.

N

: number of responses per sample.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
May 10, 2026Rui Liu, Dian Yu, Zhenwen Liang +6Multimodal Understanding Multimodal Reward Modeling
1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill
Unsupervised Multimodal Reward Modeling
Oct 8, 2026Yueying Li, Jingxuan Fan, Kianté Brantley +2Multimodal Reward Modeling VLM Evaluation
Massachusetts Institute of Technology · Cornell University · Scale AI +1
Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation
Oct 16, 2026Ziyun Zou, Jiwoo Hong, Yifan Lu +5Preference Alignment Preference-Based RL
Mohamed bin Zayed University of Artificial Intelligence · Nubank · Dartmouth College

May 10, 2026J/K move · Enter open · S save

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Rui Liu, Dian Yu, Zhenwen Liang, Yucheng Shi+5

1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill

Multimodal Understanding Multimodal Reward Modeling Multimodal Large Language Models Rubric-Based Evaluation Multimodal Foundation Models Vision-Language Model-as-a-Judge+3

Open paper AskPDF arXiv

Aligning Multimodal Large Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce

\textbf{DeltaRubric}

, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a

\textit{Disagreement Planner}

, the model generates a neutral, instance-specific verification checklist. Transitioning into a

\textit{Checklist Verifier}

, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by

\textbf{+22.6}

(4B) and

\textbf{+18.8}

(8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

Abstract

Reward Modeling

Explore similar work

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Unsupervised Multimodal Reward Modeling

Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation