You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Yinuo Yang; Zixian Ma; Manasi Ganti; Jieyu Zhang; Ranjay Krishna

cs.CVApr 13, 2026

You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Authors: Yinuo Yang, Zixian Ma, Manasi Ganti, Jieyu Zhang, Ranjay Krishna

Organizations: Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195, USA

Abstract

We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward models evaluate each response independently, requiring multiple forward passes, one for each potential response. Our approach concatenates multiple responses with separator tokens and applies cross-entropy over their scalar scores, enabling direct comparative reasoning and efficient $N$ -way preference learning. The multi-response design also yields up to $N\times$ wall-clock speedup and FLOPs reduction over conventional single-response scoring. To enable $N$ -way reward evaluation beyond existing pairwise benchmarks, we construct two new benchmarks: (1) MR $^2$ Bench-Image contains human-annotated rankings over responses from 8 diverse models; (2) MR $^2$ Bench-Video is a large-scale video-based reward benchmark derived from 94K crowdsourced pairwise human judgments over video question-answering spanning 19 models, denoised via preference graph ensemble. Both benchmarks provide 4-response evaluation variants sampled from the full rankings. Built on a 4B vision-language backbone with LoRA fine-tuning and a lightweight MLP value head, our model achieves state-of-the-art results on six multimodal reward benchmarks, including MR $^2$ Bench-Image, MR $^2$ Bench-Video, and four other existing benchmarks. Our model outperforms existing larger generative and discriminative reward models. We further demonstrate that our reward model, when used in reinforcement learning with GRPO, produces improved policy models that maintain performance across standard multimodal benchmarks while substantially improving open-ended generation quality, outperforming a single-response discriminative reward model (RM) baseline by a large margin in both training stability and open-ended generation quality.

Figures & tables

Figure 1: Comparison of reward model architectures. Left: Single-Response discriminative RM scores each

(x,y_{i})

pair independently via separate forward passes. Center: Generative RM prompts a VLM to output a preference distribution

p(I\mid x,y_{1},y_{2})

autoregressively. Right: Our Multi-Response discriminative RM concatenates all

N

candidates into a single sequence

(x,y_{1},y_{2},\ldots,y_{N})

and uses a multi-response scoring head to produce scores for all candidates in one forward pass.

Proprietary Models †
		Image				Video
Model	Size	VL-RB	MM-RB	MMRLHF	MR 2 B-I	VRB	MR 2 B-V	Avg
GPT-5 ( OpenAI, 2025 )	–	75.0	64.6	71.8	87.1	68.2	50.1	69.5
Claude-Sonnet-4.5 ( Anthropic, 2025 )	–	68.6	78.2	70.0	72.9	67.5	49.1	67.7
Gemini-2.5-Pro ( Comanici et al., 2025 )	–	70.5	82.4	70.6	71.2	63.2	49.7	67.9
Open-Source General VLMs †

Table 1: Main results on multimodal reward benchmarks. Our Molmo2-4B RM (4B) achieves the highest average across all open-source models, outperforming larger generative and discriminative baselines. VL-RB : VL-RewardBench (macro pairwise acc.); MM-RB : Multimodal RewardBench (pairwise acc.); MMRLHF : MM-RLHF RewardBench (pairwise acc.); MR 2 B-I : MR 2 Bench-Image (best-of-4 acc.); VRB : VideoRewardBench (macro pairwise acc.); MR 2 B-V : MR 2 Bench-Video (best-of-4 acc.). † Generative judge. ∗ Paper-reported score.

Base Model	Scoring	VL-RB	MM-RB	MMRLHF	MR 2 B-I	VRB	MR 2 B-V	Avg
Molmo2-4B	Multiple (CE)	62.1	73.8	88.8	52.5	64.3	47.1	64.8
Molmo2-4B	Single (BT)	57.7	61.9	64.7	41.2	60.8	37.6	54.0
Qwen3-VL-4B	Multiple (CE)	63.6	71.2	84.7	58.8	64.9	47.5	65.1
Qwen3-VL-4B	Single (BT)	67.4	73.5	88.8	49.6	58.1	40.6	63.0

Table 2: Multi-response vs. single-response scoring. Multi-response CE outperforms single-response BT on average, with a large gap on Molmo2-4B (64.8% vs. 54.0%).

Figure 2: Inference efficiency of multi-response vs. single-response scoring on Molmo2-4B (single NVIDIA H100 80 GB GPU). Per-sample latency and FLOPs grouped by

N

and modality, achieving up to

3.9\times

latency and

4.0\times

FLOPs reduction when

N=4

Figure 4: Validation reward during GRPO training. The multi-response RM provides a steadily increasing reward signal, while the single-response RM’s reward is unstable. The y-axis scales differ because the two reward models produce differently scaled outputs.

Model	WildVision	LLaVA-W	MMHal
Molmo2-4B (base)	54.6	92.4	3.98
+ GRPO (Multi-RM)	60.2 ( $+$ 5.6)	97.0 ( $+$ 4.6)	4.25 ( $+$ 0.27)
+ GRPO (Single-RM)	55.8 ( $+$ 1.2)	91.6 ( $-$ 0.8)	4.17 ( $+$ 0.19)

Table 3: GRPO on open-ended generation. Multi-RM substantially improves three open-ended benchmarks, while Single-RM shows little gains and even hurts LLaVA-Bench.

	VQAv2	TextVQA	ChartQA	DocVQA	InfoVQA	AI2D	MMMU	RWQA	MathVista	CountBench	PixMoCount	MuirBench	MMIU
Molmo2-4B (base)	86.7	84.9	86.0	92.4	78.6	95.6	50.9	75.4	56.7	94.1	88.1	60.5	55.5
Molmo2-4B + GRPO (Multi-RM)	86.7	84.7	86.2	92.5	78.6	95.7	50.6	75.9	56.5	94.3	88.3	60.7	55.9
Molmo2-4B + GRPO (Single-RM)	86.7	84.9	83.1	91.6	78.6	95.3	50.7	75.7	56.5	94.1	88.3	54.2	55.5

Table 4: GRPO on standard benchmarks. Multi-RM preserves performance across all 24 standard image and video benchmarks, while Single-RM degrades on several.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Proprietary Models
		MR 2 B-Image			MR 2 B-Video
Model	Size	Pairwise	Direct	$\Delta$	Pairwise	Direct	$\Delta$
GPT-5	–	87.1	87.5	+0.4	50.1	50.5	+0.4
Claude-Sonnet-4.5	–	72.9	72.5	-0.4	49.1	49.3	+0.2
Gemini-3-Flash	–	72.1	70.0	-2.1	52.3	52.5	+0.2
Open-Source General VLMs

Appendix

Table 6: Comparison of pairwise aggregation vs. direct best-of-4 selection on MR 2 Bench-Image and MR 2 Bench-Video. Pairwise : the model evaluates all

\binom{4}{2}=6

response pairs and selects the response with the highest win count (as used in Table 1 ). Direct : the model receives all 4 responses simultaneously and directly selects the best one.

\Delta

= Direct

-

Pairwise.

Proprietary Models †
Model	Size	VQA	Reason.	Safety	MR 2 B-I
GPT-5 ( OpenAI, 2025 )	–	80.0	87.5	93.8	87.1
Claude-Sonnet-4.5 ( Anthropic, 2025 )	–	62.5	76.2	80.0	72.9
Gemini-2.5-Pro ( Comanici et al., 2025 )	–	61.3	77.5	75.0	71.2
Open-Source General VLMs †
InternVL3-8B ( Zhu et al., 2025 )	8B	38.8	57.5	70.0	55.4

Appendix

Table 7: Per-category results on MR 2 Bench-Image (best-of-4 accuracy, 240 samples: 80 VQA, 80 reasoning, 80 safety, chance = 25%). † Generative judge (LLM-as-a-judge). Best result per category in bold , second best underlined .

Proprietary Models †
		VideoRewardBench
Model	Size	Perc-S	Perc-L	Know.	Reason	Safety	VRB Macro	MR 2 B-V
GPT-5 ( OpenAI, 2025 )	–	57.4	68.9	67.6	62.6	84.6	68.2	50.1
Claude-Sonnet-4.5 ( Anthropic, 2025 )	–	55.4	73.9	73.1	61.5	73.8	67.5	49.1
Gemini-2.5-Pro ( Comanici et al., 2025 )	–	62.7	47.7	72.7	68.0	64.7	63.2	49.7
Open-Source General VLMs †

Appendix

Table 8: Results on video reward benchmarks . VideoRewardBench (VRB) : VideoRewardBench ( Zhang et al., 2025e ) (pairwise accuracy across five categories, 1,563 pairs, chance = 50%); MR 2 B-V : MR 2 Bench-Video (best-of-4 accuracy, 495 samples, chance = 25%). † Generative judge (LLM-as-a-judge). Best result per category in bold , second best underlined .

Category	Benchmark	Split	Metric	Notes
Image Native	VQAv2	test-standard	VQA score	Server submission (EvalAI)
	TextVQA	val	VQA score
	ChartQA	test	Relaxed correctness
	DocVQA	test	ANLS	Server submission (RRC)
	InfoVQA	test	ANLS	Server submission (RRC)
	AI2D	test	Accuracy (transparent)

Appendix

Table 9: Per-benchmark evaluation configuration for GRPO policy evaluation (Section 5.2 ), following the Molmo2 technical report ( Clark et al., 2026 ) . Deviations: (1) validation splits used where test sets are unavailable; (2) EgoSchema val/500 instead of test/5000; (3) video evaluation uses decord (vs. torchcodec ), max_frames =376 (vs. 384), and 10K subtitle token cap.

Figure 5: Per-sample inference latency ( left , ms) and average FLOPs ( right ) for Qwen3-VL-4B on a single NVIDIA H100 80 GB GPU. Same grouping as Figure 3 .

Figure 6: Per-sample FLOPs comparison between our Molmo2-4B RM and open-source baselines, averaged over image benchmarks ( left : VL-RB, MM-RB, MMRLHF, MR 2 B-I) and video benchmarks ( right : VRB, MR 2 B-V). MM-RLHF-Reward on video benchmarks ( ∗ , hatched) are theoretically estimated due to incomplete automated measurement of LLaVA’s internal vision token expansion (see text).

Open-Source General VLMs
Model	Size	BoN	Pair	$\tau$
InternVL3-8B	8B	40.4	71.6	0.466
Qwen2.5-VL-7B	7B	44.4	73.6	0.504
Qwen3-VL-4B	4B	47.9	75.8	0.538
Qwen3-VL-8B	8B	47.7	77.2	0.563
Qwen3-VL-32B	32B	49.9	77.8	0.592

Appendix

Table 10: Full metrics on our MR 2 Bench-Video (4-response, 495 samples). BoN = best-of-N accuracy (%, primary metric from Table 1 ); Pair = pairwise accuracy (%);

\tau

= Kendall’s

\tau

rank correlation. Models: Skywork-VL-Reward ( Wang et al., 2025 ) , IXC-2.5-Reward ( Zang et al., 2025 ) , R1-Reward ( Zhang et al., 2025a ) , MM-RLHF-Reward ( Zhang et al., 2025c ) , LLaVA-Critic ( Xiong et al., 2025 ) , Molmo2 ( Clark et al., 2026 ) , Qwen3-VL ( Bai et al., 2025a ) . Best per section in bold , second best underlined .

Dataset	Collected	Used	$N$
Multimodal
MMHAL/MM-RLHF	16,321	16,321	$N{=}3\sim 5$
lmms-lab/LLaVA-Critic-113k	71,331	56,635	$N{=}2\sim 13$
openbmb/RLAIF-V-Dataset	83,132	64,097	$N{=}2$
MMInstruction/VLFeedback	80,258	63,448	$N{=}4$
YiyangAiLab/POVID_preference_data_for_VLLMs	17,184	17,184	$N{=}2$

Appendix

Table 11: Training data composition. We curate 436K samples from 10 datasets, with 35% containing

N{>}2

responses for listwise training.

N

: number of responses per sample.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

May 10, 2026

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Aligning Multimodal Large Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce

\textbf{DeltaRubric}

, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a

\textit{Disagreement Planner}

, the model generates a neutral, instance-specific verification checklist. Transitioning into a

\textit{Checklist Verifier}

, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by

\textbf{+22.6}

(4B) and

\textbf{+18.8}

(8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

Rui Liu, Dian Yu, Zhenwen Liang +6

1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill

Multimodal Understanding Multimodal Reward Modeling Multimodal Large Language Models Rubric-Based Evaluation Multimodal Foundation Models Vision-Language Model-as-a-Judge+3

Ask PDF

Oct 8, 2026

Unsupervised Multimodal Reward Modeling

Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy's own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark. Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.

Yueying Li, Jingxuan Fan, Kianté Brantley +2

Massachusetts Institute of Technology · Cornell University · Scale AI +1

Multimodal Reward Modeling VLM Evaluation

Ask PDF

Oct 16, 2026

Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation

Multimodal preference alignment often follows a score-based RLHF pipeline: a reward model is first trained to learn human preferences, then used to score on-policy responses and optimize the response model's generation. This turns preference supervision into a scalar signal for generation: a model may learn to produce answers with higher preference scores, but is rarely trained explicitly to learn the reasons underlying human preferences. We therefore ask whether making preference understanding an internal capability of the response model can improve its generation. To investigate this question, we propose PrefMind, an online dual-agent reinforcement learning framework with two trainable agents: a response Generator and a preference Critic. Given a pair of candidate responses, the agents analyze and discuss which response better aligns with human preference. Human labels supervise their judgments and corrections. For pairs on which both the Generator and revised Critic correctly identify the human-preferred response, the Generator produces an improved answer, and the Critic compares it against the original preferred response to provide a generation reward. Both agents are jointly updated from interactions sampled with their current policies. Through this process, PrefMind trains the Generator to understand human preferences via explicit judgment and correction, and to translate this understanding into better responses. Experiments across three model scales show improvements in both standalone preference judgment and downstream response generation.

Ziyun Zou, Jiwoo Hong, Yifan Lu +5

Mohamed bin Zayed University of Artificial Intelligence · Nubank · Dartmouth College

Preference Alignment Preference-Based RL Preference Modeling Multi-Agent LLM Systems Multimodal Generation

Ask PDF

Abstract

Reward Modeling

Explore similar work

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Unsupervised Multimodal Reward Modeling

Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation