MJ1: Multimodal Judgment via Grounded Verification

Bhavesh Kumar; Dylan Feng; Leonard Tang

cs.LGMar 9, 2026

MJ1: Multimodal Judgment via Grounded Verification

Authors: Bhavesh Kumar, Dylan Feng, Leonard Tang

Organizations: Haize Labs

Abstract

Multimodal judges struggle to ground decisions in visual evidence. We present MJ1, a multimodal judge trained with reinforcement learning that enforces visual grounding through a structured grounded verification chain (observations $\rightarrow$ claims $\rightarrow$ verification $\rightarrow$ evaluation $\rightarrow$ scoring) and a counterfactual consistency reward that penalizes position bias. Even without training, our mechanism improves base-model accuracy on MMRB2 by +3.8 points on Image Editing and +1.7 on Multimodal Reasoning. After training, MJ1, with only 3B active parameters, achieves 77.0% accuracy on MMRB2 and surpasses orders-of-magnitude larger models like Gemini-3-Pro. These results show that grounded verification and consistency-based training substantially improve multimodal judgment without increasing model scale.

Figures & tables

Panel of Figure 1 — Figure 1: MJ 1 grounded verification chain. Judgement scores are generated based on verifying response claims against visual observations. Explicit visual grounding of the reasoning chain mitigates visual attention degradation.

Panel of Figure 2 — Figure 2: Two-phase training pipeline. Cold-start SFT on distilled reasoning traces establishes format and basic judgment capability. GRPO then optimizes a composite reward that incentivizes both correctness and position invariance.

Panel of Figure 3 — Figure 3: Computational structure comparison. (a) Standard judgment permits a shortcut path (dashed red) where scores depend minimally on images. (b) MJ 1 forces computation through observations $O$ , claim extraction $C$ , and verification $V$ . The dashed arrow indicates the forced back-reference from verification to observations. The consistency reward $R_{\text{cons}}$ couples verification to scores, requiring coherent image-grounded reasoning.

Prompting Strategy	Image Editing	Multimodal Reasoning
Open-ended reasoning	62.4	53.4
MJ 1 grounded verification	66.2 (+3.8)	55.1 (+1.7)

Table 1: Effect of grounded verification prompting on untrained base model. Accuracy (%) on 500 samples each from MMRB2 Image Editing and Multimodal Reasoning. Structured grounding improves accuracy without any training.

Panel of Figure 4 — Figure 4: Consistency as a grounding signal. (a) Mean $R_{\text{cons}}$ under three image conditions on an untrained base model. Shuffled images yield the lowest consistency, below even the no-image baseline. (b) Mean $R_{\text{correct}}$ shows degraded performance when visual grounding is disrupted, with both shuffled and blank conditions approaching random chance.

Judge	T2I	Editing	Interleaved	Reasoning	Avg.
Open-source multimodal LLMs
Gemma 3 4B ( Gemma Team, 2025 )	51.7	51.0	51.3	48.8	50.7
Gemma 3 12B ( Gemma Team, 2025 )	56.0	58.0	58.0	49.3	55.3
Gemma 3 27B ( Gemma Team, 2025 )	58.3	60.2	61.1	49.4	57.3
Qwen2.5-VL-7B ( Bai et al., 2025 )	50.4	57.1	48.4	47.5	50.9
Qwen2.5-VL-72B ( Bai et al., 2025 )	59.1	64.6	62.3	50.0	59.0

Table 2: Main results on MMRB2. Accuracy (%) across four subtasks. MJ 1 achieves state-of-the-art with only 3B active parameters, surpassing all API-based and open-source models. Best results in bold , second-best underlined .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Source Datasets	Samples
Text-to-Image	Rapidata Coherence [ Rapidata, 2024 ]	12K
Image Editing	EditReward [ EditReward Team, 2024 ]	38K
Reasoning	RLAIF-V [ Yu and others, 2024 ]	20K
Total		70K

Appendix

Table 3: Training data composition. Samples, sources, and citations for each domain.

Figure 5: Cold-start SFT training loss.

Panel of Figure 6 — Figure 6: GRPO reward curves.

Panel of Figure 8 — Figure 8: Task-specific evaluation criteria for Text-to-Image Generation.

Panel of Figure 9 — Figure 9: Task-specific evaluation criteria for Image Editing.

Panel of Figure 10 — Figure 10: Task-specific evaluation criteria for Interleaved Generation.

Panel of Figure 11 — Figure 11: Task-specific evaluation criteria for Visual Reasoning.

Position Bias in VLMs

See all 9 papers

VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM Evaluation Multimodal Grounding
Imperial College London
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
May 10, 2026Rui Liu, Dian Yu, Zhenwen Liang +6Multimodal Understanding Multimodal Reward Modeling
1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill
VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus
Aug 11, 2026Rohit Sinha, Kunal Tilaganji, Tanuja Ganu +3Multimodal Large Language Models Multimodal CoT Reasoning
Indian Institute of Technology, Hyderabad · Microsoft Research

Jun 5, 2026J/K move · Enter open · S save

VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

Didi Zhu, Changrui Chen, Stefanos Zafeiriou, Jiankang Deng

Imperial College London

VLM Evaluation Multimodal Grounding VLM Robustness Counterfactual Evaluation of VLMs Multimodal Reasoning Visually Grounded Reasoning+3

Open paper AskPDF arXiv

When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence? Correct answers can coexist with flawed reasoning, making accuracy alone an incomplete test of grounding. We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair keeps the question fixed but minimally changes the evidence so the gold answer deterministically flips. We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images. Together, these metrics show that paired correctness and evidence dependence are related but distinct: capable models can still fail to update after task-critical visual changes, and collapse becomes more severe for some models when the edited image follows an earlier answer in a sequential setting. Further details are available on our project page: https://didizhu-judy.github.io/VisualFLIP/

Abstract

Position Bias in VLMs

Explore similar work

VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus