Grounding Multimodal Reasoning with Evidence-Ablated Negatives

Chenxi Liu; Tianyi Xiong; Yanshuo Chen; Heng Huang

cs.CLOct 29, 2026

Grounding Multimodal Reasoning with Evidence-Ablated Negatives

Authors: Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Heng Huang

Organizations: Department of Computer Science, University of Maryland, College Park, College Park, MD 20742

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful tool for improving multimodal reasoning. However, standard outcome supervision suffers from credit assignment ambiguity, as a model may receive positive rewards by exploiting language priors or dataset shortcuts rather than grounding its reasoning in the requisite visual evidence. This makes high-quality negative responses critical, since effective negatives should remain plausible under the current policy while exposing the shortcut that should be suppressed. In this paper, we propose EVAN, a multimodal reinforcement learning framework utilizing EVidence-Ablated Negatives to train LMMs against their own language-prior failures. EVAN first identifies highly visually dependent examples using a scaled image information gain metric and preserves data diversity with a mixed-ranking strategy. It then constructs evidence-ablated images by masking the most discriminative image patches, eliciting plausible but visually ungrounded responses from the current policy. During policy optimization, these negative responses are incorporated as bounded auxiliary signals under the clean image, penalizing shortcut learning while maintaining RL stability. Experiments across challenging multimodal reasoning benchmarks demonstrate that EVAN mitigates modality imbalance and improves over strong RLVR baselines. Our code is available at https://github.com/DawnLIU35/EVAN.git.

Figures & tables

Qwen2.5-VL-3B-Instruct
Model	Geometry3K	MathVista	MathVerse	We-Math	DynaMath	MathVision	MMK12	MMMU-Pro	LogicVista	Avg.
PAPO-3B	36.9	63.3	60.5	62.4	57.2	26.2	62.9	28.4	43.2	49.0
Qwen2.5-VL-3B	19.4	46.3	33.5	29.9	40.4	18.2	36.3	19.8	32.1	30.7
+DAPO	38.8	62.4	60.4	60.5	59.3	26.5	72.2	29.9	43.0	50.3
+ EVAN (ours)	39.9	62.7	62.8	64.6	60.6	27.6	71.5	31.0	43.3	51.6
Qwen2.5-VL-7B-Instruct

Table 1: Comparison with state-of-the-art open-source reasoning LMMs on multimodal reasoning benchmarks. All methods are evaluated under the same evaluation configuration. The best performance is marked in bold .

Variant	Geometry3K	MathVista	MathVerse	We-Math	DynaMath	MathVision	MMK12	MMMU-Pro	LogicVista	Avg.
Random	38.5	62.4	62.5	65.1	60.1	27.1	72.2	30.0	42.0	51.1
Global	39.8	60.8	60.9	61.9	59.2	27.5	72.3	28.8	41.7	50.3
Category	38.4	63.5	63.5	64.8	59.6	26.8	72.7	29.3	42.9	51.3
RankMix ( EVAN )	39.9	62.7	62.8	64.6	60.6	27.6	71.5	31.0	43.3	51.6

Table 2: Ablation study of RankMix Selection with Qwen2.5-VL-3B. All variants use the same 20% data budget and differ only in the selection strategy.

Variant	Geometry3K	MathVista	MathVerse	We-Math	DynaMath	MathVision	MMK12	MMMU-Pro	LogicVista	Avg.
Clean Image	39.8	63.5	60.8	61.3	59.8	26.9	72.8	29.2	42.4	50.7
Random Patch	39.9	62.0	62.1	61.4	59.6	27.2	72.9	29.9	42.8	50.8
FGSM	39.6	61.8	60.6	64.1	59.8	26.5	73.2	29.6	42.3	50.8
Adv. mask ( EVAN )	39.9	62.7	62.8	64.6	60.6	27.6	71.5	31.0	43.3	51.6

Table 3: Ablation on evidence-ablated image construction with Qwen2.5-VL-3B. All variants use the same selected examples and differ only in the image used for evidence-ablated rollouts.

Variant	Geometry3K	MathVista	MathVerse	We-Math	DynaMath	MathVision	MMK12	MMMU-Pro	LogicVista	Avg.
EVAN ( $\rho=25\%$ )	39.6	59.9	61.5	64.9	61.0	26.8	72.7	31.1	42.8	51.1
EVAN ( $\rho=50\%$ )	39.9	62.7	62.8	64.6	60.6	27.6	71.5	31.0	43.3	51.6
EVAN ( $\rho=75\%$ )	39.4	61.6	62.2	63.7	60.2	27.9	73.0	30.7	43.5	51.4
EVAN ( $\rho=100\%$ )	39.5	63.2	63.1	65.0	59.1	26.5	71.3	29.4	41.0	50.9

Table 4: Effect of the masking ratio

\rho

on Qwen2.5-VL-3B. All variants use the same training recipe and differ only in the fraction of masked patches.

\rho=100\%

masks the entire image.

Figure 1: (Left) Bounded negative advantage (BNA) ablation on Qwen2.5-VL-3B, with entropy on the left axis and gradient norm on the right axis. (Right) rollout filtering ablation on Qwen2.5-VL-7B, with PPO KL Divergence on the left axis and response length on the right axis.

Figure 2: Training dynamics of the EVidence-Ablated Negatives branch. (Left) Number of prompts with prepared evidence-ablated images in each training step. (Right) Number of retained evidence-ablated negative responses appended to the clean-image update after filtering.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Correct clean responses	1	2	3	4	5	6	7
Clean correct response	2.65	1.73	1.29	1.00	0.77	0.58	0.38
Clean incorrect response	$-0.38$	$-0.58$	$-0.77$	$-1.00$	$-1.29$	$-1.73$	$-2.65$
Evidence-ablated negative	$-0.38$	$-0.58$	$-0.77$	$-1.00$	$-0.77$	$-0.58$	$-0.38$

Appendix

Table 5: Advantages under binary rewards for a clean rollout group of

G=8

responses. Each column corresponds to a number of correct responses in the clean rollout group, and each row gives the advantage assigned to one type of response.

Model	Geometry3K	MathVista	MathVerse	We-Math	DynaMath	MathVision	MMK12	MMMU-Pro	LogicVista	Avg.
Qwen2.5-VL-3B-Instruct
DAPO	38.8 $\pm$ 0.80	62.4 $\pm$ 0.52	60.4 $\pm$ 0.28	60.5 $\pm$ 0.47	59.3 $\pm$ 0.29	26.5 $\pm$ 0.26	72.2 $\pm$ 0.24	29.9 $\pm$ 0.46	43.0 $\pm$ 0.93	50.3 $\pm$ 0.25
EVAN	39.9 $\pm$ 0.66	62.7 $\pm$ 0.45	62.8 $\pm$ 0.50	64.6 $\pm$ 0.42	60.6 $\pm$ 0.17	27.6 $\pm$ 0.58	71.5 $\pm$ 0.40	31.0 $\pm$ 0.61	43.3 $\pm$ 1.14	51.6 $\pm$ 0.18
Qwen2.5-VL-7B-Instruct
DAPO	45.5 $\pm$ 0.99	66.2 $\pm$ 0.58	69.2 $\pm$ 0.46	68.4 $\pm$ 0.32	65.8 $\pm$ 0.12	29.0 $\pm$ 0.26	81.9 $\pm$ 0.24	37.0 $\pm$ 0.44	45.8 $\pm$ 1.19	56.5 $\pm$ 0.16
EVAN	46.0 $\pm$ 1.08	69.0 $\pm$ 0.46	69.1 $\pm$ 0.58	69.7 $\pm$ 0.33	66.5 $\pm$ 0.45	31.3 $\pm$ 0.35	82.2 $\pm$ 0.24	38.8 $\pm$ 0.50	48.8 $\pm$ 1.07	57.9 $\pm$ 0.23

Appendix

Table 6: 95% confidence intervals of DAPO and EVAN on Qwen2.5-VL-3B and Qwen2.5-VL-7B. Each interval is computed from the eight stochastic pass@1 evaluations of the same checkpoint.

Model	ChartQA	GQA	Visual CounterFact (CF+this)
			Color	Size	Pooled
DAPO	81.74	54.46	84.61	79.57	81.61
EVAN	82.55	55.83	85.04	84.99	85.01

Appendix

Table 7: Transfer of the Qwen2.5-VL-3B DAPO and EVAN checkpoints beyond structured reasoning.

Method	Total training time (h)	Cumulative rollout time (h)
DAPO	36.05	$\approx$ 22.0
EVAN	36.74	$\approx$ 21.2

Appendix

Table 8: Training cost of the matched 300-step Qwen2.5-VL-7B runs on 4

\times

NVIDIA H200 GPUs, excluding the one-time preprocessing of EVAN .

Figure 3: Rollout filtering ratios during training. The fraction of candidates that pass the length filter (left) and the clean-image likelihood filter (right) for EVAN -3B and EVAN -7B.

Responses	Reviewer	Audited	Pathological	Degenerate text	Describes mask
Retained	GPT-5.5	250	0	0	0
Retained	Human	50	0	0	0
Removed by likelihood filter	GPT-5.5	100	40	30	10

Appendix

Table 9: Semantic audit of evidence-ablated responses from the EVAN run on Qwen2.5-VL-3B. Retained responses pass all filters and are used for training. Pathological responses contain degenerate text or explicitly describe the mask.

Variant	Geometry3K	MathVista	MathVerse	We-Math	DynaMath	MathVision	MMK12	MMMU-Pro	LogicVista	Avg.
EVAN ( $\tau=10\%$ )	39.5	63.2	63.2	63.9	59.3	27.4	72.4	30.1	44.6	51.5
EVAN ( $\tau=20\%$ )	39.9	62.7	62.8	64.6	60.6	27.6	71.5	31.0	43.3	51.6
EVAN ( $\tau=30\%$ )	38.8	63.7	60.3	60.3	61.1	27.5	72.1	30.3	43.0	50.8

Appendix

Table 10: Effect of the RankMix selection ratio

\tau

on Qwen2.5-VL-3B. All variants use the same training recipe and differ only in the fraction of selected visually dependent examples.

Variant	Scaled-IIG Before	Scaled-IIG After	Acc. Before (%)	Acc. After (%)
Adv. mask	0.595	0.455	88.3	66.3
FGSM	0.595	0.093	88.3	35.3

Appendix

Table 11: Scaled-IIG and model accuracy before and after image perturbation on the RankMix 20% selected data.

Model	AUC	Top 20%	Bottom 20%	Gap
Qwen2.5-VL-3B-Instruct	0.566	43.6	28.5	15.1
Qwen2.5-VL-7B-Instruct	0.583	63.5	41.0	22.6
Qwen3-VL-8B-Instruct	0.592	85.8	71.7	14.1
Qwen3-VL-8B-Thinking	0.600	86.6	70.5	16.2
Mean	0.585	69.9	52.9	17.0

Appendix

Table 12: Scaled-IIG and response correctness on 5,000 ViRL39K questions with five sampled responses per question. AUC measures how well the scaled-IIG of each response predicts its correctness. Top 20% and Bottom 20% report the accuracy (%) of the responses with the highest and lowest scaled-IIG, and Gap is their difference.

Selection	Non-Geo Math	Geometric	Tables/Charts	Science	Spatial	Broader STEM	Social Science	Commonsense	Total
Full dataset	14901	10631	6189	3475	1922	956	676	120	38870
RankMix 10%	1122	382	1349	428	298	164	121	23	3887
RankMix 20%	2471	1131	2235	857	521	304	214	41	7774
RankMix 30%	3860	2092	2974	1241	720	423	295	56	11661
Global Top-20%	1244	306	4125	720	529	463	307	80	7774

Appendix

Table 13: Category distribution of selected training examples.

Figure 4: Case study on a train-schedule question. Left: clean image. Right: evidence-ablated image after adversarial masking.

Figure 5: Case study on a counting question. Top: clean image. Bottom: evidence-ablated image after adversarial masking.

Figure 6: Case study on a table question. Left: clean image. Right: evidence-ablated image after adversarial masking.

Figure 7: Case study on a genetics question. Top: clean image. Bottom: evidence-ablated image after adversarial masking.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 512 papers

Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
May 29, 2026Ruina Hu, Chen Wang, Lai Wei +5Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning
1Harbin Institute of Technology · 2Zhongguancun Academy · 4Nankai University +4
Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning
Mar 30, 2026Ziqi Miao, Haonan Jia, Lijun Li +4Multimodal Reasoning Multimodal Large Language Models
Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness
Apr 20, 2026Mengzhao Jia, Zhihan Zhang, Meng JiangMultimodal Reasoning Multimodal Reward Modeling
University of Notre Dame

May 29, 2026J/K move · Enter open · S save

Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR

Ruina Hu, Chen Wang, Lai Wei, Jionghao Bai+4

1Harbin Institute of Technology · 2Zhongguancun Academy · 4Nankai University+4

Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning Attention Mechanisms Visual Grounding Vision-Language Alignment RL for VLMs

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves vision-language models (VLMs) by optimizing outcome rewards derived from final answers. However, such outcome-only rewards do not tell the model which image regions justify an answer. For questions that require visual grounding, these rewards cannot distinguish responses supported by relevant visual evidence from those produced by language-prior shortcuts or lucky guesses. We introduce EASE (Evidence-Anchored Spatial Attention), which augments multimodal RLVR with visual-evidence process supervision. EASE converts annotated evidence regions into a smoothed visual-token target and uses it to guide response-to-image attention during RL training, but only on high-reward trajectories. The annotations are used solely as privileged training labels, while inference requires only the original image and question. Across Qwen2.5-VL-7B, Qwen3-VL-4B, and Qwen3-VL-8B, EASE raises average scores over DAPO by 2.5 to 3.1 points on perception, hallucination, visual math, and multimodal reasoning benchmarks. Diagnostics and ablations show that EASE better aligns visual attention with annotated evidence regions.

Abstract

Multimodal Reasoning

Explore similar work

Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR

Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning

Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness