Grounding on Request: Learning Modality Control in Vision-Language Models

Maxime Dassen; Alexander Martin; Andrew Yates; Reno Kriz; William Walden

cs.CVOct 24, 2026

Grounding on Request: Learning Modality Control in Vision-Language Models

Authors: Maxime Dassen, Alexander Martin, Andrew Yates, Reno Kriz, William Walden

Organizations: University of Amsterdam · Johns Hopkins University · Human Language Technology Center of Excellence, Johns Hopkins University

Abstract

When an image and its caption disagree, can a user reliably control which one a vision-language model (VLM) follows? Accuracy alone cannot answer this question, since a model might give the correct answer from parametric knowledge without following the requested evidence. We therefore evaluate modality grounding control through conflict-preserving counterfactual interventions. Specifically, Flip measures whether an initially correct answer follows an edit to the requested modality, while Stay measures whether it remains stable when the competing modality is edited. To learn this control, we build on neologism learning by introducing new words for visual and textual grounding, training only their embeddings while keeping the pretrained model fixed. We also introduce Counterfactual Twin-Margin (CTM) supervision, which uses paired interventions to teach how answer probabilities should change when the requested evidence changes. Across three VLMs on three existing benchmarks and our multi-attribute Ground-CF benchmark, we find that learned modality words improve control over natural-language instructions. Surprisingly, a single learned embedding per modality provides strong lightweight visual control and shows little average degradation on held-out values. LoRA remains strongest overall, although its visual advantage narrows under held-out evaluation. Moreover, counterfactual supervision improves control in several settings where standard supervision is weaker. Finally, we test our method on real-world news video and find that visual control improves when the relevant evidence is made easier to access. Overall, our results show that reliable modality control requires testing not only whether an answer is correct, but whether it continues to follow edits to the modality of choice.

Figures & tables

Figure 1: Accuracy misses modality grounding control. Left: the image shows a purple frog, the caption says green, and the model correctly answers purple under an image-grounding request. Middle: after the image changes to orange, answering green fails to follow the requested modality. Right: the desired answer is orange, shown with the learned visual grounding word <look> .

	Gemma-3-4B		Qwen3.5-4B		Qwen3.5-2B
Benchmark	Image	Text	Image	Text	Image	Text
VCF	.40	.60	.50	.50	.67	.33
ColorSwap	.43	.57	.68	.32	.74	.26
COCO-CF	.32	.68	.16	.84	.36	.64
Ground-CF	.43	.50	.70	.23	.62	.24

Table 1: Default modality preference depends on both model and benchmark. We report how often each base model selects the value supported by the image or text on conflicting examples. The larger value is shown in bold.

Figure 2: Accuracy overstates reliable modality control. Results are shown for natural-language image requests (left) and text requests (right). If every initially correct answer behaves appropriately under intervention, FLIP or STAY equals Accuracy. Points below the diagonal therefore identify correct predictions that fail the corresponding counterfactual test.

	Visual request <look>						Text request <read>
Method	ID			OOD			ID			OOD
	A	F	S	A	F	S	A	F	S	A	F	S
Base model	.398	.208	.294	.419	.251	.324	.546	.422	.435	.513	.417	.405
Instruction	.555	.367	.455	.573	.401	.486	.848	.784	.799	.825	.762	.784
Prompt-tuning †	.742	.630	.683	.734	.630	.683	.979	.968	.976	.981	.971	.979
Neologism	.800	.664	.733	.779	.657	.717	.940	.917	.930	.921	.902	.912

Table 2: Learned modality requests substantially improve control, while visual grounding remains harder. Scores are averaged across three models and four benchmarks. We report Flip (F) and Stay (S) for ID and held-out evaluation. † The prompt-tuning visual average includes one collapsed Gemma-3-4B COCO-CF cell (image Accuracy

.050

), the full per-cell results are reported in Appendix D . Bold denotes the best method and underlining the runner-up.

Figure 3: CTM helps most where SFT is weaker. Ground-CF results for neologisms trained with SFT or CTM; cells show CTM and CTM $-$ SFT (bold: McNemar $p<.05$ ).

	Distinguish words	Identify source
Gemma-3-4B	.750	.500
Qwen3.5-2B	.625	.625

Figure 4: Behavioral control does not imply self-verbalizable meaning. Models often distinguish the learned words, but are substantially less reliable at identifying the modality each word denotes.

	Gemma-3-4B		Qwen3.5-4B		Qwen3.5-2B
Method	$\langle$ look $\rangle$	$\langle$ read $\rangle$	$\langle$ look $\rangle$	$\langle$ read $\rangle$	$\langle$ look $\rangle$	$\langle$ read $\rangle$
Base model	$.214\rightarrow.277$	$.657$	$.145\rightarrow.202$	$.693$	$.087\rightarrow.127$	$.670$
Instruction	$.191\rightarrow.266$	$.727$	$.162\rightarrow.364$	$.820$	$.110\rightarrow.173$	$.760$
Prompt-tuning	$.173\rightarrow\underline{.249}$	$\underline{.870}$	$.156\rightarrow.347$	$.807$	$.133\rightarrow.214$	$\underline{.863}$
Neologism	$.104\rightarrow.220$	$.863$	$.202\rightarrow\underline{.486}$	$\underline{.903}$	$.145\rightarrow\underline{.295}$	$.850$
LoRA	$.191\rightarrow\mathbf{.416}$	$\mathbf{.983}$	$.376\rightarrow\mathbf{.595}$	$\mathbf{.993}$	$.353\rightarrow\mathbf{.538}$	$\mathbf{.997}$

Table 3: Making the relevant visual evidence easier to find improves visual control on news frames. We report Flip on MultiVENT-G. For

\langle

look

\rangle

, scores compare the full frame with a crop centered on the human-annotated evidence region (Full

\rightarrow

Crop), making the relevant visual information a larger part of the input. Crops are used only at evaluation time. For

\langle

read

\rangle

, we report full-frame performance. Visual pairs have

n{=}173

and description pairs

n{=}300

per model. Bold denotes the best learned method and underlining the runner-up.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Queried value	Image edit	Text edit	Cand.	Items ID / Held-out	Held out
VCF	object colour	counterfactual-colour image of the same object	colour replaced	4	120 / 140	94 objects unseen in training
ColorSwap	colour–object binding	colours swapped between objects	colours reassigned	4	140 / 140	pink, purple
COCO-CF	entity noun	counterfactual image	noun replaced	4	140 / 140	nouns unseen in training
Ground-CF	colour, shape, count, spatial relation	queried value changed; count/spatial scenes re-rendered	queried value replaced	3	748 / 800 (707 / 800)	purple, pink; triangle, star; counts 7–9; above

Appendix

Table 4: Controlled benchmarks and counterfactual interventions. “Items” gives the evaluated conflict examples rather than the size of the original dataset. Ground-CF reports all items, with the eligible image-twin subset in parentheses.

Method	Gemma-3-4B	Qwen3.5-4B	Qwen3.5-2B
Neologism / prompt tuning	5,120	5,120	4,096
LoRA, two adapters	59,604,992	42,467,328	21,823,488
LoRA, one joint adapter	29,802,496	21,233,664	10,911,744

Appendix

Table 5: Trainable parameters in the controlled experiments. LoRA counts include both source-specific adapters; one is active for each request.

Setting	Neologism / prompt tuning	LoRA
Trainable parameters	added embedding rows	rank-16 adapters, $\alpha=32$ , dropout $.05$
Adapted modules	—	q, k, v, o, gate, up, down projections; vision tower and projector frozen
Optimizer	AdamW, weight decay $.01$	AdamW, weight decay $.01$
Peak learning rate	$.05$	$.0002$
Schedule	20 warm-up steps, cosine decay to $10\%$ of peak	same
Steps / checkpoint	800 / 800	800 / 800

Appendix

Table 6: Training settings for the controlled experiments.

Method	Position	Training example	Evaluation example
English	end	—	According to the image (and only the image), what is the answer?
Prompt tuning	start	<pt> + task sentence	<pt> + held-out task sentence
Neologism	end	Give me a <look> answer.	Answer in a <look> manner.
LoRA, separate	adapter	task sentence	held-out task sentence
LoRA, joint	end	Answer using only the image.	held-out instruction

Appendix

Table 7: How each modality request is represented. Evaluation wordings are not used during training.

Split	Interface	$F_{I}$	$S_{I}$	$F_{T}$	$S_{T}$
ID	Word, end	.725	.773	.994	.996
ID	Word, start	.716	.767	.988	.988
ID	Prompt tuning, start	.767	.809	.979	.983
Held-out	Word, end	.608	.645	.995	.997
Held-out	Word, start	.609	.647	.981	.984
Held-out	Prompt tuning, start	.630	.667	.971	.977

Appendix

Table 8: Ground-CF placement control , averaged across the three models. Words are evaluated at the position in which they were trained.

Split	Dataset	Method	$A_{I}$	$F_{I}$	$S_{I}$	$A_{T}$	$F_{T}$	$S_{T}$
ID	VCF	No request	0.167	0.017	0.100	0.833	0.800	0.683
ID	VCF	Instruction	0.608	0.367	0.517	0.983	0.967	0.967
ID	VCF	Prompt tuning	0.817	0.700	0.700	1.000	1.000	1.000
ID	VCF	Neologism	0.933	0.867	0.900	1.000	1.000	1.000
ID	VCF	LoRA (separate)	0.892	0.783	0.842	1.000	1.000	1.000
ID	ColorSwap	No request	0.407	0.129	0.250	0.571	0.471	0.493

Appendix

Table 9: Complete Gemma-3-4B results. Ground-CF visual Flip and text Stay use the eligible image-twin subset; all other metrics use the full evaluation set. Joint LoRA is evaluated only on Ground-CF.

Split	Dataset	Method	$A_{I}$	$F_{I}$	$S_{I}$	$A_{T}$	$F_{T}$	$S_{T}$
ID	VCF	No request	0.267	0.117	0.150	0.733	0.675	0.583
ID	VCF	Instruction	0.700	0.550	0.625	1.000	1.000	1.000
ID	VCF	Prompt tuning	0.850	0.733	0.758	1.000	1.000	1.000
ID	VCF	Neologism	0.908	0.817	0.875	1.000	1.000	1.000
ID	VCF	LoRA (separate)	0.883	0.767	0.825	1.000	1.000	1.000
ID	ColorSwap	No request	0.529	0.300	0.357	0.429	0.257	0.307

Appendix

Table 10: Complete Qwen3.5-4B results. Conventions follow Table 9 .

Split	Dataset	Method	$A_{I}$	$F_{I}$	$S_{I}$	$A_{T}$	$F_{T}$	$S_{T}$
ID	VCF	No request	0.417	0.150	0.317	0.583	0.483	0.317
ID	VCF	Instruction	0.633	0.367	0.525	0.833	0.758	0.683
ID	VCF	Prompt tuning	0.933	0.867	0.900	1.000	1.000	1.000
ID	VCF	Neologism	0.875	0.767	0.792	1.000	1.000	1.000
ID	VCF	LoRA (separate)	0.850	0.733	0.742	1.000	1.000	1.000
ID	ColorSwap	No request	0.557	0.143	0.471	0.400	0.279	0.357

Appendix

Table 11: Complete Qwen3.5-2B results. Conventions follow Table 9 .

Model	Benchmark	$F_{I}/A_{I}$	$S_{I}/A_{I}$	$F_{T}/A_{T}$	$S_{T}/A_{T}$
Gemma-3-4B	VCF	.603	.849	.983	.983
	ColorSwap	.627	.787	.880	.944
	COCO-CF	.447	.766	.919	.919
	Ground-CF	.639	.799	.896	.934
Qwen3.5-4B	VCF	.786	.893	1.000	1.000
	ColorSwap	.730	.775	.948	.978

Appendix

Table 12: Conditional control for natural-language instructions in ID evaluation.

F/A

and

S/A

give the fraction of initially correct predictions that remain correct after the corresponding intervention.

Split	Model	Objective	$A_{I}$	$F_{I}$	$S_{I}$	$A_{T}$	$F_{T}$	$S_{T}$
ID	G3	SFT	0.783	0.605	0.738	0.959	0.914	0.936
ID	G3	Paired SFT	0.734	0.571	0.686	0.971	0.947	0.941
ID	G3	SFT+CTM	0.759	0.632	0.693	0.996	0.983	0.987
ID	G3	SFT+CTM+Stay	0.779	0.621	0.723	0.995	0.989	0.989
ID	Q4	SFT	0.755	0.666	0.687	1.000	1.000	1.000
ID	Q4	Paired SFT	–	–	–	–	–	–

Appendix

Table 13: Ground-CF objective comparison for

L{=}1

neologisms at checkpoint 800. G3, Q4, and Q2 denote Gemma-3-4B, Qwen3.5-4B, and Qwen3.5-2B. Paired SFT is available for Gemma-3-4B and Qwen3.5-2B.

Model	Objective	Reference	Question	Carrier	Both
Gemma-3-4B	Paired SFT	.629	.635	.642	.638
Gemma-3-4B	+CTM	.641	.617	.646	.622
Qwen-4B	Paired SFT	.921	.914	.925	.914

Appendix

Table 14: Joint LoRA retains control under the tested wording changes. Image Flip in the supplementary Ground-CF wording evaluation. Question and Carrier change the question wording and natural-language modality request, respectively; Both applies the two changes together. The trained adapter remains fixed and active in every condition. Values are point estimates, so small differences are not interpreted as statistical equivalence.

	ID		Held-out values
Configuration	Caption first	Image first	Caption first	Image first
Frozen, no request	.561	.495	.527	.350
Neologism, $L=5$ , paired SFT	.953	.893	.927	.893
Neologism, $L=5$ , +CTM	.977	.977	.967	.983
Neologism, $L=5$ , +CTM+Stay	.993	.982	.989	.984
Neologism, $L=1$ , +CTM+Stay	.949	.855	.945	.907

Appendix

Table 15: The observed source-order gap depends on the control configuration and objective. Caption Flip on Ground-CF for Gemma-3-4B, separated by image–caption order. The

L=5

neologism rows and the paired-SFT prompt-tuning row average three training seeds. Values are rounded point estimates; identical scores do not establish statistical equivalence. Both source orders occur during training, so this comparison measures differences across familiar layouts rather than transfer to unseen positions.

Request	Metric	Method	Held-out $-$ ID	95% interval
look	flip	Instruction	+0.034	[+0.007, +0.062]
look	flip	Prompt tuning	-0.000	[-0.026, +0.025]
look	flip	Neologism	-0.007	[-0.033, +0.020]
look	flip	LoRA	-0.094	[-0.118, -0.069]
look	stay	Instruction	+0.031	[+0.003, +0.060]
look	stay	Prompt tuning	+0.000	[-0.025, +0.024]

Appendix

Table 16: Change from ID to held-out evaluation. We report macro-average changes with item-bootstrap

95\%

intervals.

		Full frame						Evidence crop
Model	Method	$A_{I}$	$F_{I}$	$U_{I}$	$A_{T}$	$F_{T}$	$U_{T}$	$A_{I}$	$F_{I}$	$U_{I}$	$A_{T}$	$F_{T}$	$U_{T}$
Gemma-3-4B	No request	0.569	0.214	0.343	0.824	0.657	0.728	0.601	0.277	0.353	0.824	0.657	0.642
Gemma-3-4B	Instruction	0.573	0.191	0.330	0.860	0.727	0.751	0.593	0.266	0.327	0.862	0.727	0.613
Gemma-3-4B	Prompt tuning	0.575	0.173	0.427	0.935	0.870	0.873	0.615	0.249	0.427	0.935	0.870	0.902
Gemma-3-4B	Neologism	0.537	0.104	0.507	0.931	0.863	0.919	0.602	0.220	0.483	0.939	0.877	0.873
Gemma-3-4B	LoRA	0.575	0.191	0.430	0.992	0.983	0.827	0.702	0.416	0.450	0.994	0.987	0.769

Appendix

Table 17: Complete MultiVENT-G results. We report visual and text Accuracy (

A

), Flip (

F

), and answer stability (

U

) for the full frame and evidence crop.

U

is answer agreement rather than correctness-conditioned Stay . Visual pairs have

n{=}173

and text pairs

n{=}300

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Oct 9, 2026

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples (

+27.2\%

for Qwen3-VL 235B and

+15.1\%

for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from

+9.7\%

+30.3\%

. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.

Kun Wang, Xindi Wu, Sanghyuk Chun +2

Princeton University

Visual Evidence Sensitivity in VLMs In-Context Learning Vision-Language Grounding Counterfactual Evaluation of VLMs

Ask PDF

Sep 27, 2026

MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Models

We release MoGround, a vision-language dataset spanning four visual domains in which the answer to every question is guaranteed to be available from exactly one modality. This guarantee enables us to measure modality distraction, the failure in which a model answers a question correctly from one modality alone and then flips to a wrong answer once irrelevant content from the other modality is added. Existing probes rarely establish single-modality answerability this way, making it hard to isolate distraction in the first place. Across seven open-source VLMs, we find that modality distraction is not universal but model-dependent. The weaker-grounded modality is the more distracted one (r = +0.86), and distraction scales inversely with grounding strength (r = -0.90). The single-modality guarantee also enables a mitigation method that needs to distinguish between relevant and irrelevant context. Trained on one split of MoGround alone, a weight-space robustness vector reduces distraction on all seven models by 9% to 51%, at a cost of only 0.1 average points of accuracy on standard multimodal tasks.

Luca Zhou, Bo Zhao, Rose Yu +2

Sapienza University of Rome · Harvard University · UC San Diego +2

VLM Evaluation Multimodal Grounding VLM Robustness

Ask PDF

Sep 3, 2026

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing multimodal IH alignment as a reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfers to multimodal attacks, failing when models must decode, reconstruct, or reason over instructions across modalities. Image-based training improves robustness beyond text-only supervision, while mixed-modality training performs best overall. Importantly, the benefits generalize beyond the synthetic typographic training setting to real-image and web-agent safety tasks, while largely preserving general multimodal capability, showing that lightweight, verifiable supervision can meaningfully improve VLM robustness under adversarial, cross-modal, and interactive instruction conflicts.

Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar

The Ohio State University

VLM Safety Alignment VLM Robustness Adversarial Attacks on VLMs Vision-Language Alignment

Ask PDF

Abstract

Vision-Language Grounding

Explore similar work

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Models

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models