The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models

Akshay Paruchuri; Ishan Chatterjee; Henry Fuchs; Ehsan Adeli; Piotr Didyk

cs.CLApr 15, 2026

The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models

Authors: Akshay Paruchuri, Ishan Chatterjee, Henry Fuchs, Ehsan Adeli, Piotr Didyk

Organizations: Università della Svizzera italiana · Stanford University · University of Washington · UNC Chapel Hill

Abstract

Multimodal language models systematically underperform on visual perception tasks, yet the structure underlying this failure remains poorly understood. We propose centroid replacement, mapping tokens to their nearest K-means centroid and removing within-cluster residual structure, as a controlled probe for modal dependence. Across seven models spanning four architecture families, post-image text replacement costs 4x more accuracy than visual replacement on six BLINK tasks; VPBench and MedBLINK preserve this aggregate ordering. Since text replacement also disrupts the task and answer interface, the gap measures relative dependence. Text centroid contrastive decoding (TCCD) exploits this asymmetry without retraining, recovering up to +16.9% accuracy on an individual task at oracle best-per-task $α_\text{interp}$ ; every model gains on at least one task, although gains vary across tasks and models. Together, centroid replacement provides a retraining-free audit of modal dependence on labeled data, while TCCD offers a reusable inference-time intervention. Code and artifacts are available at: https://github.com/yahskapar/centroid-erasure

Figures & tables

Figure 1: Centroid erasure exposes and exploits modal competition in multimodal language models. (a) In one BLINK example, text centroid replacement raises visual attention at L22 (3.6%

\to

11.1%), while text centroid contrastive decoding (TCCD) flips the prediction from incorrect to correct. The displayed heatmaps are qualitative; quantitative attention is computed separately after RoPE. (b) Across seven models and six BLINK tasks, post-image text replacement costs 4

\times

more accuracy than visual replacement (25.9% vs. 6.5%). Since text replacement also changes the task and answer interface, the gap measures relative dependence. (c) At oracle best-per-task

\alpha_{\text{interp}}

, TCCD recovers up to +16.9% accuracy on an individual task, with every model showing positive gains on at least one task.

Figure 2: The text–visual asymmetry is consistent across models. Per-task centroid replacement cost (visual in teal, text in red) across seven models. Text replacement is more costly for every model when averaged across tasks and in 37 of 42 model–task comparisons, with four ties and one reversal.

Model	Family	Training	Vis. Cost	Text Cost	Ratio
Qwen3-VL-8B	Qwen ( Bai et al., 2025a )	SFT+	16.1%	31.5%	2.0 $\times$
Qwen3-VL-4B	Qwen ( Bai et al., 2025a )	SFT+	16.2%	34.7%	2.1 $\times$
Qwen2.5-VL-7B	Qwen ( Bai et al., 2025b )	SFT	1.4%	27.2%	19.2 $\times$
Qwen2.5-VL-3B	Qwen ( Bai et al., 2025b )	SFT	9.1%	13.7%	1.5 $\times$
InternVL2.5-8B	InternVL ( Chen et al., 2024c )	MPO	$-$ 0.9%	29.4%	34.4 $\times$ $\dagger$
LLaVA-OV-7B	LLaVA ( Li et al., 2024 )	DPO	5.2%	25.8%	4.9 $\times$

Table 1: Centroid replacement cost across seven models. Mean accuracy cost when replacing visual or post-image text tokens with their nearest centroids (

\alpha_{\text{interp}}

=0). Text replacement costs 4.0

\times

more than visual on average. Negative values indicate that replacement improves accuracy.

\dagger

Ratio uses

|

vis. cost

|

; per-task breakdown in Figure 2 .

Task	$n$	Base	+CD	$\Delta$ best	$\Delta$ 0.4
Forensic	132	47.7%	59.1%	+11.4%	+7.6%
Vis. Sim.	135	76.3%	86.7%	+10.4%	+8.9%
Art Style	117	55.6%	62.4%	+6.8%	+6.8%
Counting	120	66.7%	69.2%	+2.5%	+0.8%
Depth	124	79.0%	80.6%	+1.6%	$-$ 2.4%
Spatial	143	88.8%	88.8%	+0%	$-$ 2.1%

Table 2: Text centroid CD results on Qwen2.5-VL-7B. Per-task accuracy and change under oracle best per-task

\alpha_{\text{interp}}

, with the fixed protocol (

\alpha_{\text{interp}}

=0.4) shown separately. The best-per-task column is an oracle upper bound; all significance tests use the fixed protocol.

Figure 3: Directional, task-selective dose–response (Qwen2.5-VL-7B,

\alpha_{\text{interp}}

=0.4; exploratory 11,218-image held-out COCO+ScienceQA cache,

K

=512). Negative

\alpha_{\text{cd}}

hurts every task; positive

\alpha_{\text{cd}}

selectively benefits the post-hoc text-competes group. On the negative-

\alpha_{\text{cd}}

side, Counting is the sole monotonicity exception.

Method	Protocol	Mean $\Delta$
Text centroid CD (ours)	fixed	+3.27%
Text-only LCD	fixed	+2.43%
VCD	fixed	+2.03%
DoLa low / high	fixed	+1.61% / +3.42%
SDCD	fixed	+0.87%

Table 3: Fixed-setting contrastive decoding baselines (Qwen2.5-VL-7B, six BLINK tasks,

n

=771). Mean accuracy change under the primary bare-continuation readout at

\alpha_{\text{cd}}

=1; TCCD also fixes

\alpha_{\text{interp}}

=0.4. Methods use separate runs on the same task rows.

Model	Train.	Base	$\Delta$ $\alpha_{\text{interp}}$ =0.4	$\Delta$ CV- $\alpha_{\text{interp}}$	$\Delta$ best- $\alpha_{\text{interp}}$	$g$
Qwen3-VL-8B	SFT+	73%	$-$ 2.1%	+0.9%	+1.6%	+0.04
Qwen3-VL-4B	SFT+	76%	$-$ 6.8%	$-$ 5.8%	$-$ 0.2%	$-$ 0.01
Qwen2.5-VL-7B	SFT	69%	+3.3%	+3.3%	+5.4%	+0.16
Qwen2.5-VL-3B	SFT	55%	+3.9%	+5.9%	+8.6%	+0.19
InternVL2.5-8B	MPO	68%	$-$ 1.0%	$-$ 1.8%	+3.1%	+0.11
LLaVA-OV-7B	DPO	68%	+2.0%	+1.5%	+4.7%	+0.14

Table 4: TCCD effect across reported training approaches. Six-task mean baseline accuracy and TCCD change under fixed

\alpha_{\text{interp}}

=0.4, leave-one-task-out selection, and oracle best-per-task selection. Here

g=(a_{\text{TCCD}}-a_{\text{base}})/(1-a_{\text{base}})

, averaged across the six tasks. Per-(model, task) Wilson CIs appear in Appendix C .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Family	Train.	Size	HuggingFace ID	Vis. Enc.
Qwen2.5-VL-3B	Qwen	SFT	3B	Qwen/Qwen2.5-VL-3B-Instruct	Qwen2-VL
Qwen2.5-VL-7B	Qwen	SFT	7B	Qwen/Qwen2.5-VL-7B-Instruct	Qwen2-VL
Qwen3-VL-4B	Qwen	SFT+	4B	Qwen/Qwen3-VL-4B-Instruct	Qwen3-VL
Qwen3-VL-8B	Qwen	SFT+	8B	Qwen/Qwen3-VL-8B-Instruct	Qwen3-VL
InternVL2.5-8B	InternVL	MPO	8B	OpenGVLab/InternVL2_5-8B-MPO-hf	InternViT
LLaVA-OV-7B	LLaVA	DPO	7B	llava-hf/llava-onevision-qwen2-7b-ov-hf	SigLIP

Appendix

Table 5: Model registry. Seven instruction-tuned MLMs spanning four architecture families, four reported training-recipe labels, and parameter scales from 3B to 8B. All use HuggingFace checkpoints.

Figure 4: Task examples from BLINK. One representative sample per task showing the input image(s), prompt with answer choices, and the correct answer. Top row: text-competes tasks (Forensic Detection, Visual Similarity, Art Style), where TCCD yields larger gains. Bottom row: text-needed tasks (Counting, Relative Depth, Spatial Relation), where gains are smaller or absent.

Task	Baseline (%)
Multi-view Reasoning	55.6
Jigsaw Puzzle Solving	55.3
Object Localization	45.1
Visual Correspondence	32.6
Semantic Correspondence	32.4
Relative Reflectance	32.1

Appendix

Table 6: BLINK tasks outside the six-task analysis. Qwen2.5-VL-7B baselines under the primary fixed four-letter readout. Several tasks have low baselines, while Multi-view Reasoning and Jigsaw are higher. Results on all fourteen tasks appear in Table 27 .

Figure 5: Per-model alpha sweep under the primary protocol. CD delta (%) versus centroid interpolation strength

\alpha_{\text{interp}}

for all seven models (

N

=2,000,

K

=256,

\alpha_{\text{cd}}

=1.0). The dashed vertical line marks the fixed protocol,

\alpha_{\text{interp}}

=0.4. SFT/DPO-labeled checkpoints (Qwen2.5, LLaVA-OV, Idefics3) show sharper task-selective peaks than SFT+/MPO checkpoints in this descriptive cross-checkpoint comparison.

Task	Base [95% CI]	+CD [95% CI]	$\Delta$	$\alpha_{\text{interp}}^{*}$
Qwen3-VL-8B
Forensic	66.7 [58.3, 74.1]	62.9 [54.4, 70.6]	$-$ 3.8	0.5
Vis. Sim.	87.4 [80.8, 92.0]	89.6 [83.3, 93.7]	+2.2	0.0
Art Style	45.3 [36.6, 54.3]	55.6 [46.5, 64.2]	+10.3	0.0
Counting	64.2 [55.3, 72.2]	65.8 [57.0, 73.7]	+1.7	0.3
Depth	88.7 [81.9, 93.2]	87.9 [81.0, 92.5]	$-$ 0.8	0.3

Appendix

Table 7: Per-(model, task) TCCD at oracle best per-task

\alpha_{\text{interp}}

(

N

=2,000 COCO images,

K

=256,

\alpha_{\text{cd}}

=1.0). Baseline and +CD accuracies and

\Delta

are in %, with Wilson 95% CIs. Same-set selection makes this an oracle analysis.

Figure 6: Task-group separation across model depth. Mean CD delta for text-competes and text-needed across 16 layers of Qwen2.5-VL-7B (

\alpha_{\text{interp}}

=0,

\alpha_{\text{cd}}

=0.65; exploratory 11,218-image held-out COCO+ScienceQA cache,

K

=512). The L12 centroid bank is injected at every layer. Group means separate from L4 through L22, indicating that the pattern is not specific to one injection layer.

Figure 7: Per-task layer sweep (Qwen2.5-VL-7B). Individual task CD deltas across 16 injection layers at

\alpha_{\text{interp}}

=0 and

\alpha_{\text{cd}}

=0.65, using the same L12 centroid bank and complementing the group means in Figure 6 . The shaded L4–L22 interval marks the range where the group means separate; individual task trajectories vary and do not all preserve the group-level direction at every layer.

Reference direction	text-competes	All six tasks
Nearest-centroid residual (TCCD)	+7.77	+3.27
Equal-norm random direction	+0.24	+0.40

Appendix

Table 8: Equal-norm matched-random specificity control (Qwen2.5-VL-7B). Mean accuracy change (%) at the fixed primary protocol: full post-image tail, L12,

\alpha_{\text{interp}}

=0.4,

\alpha_{\text{cd}}

=1.0, seed 42.

Figure 8: Dose-dependent sensitivity across positional text spans. CD effect when selectively applying centroid replacement to the full post-image tail, its first 70% or last 30%, or the pre-visual prefix (Qwen2.5-VL-7B, $\alpha_{\text{interp}}=0.4$ , $\alpha_{\text{cd}}=1.0$ ). These ranges are positional heuristics rather than semantic parses. Full-post-image replacement produces the strongest gains on text-competes tasks (Forensic $+7.6\%$ , Vis. Sim. $+8.9\%$ , Art Style $+6.8\%$ ). Pre-visual-prefix replacement yields a large gain on Forensic Detection ( $+12.1\%$ ). Suffix replacement at this dose is largely flat or mildly negative; at $\alpha_{\text{interp}}=0.3$ it recovers text-competes gains ( Table 9 ).

text-competes mean (3 tasks)
	$\alpha_{\text{interp}}$ =0.2	$\alpha_{\text{interp}}$ =0.3	$\alpha_{\text{interp}}$ =0.4	$\alpha_{\text{interp}}$ =0.6
Full post-image tail	+3.8	+0.5	+7.8	$-$ 4.2
Post-image suffix (30%)	+4.9	+5.5	$-$ 1.3	$-$ 4.3
Post-image prefix (70%)	$-$ 1.7	$-$ 1.5	$-$ 0.9	$-$ 0.6
Pre-visual prefix	+1.4	+2.1	+4.3	+2.5
All-task mean (6 tasks)

Appendix

Table 9: Positional-span-by-dose grid, summary (Qwen2.5-VL-7B). Mean CD

\Delta

(%) by dose for text-competes (top) and all six tasks (bottom). Values come from the primary full-tail sweep, suffix dose-response control, and prefix positional-grid evaluation, all using the same pipeline.

Post-image prefix (first 70%)
Task	$\alpha_{\text{interp}}$ =0.2	$\alpha_{\text{interp}}$ =0.3	$\alpha_{\text{interp}}$ =0.4	$\alpha_{\text{interp}}$ =0.6
Forensic	+0.0	$-$ 0.8	$-$ 0.8	+1.5
Vis. Sim.	+0.0	+1.5	+1.5	+0.0
Art Style	$-$ 5.1 ∗	$-$ 5.1 ∗	$-$ 3.4 ∗	$-$ 3.4
Counting	$-$ 0.8	$-$ 1.7	$-$ 0.8	+0.8
Depth	+0.0	+0.0	$-$ 0.8	+0.8

Appendix

Table 10: Positional-span-by-dose grid, per-task detail (post-image-prefix and pre-visual-prefix replacement;

\Delta

in %). ∗ marks nominal, unadjusted McNemar

p<0.05

with no multiplicity correction. Post-image-suffix summaries are in Table 9 ; full-post-image cells are in the

\alpha_{\text{interp}}

sweep of Appendix C .

Variant	Vis. Kept	Text Kept	Vis. Cost	Text Cost	Mean Best $\Delta$
Baseline	707,414	32,000	$+1.4$	$+27.2$	$+5.4$
no_dead (bottom-5% norm)	672,043	30,400	$+0.3$	$+27.2$	$+4.8$
no_sink (top-1% norm)	700,339	31,680	$+1.7$	$+27.2$	$+5.0$
no_either (both)	664,968	30,080	$+2.3$	$+27.2$	$+5.2$
$\Delta$ vs baseline ( no_dead )	—	—	$-1.08$	+0	$-0.68$
$\Delta$ vs baseline ( no_sink )	—	—	$+0.26$	+0	$-0.40$

Appendix

Table 11: Sensitivity to sink/dead-token exclusions (Qwen2.5-VL-7B, BLINK 6 tasks). Centroids are refit on 2,000 COCO activations under three exclusion variants. no_dead drops bottom-5% norm tokens; no_sink drops top-1% norm tokens; no_either applies both filters. Mean oracle best-

\alpha_{\text{interp}}

CD delta changes by at most 0.68% from baseline, and the text centroid cost is identical across all four variants.

Task	$n$	Base [95% CI]	+CD [95% CI]	$\Delta$	$h$
Forensic	132	47.7 [39.4, 56.2]	55.3 [46.8, 63.5]	+7.6	.152
Vis. Sim.	135	76.3 [68.5, 82.7]	85.2 [78.2, 90.2]	+8.9	.227
Art Style	117	55.6 [46.5, 64.2]	62.4 [53.4, 70.6]	+6.8	.139
Counting	120	66.7 [57.8, 74.5]	67.5 [58.7, 75.2]	+0.8	.018
Depth	124	79.0 [71.0, 85.3]	76.6 [68.4, 83.2]	$-$ 2.4	$-$ .058
Spatial	143	88.8 [82.6, 93.0]	86.7 [80.2, 91.3]	$-$ 2.1	$-$ .064

Appendix

Table 12: Wilson 95% confidence intervals (Qwen2.5-VL-7B, fixed

\alpha_{\text{interp}}

=0.4,

\alpha_{\text{cd}}

=1.0,

K

=256). Baseline and +CD accuracies and

\Delta

are in %. Greedy decoding produces deterministic predictions; intervals reflect finite-sample binomial uncertainty, not run-to-run variance. Cohen’s

h

clears the conventional “small” threshold of 0.2 on Visual Similarity and sits near it on Forensic Detection and Art Style.

Task	Fixed 0.4	Oracle	$\alpha_{\text{interp}}^{*}$
Forensic	+7.6	+11.4	0.5
Vis. Sim.	+8.9	+10.4	0.1
Art Style	+6.8	+6.8	0.4
Counting	+0.8	+2.5	0.5
Depth	$-$ 2.4	+1.6	0.8
Spatial	$-$ 2.1	+0	0.8

Appendix

Table 13: Fixed

\alpha_{\text{interp}}

=0.4 vs. per-task oracle (Qwen2.5-VL-7B;

\Delta

in %). Mean gains are +3.3% at fixed

\alpha_{\text{interp}}

=0.4 and +5.4% at the same-set per-task optimum; their ratio is 0.60 using unrounded means. The task-level pattern ( text-competes

\gg

text-needed ) appears under both protocols.

Task	km=42	km=800	km=1337	km=2024	km=8320	$\sigma$
Forensic	+11.4	+9.1	+3.8	+8.3	+9.1	2.5
Vis. Sim.	+10.4	+10.4	+9.6	+9.6	+10.4	0.4
Art Style	+6.8	+6.8	+6.8	+4.3	+7.7	1.2
Counting	+2.5	+2.5	+3.3	+2.5	+3.3	0.4
Depth	+1.6	+0.8	$-$ 0.8	$-$ 1.6	+0.8	1.2
Spatial	+0	+0	+0	+0	+0	0.0

Appendix

Table 14: Refit stability of the reported per-task deltas (Qwen2.5-VL-7B, primary implementation, data seed 1337). CD delta (%) across five independent

K

-means fits (seeds 42, 800, 1337, 2024, 8320), evaluated at the frozen per-task

\alpha_{\text{interp}}^{*}

selected on the primary fit. The seed-42 column reproduces Table 2 ;

\alpha_{\text{interp}}

is not reselected for each fit. Forensic Detection has the largest sensitivity (

\sigma

=2.5%).

Task	km=42	km=800	km=1337	km=2024	km=8320	$\sigma$
Forensic	+7.6	+11.4	+12.1	+11.4	+8.3	1.8
Vis. Sim.	+8.9	+7.4	+8.2	+8.2	+8.2	0.5
Art Style	+6.8	+6.8	+6.8	+4.3	+7.7	1.2
Counting	+0.8	+2.5	+2.5	+1.7	+2.5	0.7
Depth	$-$ 2.4	$-$ 3.2	$-$ 3.2	$-$ 2.4	$-$ 3.2	0.4
Spatial	$-$ 2.1	$-$ 0.7	$-$ 0.7	$-$ 1.4	$-$ 2.1	0.6

Appendix

Table 15: Refit stability at the fixed deployment protocol (Qwen2.5-VL-7B, primary implementation,

\alpha_{\text{interp}}

=0.4). Same five refits as Table 14 ; the seed-42 column reproduces Table 12 . The maximum unrounded per-task

\sigma

is 1.83%, and the text-competes

>

text-needed group separation holds in all five refits.

Task	$\sigma_{\text{kmeans}}$	$\sigma_{\text{data}}$	$\sigma_{\text{total}}$
Forensic	1.4	1.3	1.5
Vis. Sim.	0.5	0.4	0.5
Art Style	0.9	0.8	1.0
Counting	0.6	0.4	0.7
Depth	0.5	0.6	0.8
Spatial	0.5	0.5	0.6

Appendix

Table 16: Sensitivity summary over the 3

\times

5 fit grid (fixed

\alpha_{\text{interp}}

=0.4,

\alpha_{\text{cd}}

=1.0).

\sigma_{\text{kmeans}}

is the within-data-seed SD across five

K

-means seeds, averaged over three data seeds;

\sigma_{\text{data}}

is the across-data-seed SD averaged over

K

-means seeds;

\sigma_{\text{total}}

is the SD across all 15 fits. These descriptive SDs are not formal factorial variance components.

Task	ECE B	ECE CD	$\Delta$ ECE	Conf. B	Conf. CD
Forensic	.103	.110	+.007	.522	.620
Vis. Sim.	.093	.085	$-$ .008	.856	.907
Art Style	.277	.299	+.022	.833	.880
Counting	.158	.185	+.027	.820	.874
Depth	.024	.061	+.038	.785	.845
Spatial	.037	.055	+.018	.894	.919

Appendix

Table 17: Calibration analysis under the earlier exploratory protocol (Qwen2.5-VL-7B, 11,218-image held-out COCO+ScienceQA cache,

K

=512,

\alpha_{\text{interp}}

=0.4,

\alpha_{\text{cd}}

=0.65; 10 equal-width bins). Text centroid CD increases ECE on five of six tasks (mean

\Delta

ECE

=+0.017

Task	COCO	COMP.	NEED.	All 6
Forensic	+11.4	+2.3	+7.6	+2.3
Vis. Sim.	+10.4	+0.7	+3.0	+0.7
Art Style	+6.8	+0	+0.9	+0
Counting	+2.5	+0.8	$-$ 0.8	+1.7
Depth	+1.6	+4.0	$-$ 0.8	+0
Spatial	+0	$-$ 1.4	$-$ 2.1	$-$ 2.1

Appendix

Table 18: Cross-task centroid-source comparison (Qwen2.5-VL-7B;

\Delta

in %). COCO uses the primary protocol (

N

=2,000,

K

=256,

\alpha_{\text{cd}}

=1, per-task best

\alpha_{\text{interp}}

); task-derived columns use earlier exploratory

K

=512 fits from the first 50 evaluation inputs per task, at fixed

\alpha_{\text{interp}}

=0.4 and

\alpha_{\text{cd}}

=0.65. Because the protocols and fit/evaluation overlap differ, the comparison is descriptive.

Benchmark	#Models	Vis. Cost	Text Cost	Asym	#Sig. Wins	#Points
BLINK (primary)	7/7	$+6.5$	$+25.9$	4 $\times$	10	42
CV-Bench+MMStar+MMVP †	7/7	—	$+23.2$	—	—	28
VPBench	7/7	$+3.7$	$+11.4$	3.1 $\times$	3	14
MedBLINK (general)	7/7	$+5.0$	$+9.7$	1.9 $\times$	11	56
MedBLINK (medical)	2/2	$+3.1$	$+8.0$	2.5 $\times$	2	16

Appendix

Table 19: Cross-benchmark summary (costs and deltas in %). Aggregate asymmetry is observed in every row where both modalities were measured. † The exploratory CV-Bench/MMStar/MMVP sweep measured text cost only and therefore does not establish an asymmetry. “Wins” are nominal

p<0.05

counts after oracle best-

\alpha_{\text{interp}}

selection and are not family-wise corrected.

Model	Depth	Semantic	Vis. Cost	Text Cost	Asym	Best $\Delta$	#Sig.
Qwen2.5-VL-7B (SFT)	64.7	32.9	$+0.6$	$+11.0$	17.3 $\times$	$-0.6$	0/2
Qwen2.5-VL-3B (SFT)	54.0	30.4	$+4.5$	$+4.7$	1.0 $\times$	$+3.5$	1/2
Qwen3-VL-8B (SFT+)	75.2	47.5	$+10.5$	$+23.5$	2.2 $\times$	+0	0/2
Qwen3-VL-4B (SFT+)	72.5	41.4	$+7.9$	$+19.2$	2.4 $\times$	$+2.3$	1/2
InternVL2.5-8B (MPO)	59.8	30.8	+0	$+7.3$	vis $\approx$ 0	$+1.0$	1/2
LLaVA-OV-7B (DPO)	67.6	30.6	$+2.0$	$+11.3$	5.6 $\times$	$+0.7$	0/2

Appendix

Table 20: VPBench summary (7 models

\times

2 tasks; baseline accuracies, costs, and deltas in %). Aggregate text cost is +11.4% versus +3.7% visual. Because marker variants were not tested, this table does not assess VPBench’s known marker sensitivity. Best-

\alpha_{\text{interp}}

deltas and nominal win counts use oracle selection.

Model	Mean Base	Vis. Cost	Text Cost	Asym	Best $\Delta$	Fixed $\Delta$	#Sig.
Qwen2.5-VL-7B (SFT)	54.3	$+0.8$	$+7.9$	10.4 $\times$	$+2.4$	$-3.7$	1/8
Qwen2.5-VL-3B (SFT)	52.0	$+5.3$	$+5.6$	1.1 $\times$	$+3.4$	$-0.1$	2/8
Qwen3-VL-8B (SFT+)	70.0	$+11.0$	$+23.8$	2.2 $\times$	$+3.2$	$-1.4$	1/8
Qwen3-VL-4B (SFT+)	65.6	$+11.4$	$+18.9$	1.7 $\times$	$+1.4$	$-2.2$	2/8
InternVL2.5-8B (MPO)	49.3	$+0.6$	$+2.9$	4.8 $\times$	$+3.2$	$-2.3$	1/8
LLaVA-OV-7B (DPO)	52.8	$+7.0$	$+6.5$	0.9 $\times$	$+3.2$	$-2.1$	2/8

Appendix

Table 21: MedBLINK summary (9 models

\times

8 tasks; baseline accuracy, costs, and deltas in %). Best-

\alpha_{\text{interp}}

is an oracle upper bound; #Sig. counts nominal, unadjusted

p<0.05

wins after selection. Aggregate text cost exceeds visual cost for 8 of 9 models (LLaVA is the exception).

\dagger

The Idefics ratio uses

|\text{visual cost}|

. The MedGemma comparison is observational and does not isolate a fine-tuning effect.

Benchmark	$n$	Text cost (%)	Vis. cost (%)	Recovery (%)	Nominal sig. +/ $-$
A-OKVQA	1000	+55 to +67	0 to +31	$-$ 0.9	0 / 3
SEED-Bench	1000	+38 to +52	0 to +36	$-$ 0.6	0 / 2
ScienceQA-IMG	1000	+37 to +62	0 to +9	$-$ 0.3	0 / 2
RealWorldQA	436	+17 to +39	0 to +19	$-$ 0.9	0 / 0
MMStar	1000	+19 to +38	0 to +27	$-$ 1.1	0 / 3
CV-Bench	1000	+5 to +45	$-$ 1 to +22	$-$ 5.1	1 / 8

Appendix

Table 22: Auxiliary answer-token-scoring sensitivity (10 models

\times

8 benchmarks). Costs and recovery are reported in %. Seven MCQA rows use the alternative scorer defined above and therefore measure scoring sensitivity rather than primary-protocol accuracy. MME uses protocol-matched binary scoring. Significance counts are nominal and unadjusted; discordant counts were not retained.

Model	Text cost (%)	Vis. cost (%)	Ratio
Qwen2.5-VL-7B	$+36.2$	$+1.8$	19.8 $\times$
Qwen3-VL-8B	$+38.3$	$+15.8$	2.4 $\times$
Qwen2.5-VL-3B	$+30.5$	$+19.3$	1.6 $\times$
Qwen3-VL-4B	$+39.0$	$+14.7$	2.7 $\times$
LLaVA-OV-7B	$+31.9$	$+0.5$	69.5 $\times$
InternVL2.5-8B	$+35.8$	$+0.2$	vis $\approx$ 0

Appendix

Table 23: Auxiliary RealWorldQA sensitivity under the alternative answer-token scorer across ten models with validated visual-token spans (

n

=436, full replacement,

\alpha_{\text{interp}}=0

; costs in %). Ratios with visual cost within one item of zero are reported as “vis

\approx

0.”

Qwen2.5-VL-7B (alt.)	$\alpha_{\text{cd}}$ =.4	CV- $\alpha_{\text{cd}}$	Oracle
TCCD (ours)	$+1.65$	$+1.67$	$+3.22$
Blank-image LCD	$+2.68$	$+2.55$	$+4.54$
VCD	$-0.52$	$-1.17$	$+0.27$

Appendix

Table 24: Extended CD baselines. Left: Qwen2.5-VL-7B under the alternative answer scorer at fixed

\alpha_{\text{interp}}

=0.4, selecting

\alpha_{\text{cd}}

at 0.4, by leave-one-task-out CV, or per-task oracle from

\{0.2,\ldots,1.0\}

. Right: primary-scored LCD/VCD on the other six models at fixed

\alpha_{\text{cd}}

=1 (mean

\Delta

in %, six BLINK tasks,

n

=771 each). Compare methods only within each block.

Method	Vote@2	Vote@5	Vote@10
Mixed draws ( $\approx$ 1 $\times$ )	0.0	$-$ 0.1	$-$ 0.2
Stochastic draws ( $\approx$ 1 $\times$ )	$-$ 3.2	$-$ 3.4	$-$ 0.7
Greedy repeats ( $\approx$ 1 $\times$ )	0.0	0.0	0.0
Text centroid CD, fixed $\alpha_{\text{interp}}$ =0.4 (2 $\times$ )	+3.3
Text centroid CD, oracle best-per-task $\alpha_{\text{interp}}$ (2 $\times$ )	+5.4

Appendix

Table 25: TCCD vs. seed-pinned answer-logit resampling (Qwen2.5-VL-7B). Mean

\Delta

vs. greedy (%), six BLINK tasks (

n

=771). All votes reuse one frozen softmax (

\approx

\times

model compute); TCCD uses two forward passes.

	Mixed			Pure-stochastic
Task	@2	@5	@10	@2	@5	@10
Forensic	0.00	$-$ 0.76	+0.76	$-$ 9.09	$-$ 4.55	0.00
Vis. Sim.	0.00	+0.74	$-$ 1.48	$-$ 0.74	$-$ 2.96	+0.74
Art Style	0.00	+0.85	+0.85	0.00	$-$ 1.71	+0.85
Counting	0.00	0.00	0.00	$-$ 4.17	$-$ 0.83	0.00
Depth	0.00	$-$ 0.81	0.00	$-$ 2.42	$-$ 6.45	$-$ 3.23

Appendix

Table 26: Per-task answer-logit-resampling deltas (% vs. greedy, Qwen2.5-VL-7B). Mixed = one greedy plus

k-1

stochastic draws; stochastic =

k

draws.

		Qwen2.5-VL-7B			Qwen3-VL-8B
Task	$n$	Base (%)	$\Delta$ (%)	$p$	Base (%)	$\Delta$ (%)	$p$
Forensic Detection	132	47.7	+7.6	.096	66.7	$-$ 6.1	.011
Visual Similarity	135	76.3	+8.9	.014	87.4	+1.5	.317
Art Style	117	55.6	+6.8	.088	45.3	+7.7	.003
Counting	120	66.7	+0.8	.655	64.2	$-$ 0.8	.739
Relative Depth	124	79.0	$-$ 2.4	.491	88.7	$-$ 4.8	.034

Appendix

Table 27: All fourteen BLINK tasks at fixed

\alpha_{\text{interp}}

=0.4 under the fixed four-letter readout (baseline and

\Delta

in %; McNemar

p

). Top: the six analyzed tasks; bottom: the remaining eight. Several Qwen3-VL-8B baselines in the lower block are substantially higher than the corresponding Qwen2.5-VL-7B baselines.

Method	Operates on	What it does
Centroid replacement / TCCD (ours)	hidden states (mid-stack)	measures text-vs-visual dependence; contrasts a text-erased pass
LCD ( Manevich and Tsarfaty, 2024 )	output logits	contrasts the text-only LLM distribution
VCD ( Leng et al., 2024 )	input pixels	contrasts a noise-distorted image
Fu et al. (2025)	analysis	visual information encoded but unused
Wu et al. (2025)	training time	instance-level modality mixing
Deng et al. (2025) ; Li et al. (2025)	behavior; quantization	text-over-vision bias; balanced quantization

Appendix

Table 28: Method positioning. Centroid replacement operates on intermediate hidden-state centroids, used first as a measurement instrument for modal dependence; prior methods operate at other loci or are descriptive.

Modality Reliance in Multimodal Models

See all 21 papers

May 3, 2026

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues can be addressed by exploiting the shared information between modalities to compensate for the impaired one. To this end, we analyze multimodal interactions -- redundant (shared), unique (exclusive), and synergistic (emergent) task-relevant information provided by the modalities -- to determine their impacts on model reliability. Specifically, amplifying redundant interactions would increase this exploitable shared information to resolve these issues; yet, modern instruction datasets often eliminate redundancies to prioritize visual grounding. We bridge this gap through a self-captioning workflow featuring a Multimodal Interaction Gate: a mechanism to convert unique interactions into redundant interactions. Our findings suggest that increasing redundancy can reduce visual induced errors by 38.3% and improve consistency by 16.8%.

Yuriel Ryan, Hei Man Ip, Adriel Kuek +2

Singapore University of Technology and Design · DSO National Laboratories · Massachusetts Institute of Technology

VLM Robustness Multimodal Hallucination Multimodal Instruction Tuning Multimodal Understanding

Ask PDF

May 20, 2026

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Multimodal large language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks (e.g., MMLU) into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models, suggesting greater robustness in multimodal reasoning across equivalent textual and visual representations. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap, improving cross-modal alignment while preserving strong text-only and vision-language model performance. The project page is available at https://dongxzz.github.io/TokenSwap/.

Andong Hua, Colton Bishop, Igor Mordatch +5

University of California, Santa Barbara · Google DeepMind

Multimodal Large Language Models Modality Gap in VLMs Multimodal Learning Cross-Modal Alignment Multimodal Model Evaluation

Ask PDF

Aug 27, 2026

Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models

Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs. speech) and language (English vs. Arabic). We introduce a speech-augmented visually grounded contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented reasoning from complete failure. Evaluating recent multimodal models under text and speech in English and Arabic, we find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with speech amplifying partial failures. We make the benchmark publicly available to the community.

Basel Mousi, Fahim Dalvi, Shammur Chowdhury +2

Qatar Computing Research Institute, HBKU, Qatar

Multimodal Model Evaluation Multimodal Robustness Multimodal Learning Multimodal Grounding Audio-Language Model Evaluation Multilingual VLM Evaluation

Ask PDF

Abstract

Modality Reliance in Multimodal Models

Explore similar work

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models