Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

Yihang Du; Juhao Liang; Zhengzhao Lai; Siyu Li; Yan Hu

cs.CLAug 15, 2026

Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

Authors: Yihang Du, Juhao Liang, Zhengzhao Lai, Siyu Li, Yan Hu

Organizations: The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · National Health Data Institute (Shenzhen)

Abstract

Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning, despite the strong multilingual competence of their text-only backbones. While mechanistic evidence from text-only models suggests that non-English inputs are routed through an English-centric latent space, the multimodal implications of this phenomenon remain unexplored. Through rigorous mechanistic analysis, we identify the Ghost Anchor phenomenon: a temporal modality asynchrony where linguistic translation to the English semantic manifold completes in early layers, while visual semanticization remains immature. Consequently, visual signals are physically present yet functionally invisible during the early alignment window. To rectify this, we propose ANCHOR, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence, ensuring visual representations proactively guide linguistic translation. Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation. Furthermore, extensive experiments on XMMMU, MaXM, and CVQA demonstrate that ANCHOR consistently outperforms standard baselines, achieving robust visual reasoning across both fine-tuned and zero-shot languages.

Figures & tables

Figure 1: The Ghost Anchor phenomenon. (a) Ideal synchronized cross-modal alignment. (b) The reality of modality asynchrony: rapid linguistic translation bypasses delayed visual semantic emergence in early layers.

Figure 2: Modality Asynchrony . (a-b) Layer-wise English Similarity ( $\text{Sim}_{en}$ ) and English Translation Ratio ( $R_{ET}$ ) in text stream. (c) Visual Grounding Score ( $S_{VG}$ ) in vision stream. Shaded regions indicate the standard error of the mean (SEM).

Figure 3: Visual Causal Effect (LLaVA-1.5-7B left, 13B right). Difference between Normal and Noise inputs ( $\Delta=\text{Normal}-\text{Noise}$ ) across layers. Shaded regions indicate the standard error of the mean (SEM).

Size	Models	en	SFT Languages				Zero-shot		Avg.
Size	Models		ar	fr	hi	pt	id	ja
7B	LLaVA-1.5	35.2	29.9	34.6	28.2	33.7	31.6	32.0	32.2
	+ Std. LoRA	34.9	32.9	36.6	30.2	36.0	32.0	30.9	33.4
	+ Early-Layer FFT	32.6	28.9	32.2	24.1	31.6	34.3	32.0	30.8
	+ ANCHOR	35.0	34.8	39.1	32.6	39.2	33.9	34.4	35.6
13B	LLaVA-1.5	34.8	30.9	35.6	29.2	37.0	33.7	30.9	33.2

Table 1: Performance comparison on multilingual multimodal benchmarks: (a) xMMMU and (b) MaXM. Best results are highlighted in bold .

Size	Models	SFT Languages			Zero-shot			Avg.
Size	Models	ar-EG	es-ES	zh-CN	am-ET	ja-JP	ro-RO	Avg.
7B	LLaVA-1.5	32.5	67.0	48.6	26.5	35.0	51.0	43.4
	+ Std. LoRA	31.5	62.0	44.7	26.1	35.5	46.7	41.1
	+ Early-Layer FFT	29.6	50.3	42.8	27.4	35.5	44.4	38.3
	+ ANCHOR	37.3	69.3	49.1	26.4	36.9	37.4	42.7
13B	LLaVA-1.5	38.4	70.4	54.3	25.2	45.8	57.0	48.5

Table 2: Performance on CVQA. We show six language–region pairs whose languages include both those used in SFT and those unseen during training; full results are reported in Table 8 . The Avg. column reports the average accuracy across the six displayed language–region pairs. Best results are highlighted in bold .

Figure 4: (a) Average visual causal effect ( $\Delta R_{ET}$ and $\Delta\text{Sim}_{en}$ ) across 7 languages. (b) Layer-wise hidden state dynamics in a cross-lingual VQA case, showing how ANCHOR enables visual grounding to override textual priors during translation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Abbreviations
Chinese	zh
Spanish	es
Hindi	hi
Arabic	ar
Russian	ru
Swahili	sw

Appendix

Table 3: Selected languages used in the mechanistic experiments and their corresponding ISO 639-1 codes.

Figure 5: Illustrative example of the input construction. This figure demonstrates the parallel input constructions across 7 structurally diverse target languages. To enforce a strict monolingual context and prevent English prior leakage during inference, both the standardized verification instruction and the ground-truth image caption are translated into the target language and paired with identical visual tokens from the COCO dataset.

Figure 6: (a) Original segmentation masks and semantic labels from the COCO dataset. (b) The mapped $24\times 24$ visual patches and their corresponding labels.

Figure 7: Visual comparison of intervention conditions. Left: Normal RGB input with intact semantic content. Right: Gaussian noise with matched per-channel statistics ( $\mu$ , $\sigma$ ) but no recognizable objects or scenes.

Model	$\Delta R_{\text{ET}}\uparrow$	$\Delta\text{Sim}_{\text{en}}\uparrow$
Qwen2.5-VL 3B	$4.12\times 10^{-3}$	$-2.31\times 10^{-4}$
Qwen2.5-VL 7B	$5.34\times 10^{-3}$	$-4.51\times 10^{-3}$
Gemma-3-4B	$-1.13\times 10^{-2}$	$-4.74\times 10^{-4}$
Gemma-3-13B	$6.95\times 10^{-3}$	$7.37\times 10^{-3}$
LLaVA-NeXT-7B	$-3.19\times 10^{-3}$	$-4.71\times 10^{-4}$
LLaVA-NeXT-13B	$8.61\times 10^{-3}$	$-8.28\times 10^{-3}$

Appendix

Table 4: Visual Causal Effect on English Translation Ratio (

\Delta R_{\text{ET}}

) and English Similarity (

\Delta\text{Sim}_{\text{en}}

Backbone	Size	$\boldsymbol{\Delta}$ En	$\boldsymbol{\Delta}$ SFT Avg.	$\boldsymbol{\Delta}$ Zero-shot Avg.	$\boldsymbol{\Delta}$ Avg.
LLaVA-NeXT	7B	+0.4	+3.1	+1.7	+2.3
LLaVA-NeXT	13B	+0.5	+4.0	+1.9	+2.9
InternVL3	8B	+0.2	+2.3	+1.1	+1.7
InternVL3	14B	+0.5	+2.6	+1.3	+1.9

Appendix

Table 5: Generalization to newer MLLM backbones on xMMMU. Each entry reports the accuracy difference in percentage points between ANCHOR and the corresponding Std. LoRA baseline (

\text{ANCHOR}-\text{Std. LoRA}

). SFT Avg. and Zero-shot Avg. are macro-averages over their respective language groups, while Avg. is the macro-average over all seven evaluated language subsets, including English.

Backbone	Size	$\boldsymbol{\Delta}$ En	$\boldsymbol{\Delta}$ SFT Avg.	$\boldsymbol{\Delta}$ Zero-shot Avg.	$\boldsymbol{\Delta}$ Avg.
LLaVA-NeXT	7B	+0.8	+3.4	+1.2	+2.4
LLaVA-NeXT	13B	+1.0	+3.8	+1.3	+2.7
InternVL3	8B	+0.5	+2.6	+0.8	+1.8
InternVL3	14B	+0.6	+3.0	+0.9	+2.1

Appendix

Table 6: Generalization to newer MLLM backbones on MaXM. Each entry reports the accuracy difference in percentage points between ANCHOR and the corresponding Std. LoRA baseline (

\text{ANCHOR}-\text{Std. LoRA}

). SFT Avg. and Zero-shot Avg. are macro-averages over their respective language groups, while Avg. is the macro-average over all seven evaluated language subsets, including English.

Backbone	Size	$\boldsymbol{\Delta}$ SFT Avg.	$\boldsymbol{\Delta}$ Zero-shot Avg.	$\boldsymbol{\Delta}$ Avg.
LLaVA-NeXT	7B	+3.4	+1.4	+2.1
LLaVA-NeXT	13B	+3.8	+1.8	+2.5
InternVL3	8B	+2.6	+1.0	+1.6
InternVL3	14B	+2.9	+1.1	+1.7

Appendix

Table 7: Generalization to newer MLLM backbones on CVQA. Each entry reports the accuracy difference in percentage points between ANCHOR and the corresponding Std. LoRA baseline (

\text{ANCHOR}-\text{Std. LoRA}

). SFT Avg. and Zero-shot Avg. are macro-averages over their respective language–region groups, while Avg. is the macro-average over all 39 language–region pairs.

Models	ar-EG	es-AR	es-CL	es-CO	es-EC	es-ES	es-MX	es-UY	hi-IN	pt-BR
LLaVA-1.5-7B	32.5	54.3	60.7	56.4	53.6	67.0	48.3	43.5	40.3	57.4
+ Std. LoRA	31.5	51.7	56.4	55.2	49.5	62.0	46.1	40.6	38.8	56.3
+ Early-Layer FFT	29.6	41.5	44.9	46.5	42.3	50.3	43.7	37.8	37.3	45.8
+ ANCHOR	37.3	55.8	60.3	58.9	53.7	69.3	51.2	46.7	42.5	59.0
LLaVA-1.5-13B	38.4	57.4	62.0	61.4	53.6	70.4	51.1	39.7	49.3	63.4
+ Std. LoRA	36.0	54.7	57.7	61.4	55.0	67.9	50.8	39.7	46.8	59.9

Appendix

Table 8: CVQA benchmark performance. Pink columns denote language–region pairs whose languages are included in SFT, while Cyan columns denote languages unseen during training. The Avg. column reports the macro-average accuracy across all 39 pairs.

$L_{early}$	xMMMU	MaXM
6	32.8	28.5
8	34.2	30.7
10	35.6	32.4
12	34.7	31.8
14	33.5	30.2
16	32.1	28.9

Appendix

Table 9: Ablation study on the target layer

L_{early}

. xMMMU and MaXM results are macro-averaged across all evaluated languages. All experiments are conducted on LLaVA-1.5-7B.

Intervention Scope	xMMMU	MaXM
Full SFT + PVA	32.8	27.5
ANCHOR (Ours)	35.6	32.4

Appendix

Table 10: Ablation on intervention scope using LLaVA-1.5-7B. Full-parameter fine-tuning augmented with PVA at layer 10 yields lower aggregate performance than the early-layer intervention on both benchmarks, suggesting that targeted early-layer updates better preserve cross-lingual generalization in this setting.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,084 papers

LanteRn: Latent Visual Structured Reasoning
Mar 26, 2026André G. Viveiros, Nuno Gonçalves, Matthias Lindemann +1Latent Visual Reasoning VLM Reasoning
Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University
Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
Jun 24, 2026Xi Xiao, Chen Liu, Chih-Ting Liao +9Multimodal Large Language Models Visual Grounding
UAB · Yale · UNSW +6
Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs
Feb 9, 2026Siqu Ou, Tianrui Wan, Zhiyuan Zhao +2Reasoning for VLMs Attention Intervention in VLMs
TeleAI · Shanghai Jiao Tong University · Northwestern Polytechnical University

Mar 26, 2026J/K move · Enter open · S save

LanteRn: Latent Visual Structured Reasoning

André G. Viveiros, Nuno Gonçalves, Matthias Lindemann, André Martins

Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University

Latent Visual Reasoning VLM Reasoning Large Vision-Language Models Visual Grounding

Open paper AskPDF arXiv

While language reasoning models excel in many tasks, visual reasoning remains challenging for current large multimodal models (LMMs). As a result, most LMMs default to verbalizing perceptual content into text, a strong limitation for tasks requiring fine-grained spatial and visual understanding. While recent approaches take steps toward thinking with images by invoking tools or generating intermediate images, they either rely on external modules, or incur unnecessary computation by reasoning directly in pixel space. In this paper, we introduce LanteRn, a framework that enables LMMs to interleave language with compact latent visual representations, allowing visual reasoning to occur directly in latent space. LanteRn augments a vision-language transformer with the ability to generate and attend to continuous visual thought embeddings during inference. We train the model in two stages: supervised fine-tuning to ground visual features in latent states, followed by reinforcement learning to align latent reasoning with task-level utility. We evaluate LanteRn on three perception-centric benchmarks (VisCoT, V*, and Blink), observing consistent improvements in visual grounding and fine-grained reasoning. These results suggest that internal latent representations provide a promising direction for more efficient multimodal reasoning.

Abstract

Multimodal Large Language Models

Explore similar work

LanteRn: Latent Visual Structured Reasoning

Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs