Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.
Figures & tables
Figure 1: Humans read text visually across diverse sources (e.g., books, webpages, documents), whereas MLLMs may produce different predictions for identical content presented in different formats.
Figure 2: Diagnosing the modality gap in visual text understanding. We evaluate MLLMs across five input modes, including pure text, rendered text images, real-world visual text, and two OCR-based diagnostic settings ( OCR-1P and OCR-2P ). Our error taxonomy reveals that while image modality amplifies reading and calculation errors, it leaves underlying reasoning capabilities largely intact but also reduces the likelihood of triggering a reasoning chain. We further propose remedies to bridge the performance gap, including rendering specification control, resolution-aware preprocessing, and LM-only self-distillation.
Table 1: Performance of different input modalities on four synthetic-image datasets (MMLU, ARC, GPQA, GSM8K). Cell shading indicates the performance difference relative to Pure Text : red indicates the modality outperforms text, blue indicates it underperforms text, and stronger shading reflects larger gaps. All HumanEval results throughout this paper are reported in Appendix E due to small dataset size and high pass@1 variance ( Chen et al., 2021 ) .
Table 2: Performance of different input modalities on two real-image datasets Qasper and SQuAD v2. Cell shading indicates the performance difference relative to Pure Text : red indicates the modality outperforms text, blue indicates it underperforms text. Image-based inputs such as Pure Image and Instr.+Image outperform Pure Text on Qasper , indicating that MLLMs can benefit from naturally occurring visual text. 4 4 4 OCR-2P is omitted for QASPER due to the computational cost of running OCR on extremely long documents. Pixtral-12B is excluded from QASPER due to insufficient context length.
Figure 3: Performance difference between Pure Text and Pure Image with different renderings. Each point is a dataset–model pair. Handwriting consistently causes larger negative drops than all the other settings.
Figure 4: Performance vs. image resolution scale on ARC. The black vertical dashed line indicates the point where Pure Image consumes the same FLOPs as Pure Text . Other datasets are in Appendix F .
Figure 5: Error taxonomy derived from grounded-theory coding of 4,195 errors across all models and datasets. Percentages indicate overall frequency.
Figure 6: Distribution of error categories across input modes (left) and datasets (right). Cell values show raw counts and within-column percentages.
Table 3: Self-distillation results across two architectures and two training datasets. All configurations use Pure-Image-input + Text-mode-Teacher with LM-only LoRA. ∣Δ∣ : text–image gap (baseline → distilled). ⋆ MMMU-Pro is a fully held-out benchmark that uses different evaluation protocols, where T and I corresponds to standard and vision modes in MMMU-Pro evaluation.
Table 4: Ablation on Qwen3-VL-8B + GSM8K: filtered vs. unfiltered traces, and ViT+LM / ViT-only / LM-only adaptation. LM-only with filtered traces is the canonical recipe. ∣Δ∣ : text–image gap.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : An example input of MMLU.
Figure S2 : An example input of ARC.
Figure S3 : An example input of GPQA.
Figure S4 : An example input of HumanEval.
Figure S5 : An example input of GSM8K.
Figure S6 : An example input of SQUAD v2.
Figure S7 : An example input of QASPER.
Figure S8 : Prompt used for OCR-style transcription from images.
Dataset
Domain
Metric
#Samples
Proportion (%)
MMLU
Knowledge (57 subjects)
Acc.
14,042
83.0
ARC
Science reasoning
Acc.
1,172
6.9
GPQA
Graduate-level QA
Acc.
198
1.2
GSM8K
Math reasoning
Acc.
1,319
7.8
HumanEval
Code generation
Pass @ 1
164
1.0
QASPER
Document QA
F1 / Acc.
1,445
—
Appendix
Table S1 : Dataset statistics and composition.
Table S2 : HumanEval (pass@1) results across input modalities. Cell shading indicates performance difference relative to Pure Text : red = outperforms text, blue = underperforms text. Results exhibit high variance due to the small dataset size (164 instances) and the inherent instability of pass@1 estimation from limited samples [ 29 ] . OCR-2P destroys performance for several models, suggesting that OCR strips structural cues (indentation, whitespace) critical for code comprehension.
Figure S9 : Performance vs. image resolution scale on HumanEval (top) and ARC (bottom). The black vertical dashed line indicates the point where Pure Image consumes the same FLOPs as Pure Text .
Note: T : text modality; I : image modality. Accuracy and F1 are percentages. QASPER (Full) : all questions; QASPER (Vis. Subset) : figure/table-dependent questions.
Dataset
InternVL3-8B
Pixtral-12B
Qwen2.5-7B-VL
Qwen2.5-32B-VL
Qwen3-VL-8B
Gemma3-4B
ARC
3.68
13.42
3.92
2.45
3.81
4.68
GSM8K
1.98
13.65
4.01
1.56
2.82
2.23
HumanEval
1.52
6.21
1.80
1.36
1.62
1.72
MathVista
3.03
14.67
4.39
1.68
3.28
4.52
MMLU
3.45
12.78
–
2.28
2.74
4.11
SQuAD
8.64
20.70
12.37
7.54
8.98
8.98
Appendix
Table S4 : Image-to-Text FLOPs ratio across models and datasets. A ratio of k indicates that processing the visual input requires k× the FLOPs of processing the equivalent text input. Missing entries indicate that the model does not support the corresponding configuration.
Figure S10 : Correlation between OCR quality and task accuracy under the OCR-2Pass setting. Each point represents one model–dataset pair. The dashed line shows the linear trend.
Figure S11 : Example of reasoning collapse on GSM8K (Qwen3-VL-8B). In text mode, the model produces detailed step-by-step reasoning (618 characters) and arrives at the correct answer. In image mode, the model skips all reasoning and outputs only an incorrect answer (12 characters).
Figure S12 : Prompt used for error classification in the inter-annotator agreement study.
Annotator Pair
Cohen’s κ
Raw Agreement
Human vs. GPT-5.2
0.57 (Moderate)
75.0%
Human vs. Claude
0.69 (Substantial)
81.8%
GPT-5.2 vs. Claude
0.64 (Substantial)
79.5%
Three-way agreement
—
68.2%
Appendix
Table S5: Inter-annotator agreement for error classification. Cohen’s κ values indicate moderate to substantial agreement across all annotator pairs.
Multimodal large language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks (e.g., MMLU) into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models, suggesting greater robustness in multimodal reasoning across equivalent textual and visual representations. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap, improving cross-modal alignment while preserving strong text-only and vision-language model performance. The project page is available at https://dongxzz.github.io/TokenSwap/.
Andong Hua, Colton Bishop, Igor Mordatch +5
University of California, Santa Barbara · Google DeepMind
Multimodal large language models (MLLMs) are increasingly provided with contextual evidence in heterogeneous forms: as a text passage, as a rendered image of the same passage, or as both together. However, it remains unclear how consistently these surface forms are processed, especially when the evidence conflicts with the model's parametric knowledge. We study modality robustness under knowledge conflict across 13 MLLMs and two datasets, and find them far from robust. (1) Contrary to common belief, models favor a context that contradicts parametric knowledge more readily in image form than in text form; (2) when a contradicting text and image are presented together, the preferred modality is essentially arbitrary, varying with input order, model, and dataset. We further demonstrate that this instability has practical consequences: it degrades performance in multimodal RAG and can be exploited by adversarial attacks. To alleviate this brittleness, we examine several simple techniques---prompting, steering, supervised fine-tuning (SFT), and direct preference optimization; the majority prove ineffective, whereas SFT achieves moderate success. We therefore call for greater awareness of this inconsistency and argue that it is fundamental, demanding attention at multiple training stages.
Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.
Hong-Han Wang, Yuntao Wang, Hu Ding
University of Science and Technology of China, Hefei, China