Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Kaiser Sun; Xiaochuang Yuan; Hongjun Liu; Chen Zhao; Cheng Zhang; Mark Dredze; Fan Bai

cs.CLMar 10, 2026

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Authors: Kaiser Sun, Xiaochuang Yuan, Hongjun Liu, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai

Organizations: Johns Hopkins University · Amazon · New York University · Texas A&M University

Abstract

Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.

Figures & tables

Figure 1: Humans read text visually across diverse sources (e.g., books, webpages, documents), whereas MLLMs may produce different predictions for identical content presented in different formats.

Figure 2: Diagnosing the modality gap in visual text understanding. We evaluate MLLMs across five input modes, including pure text, rendered text images, real-world visual text, and two OCR-based diagnostic settings ( OCR-1P and OCR-2P ). Our error taxonomy reveals that while image modality amplifies reading and calculation errors, it leaves underlying reasoning capabilities largely intact but also reduces the likelihood of triggering a reasoning chain. We further propose remedies to bridge the performance gap, including rendering specification control, resolution-aware preprocessing, and LM-only self-distillation.

Panel of Table 1 — Table 1: Performance of different input modalities on four synthetic-image datasets (MMLU, ARC, GPQA, GSM8K). Cell shading indicates the performance difference relative to Pure Text : red indicates the modality outperforms text, blue indicates it underperforms text, and stronger shading reflects larger gaps. All HumanEval results throughout this paper are reported in Appendix E due to small dataset size and high pass@1 variance ( Chen et al., 2021 ) .

Panel of Table 2 — Table 2: Performance of different input modalities on two real-image datasets Qasper and SQuAD v2. Cell shading indicates the performance difference relative to Pure Text : red indicates the modality outperforms text, blue indicates it underperforms text. Image-based inputs such as Pure Image and Instr.+Image outperform Pure Text on Qasper , indicating that MLLMs can benefit from naturally occurring visual text. 4 4 4 OCR-2P is omitted for QASPER due to the computational cost of running OCR on extremely long documents. Pixtral-12B is excluded from QASPER due to insufficient context length.

Figure 3: Performance difference between Pure Text and Pure Image with different renderings. Each point is a dataset–model pair. Handwriting consistently causes larger negative drops than all the other settings.

Figure 4: Performance vs. image resolution scale on ARC. The black vertical dashed line indicates the point where Pure Image consumes the same FLOPs as Pure Text . Other datasets are in Appendix F .

Figure 5: Error taxonomy derived from grounded-theory coding of 4,195 errors across all models and datasets. Percentages indicate overall frequency.

Panel of Figure 6 — Figure 6: Distribution of error categories across input modes (left) and datasets (right). Cell values show raw counts and within-column percentages.

Panel of Table 3 — Table 3: Self-distillation results across two architectures and two training datasets. All configurations use Pure-Image-input + Text-mode-Teacher with LM-only LoRA. $|\Delta|$ : text–image gap (baseline $\to$ distilled). ⋆ MMMU-Pro is a fully held-out benchmark that uses different evaluation protocols, where T and I corresponds to standard and vision modes in MMMU-Pro evaluation.

Panel of Table 4 — Table 4: Ablation on Qwen3-VL-8B + GSM8K: filtered vs. unfiltered traces, and ViT+LM / ViT-only / LM-only adaptation. LM-only with filtered traces is the canonical recipe. $|\Delta|$ : text–image gap.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure S1 : An example input of MMLU.

Panel of Figure 12 — Figure S2 : An example input of ARC.

Panel of Figure 13 — Figure S3 : An example input of GPQA.

Panel of Figure 14 — Figure S4 : An example input of HumanEval.

Panel of Figure 15 — Figure S5 : An example input of GSM8K.

Panel of Figure 16 — Figure S6 : An example input of SQUAD v2.

Panel of Figure 17 — Figure S7 : An example input of QASPER.

Panel of Figure 18 — Figure S8 : Prompt used for OCR-style transcription from images.

Dataset	Domain	Metric	#Samples	Proportion (%)
MMLU	Knowledge (57 subjects)	Acc.	14,042	83.0
ARC	Science reasoning	Acc.	1,172	6.9
GPQA	Graduate-level QA	Acc.	198	1.2
GSM8K	Math reasoning	Acc.	1,319	7.8
HumanEval	Code generation	Pass $@$ 1	164	1.0
QASPER	Document QA	F1 / Acc.	1,445	—

Appendix

Table S1 : Dataset statistics and composition.

Panel of Table 20 — Table S2 : HumanEval (pass@1) results across input modalities. Cell shading indicates performance difference relative to Pure Text : red = outperforms text, blue = underperforms text. Results exhibit high variance due to the small dataset size (164 instances) and the inherent instability of pass@1 estimation from limited samples [ 29 ] . OCR-2P destroys performance for several models, suggesting that OCR strips structural cues (indentation, whitespace) critical for code comprehension.

Figure S9 : Performance vs. image resolution scale on HumanEval (top) and ARC (bottom). The black vertical dashed line indicates the point where Pure Image consumes the same FLOPs as Pure Text .

Panel of Table 22 — Note: T : text modality; I : image modality. Accuracy and F1 are percentages. QASPER (Full) : all questions; QASPER (Vis. Subset) : figure/table-dependent questions.

Dataset	InternVL3-8B	Pixtral-12B	Qwen2.5-7B-VL	Qwen2.5-32B-VL	Qwen3-VL-8B	Gemma3-4B
ARC	3.68	13.42	3.92	2.45	3.81	4.68
GSM8K	1.98	13.65	4.01	1.56	2.82	2.23
HumanEval	1.52	6.21	1.80	1.36	1.62	1.72
MathVista	3.03	14.67	4.39	1.68	3.28	4.52
MMLU	3.45	12.78	–	2.28	2.74	4.11
SQuAD	8.64	20.70	12.37	7.54	8.98	8.98

Appendix

Table S4 : Image-to-Text FLOPs ratio across models and datasets. A ratio of

k

indicates that processing the visual input requires

k\times

the FLOPs of processing the equivalent text input. Missing entries indicate that the model does not support the corresponding configuration.

Panel of Figure 24 — Figure S10 : Correlation between OCR quality and task accuracy under the OCR-2Pass setting. Each point represents one model–dataset pair. The dashed line shows the linear trend.

Panel of Figure 25 — Figure S11 : Example of reasoning collapse on GSM8K (Qwen3-VL-8B). In text mode, the model produces detailed step-by-step reasoning (618 characters) and arrives at the correct answer. In image mode, the model skips all reasoning and outputs only an incorrect answer (12 characters).

Panel of Figure 26 — Figure S12 : Prompt used for error classification in the inter-annotator agreement study.

Annotator Pair	Cohen’s $\kappa$	Raw Agreement
Human vs. GPT-5.2	0.57 (Moderate)	75.0%
Human vs. Claude	0.69 (Substantial)	81.8%
GPT-5.2 vs. Claude	0.64 (Substantial)	79.5%
Three-way agreement	—	68.2%

Appendix

Table S5: Inter-annotator agreement for error classification. Cohen’s

\kappa

values indicate moderate to substantial agreement across all annotator pairs.

Explore similar work

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
May 20, 2026Andong Hua, Colton Bishop, Igor Mordatch +5Multimodal Large Language Models Modality Gap in VLMs
University of California, Santa Barbara · Google DeepMind
Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
Sep 1, 2026Jungyeon Lee, Yejin Yoon, Taeuk KimMultimodal Robustness Knowledge Conflicts in Language Models
Hanyang University, Seoul, Republic of Korea
The Alignment Illusion in Multimodal Large Language Models
Sep 24, 2026Hong-Han Wang, Yuntao Wang, Hu DingRepresentational Similarity Analysis Multimodal Large Language Models
University of Science and Technology of China, Hefei, China

May 20, 2026J/K move · Enter open · S save

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini+4

University of California, Santa Barbara · Google DeepMind

Multimodal Large Language Models Modality Gap in VLMs Multimodal Learning Cross-Modal Alignment Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks (e.g., MMLU) into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models, suggesting greater robustness in multimodal reasoning across equivalent textual and visual representations. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap, improving cross-modal alignment while preserving strong text-only and vision-language model performance. The project page is available at https://dongxzz.github.io/TokenSwap/.

Abstract

Explore similar work

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

The Alignment Illusion in Multimodal Large Language Models