TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Andong Hua; Colton Bishop; Igor Mordatch; Arian Hosseini; Jindong Gu; Aleksandra Faust; Rebecca Roelofs; Yao Qin

cs.CLMay 20, 2026

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Authors: Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin

Organizations: University of California, Santa Barbara · Google DeepMind

Abstract

Multimodal large language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks (e.g., MMLU) into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models, suggesting greater robustness in multimodal reasoning across equivalent textual and visual representations. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap, improving cross-modal alignment while preserving strong text-only and vision-language model performance. The project page is available at https://dongxzz.github.io/TokenSwap/.

Figures & tables

Figure 1: A pervasive modality gap exists: all models perform worse on image-interleaved inputs than on text-only inputs, with gaps ranging from 4.2% to 47.4%. Each point represents a model, colored by family, with triangles indicating reasoning models. The diagonal

y=x

denotes equal performance, and dashed lines indicate constant gap levels (4.0%, and 15.0%). All models lie below the diagonal, indicating consistently lower performance under image-interleaved inputs. The right panel zooms into high-performing models. Gemini-3-Flash achieves the smallest gap (4.2%). Reasoning models typically exhibit smaller gaps (4.0%–15.0%), while non-reasoning models tend to have substantially larger gaps.

Figure 2: TokenSwap constructs semantically aligned image-interleaved questions, where Step 1 panel shows the original MMLU question and the Final Benchmark panel shows its image-interleaved counterpart. We first extract visualizable concepts and generate or retrieve images representing them, followed by rigorous filtering to ensure validity, task relevance, and semantic consistency, resulting in semantically aligned image-interleaved questions.

Figure 3: Reasoning models exhibit a smaller modality gap, while prompting strategies fail to reduce it. (a) Distribution of modality gap for reasoning and non-reasoning models. (b) We compare different prompting strategies. Each pair shows text accuracy (hollow) and image-interleaved accuracy (filled), with the modality gap

\Delta

indicated.

Figure 4: Scaling provides limited improvement in modality gap, and generation-based benchmarks exhibit smaller gaps. (a) plots absolute modality gap versus training FLOPs. FLOPs are collected for all open-source non-reasoning models with publicly available training compute estimates, based on original papers and Epoch AI 2 2 2 https://epoch.ai/data/ai-models/ . (b) Each blue/red pair corresponds to the same model evaluated with retrieved versus generated images, sharing identical text accuracy since only the image source differs. Red points consistently lie above blue ones, indicating that generated images yield higher image-interleaved accuracy and smaller modality gaps.

Figure 5: TokenSwap-Bench is complementary to the existing benchmark. We plot 8 overlapping models, with SEAM [ 37 ] results taken from the original paper; numerical values are provided in the Appendix B.7 . We observe high Pearson correlation with SEAM in both text accuracy (

r=0.811

) and image-interleaved accuracy (

r=0.914

), but near-zero correlation in modality gap (

r=-0.028

Figure 6: TokenSwap reduces the modality gap while preserving text performance. We compare Text training (blue) and TokenSwap training (red) across both pre-training and post-training, using either generated (Gen) or retrieved (Ret) images. Text training includes Gen and Ret variants, as it is paired with the TokenSwap variants constructed from the same samples. Each pair shows text accuracy (hollow) and image-interleaved accuracy (filled), with the modality gap

\Delta

indicated.

Method	Accuracy
Baseline	0.9003
Text Training	0.8947
TokenSwap Training	0.9270

Table 1: TokenSwap training improves OCR performance. We report accuracy on the IIIT 5K-Word dataset.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Text	Image	GPT Proxy Gap	Gemini Proxy Gap
Gemini-3-Flash	0.983	0.906	0.077	0.042
Claude-Opus-4.6	0.982	0.901	0.081	0.057
GPT-5.1	0.982	0.897	0.085	0.077
Qwen3-VL-4B-Thinking	0.859	0.729	0.130	0.109
GPT-4.1	0.886	0.726	0.160	0.164
Qwen3-VL-4B	0.704	0.509	0.195	0.184

Appendix

Table 2: Detailed results using GPT-5-mini and Gemini-2.0-Flash as filtering proxies.

Figure 7: Example of a multiple-choice question used in the human study.

Figure 8: Images in few-shot demonstrations. Images are generated by Gemini to represent corresponding textual concepts. In the image-interleaved version, these textual concepts are replaced with the corresponding images.

Figure 9: Examples of retrieved and generated images for the same concepts. Each column corresponds to a replaced word, with a generated image shown in the top row and a retrieved image shown in the bottom row.

Model	Text	Image	GSM8K Gap	MMLU Gap
Gemini-3-Flash	1.000	0.945	0.055	0.042
Claude-Opus-4.6	1.000	0.959	0.041	0.057
GPT-5.1	1.000	0.945	0.055	0.077
Qwen3-VL-4B-Thinking	0.973	0.753	0.220	0.109
GPT-4.1	0.986	0.795	0.191	0.164
Qwen3-VL-4B	0.904	0.575	0.329	0.184

Appendix

Table 3: Modality gap on GSM8K compared with TokenSwap-Bench .

Figure 10: Strong scaling fit but limited improvement in relative modality gap. While the linear fit achieves a high

R^{2}=0.811

, indicating a consistent scaling trend, the slope remains small, suggesting limited gains from increasing training compute. Each point represents a model, plotting relative modality gap against estimated training FLOPs.

Figure 11: Per-subject accuracy on MMLU comparing text-only and image-interleaved inputs.

Figure 12: Distribution of relative modality gap for reasoning and non-reasoning models.

Figure 13: The modality gap generally increases with the number of image replacements, while the white-image gap remains near zero. We average results across nine models and group samples by the number of images. The natural-image gap ( Text

-

Image ) generally increases as more textual concepts are replaced with images, whereas the white-image gap ( Text

-

White Image ) remains close to zero, indicating that the trend cannot be explained by additional visual tokens alone. The bottom panel shows the number of samples in each group.

Model	Text	White Image	Image	Text $-$ White Image	Text $-$ Image (Modality Gap)
Gemini-3-Flash	0.925	0.935	0.884	$-0.010$	0.041
Claude-Opus-4.6	0.984	0.983	0.926	0.001	0.058
GPT-5.1	0.980	0.981	0.903	$-0.001$	0.077
Qwen3-VL-4B-Thinking	0.889	0.890	0.780	$-0.001$	0.109
GPT-4.1	0.925	0.923	0.761	0.002	0.164
Qwen3-VL-4B	0.707	0.714	0.523	$-0.007$	0.184

Appendix

Table 4: Detailed results for the white-image control.

Model	Text	Rendered	Image	Text $\rightarrow$ Rendered	Rendered $\rightarrow$ Image
Gemini-3-Flash	0.925	0.917	0.884	19.5%	80.5%
Claude-Opus-4.6	0.984	0.976	0.926	13.8%	86.2%
GPT-5.1	0.980	0.976	0.903	5.2%	94.8%
Qwen3-VL-4B-Thinking	0.889	0.876	0.780	11.9%	88.1%
GPT-4.1	0.925	0.906	0.761	11.6%	88.4%
Qwen3-VL-4B	0.707	0.690	0.523	9.2%	90.8%

Appendix

Table 5: Detailed results for the rendered-text-image control.

Model	Text	Image	Gap
GPT-4.1	0.925	0.761	+0.164
GPT-4.1-mini	0.892	0.686	+0.206
GPT-4o	0.913	0.693	+0.220
GPT-4o-mini	0.832	0.545	+0.288
GPT-5	0.952	0.856	+0.096
GPT-5.1	0.980	0.903	+0.077

Appendix

Table 6: Detailed numerical results for TokenSwap-Bench .

Model	SEAM		Ours		Gap
	Text	Image	Text	Image	SEAM	Ours
GPT-4o	0.635	0.482	0.913	0.693	+0.153	+0.220
GPT-4o-mini	0.555	0.411	0.832	0.545	+0.144	+0.288
InternVL2.5-78B	0.448	0.414	0.920	0.600	+0.034	+0.320
InternVL2.5-8B	0.324	0.337	0.756	0.382	-0.013	+0.374
InternVL3-78B	0.525	0.427	0.943	0.685	+0.098	+0.258

Appendix

Table 7: Comparison of modality gap between SEAM and our evaluation on overlapping models.

Training Data	Text	Image	Gap
Baseline	0.741	0.438	0.303
Text Training	0.753	0.429	0.324
TokenSwap Training	0.748	0.518	0.230

Appendix

Table 8: TokenSwap training results on InternVL3-8B-Pretrained.

Figure 14: Applying TokenSwap to the Magpie-Pro dataset. We construct image-interleaved instructions by replacing key textual concepts in the original queries with semantically aligned images. The red text highlights the concepts that are replaced by images.

	GQA [ 19 ]	TextVQA [ 35 ]	MME [ 14 ]
Baseline	63.8	62.5	1542.4
pre-train + Generate	63.5	61.7	1535.9
pre-train + Retrieval	63.7	61.9	1587.0
post-train + Generate	63.9	62.3	1560.0
post-train + Retrieval	63.8	62.0	1544.1

Appendix

Table 9: Evaluation on standard vision-language benchmarks (GQA, TextVQA, and MME-Peception). TokenSwap training achieves comparable performance to the baseline across all settings, indicating no noticeable degradation.

Figure 16: Applying TokenSwap to the IIIT 5K-Word dataset. We transform standard image–text pairs into image-interleaved instructions by replacing target words with images. The queries are generated from the ground-truth text using an LLM, where the OCR word is substituted with its image representation.

Training Domain	Natural Image-Interleaved Accuracy	OCR Accuracy
No Train	0.4466	0.9003
Natural Image	0.5132	0.9196
OCR Image	0.4393	0.9270

Appendix

Table 10: TokenSwap training is sensitive to domain alignment . We consider two image domains: natural images and OCR. We use image-interleaved performance on TokenSwap-Bench to represent the natural-image domain, as it involves natural images rather than OCR-style text images. TokenSwap improves performance only when the training and evaluation domains are aligned.

Explore similar work

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1
Attending to Multimodal Generation One Token at a Time
Jul 4, 2026Varun Gupta, Vineet Gandhi, Makarand TapaswiMultimodal Large Language Models Multimodal Generation
CVIT, IIIT Hyderabad
MIBench: Evaluating LMMs on Multimodal Interaction
Mar 13, 2026Yu Miao, Zequn Yang, Yake Wei +5Multimodal Large Language Models Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China +1

Mar 10, 2026J/K move · Enter open · S save

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Kaiser Sun, Xiaochuang Yuan, Hongjun Liu, Chen Zhao+3

Johns Hopkins University · Amazon · New York University+1

Multimodal Large Language Models Modality Gap in VLMs Multimodal Reasoning Document Image Understanding Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.

Abstract

Explore similar work

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Attending to Multimodal Generation One Token at a Time

MIBench: Evaluating LMMs on Multimodal Interaction