When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
Organizations: Institute of Computing Technology, Chinese Academy of Sciences · Department of Computer Science, Aberystwyth University · Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a diverse range of multimodal tasks. However, these models suffer from a core problem known as text dominance: they depend heavily on text for their inference, while underutilizing other modalities. While prior work has acknowledged this phenomenon in vision-language tasks, often attributing it to data biases or model architectures. In this paper, we conduct the first systematic investigation of text dominance across diverse data modalities, including images, videos, audio, time-series, and graphs. To measure this imbalance, we propose two evaluation metrics: the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI). Our comprehensive analysis reveals that text dominance is both significant and pervasive across all tested modalities. Our in-depth analysis identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. Furthermore, we propose a simple token compression method that effectively rebalances model attention. Applying this method to LLaVA-7B, for instance, drastically reduces its MDI from 10.23 to a well-balanced value of 0.86. Our analysis and methodological framework offer a foundation for the development of more equitable and comprehensive multimodal language models.
Figures & tables
| Model | Modality | Dataset | Early | Middle | Late | |||
| MDI | AEI | MDI | AEI | MDI | AEI | |||
| Qwen2.5-VL-7B | Image | MMMU_Pro | 2.26 | 1.42 | 21.12 | 10.86 | 33.10 | 14.24 |
| Qwen2.5-VL-32B | 3.84 | 2.82 | 54.96 | 21.88 | 26.03 | 13.95 | ||
| Qwen2.5-VL-72B | 9.33 | 6.15 | 92.21 | 60.43 | 24.46 | 14.60 | ||
| LLaVA-1.5-7B | 1.91 | 1.55 | 8.07 | 2.92 | 15.63 | 3.36 | ||
| Kimi-VL-A3B-Instruct | 2.27 | 3.91 | 3.78 | 2.99 | 28.39 | 2.59 | ||
| Model | Reduction | Method | Early | Middle | Late | Acc | Time | |||
| MDI | AEI | MDI | AEI | MDI | AEI | |||||
| LLaVA-7B | 0 % | No reduction | 1.91 | 1.55 | 8.07 | 2.92 | 15.63 | 3.36 | 33.23 | 0.42 |
| 50 % | Random | 1.38 | 1.30 | 4.14 | 1.84 | 8.98 | 2.28 | 31.30 | 0.15 | |
| FastV | 1.28 | 1.23 | 4.11 | 1.84 | 8.91 | 2.23 | 33.23 | 0.17 | ||
| FasterVLM | 1.21 | 1.11 | 4.08 | 1.83 | 8.84 | 2.14 | 33.30 | 0.15 | ||
| 75 % | Random | 0.96 | 1.03 | 2.38 | 1.41 | 5.74 | 1.64 | 31.79 | 0.14 | |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Modality | Model | Total Attn | Avg Attn/Tok | Avg Token Count | ||||
| Text | Non-t | Text | Non-t | Text | Non-t | Out | ||
| Image | Qwen2.5-VL-7B | 0.049 | 0.951 | 6.0e-4 | 2.7e-4 | 118.6 | 3584.0 | 208.0 |
| Kimi-VL-A3B | 0.715 | 0.285 | 1.1e-2 | 1.3e-3 | 92.6 | 413.6 | 138.7 | |
| LLaVA-1.5-7B | 0.175 | 0.825 | 2.3e-3 | 1.4e-3 | 117.3 | 576.0 | 67.7 | |
| [1pt/5pt] Video | Qwen2.5-VL-7B | 0.092 | 0.908 | 2.7e-3 | 2.5e-4 | 34.8 | 3584.0 | 77.6 |
| VideoLLaMA3-7B | 0.109 | 0.891 | 8.6e-3 | 4.5e-4 | 14.1 | 2288.4 | 15.8 | |
| Modality | Model | Total Attn | Avg Attn/Tok | Avg Token Count | ||||
| Text | Non-t | Text | Non-t | Text | Non-t | Out | ||
| Image | Qwen2.5-VL-7B | 0.348 | 0.652 | 3.8e-3 | 1.8e-4 | 118.6 | 3584.0 | 208.0 |
| Kimi-VL-A3B | 0.547 | 0.453 | 8.1e-3 | 2.1e-3 | 92.6 | 413.6 | 138.7 | |
| LLaVA-1.5-7B | 0.595 | 0.405 | 7.2e-3 | 7.0e-4 | 117.3 | 576.0 | 67.7 | |
| [1pt/5pt] Video | Qwen2.5-VL-7B | 0.423 | 0.577 | 1.2e-2 | 2.0e-4 | 36.7 | 3584.0 | 86.4 |
| VideoLLaMA3-7B | 0.474 | 0.526 | 3.7e-2 | 2.6e-4 | 14.8 | 2283.0 | 15.8 | |
| Modality | Model | Total Attn | Avg Attn/Tok | Avg Token Count | ||||
| Text | Non-t | Text | Non-t | Text | Non-t | Out | ||
| Image | Qwen2.5-VL-7B | 0.456 | 0.544 | 5.0e-3 | 1.5e-4 | 118.6 | 3584.0 | 208.0 |
| Kimi-VL-A3B | 0.474 | 0.526 | 7.0e-3 | 2.5e-3 | 92.6 | 413.6 | 138.7 | |
| LLaVA-1.5-7B | 0.716 | 0.284 | 8.6e-3 | 4.9e-4 | 117.3 | 576.0 | 67.7 | |
| [1pt/5pt] Video | Qwen2.5-VL-7B | 0.448 | 0.553 | 1.3e-2 | 1.5e-4 | 33.8 | 3584.0 | 83.8 |
| VideoLLaMA3-7B | 0.491 | 0.509 | 3.7e-2 | 2.4e-4 | 15.2 | 2338.9 | 14.1 | |
| Condition | Removed | Accuracy (%) |
| Full visual tokens | 0% | 73.4 |
| Informed selection | 50% | 73.8 |
| Informed selection | 75% | 73.6 |
| Random dropping | 75% | 70.6 |
| Informed selection | 90% | 72.6 |
| Condition | Retained | Accuracy (%) | MDI |
| Full video | 100% | 61 | 523.83 |
| Informed selection | 50% | 64 | 271.68 |
| Random selection | 50% | 58 | 313.70 |