Organizations: Institute of Computing Technology, Chinese Academy of Sciences · Department of Computer Science, Aberystwyth University · Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a diverse range of multimodal tasks. However, these models suffer from a core problem known as text dominance: they depend heavily on text for their inference, while underutilizing other modalities. While prior work has acknowledged this phenomenon in vision-language tasks, often attributing it to data biases or model architectures. In this paper, we conduct the first systematic investigation of text dominance across diverse data modalities, including images, videos, audio, time-series, and graphs. To measure this imbalance, we propose two evaluation metrics: the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI). Our comprehensive analysis reveals that text dominance is both significant and pervasive across all tested modalities. Our in-depth analysis identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. Furthermore, we propose a simple token compression method that effectively rebalances model attention. Applying this method to LLaVA-7B, for instance, drastically reduces its MDI from 10.23 to a well-balanced value of 0.86. Our analysis and methodological framework offer a foundation for the development of more equitable and comprehensive multimodal language models.
Figures & tables
Figure 1: Overview of text dominance in MLLMs. Each modality is tokenized and fed to the LLM backbone. The thinner arrows from non-text tokens reflect their lower per-token attention: despite comprising the majority of input tokens, non-text modalities receive substantially less attention per token than text.
Model
Modality
Dataset
Early
Middle
Late
MDI
AEI
MDI
AEI
MDI
AEI
Qwen2.5-VL-7B
Image
MMMU_Pro
2.26
1.42
21.12
10.86
33.10
14.24
Qwen2.5-VL-32B
3.84
2.82
54.96
21.88
26.03
13.95
Qwen2.5-VL-72B
9.33
6.15
92.21
60.43
24.46
14.60
LLaVA-1.5-7B
1.91
1.55
8.07
2.92
15.63
3.36
Kimi-VL-A3B-Instruct
2.27
3.91
3.78
2.99
28.39
2.59
Table 1: MDI and AEI across models, modalities, and benchmarks. Image and video rows use the original input without replication. The notation ×n denotes the replication factor applied to non-textual tokens. Early, Middle, and Late report statistics from the first two, middle two, and last two model layers, respectively. GraphGPT-Eval and TS-Reasoning abbreviate GraphGPT-Eval-Instruction and TimeSeries-Reasoning. Rows with replication factors ( × 5, × 10) are analyzed in § 4 .
Figure 2: Text dominance across MLLMs of different modalities. The dashed line marks MDI = 1; points to its right indicate text-dominant attention. The graph modality (MDI < 1) is a boundary case discussed in § 4.3 , where compact, information-dense tokens reverse the dominant modality.
Figure 3: Accuracy under multimodal ( With Image ) and text-only ( Text Only ) inputs across MDI buckets on LLaVA-1.5-7B / MMMU-Pro. The accuracy gap Δ increases with MDI, indicating that the model under-attends to images most severely on samples where visual information is most critical for correct prediction.
Figure 4: MDI and AEI of audio and time-series models under × 1, × 5, and × 10 token replication. Even at × 1, both models show AEI > 1 in late layers, suggesting that task formulation contributes to text dominance beyond token scaling alone.
Model
Reduction
Method
Early
Middle
Late
Acc
Time
MDI
AEI
MDI
AEI
MDI
AEI
LLaVA-7B
0 %
No reduction
1.91
1.55
8.07
2.92
15.63
3.36
33.23
0.42
50 %
Random
1.38
1.30
4.14
1.84
8.98
2.28
31.30
0.15
FastV
1.28
1.23
4.11
1.84
8.91
2.23
33.23
0.17
FasterVLM
1.21
1.11
4.08
1.83
8.84
2.14
33.30 ↑
0.15
75 %
Random
0.96
1.03
2.38
1.41
5.74
1.64
31.79
0.14
Table 2: Effect of token-reduction methods and ratios on MDI, AEI, accuracy, and latency (LLaVA-1.5 on MMMU-Pro). 0% denotes the original setting. Under each non-zero ratio, we compare Random, FastV, and FasterVLM. ↑ indicates FasterVLM outperforms both Random and FastV in accuracy.
Figure 5: Attention maps under 50% token compression on two representative examples. From left to right: input image, baseline attention map, and attention map with 50% compression. In both cases, the baseline prediction is incorrect, while token compression yields the correct answer.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Modality
Model
Total Attn
Avg Attn/Tok
Avg Token Count
Text
Non-t
Text
Non-t
Text
Non-t
Out
Image
Qwen2.5-VL-7B
0.049
0.951
6.0e-4
2.7e-4
118.6
3584.0
208.0
Kimi-VL-A3B
0.715
0.285
1.1e-2
1.3e-3
92.6
413.6
138.7
LLaVA-1.5-7B
0.175
0.825
2.3e-3
1.4e-3
117.3
576.0
67.7
[1pt/5pt] Video
Qwen2.5-VL-7B
0.092
0.908
2.7e-3
2.5e-4
34.8
3584.0
77.6
VideoLLaMA3-7B
0.109
0.891
8.6e-3
4.5e-4
14.1
2288.4
15.8
Appendix
Table 3: Attention and token statistics for the first two layers .
Modality
Model
Total Attn
Avg Attn/Tok
Avg Token Count
Text
Non-t
Text
Non-t
Text
Non-t
Out
Image
Qwen2.5-VL-7B
0.348
0.652
3.8e-3
1.8e-4
118.6
3584.0
208.0
Kimi-VL-A3B
0.547
0.453
8.1e-3
2.1e-3
92.6
413.6
138.7
LLaVA-1.5-7B
0.595
0.405
7.2e-3
7.0e-4
117.3
576.0
67.7
[1pt/5pt] Video
Qwen2.5-VL-7B
0.423
0.577
1.2e-2
2.0e-4
36.7
3584.0
86.4
VideoLLaMA3-7B
0.474
0.526
3.7e-2
2.6e-4
14.8
2283.0
15.8
Appendix
Table 4: Attention and token statistics for the middle two layers .
Modality
Model
Total Attn
Avg Attn/Tok
Avg Token Count
Text
Non-t
Text
Non-t
Text
Non-t
Out
Image
Qwen2.5-VL-7B
0.456
0.544
5.0e-3
1.5e-4
118.6
3584.0
208.0
Kimi-VL-A3B
0.474
0.526
7.0e-3
2.5e-3
92.6
413.6
138.7
LLaVA-1.5-7B
0.716
0.284
8.6e-3
4.9e-4
117.3
576.0
67.7
[1pt/5pt] Video
Qwen2.5-VL-7B
0.448
0.553
1.3e-2
1.5e-4
33.8
3584.0
83.8
VideoLLaMA3-7B
0.491
0.509
3.7e-2
2.4e-4
15.2
2338.9
14.1
Appendix
Table 5: Attention and token statistics for the last two layers .
Figure S1: Case study of text dominance on two Biology questions (Qwen2.5-VL-7B, MMMU-Pro). Left : low MDI (2.3), the model attends to the image and answers correctly. Right : high MDI (7.1), the model over-relies on textual priors and misidentifies the labeled structure. The bottom bars show the text vs. image attention share.
Condition
Removed
Accuracy (%)
Full visual tokens
0%
73.4
Informed selection
50%
73.8
Informed selection
75%
73.6
Random dropping
75%
70.6
Informed selection
90%
72.6
Appendix
Table 6: Image compression on 500 MMBench-EN items.
Condition
Retained
Accuracy (%)
MDI
Full video
100%
61
523.83
Informed selection
50%
64
271.68
Random selection
50%
58
313.70
Appendix
Table 7: Video compression on 2,000 randomly sampled MVBench items.
University of Chinese Academy of Sciences, Beijing, China · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China
OLIVES at the Center for Signal and Information Processing CSIP, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA