High-resolution Multimodal Large Language Models (MLLMs) face prohibitive computational costs during inference due to the explosion of visual tokens. Existing acceleration strategies, such as token pruning or layer sparsity, suffer from severe "backbone dependency", performing well on Vicuna or Mistral architectures (e.g., LLaVA) but causing significant performance degradation when transferred to architectures like Qwen. To address this, we leverage truncated matrix entropy to uncover a universal three-stage inference lifecycle, decoupling visual redundancy into universal Intrinsic Visual Redundancy (IVR) and architecture-dependent Secondary Saturation Redundancy (SSR). Guided by this insight, we propose HalfV, a framework that first mitigates IVR via a unified pruning strategy and then adaptively handles SSR based on its specific manifestation. Experiments demonstrate that HalfV achieves superior efficiency-performance trade-offs across diverse backbones. Notably, on Qwen25-VL, it retains 96.8% performance at a 4.1× FLOPs speedup, significantly outperforming state-of-the-art baselines. Our code is available at https://github.com/civilizwa/HalfV.
Figures & tables
Figure 1: (a): Comparison of token-level methods (HoloV, DART) at a 77.8% pruning ratio and the layer-level method (ShortV) across Vicuna, Mistral and Qwen backbones. Results represent the average relative performance compared to the baseline (%) across POPE, MME, MMBench, and SQA datasets; (b): We find that models with different backbones all exhibit a universal three-stage lifecycle: Modality alignment, global aggregation and visual saturation.
Figure 2: (a) The average truncated matrix entropy for different architectures on GQA dataset. The x-axis is the depth percentage of the layer; (b) The average truncated matrix entropy for different types of tokens on LLaVA-1.5v-7B.; (c) 3D Line Plot of the Average Spectrum of G across Layers on LLaVA-1.5v-7B. Additional results see Appendix ; (d) KL divergence for Vicuna and Qwen2.5 backbones on AI2D dataset.
Figure 3: Comparison of attention heatmaps at the initial and final layers of Stage I on LLaVA-1.5v-7B.
Figure 4: Suppressing only 1% of visual tokens leads to a performance comparable to pruning 70% of them in Stage II. This experiment is conducted on the MME (top) and AI2D (bottom) datasets, using cosine similarity (left) and attention scores (right) as criteria. For comparison, we also report the baseline results and the performance when pruning only 1% of visual tokens.
Figure 5: Overview of HalfV. (a) Preliminary analysis. We use a small subset of the dataset (100 samples) to identify the onset of the three internal stages in the LLM. (b) Prefill acceleration. Implementation details of our two-step acceleration strategy in the LLM prefill stage.
Pruning Layer
Stage
MU↓
Latency Gain ↑
Early Layer
I
0.87
46.8%
Stage II Start
II
0.21
41.5%
Stage II Mid
II
0.29
37.8%
Deep Layer
III
0.65
22.4%
Table 1: Marginal utility (MU) of one-shot token pruning at different layers.
Setting
GQA
OCRBench
ChartQA
LLaVA-1.5v-7B
61.9
275
18.2
(a) Withdraw
54.7( -11.6% )
52( -81.1% )
13.6( -25.3% )
(b) Suppress
60.9( -1.6% )
311( +13.1% )
17.0( -6.5% )
Qwen25-VL-7B
60.7
848
84.0
(c) Withdraw
59.7( -1.6% )
698( -17.6% )
75.4( -10.2% )
(d) Suppress
41.2( -32.1% )
117( -86.2% )
67.2( -20.0% )
Table 2: Comparative experiments on the LLaVA-1.5v-7B and Qwen25-VL-7B models. We evaluate their performance on the GQA, OCRBench, and ChartQA datasets.
Method
TFLOPS
GQA
MME
POPE
SQA
VQA text
VizWiz
MMB en
AI2D
Average
LLaVA-1.5v-7B Liu et al. (2023) , Backbone: Vicuna
Vanilla
8.31
62.0
1859
85.9
70.4
58.2
54.4
64.8
54.8
100.0%
FastV Chen et al. (2024a)
3.48
57.6
1730
81.0
68.9
52.5
51.3
61.6
49.7
94.6%
VTW Lin et al. (2025)
4.48
58.7
1792
83.4
69.6
49.7
51.8
63.0
55.4
96.7%
FitPrune Ye et al. (2025)
3.16
58.5
1776
77.9
68.0
57.4
51.7
62.7
52.4
96.6%
HiRED Arif et al. (2025)
-
58.7
1737
82.8
68.4
47.4
50.1
62.8
51.7
93.9%
Table 3: Comparison with state-of-the-art inference acceleration methods on LLaVA-1.5-7B/13B, LLaVA-NeXT-7B and Qwen25-VL-7B models. The best result in each group is highlighted in bold , and the second-best result is underlined . Results that are higher than the original model are marked in red . † method focuses on layer-level redundancy, while others focus on token-level redundancy.
Method
GQA
MMB
MME
POPE
SQA
VQA Text
AI2D
Avg.
Qwen2-VL-7B
Upper Bound, All Tokens ( 100% )
Vanilla
60.7
82.5
2307
86.5
88.7
83.2
81.4
100%
Qwen2-VL-7B
Flops Ratio Reduction ( ↓ 88.9% )
+ FastV (ECCV24)
50.1
69.2
1940
78.6
77.4
60.3
68.7
83.6%
+ DART (EMNLP25)
54.3
74.8
2086
76.9
81.7
62.5
72.1
87.9%
+ HoloV (NIPS25)
52.8
72.4
2006
80.7
79.5
61.8
72.6
86.8%
Table 4: More comparative experiments on Qwen2-VL-7B and Qwen3-VL-4B models.
Method
TGIF
MSVD
MSRVT
Avg.
Acc.
Score
Acc.
Score
Acc.
Score
Acc.
Score
VideoChat-7B
34.4
2.3
56.3
2.8
45.0
2.5
45.1
2.5
LLaMA-Adapter-7B
-
-
54.9
3.1
43.8
2.7
-
-
Video-ChatGPT
51.4
3.0
64.9
3.3
49.3
2.8
55.2
3.0
Video-LLaVA-7B
47.0
3.4
70.2
3.9
57.3
3.5
58.2
3.6
+ FastV ( ECCV24 )
45.2
3.1
71.0
3.9
55
3.5
57.1
3.5
Table 5: Comparative experiments on video understanding tasks.
Figure 6: Performance comparison of different models under different pruning layers and anchor retention ratios.
Method
Total Time
TTFT
Prefill Time
Latency
Perf.
(Min:Sec)
(Millisecond)
(Millisecond)
(Millisecond)
LLaVA-NeXT-7B (Mistral)
Vanilla
73:44
391.9
391.0
495.1
86.8
FastV
44:21 ( ↓39.8% )
202.7 ( ↓48.2% )
201.9 ( ↓48.3% )
249.1 ( ↓49.6% )
85.5 ( ↓1.5% )
DART
46:17 ( ↓37.2% )
205.4 ( ↓47.5% )
205.2 ( ↓47.5% )
277.4 ( ↓43.9% )
86.0 ( ↓0.9% )
ShortV
59:48 ( ↓18.8% )
284.7 ( ↓27.3% )
284.1 ( ↓27.3% )
382.4 ( ↓22.7% )
86.5 ( ↓0.3% )
Table 6: Evaluation of efficiency for different models on POPE. Inference costs of Total Time, Time To First Token (TTFT), Prefill Time, Latency on POPE dataset for LLaVA-NeXT-7B and Qwen25-VL-7B models.
#
IVR
SSR
Perf.
TTFT
Speedup
MO
MO/TTFT
LLaVA-NeXT-7B (Mistral)
(a)
✔
67.1 ( ↓0.3% )
280.9 ms
1.48×
20.6 ms
7.3%
(b)
✔
67.3 ( ↓0.0% )
332.7 ms
1.26×
0.8 ms
0.2%
(c)
✔
✔
67.0 ( ↓0.4% )
220.9 ms
1.89×
21.7 ms
9.8%
Qwen25-VL-7B (Qwen2.5)
(d)
✔
79.5 ( ↓2.3% )
173.0 ms
1.84×
10.2 ms
5.8%
Table 7: Ablation study of module efficiency on AI2D dataset. IVR denotes the module that mitigates Intrinsic Visual Redundancy ; SSR denotes the module that suppresses Semantic Saturation Redundancy . MO abbreviates module overhead , and MO/TTFT measures the module’s relative time overhead as a fraction of TTFT.
Figure 7: Comparison of feature space coverage
Figure 8: Impact of SSR start layer on different models
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Illustration of Handling Inactivity Layers
Setting
IVR
SSR
w/ rope
POPE
ChartQA
Accuracy
F1 Score
Precision
Recall
Augmented Split
Human Split
Overall
RIVR=0.5,RS=0.2
✔
✗
✔
86.94
85.50
96.14
76.98
92.64 ± 0.74
64.56 ± 1.35
78.60 ± 0.82
RIVR=0.5,RS=0.2
✔
✗
✗
87.12
85.71
96.24
77.27
92.16 ± 0.76
65.12 ± 1.35
78.64 ± 0.82
RIVR=0.25,RS=0.1
✔
✗
✔
86.14
84.41
96.46
75.04
83.20 ± 1.06
48.16 ± 1.41
65.68 ± 0.95
RIVR=0.25,RS=0.1
✔
✗
✗
86.32
84.65
96.45
75.42
82.8 ± 1.07
49.12 ± 1.41
66.00 ± 0.95
RSSR=0.15
✗
✔
✔
87.70
86.45
96.34
78.40
93.60 ± 0.69
72.32 ± 1.27
82.96 ± 0.75
Appendix
Table 8: Performance comparison of Qwen2.5-VL-7B under different settings when enabling or disabling RoPE in the IVR and SSR stages.
Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing 33.7% TFLOPs on Qwen3-VL while retaining 99.5% of the vanilla model performance.
Zhaoyang Luo, Runmin Dong, Miao Yang +4
Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China +1
Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead. Existing visual token compression methods reduce this cost by pruning low-saliency tokens in shallow layers, but this practice couples temporary computational deactivation with irreversible information deletion. We show that this coupling breaks an intrinsic inference pattern in MLLMs. Through layer-wise attention analysis, we identify Visual Backtracking: visual pivots remain spatially stable, yet in middle layers they actively retrieve context from tokens that appeared unimportant earlier. Thus, early low-saliency tokens can be inactive for computation while still being useful for later reasoning. Motivated by this finding, we propose ReTrace, a backtracking-aware compression framework that introduces a dormant token state. ReTrace caches non-pivot tokens outside expensive layer-wise computation, restores them during the backtracking window, and prunes only after their information has been absorbed. Experiments on Qwen2.5-VL and InternVL3 demonstrate nearly 90% reductions in FLOPs and KV cache, a 2.83× prefilling speedup, and strong performance retention across complex vision-language benchmarks.
Guangzong Si, Dong Wang, Yifan Yu +3
University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University +2
Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.
Surendra Pathak, Bo Han
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA