From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration

Jiaqi Shi; Yuechan Li; Xulong Zhang; Xiaoyang Qu; Jianzong Wang

cs.CVApr 8, 2026

From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration

Authors: Jiaqi Shi, Yuechan Li, Xulong Zhang, Xiaoyang Qu, Jianzong Wang

Organizations: University of Science and Technology of China · Wuhan University · Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China

Abstract

High-resolution Multimodal Large Language Models (MLLMs) face prohibitive computational costs during inference due to the explosion of visual tokens. Existing acceleration strategies, such as token pruning or layer sparsity, suffer from severe "backbone dependency", performing well on Vicuna or Mistral architectures (e.g., LLaVA) but causing significant performance degradation when transferred to architectures like Qwen. To address this, we leverage truncated matrix entropy to uncover a universal three-stage inference lifecycle, decoupling visual redundancy into universal Intrinsic Visual Redundancy (IVR) and architecture-dependent Secondary Saturation Redundancy (SSR). Guided by this insight, we propose HalfV, a framework that first mitigates IVR via a unified pruning strategy and then adaptively handles SSR based on its specific manifestation. Experiments demonstrate that HalfV achieves superior efficiency-performance trade-offs across diverse backbones. Notably, on Qwen25-VL, it retains 96.8% performance at a 4.1 $\times$ FLOPs speedup, significantly outperforming state-of-the-art baselines. Our code is available at https://github.com/civilizwa/HalfV.

Figures & tables

Figure 1: (a): Comparison of token-level methods (HoloV, DART) at a 77.8% pruning ratio and the layer-level method (ShortV) across Vicuna, Mistral and Qwen backbones. Results represent the average relative performance compared to the baseline (%) across POPE, MME, MMBench, and SQA datasets; (b): We find that models with different backbones all exhibit a universal three-stage lifecycle: Modality alignment, global aggregation and visual saturation.

Panel of Figure 2 — Figure 2: (a) The average truncated matrix entropy for different architectures on GQA dataset. The x-axis is the depth percentage of the layer; (b) The average truncated matrix entropy for different types of tokens on LLaVA-1.5v-7B.; (c) 3D Line Plot of the Average Spectrum of $\mathcal{G}$ across Layers on LLaVA-1.5v-7B. Additional results see Appendix ; (d) KL divergence for Vicuna and Qwen2.5 backbones on AI2D dataset.

Panel of Figure 3 — Figure 3: Comparison of attention heatmaps at the initial and final layers of Stage I on LLaVA-1.5v-7B.

Figure 4: Suppressing only 1% of visual tokens leads to a performance comparable to pruning 70% of them in Stage II. This experiment is conducted on the MME (top) and AI2D (bottom) datasets, using cosine similarity (left) and attention scores (right) as criteria. For comparison, we also report the baseline results and the performance when pruning only 1% of visual tokens.

Figure 5: Overview of HalfV. (a) Preliminary analysis. We use a small subset of the dataset (100 samples) to identify the onset of the three internal stages in the LLM. (b) Prefill acceleration. Implementation details of our two-step acceleration strategy in the LLM prefill stage.

Pruning Layer	Stage	$\mathbf{MU}$ $\downarrow$	Latency Gain $\uparrow$
Early Layer	I	0.87	46.8%
Stage II Start	II	0.21	41.5%
Stage II Mid	II	0.29	37.8%
Deep Layer	III	0.65	22.4%

Table 1: Marginal utility (MU) of one-shot token pruning at different layers.

Setting	GQA	OCRBench	ChartQA
LLaVA-1.5v-7B	61.9	275	18.2
(a) Withdraw	54.7( -11.6% )	52( -81.1% )	13.6( -25.3% )
(b) Suppress	60.9( -1.6% )	311( +13.1% )	17.0( -6.5% )
Qwen25-VL-7B	60.7	848	84.0
(c) Withdraw	59.7( -1.6% )	698( -17.6% )	75.4( -10.2% )
(d) Suppress	41.2( -32.1% )	117( -86.2% )	67.2( -20.0% )

Table 2: Comparative experiments on the LLaVA-1.5v-7B and Qwen25-VL-7B models. We evaluate their performance on the GQA, OCRBench, and ChartQA datasets.

LLaVA-1.5v-7B Liu et al. (2023) , Backbone: Vicuna
Method	TFLOPS	GQA	MME	POPE	SQA	VQA ${}^{\textit{text}}$	VizWiz	MMB ${}^{\textit{en}}$	AI2D	Average
Vanilla	8.31	62.0	1859	85.9	70.4	58.2	54.4	64.8	54.8	100.0%
FastV Chen et al. (2024a)	3.48	57.6	1730	81.0	68.9	52.5	51.3	61.6	49.7	94.6%
VTW Lin et al. (2025)	4.48	58.7	1792	83.4	69.6	49.7	51.8	63.0	55.4	96.7%
FitPrune Ye et al. (2025)	3.16	58.5	1776	77.9	68.0	57.4	51.7	62.7	52.4	96.6%
HiRED Arif et al. (2025)	-	58.7	1737	82.8	68.4	47.4	50.1	62.8	51.7	93.9%

Table 3: Comparison with state-of-the-art inference acceleration methods on LLaVA-1.5-7B/13B, LLaVA-NeXT-7B and Qwen25-VL-7B models. The best result in each group is highlighted in bold , and the second-best result is underlined . Results that are higher than the original model are marked in red .

{\dagger}

method focuses on layer-level redundancy, while others focus on token-level redundancy.

Method	GQA	MMB	MME	POPE	SQA	VQA ${}^{\textit{Text}}$	AI2D	Avg.
Qwen2-VL-7B	Upper Bound, All Tokens ( 100% )
Vanilla	60.7	82.5	2307	86.5	88.7	83.2	81.4	100%
Qwen2-VL-7B	Flops Ratio Reduction ( $\downarrow$ 88.9% )
+ FastV (ECCV24)	50.1	69.2	1940	78.6	77.4	60.3	68.7	83.6%
+ DART (EMNLP25)	54.3	74.8	2086	76.9	81.7	62.5	72.1	87.9%
+ HoloV (NIPS25)	52.8	72.4	2006	80.7	79.5	61.8	72.6	86.8%

Table 4: More comparative experiments on Qwen2-VL-7B and Qwen3-VL-4B models.

Method	TGIF		MSVD		MSRVT		Avg.
Method	Acc.	Score	Acc.	Score	Acc.	Score	Acc.	Score
VideoChat-7B	34.4	2.3	56.3	2.8	45.0	2.5	45.1	2.5
LLaMA-Adapter-7B	-	-	54.9	3.1	43.8	2.7	-	-
Video-ChatGPT	51.4	3.0	64.9	3.3	49.3	2.8	55.2	3.0
Video-LLaVA-7B	47.0	3.4	70.2	3.9	57.3	3.5	58.2	3.6
+ FastV ( ECCV24 )	45.2	3.1	71.0	3.9	55	3.5	57.1	3.5

Table 5: Comparative experiments on video understanding tasks.

Panel of Figure 6 — Figure 6: Performance comparison of different models under different pruning layers and anchor retention ratios.

Method	Total Time	TTFT	Prefill Time	Latency	Perf.
Method	(Min:Sec)	(Millisecond)	(Millisecond)	(Millisecond)	Perf.
LLaVA-NeXT-7B (Mistral)
Vanilla	73:44	391.9	391.0	495.1	86.8
FastV	44:21 ( $\downarrow 39.8\%$ )	202.7 ( $\downarrow 48.2\%$ )	201.9 ( $\downarrow 48.3\%$ )	249.1 ( $\downarrow 49.6\%$ )	85.5 ( $\downarrow 1.5\%$ )
DART	46:17 ( $\downarrow 37.2\%$ )	205.4 ( $\downarrow 47.5\%$ )	205.2 ( $\downarrow 47.5\%$ )	277.4 ( $\downarrow 43.9\%$ )	86.0 ( $\downarrow 0.9\%$ )
ShortV	59:48 ( $\downarrow 18.8\%$ )	284.7 ( $\downarrow 27.3\%$ )	284.1 ( $\downarrow 27.3\%$ )	382.4 ( $\downarrow 22.7\%$ )	86.5 ( $\downarrow 0.3\%$ )

Table 6: Evaluation of efficiency for different models on POPE. Inference costs of Total Time, Time To First Token (TTFT), Prefill Time, Latency on POPE dataset for LLaVA-NeXT-7B and Qwen25-VL-7B models.

#	IVR	SSR	Perf.	TTFT	Speedup	MO	MO/TTFT
LLaVA-NeXT-7B (Mistral)
(a)	✔		67.1 ( $\downarrow 0.3\%$ )	280.9 ms	$1.48\times$	20.6 ms	7.3%
(b)		✔	67.3 ( $\downarrow 0.0\%$ )	332.7 ms	$1.26\times$	0.8 ms	0.2%
(c)	✔	✔	67.0 ( $\downarrow 0.4\%$ )	220.9 ms	$1.89\times$	21.7 ms	9.8%
Qwen25-VL-7B (Qwen2.5)
(d)	✔		79.5 ( $\downarrow 2.3\%$ )	173.0 ms	$1.84\times$	10.2 ms	5.8%

Table 7: Ablation study of module efficiency on AI2D dataset. IVR denotes the module that mitigates Intrinsic Visual Redundancy ; SSR denotes the module that suppresses Semantic Saturation Redundancy . MO abbreviates module overhead , and MO/TTFT measures the module’s relative time overhead as a fraction of TTFT.

Figure 7: Comparison of feature space coverage

Panel of Figure 8 — Figure 8: Impact of SSR start layer on different models

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Illustration of Handling Inactivity Layers

Setting	IVR	SSR	w/ rope	POPE				ChartQA
Setting	IVR	SSR	w/ rope	Accuracy	F1 Score	Precision	Recall	Augmented Split	Human Split	Overall
$R_{\text{IVR}}=0.5,R_{\mathcal{S}}=0.2$	✔	✗	✔	86.94	85.50	96.14	76.98	92.64 $\pm$ 0.74	64.56 $\pm$ 1.35	78.60 $\pm$ 0.82
$R_{\text{IVR}}=0.5,R_{\mathcal{S}}=0.2$	✔	✗	✗	87.12	85.71	96.24	77.27	92.16 $\pm$ 0.76	65.12 $\pm$ 1.35	78.64 $\pm$ 0.82
$R_{\text{IVR}}=0.25,R_{\mathcal{S}}=0.1$	✔	✗	✔	86.14	84.41	96.46	75.04	83.20 $\pm$ 1.06	48.16 $\pm$ 1.41	65.68 $\pm$ 0.95
$R_{\text{IVR}}=0.25,R_{\mathcal{S}}=0.1$	✔	✗	✗	86.32	84.65	96.45	75.42	82.8 $\pm$ 1.07	49.12 $\pm$ 1.41	66.00 $\pm$ 0.95
$R_{\text{SSR}}=0.15$	✗	✔	✔	87.70	86.45	96.34	78.40	93.60 $\pm$ 0.69	72.32 $\pm$ 1.27	82.96 $\pm$ 0.75

Appendix

Table 8: Performance comparison of Qwen2.5-VL-7B under different settings when enabling or disabling RoPE in the IVR and SSR stages.

Model	$L_{\text{IVR}}$	$R_{\text{IVR}}$	$R_{\mathcal{S}}$	$L_{\text{SSR}}$
LLaVA-1.5v-7B (32 Layers)	3	50%	0.2	15
LLaVA-1.5v-13B (40 Layers)	3	50%	0.2	15
LLaVA-NeXT-7B (32 Layers)	2	50%	0.1	16
Qwen25-VL-7B (28 Layers)	2	[25%,5%]	0.1	21

Appendix

Table 9: Hyperparameters for HalfV

Model	Method	$R$	$K$	$N$
LLaVA-1.5v-7B	Single-stage	33.3%	2	-
	Multi-stage	[50%,75%,87.5%]	[5,10,20]	-
	ShortV	-	-	19
LLaVA-1.5v-13B	Single-stage	33.3%	2	-
	Multi-stage	[50%,75%,87.5%]	[5,10,20]	-
	ShortV	-	-	24

Appendix

Table 10: Baseline Settings

Model	$t$	$v$	$h$	$m$
LLaVA-1.5v-7B	50	576	4096	11008
LLaVA-1.5v-13B	50	576	5120	13824
LLaVA-NeXT-7B	50	2352	4096	14336
Qwen25-VL-7B	50	2352	3584	18944

Appendix

Table 11: Computational Settings

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
Jun 30, 2026Zhaoyang Luo, Runmin Dong, Miao Yang +4Efficient VLM Inference Multimodal Large Language Models
Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China +1
ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models
Oct 18, 2026Guangzong Si, Dong Wang, Yifan Yu +3Efficient VLM Inference Visual Token Compression
University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University +2
Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
Mar 30, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Compression
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA

Jun 30, 2026J/K move · Enter open · S save

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

Zhaoyang Luo, Runmin Dong, Miao Yang, Fan Wei+3

Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China+1

Efficient VLM Inference Multimodal Large Language Models Visual Token Pruning Multimodal Understanding Multimodal Transformer

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing 33.7% TFLOPs on Qwen3-VL while retaining 99.5% of the vanilla model performance.

Abstract

Efficient VLM Inference

Explore similar work

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies