RCP: Representation Consistency Pruner for Mitigating Distribution Shift in Large Vision-Language Models
Authors: Jianwei Zhang, Chaoning Zhang, Sihan Cao, Wang Liu, Pengcheng Zheng, Jiaxin Huang, Caiyan Qin, Yalan Ye, Wei Dong, Yang Yang
Organizations: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Qingshuihe Campus, No. 2006 Xiyuan Avenue, High-Tech Zone (West), Chengdu 611731, China · Machine Learning Department, Mohamed bin Zayed University of Artificial Intelligence, Building 1B, Masdar City, Abu Dhabi, United Arab Emirates · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, University Town of Shenzhen, Nanshan District, Shenzhen 518055, China · School of Information and Control Engineering, Xi’an University of Architecture and Technology, No. 13 Yanta Road, Xi’an 710055, China
Large Vision-Language Models (LVLMs) suffer from prohibitive inference costs due to the massive number of visual tokens processed by the language decoder. Existing pruning methods often lead to significant performance degradation because the irreversible removal of visual tokens causes a distribution shift in the hidden states that deviates from the pre-trained full-token regime. To address this, we propose Representation Consistency Pruner, which we refer to as RCP, as a novel framework that integrates cumulative visual token pruning with a delayed repair mechanism. Specifically, we introduce a cross-attention pruner that leverages the intrinsic attention of the LLM as a baseline to predict cumulative masks, ensuring consistent and monotonic token reduction across layers. To compensate for the resulting information loss, we design a delayed repair adapter denoted as DRA, which caches the essence of pruned tokens and applies FiLM-based modulation specifically to the answer generation tokens. We employ a repair loss to match the first and second-order statistics of the pruned representations with a full-token teacher. RCP is highly efficient because it trains only lightweight plug-in modules while allowing for physical token discarding at inference. Extensive experiments on LVLM benchmarks demonstrate that RCP removes up to 88.9% of visual tokens and reduces FLOPs by up to 85.7% with only a marginal average accuracy drop, and outperforms prior methods that avoid fine-tuning the original model on several widely used benchmarks.
Figures & tables
Fig. 1: Layer-wise representation drift between the original and pruned models, measured by W22 .
Fig. 2: Overview of the proposed Representation Consistency Pruner framework. (a) Overall Architecture illustrates the global system where one or more pruning modules precede a delayed repair adapter separated by multiple language model decoder layers. (b) Pruner Layer & Delayed Repair Adapter provides the structural details for the internal components of both the pruning and repair modules.
Method
GQA
MME
POPE
SQA
VQA V2
MMB
VizWiz
Avg Acc. (%)
Upper Bound (576 Tokens) ( 100% )
Vanilla
62
1763.7
85.9
69.5
78.5
64.3
50
100
100%
100%
100%
100%
100%
100%
100%
Retain 192 Tokens ( ↓ 66.7%)
ToMe [ 37 ]
54.39
1480.49
72.4
65.2
68.0
60.13
-
88.32 ( ↓ 11.68)
87.72%
83.94%
84.28%
93.81%
86.62%
93.51%
-
TABLE I: Performance comparison on multiple benchmarks with LLaVA-1.5-7B backbone. We report absolute scores and the compression retention rates to Upper Bound model. Acc. means the average of normalized relative percentages accuracy across 7 benchmarks.
Method
Token
GQA
MME
POPE
Avg.
LLaVA-1.5-13B
576
63.3
1789.68
85.99
-
100%
100%
100%
100%
FastV [ 34 ]
192
54.25
1515.86
64.66
-
85.7%
84.7%
75.2%
81.87%
Ours
192
60.20
1786.10
85.73
-
95.11%
99.8%
99.7%
98.20%
TABLE II: Performance comparison on multiple benchmarks with LLaVA-1.5-13B backbone.
Method
Avg. Token
Relative Accuracy
Storage (MB)
Δ
CUDA Times (ms)
Δ
FLOPs (T)
Δ
Upper Bound
576
100%
302.4
-
403.1
-
9.6
-
FastV
192
88.71%
100.8
66.7%
230.1
42.9%
2.0
79.2%
192
99.06%
100.8
66.7%
263.7
34.6%
3.52
63.3%
Ours
128
96.73%
67.2
77.8%
196.0
51.4%
2.4
75.0%
64
95.00%
33.6
88.9%
169.6
57.9%
1.37
85.7%
TABLE III: Efficiency analysis including cache storage memory, latency, CUDA times, and FLOPs, where Δ denotes the reduction ratio.
Fig. 3: Comparison of the repair loss between the baseline variant using only the pruner and the full RCP framework to demonstrate the stabilization effect provided by the repair adapter and the repair loss.
Method
VQAv2
POPE
Avg.
Upper Bound (Full)
78.5
85.9
82.20
RCP (Full)
74.53
80.49
77.51
w/o Pruner (Top-K)
67.13
70.32
68.73
w/o Adapter
74.13
79.62
76.88
w/o Repair Loss
72.98
79.03
76.00
Mean-only Repair Loss
73.06
79.42
76.24
TABLE IV: Ablation study of individual modules in the RCP framework.
Pruning Layer Indexes
VQAv2
POPE
Avg.
[2]
71.34
72.52
71.93
[2, 14]
74.11
80.21
77.16
[2, 14, 26]
74.53
80.49
77.51
[3, 15, 27]
74.11
79.41
76.76
[4, 16, 28]
74.13
80.53
77.33
TABLE V: Effect of pruner quantity and layer placement.
Target
Layer 5 Rate
Layer 15 Rate
Layer 25 Rate
192 Tokens
60.21%
7.47%
1.56%
128 Tokens
25.84%
4.13%
1.54%
Target
Layer 2 Rate
Layer 14 Rate
Layer 26 Rate
64 Tokens
16.72%
4.59%
1.44%
TABLE VI: Target budgets compared with actual retention rates at various layer depths.
Post Layer Indexes
VQAv2
POPE
Avg.
[23]
74.46
77.73
76.10
[23, 30]
74.53
80.49
77.51
[22, 29]
74.42
80.52
77.47
[14, 26]
74.27
80.23
77.25
TABLE VII: Ablation of adapter quantity and placement.
Fig. 4: Visualization of pruning masks at Layer 5, Layer 15, and Layer 25 under the 128 token setting which shows the progressive transition from background suppression to salient region focus.
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China
Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose VPRune, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.
Guangchuan Lv, Dianxing Shi, Dingjie Fu
Northeastern University · Beihang University · Huazhong University of Science and Technology
Large Vision-Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual tokens. Existing methods typically struggle to model progressive visual token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance. We formulate visual token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning. The pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7% of visual tokens and achieves up to a 54.2% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7%. Code is released at mypink{https://github.com/MagicVicCoder/TPRL}.
Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China