Large Vision-Language Models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, incurring substantial computation and memory costs. Existing visual-token reduction methods often irreversibly discard tokens despite depth-varying query relevance. Although recoverable routing allows excluded tokens to return, re-admitted tokens retain stale hidden states after bypassing intermediate decoder transformations. Thus, token recoverability does not guarantee representation readiness at the current decoder depth. Under a fixed routing trajectory at a 64-token budget, replacing stale re-admitted states with current-depth oracle representations improves average RefCOCO [email protected] from 18.96 to 23.18, revealing the impact of representation staleness. We propose CatchUp, a training-free framework combining reversible visual-token routing with Local Transition Correction (LTC). CatchUp repeatedly re-evaluates query-conditioned relevance over the complete visual-token set, preserving deferred tokens for later re-admission. LTC approximates their skipped representation evolution by transferring reference-to-current hidden-state transitions from semantically and spatially related active anchors. This correction improves alignment with current-depth oracle representations without recomputing the skipped decoder layers. Across three referring-expression grounding datasets and six multimodal benchmarks, CatchUp achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets. At a 64-token budget, CatchUp processes 88.9% fewer active visual tokens per post-routing decoder interval, retains 92.17% of dense-model multimodal performance, and reduces end-to-end latency by 8.0%. Code is available at https://github.com/pej0918/CatchUp
Figures & tables
Method
Routing and token handling
Repeated selection
Original-token recoverability
Re-admission correction
FastV [ 3 ]
One-shot permanent pruning
✗
✗
✗
SparseVLM [ 26 ]
Adaptive pruning and recycling
✓
✗
✗
PyramidDrop [ 17 ]
Progressive stage-wise pruning
✓
✗
✗
Nüwa [ 9 ]
Spatial aggregation and pruning
✗
✗
✗
VScan [ 25 ]
Encoder-side merging and intermediate-layer pruning
✗
✗
✗
ALVTS [ 4 ]
Per-layer bypass and reintegration
✓
✓
✗
Table 1: Structural comparison of representative visual-token reduction methods. Repeated selection denotes reassessment of visual-token participation at multiple decoder depths. Original-token recoverability indicates whether excluded tokens remain independently eligible for later computation. Re-admission correction indicates explicit compensation for decoder transformations skipped by returning tokens.
Figure 1: Late-emerging visual evidence for right zebra . The target region (red) contains a token (blue) that first enters the top-64 relevance set at layer 25. The curve shows its relevance percentile; the dashed line marks the top-64 threshold, and stars denote top-64 membership.
Figure 2: Evidence dynamics over 500 RefCOCO samples. (a) Mean relevance volatility is 28.85% across adjacent layers (dashed line). (b) LER relative to layer 2 reaches 99.42% at layer 31.
Figure 3: Fixed-routing oracle intervention on re-admitted token states. Replacing stale states with current-depth oracle states improves average RefCOCO [email protected] from 18.96 to 23.18. For right truck , the red box marks the ground truth; orange (a) and blue (b) indicate stale and oracle predictions.
Figure 4: Overview of CatchUp. Reversible routing re-evaluates visual-token relevance across decoder layers, processing only selected tokens while preserving deferred tokens for later re-admission. LTC corrects newly re-admitted representations by transferring reference-to-current hidden-state changes from semantically and spatially related active anchors.
Method
Visual Grounding
MLLM Benchmarks
RefCOCO
RefCOCO+
RefCOCOg
Avg.
POPE
VQAv2
VizWiz
GQA
MM-Vet
MMBench
Avg.
val
A
B
val
A
B
val
test
Retention
Retention
Dense Reference, 576 Tokens (100%)
LLaVA-1.5-7B
54.15
59.27
47.97
46.53
53.93
39.05
46.89
47.67
100.00%
85.90
75.84
54.30
60.47
28.67
62.37
100.00%
Routing Budget K=192 ( ↓ 67%)
FastV (ECCV’24)
25.38
27.91
21.94
21.09
23.68
18.04
21.55
20.52
45.52%
79.66
69.90
54.72
55.04
25.09
58.68
93.05%
Table 3: Main results on eight referring-expression grounding splits and six multimodal benchmarks. CatchUp leads all grounding splits at every budget and retains 92.17% of dense-model multimodal performance at K=64 . Avg. Retention averages per-benchmark or per-split performance ratios relative to full LLaVA-1.5-7B; bold denotes the best efficient method at each budget.
Figure 5: Reversible visual-token routing for the referring expression player . FastV shows the final tokens after irreversible pruning, while CatchUp shows selected tokens at L2 and L7, including newly re-admitted tokens. The red box marks the ground-truth target; whitened regions denote deferred tokens, and blue outlines mark tokens re-admitted at L7.
Figure 6: RefCOCO accuracy versus (a) decoder KV-cache memory and (b) end-to-end latency on LLaVA-1.5-7B at matched prefill TFLOPs (within 1%). CatchUp lies on the non-dominated accuracy–efficiency frontier. Dashed lines mark the Pareto frontier; shaded regions are dominated. Dense inference requires 8.673 TFLOPs.
Variant
RefCOCO ↑
TFLOPs ↓
E2E (ms) ↓
Peak CUDA (MiB) ↓
w/o LTC
18.96
2.131
1589.28
17799.3
CatchUp
21.26
2.194
1611.17
17820.5
Table 6: LTC overhead at K=64 . Transition correction improves RefCOCO by 2.30 points with 2.96% additional prefill TFLOPs, 1.38% higher end-to-end latency, and 21.2 MiB additional peak PyTorch CUDA allocation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Setting
Routing layers
{2,7,15,23}
Number of anchors M
8
Spatial weight λ
0.15
Temperature τ
0.07
Transition scale γ
0.80
Snapshot retention
Need-based
Appendix
Table 7: Fixed CatchUp implementation configuration shared across evaluated datasets, backbones, and routing budgets. Hyperparameter sensitivity is evaluated in Section C.8 .
Gap (layers)
Stale
Absolute
LTC
5 – 8
0.6914
0.6044
0.4219
9 – 12
0.9363
1.3429
0.8757
13+
1.0834
1.4216
1.0640
Appendix
Table 8: Normalized ℓ2 discrepancy from current-depth oracle states by re-admission gap ( K=64 ). LTC yields the lowest discrepancy in all three ranges, with a smaller margin for gaps of 13+ layers. Lower is better.
Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive token reduction while maintaining general VQA performance. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/
Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu
National Yang Ming Chiao Tung University · National Taiwan University
Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We present RETOKEN, a single learnable embedding that extracts retrieval signals from the VLM's internal representations to select query-relevant visual tokens from the pre-filled KV cache. This enables retrieval within the answering VLM, without a separate retriever or re-encoding. Despite being trained on only a small image-QA dataset, RETOKEN generalizes across image and video benchmarks. On Visual Haystacks, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative). On LVBench, it transfers zero-shot to long video and improves Qwen3VL-8B by 8.0 points. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken
Yao Xiao, Reuben Tan, Zhen Zhu +3
University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China