CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

Eunju Park; Uiwon Hwang

cs.CVOct 16, 2026

CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

Authors: Eunju Park, Uiwon Hwang

Organizations: Division of Artificial Intelligence & Software, Ewha Womans University

Abstract

Large Vision-Language Models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, incurring substantial computation and memory costs. Existing visual-token reduction methods often irreversibly discard tokens despite depth-varying query relevance. Although recoverable routing allows excluded tokens to return, re-admitted tokens retain stale hidden states after bypassing intermediate decoder transformations. Thus, token recoverability does not guarantee representation readiness at the current decoder depth. Under a fixed routing trajectory at a 64-token budget, replacing stale re-admitted states with current-depth oracle representations improves average RefCOCO [email protected] from 18.96 to 23.18, revealing the impact of representation staleness. We propose CatchUp, a training-free framework combining reversible visual-token routing with Local Transition Correction (LTC). CatchUp repeatedly re-evaluates query-conditioned relevance over the complete visual-token set, preserving deferred tokens for later re-admission. LTC approximates their skipped representation evolution by transferring reference-to-current hidden-state transitions from semantically and spatially related active anchors. This correction improves alignment with current-depth oracle representations without recomputing the skipped decoder layers. Across three referring-expression grounding datasets and six multimodal benchmarks, CatchUp achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets. At a 64-token budget, CatchUp processes 88.9% fewer active visual tokens per post-routing decoder interval, retains 92.17% of dense-model multimodal performance, and reduces end-to-end latency by 8.0%. Code is available at https://github.com/pej0918/CatchUp

Figures & tables

Method	Routing and token handling	Repeated selection	Original-token recoverability	Re-admission correction
FastV [ 3 ]	One-shot permanent pruning	✗	✗	✗
SparseVLM [ 26 ]	Adaptive pruning and recycling	✓	✗	✗
PyramidDrop [ 17 ]	Progressive stage-wise pruning	✓	✗	✗
Nüwa [ 9 ]	Spatial aggregation and pruning	✗	✗	✗
VScan [ 25 ]	Encoder-side merging and intermediate-layer pruning	✗	✗	✗
ALVTS [ 4 ]	Per-layer bypass and reintegration	✓	✓	✗

Table 1: Structural comparison of representative visual-token reduction methods. Repeated selection denotes reassessment of visual-token participation at multiple decoder depths. Original-token recoverability indicates whether excluded tokens remain independently eligible for later computation. Re-admission correction indicates explicit compensation for decoder transformations skipped by returning tokens.

Figure 1: Late-emerging visual evidence for right zebra . The target region (red) contains a token (blue) that first enters the top-64 relevance set at layer 25. The curve shows its relevance percentile; the dashed line marks the top-64 threshold, and stars denote top-64 membership.

Figure 2: Evidence dynamics over 500 RefCOCO samples. (a) Mean relevance volatility is 28.85% across adjacent layers (dashed line). (b) LER relative to layer 2 reaches 99.42% at layer 31.

Figure 3: Fixed-routing oracle intervention on re-admitted token states. Replacing stale states with current-depth oracle states improves average RefCOCO Acc@0.5 from 18.96 to 23.18. For right truck , the red box marks the ground truth; orange (a) and blue (b) indicate stale and oracle predictions. — Figure 3: Fixed-routing oracle intervention on re-admitted token states. Replacing stale states with current-depth oracle states improves average RefCOCO [email protected] from 18.96 to 23.18. For right truck , the red box marks the ground truth; orange (a) and blue (b) indicate stale and oracle predictions.

Figure 4: Overview of CatchUp. Reversible routing re-evaluates visual-token relevance across decoder layers, processing only selected tokens while preserving deferred tokens for later re-admission. LTC corrects newly re-admitted representations by transferring reference-to-current hidden-state changes from semantically and spatially related active anchors.

Method	Visual Grounding									MLLM Benchmarks
	RefCOCO			RefCOCO+			RefCOCOg		Avg.	POPE	VQAv2	VizWiz	GQA	MM-Vet	MMBench	Avg.
	val	A	B	val	A	B	val	test	Retention							Retention
Dense Reference, 576 Tokens (100%)
LLaVA-1.5-7B	54.15	59.27	47.97	46.53	53.93	39.05	46.89	47.67	100.00%	85.90	75.84	54.30	60.47	28.67	62.37	100.00%
Routing Budget $K=192$ ( $\downarrow$ 67%)
FastV (ECCV’24)	25.38	27.91	21.94	21.09	23.68	18.04	21.55	20.52	45.52%	79.66	69.90	54.72	55.04	25.09	58.68	93.05%

Table 3: Main results on eight referring-expression grounding splits and six multimodal benchmarks. CatchUp leads all grounding splits at every budget and retains 92.17% of dense-model multimodal performance at

K=64

. Avg. Retention averages per-benchmark or per-split performance ratios relative to full LLaVA-1.5-7B; bold denotes the best efficient method at each budget.

Figure 5: Reversible visual-token routing for the referring expression player . FastV shows the final tokens after irreversible pruning, while CatchUp shows selected tokens at L2 and L7, including newly re-admitted tokens. The red box marks the ground-truth target; whitened regions denote deferred tokens, and blue outlines mark tokens re-admitted at L7.

Figure 6: RefCOCO accuracy versus (a) decoder KV-cache memory and (b) end-to-end latency on LLaVA-1.5-7B at matched prefill TFLOPs (within 1%). CatchUp lies on the non-dominated accuracy–efficiency frontier. Dashed lines mark the Pareto frontier; shaded regions are dominated. Dense inference requires 8.673 TFLOPs.

Variant	RefCOCO $\uparrow$	TFLOPs $\downarrow$	E2E (ms) $\downarrow$	Peak CUDA (MiB) $\downarrow$
w/o LTC	18.96	2.131	1589.28	17799.3
CatchUp	21.26	2.194	1611.17	17820.5

Table 6: LTC overhead at

K=64

. Transition correction improves RefCOCO by 2.30 points with 2.96% additional prefill TFLOPs, 1.38% higher end-to-end latency, and 21.2 MiB additional peak PyTorch CUDA allocation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Setting
Routing layers	$\{2,7,15,23\}$
Number of anchors $M$	8
Spatial weight $\lambda$	0.15
Temperature $\tau$	0.07
Transition scale $\gamma$	0.80
Snapshot retention	Need-based

Appendix

Table 7: Fixed CatchUp implementation configuration shared across evaluated datasets, backbones, and routing budgets. Hyperparameter sensitivity is evaluated in Section C.8 .

Gap (layers)	Stale	Absolute	LTC
$5$ – $8$	0.6914	0.6044	0.4219
$9$ – $12$	0.9363	1.3429	0.8757
$13+$	1.0834	1.4216	1.0640

Appendix

Table 8: Normalized

\ell_{2}

discrepancy from current-depth oracle states by re-admission gap (

K=64

). LTC yields the lowest discrepancy in all three ranges, with a smaller margin for gaps of 13+ layers. Lower is better.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
Jun 10, 2026Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun LiuEfficient VLM Inference Visual Token Pruning
National Yang Ming Chiao Tung University · National Taiwan University
ReToken: Improving Long-Context VLMs with Visual Retrieval Token
Jul 30, 2026Yao Xiao, Reuben Tan, Zhen Zhu +3Vision-Language Models Vision-Language Retrieval
University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind
ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs
Sep 28, 2026Xu Li, Yuxuan Liang, Yi Zheng +6Efficient VLM Inference Large Vision-Language Models
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China

Jun 10, 2026J/K move · Enter open · S save

Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

National Yang Ming Chiao Tung University · National Taiwan University

Efficient VLM Inference Visual Token Pruning

Open paper AskPDF arXiv

Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive token reduction while maintaining general VQA performance. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/

Abstract

Efficient VLM Inference

Explore similar work

Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

ReToken: Improving Long-Context VLMs with Visual Retrieval Token

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs