LoopZip: Visual Token Compression for Efficient Looped Vision-Language Models
Authors: Weizhong Huang, Zhicheng Liu, Yan Zhang, Fei Chao, Xiawu Zheng, Liujuan Cao
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Looped vision-language models (VLMs) deepen computation by reusing shared layers, approaching larger models' capabilities with fewer parameters and easing the storage and deployment burden of stacking independent layers. However, repeatedly processing long visual sequences still incurs substantial compute and GPU memory costs. Visual token compression reduces these costs by shortening sequences. Existing methods mainly select original tokens directly or combine selection with aggregation. In looped models, the former may overlook changing token importance across calls and discard information needed later; the latter lack joint consideration of output changes across calls, limiting information preservation. We propose LoopZip, a training-free visual token compression algorithm addressing both issues. Visual aggregation builds a compact candidate pool before the looped backbone, and cross-invocation selection averages instruction attention from the first two existing calls of the same shared attention layer to retain tokens needed by both states. Weight-tied value correction then jointly fits shared visual-value adjustments to both pre-selection layer outputs, reducing output deviations across states. To validate LoopZip's effectiveness, we conduct extensive experiments covering two open-source looped VLMs, eleven benchmarks, and three pruning ratios. On LoopVL-1B, LoopZip retains 97.16% of the original model's performance at 50% pruning ratio and exceeds the strongest baseline by 5.36% at 87.5% pruning ratio. Across Hugging Face Transformers, vLLM, and SGLang, LoopVL-1B with LoopZip reduces peak allocated GPU memory by 58.36%-58.98% and achieves 2.01-2.47x end-to-end speedups over the original model.
Figures & tables
Figure 1: Performance and efficiency on LoopVL-1B. LoopZip maintains high performance under pruning while reducing inference latency and GPU memory usage.
Figure 2: Why should token compression consider different calls? (a) Combining two calls improves later-token coverage. (b) Right: removal-induced output deviation for one fixed first-call selection, before value correction; the horizontal axis counts calls of the same shared layer. Left: the motivated joint correction. Diagnostics use LoopVL-1B, 25% retention, and means over 160 questions; Section 5.5 gives the full analysis.
Figure 3: LoopZip’s forward path. Aggregation forms K0 candidates from N patches (Sec. 4.2 ). Both first calls receive K0 candidates; their mean attention selects K1 (Sec. 4.3 ). Their outputs O0,O1 jointly supervise correction X⋆ , applied only to retained values at call 2 (Sec. 4.4 ). After that block, later calls use K1 visual tokens (Sec. 4.5 ); all text tokens remain.
Figure 4: Visual aggregation (schematic). High-attention patches are retained directly (image tiles); similar remaining features form projected group summaries (diamonds). Both types fill the K0 -token candidate pool.
Figure 5: Cross-invocation selection. The car image overlays first-call-only (blue), second-call-only (purple), and shared (gray) top-attention patches. The schematic combines the two calls' candidate attention scores and retains K1 tokens.
Figure 6: Shared value correction (schematic). The two calls’ fitting terms (blue/purple) yield one regularized joint solution X⋆ (orange). The LoopVL timeline shows that only call t=1 receives the correction.
Visual Understanding
Hallucination
Reasoning
Method
MM Star
RWQA
VMCB
AI2D
Chart QA
HR Bench 4K
POPE
Hallu.
MMMU -Pro
MathV
LogicV
Avg. ↑
Rel. Avg. ↑
LoopVL-1B [ 39 ]
Vanilla
63.13
70.85
70.50
75.78
74.60
56.50
90.84
44.41
38.32
38.22
43.18
60.58
100.00%
FastV [ 7 ] (ECCV’24)
53.00
59.87
66.00
69.95
45.16
51.25
85.80
38.65
35.03
34.51
38.26
52.50
87.04%
SparseVLM [ 57 ] (ICML’25)
54.60
58.69
65.30
70.01
48.48
51.62
85.69
38.45
35.90
34.41
37.81
52.81
87.53%
CDPruner [ 56 ] (NeurIPS’25)
56.07
67.06
68.20
73.41
49.04
52.62
89.56
38.86
35.49
32.43
37.14
54.53
89.59%
Table 1: Two looped VLMs at 50% pruning ratio. Scores are percentages; Rel. Avg. averages ratios to Vanilla. Bold and underlined values indicate the best and second-best results among compression methods, respectively.
Figure 7: Aggregate performance at 87.5% (a) and 75% (b) pruning. Light/dark bars show Avg. and Rel. Avg. (%), with values above bars and Vanilla as the unpruned reference. Full results: Table 10, Appendix E.
Figure 8: Vanilla versus LoopZip at 8192 visual tokens and 87.5% pruning. (a)–(d) Prefill, decode, end-to-end latency, and peak allocated GPU memory; lower is better. Colors identify HF, vLLM, and SGLang; light/dark bars show Vanilla/LoopZip. Arrows report Vanilla/LoopZip ratios; labels average five inputs. Full comparisons: Appendix F.
Figure 9: Hyperparameter comparison on LoopVL-1B at 87.5% pruning. Avg. (3) averages MMStar, MMMU-Pro, and POPE. Each panel varies one parameter. Blue circles denote other settings; orange diamonds denote the default setting.
Figure 10: LoopVL-1B’s first shared attention layer, calls 1 – 6 . (a) First-two top- 25% visual-token masks for an MMStar car example; rectangles mark the same windshield region. (b) Number of top-attention patches within that region. At 75% pruning, (c) top- 25% visual-token coverage and (d) layer-output error relative to each call’s uncompressed output average 160 questions. Blue uses first-call attention, green the first two calls’ mean attention, and gray random selection. Selected sets stay fixed across calls; stars mark a separate train diagnostic. (e) With first-call selection, typical output-error variation is 4.2× , 2.5× , 3.4× , and 5.1× on HRBench4K, ChartQA, MMStar, and POPE; the dashed line denotes equal errors. See Appendices B and A.
Variant
MMStar
MMMU -Pro
POPE
Avg. (3)
Full LoopZip
51.00
31.62
86.98
56.53
Visual Aggregation Only
50.27
30.12
86.61
55.67
w/o Visual Aggregation
40.80
30.40
78.08
49.76
w/o Cross-Invocation Selection
48.33
31.21
86.79
55.45
w/o Joint Fitting
47.93
30.81
86.76
55.17
w/o Value Correction
48.00
31.21
86.61
55.28
Table 2: Ablation results for individual components (%).
m
MMStar
MMMU -Pro
POPE
Avg. (3)
Tc
Tp
1
48.27
31.27
86.65
55.40
31.10
209.66
2
51.00
31.62
86.98
56.53
42.05
219.43
3
47.07
30.75
86.34
54.72
55.47
235.09
4
47.07
31.27
86.23
54.85
68.64
251.16
5
47.20
31.85
86.41
55.15
81.40
257.71
6
47.20
31.79
87.10
55.36
95.67
272.66
Table 3: Observation-count ablation. Scores (%) and compression/prefill times ( Tc/Tp , ms) with m calls.
Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.
Yupeng Zheng, Kai Zou, Bin Liu +1
Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China Hefei, China
Large Vision-Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual tokens. Existing methods typically struggle to model progressive visual token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance. We formulate visual token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning. The pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7% of visual tokens and achieves up to a 54.2% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7%. Code is released at mypink{https://github.com/MagicVicCoder/TPRL}.
Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China
While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.