Multimodal large language models suffer from severe computational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projector modules to align and compress visual tokens into text-aligned features, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resampler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and textual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual Token Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time.
Figures & tables
Figure 1: Qualitative comparisons of three representative visual token compression methods under different scenarios.
Figure 2: Comparison of compression strategies across tasks. Pooling excels on global understanding, pruning leads on local details, and resampling performs best on style and emotion, highlighting task-dependent advantages.
Figure 3: The architecture of the proposed QMoP.
Figure 4: Overview of VTCBench and the quantitative analysis of compression strategies.
Methods
VQA T
POPE
VQA v2
MM-Vet
GQA
MME
MMB
Seed
MMStar
MMMU
Avg.
LLaVA-1.5-7B, Vision token = 144
Vanilla [ 21 ]
58.2
85.9
78.5
31.6
62.0
1511
64.3
66.1
31.9
34.6
100.00%
Pixel-Shuffle [ 8 ]
54.2
87.0
75.6
29.1
60.2
1422
64.5
61.6
31.7
32.7
96.14%
FasterVLM [ 40 ]
57.1
83.5
76.2
32.2
58.0
1440
64.3
62.3
31.3
34.8
97.64%
C-Abstractor [ 5 ]
54.1
85.9
75.7
28.2
60.4
1392
63.3
62.3
31.1
33.4
95.48%
MQT-LLaVA [ 13 ]
52.6
86.2
76.8
29.8
61.4
1445
64.4
62.0
31.9
34.8
97.22%
Table 1: Performance comparison with pre-LLM compression methods on LLaVA-1.5-7B. The best and second-best results are highlighted in bold and underlined .
Method
Vision
VQA T
POPE
VQA v2
GQA
MMB
MMB-
OCR
Avg.
Token
CN
Bench
LLaVA-1.5-7B
576
58.2
85.9
78.5
62.0
64.3
58.3
297.0
100.00%
FastV [ 6 ]
192
52.5
64.8
67.1
52.7
61.2
57.0
291
89.58%
HiRED [ 2 ]
192
47.4
82.8
74.9
58.7
62.8
54.7
190
89.06%
FitPrune [ 35 ]
192
57.4
83.4
-
60.4
63.3
56.4
-
-
LLaVA-PruMerge [ 28 ]
192
54.3
71.3
70.6
54.3
59.6
52.9
253
88.92%
Table 2: Performance comparison with intra-LLM visual token compression methods on LLaVA-1.5-7B.
Method
Type
GU
SC
RC
SE
LD
Avg.
LLaVA-1.5-7B [ 21 ]
-
56.1
47.4
65.9
72.0
60.9
61.0
MQT-LLaVA [ 13 ]
A
54.7
46.1
64.3
74.7
54.9
59.5
Pixel-Shuffle [ 8 ]
B
53.6
47.0
61.2
70.0
55.3
57.9
C-Abstract [ 5 ]
B
53.2
47.8
59.6
70.3
55.2
57.7
TokenPacker [ 18 ]
B
55.8
47.8
64.7
74.0
57.5
60.5
LDP-V2 [ 9 ]
B
56.8
48.3
63.9
74.3
56.4
60.5
Table 3: Performance comparison on VTCBench. The results are reported across five dimensions: GU (Global Understanding), SC (Spatial and Counting), RC (Reasoning and Common Sense), SE (Style and Emotion), and LD (Local Details). A, B, and C indicate whether the method is based on resampler, pooling, or pruning, respectively.
Method
GU
SN
RC
SE
LD
Avg.
LLaVA-1.5-7B
56.1
47.4
65.9
72.0
60.9
61.0
A
54.7
46.1
64.3
74.7
54.9
59.5
B
56.5
47.8
61.2
71.7
56.2
59.2
C
55.8
47.4
61.6
74.3
60.5
60.3
A+B
56.8
47.8
63.1
72.3
57.6
60.1
A+C
57.2
47.8
62.7
73.0
59.2
60.4
Table 4: Ablation study on different projector combinations across VTCBench. The results are reported across five dimensions: GU (Global Understanding), SC (Spatial and Counting), RC (Reasoning and Common Sense), SE (Style and Emotion), and LD (Local Details). A, B, and C denote the three compression methods: resampler, pooling, and pruning, respectively.
Tokens
VQA T
VQA v2
GQA
MMB
POPE
MM-Vet
Avg.(%)
576
58.2
78.5
62.0
64.3
85.9
31.6
100.00
144
57.1
78.1
62.0
64.8
87.7
33.3
99.49
64
56.0
76.4
60.2
64.6
87.2
30.8
96.91
36
54.2
74.1
58.4
62.3
86.8
28.7
93.67
16
51.0
73.2
58.2
61.3
86.5
26.3
92.34
4
51.1
70.8
56.7
59.6
82.9
26.4
89.03
Table 5: Ablation on the number of visual tokens in QMoP.
Tokens
TFLOPs
KVcache
Train
Inference
Performance
time
time
576
3.82
302.0M
102.3h
3h05m
100%
144
0.94
75.5M
75.6h
2h29m
100.73%
64
0.42
33.6M
65.5h
2h20m
97.97%
36
0.23
18.9M
61.3h
2h20m
94.97%
16
0.10
8.4M
58.0h
2h18m
92.78%
Table 6: Complexity comparison between QMoP and baseline (LLaVA-1.5-7B).
Method
TokenPacker
LDP-V2
QGR
QMoP
FLOPs
10.73G
12.1G
3.57G
7.29G+3.57G
Table 7: Efficiency breakdown of the projector overhead.
Figure 12
Figure 5: Qualitative comparison between LLaVA-1.5-7B and our proposed QMoP.
Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.
Yupeng Zheng, Kai Zou, Bin Liu +1
Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China Hefei, China
Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language understanding tasks, yet their inference efficiency is often hampered by the large number of visual tokens, particularly in high-resolution or multi-image scenarios. To address this issue, we propose EvoComp, a visual token compression framework that significantly reduces token count while preserving task accuracy. EvoComp introduces a lightweight encoder-only transformer-based compressor that selects the most informative and non-redundant visual tokens by jointly considering visual and textual contexts. A core challenge lies in providing effective supervision for training the compressor. To this end, we design an evolutionary labeling strategy that searches for token subsets minimizing the MLLM's output loss, while enforcing semantic diversity through vocabulary-based token grouping. We further train the compressor using a tailored loss function combining the GHM loss to mitigate class and difficulty imbalance, and a cosine similarity regularization to encourage semantic separation between retained and discarded tokens. Extensive experiments across multiple vision-language benchmarks show that EvoComp outperforms existing methods based on attention or similarity heuristics. Notably, it retains 99.3% of the original accuracy under 3x token compression and delivers up to 1.6x speedup on mobile devices.
Large vision-language models (VLMs), which process both visual inputs and text queries, incur high inference costs due to the large number of visual tokens, making visual token pruning a natural approach for improving efficiency. Recent work has leveraged query information to guide token selection, often through heuristics derived from attention patterns or token dynamics. However, these approaches do not explicitly exploit the underlying structure of query-vision interactions. We address this gap by observing that the query-vision interaction matrix, naturally induced by the query and key projections already present in transformer attention, is effectively low-rank, with a small number of dominant latent interaction modes capturing query-relevant semantics. Motivated by this observation, we propose Query-Vision Decomposition (QViD), a novel training-free query-aware visual token pruning method that exploits this structure. Extensive experiments on both image and video understanding benchmarks show that QViD consistently improves performance under matched token budgets, with the largest gains appearing under aggressive compression regimes.
Harshithanjani Athi, Sravan Kumar Ankireddy, Jianzhong Charlie Zhang +1
1The University of Texas at Austin · 2Samsung Research America