Multimodal large language models suffer from severe computational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projector modules to align and compress visual tokens into text-aligned features, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resampler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and textual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual Token Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time.
Figures & tables
Figure 1: Qualitative comparisons of three representative visual token compression methods under different scenarios.
Figure 2: Comparison of compression strategies across tasks. Pooling excels on global understanding, pruning leads on local details, and resampling performs best on style and emotion, highlighting task-dependent advantages.
Figure 3: The architecture of the proposed QMoP.
Figure 4: Overview of VTCBench and the quantitative analysis of compression strategies.
Methods
VQA T
POPE
VQA v2
MM-Vet
GQA
MME
MMB
Seed
MMStar
MMMU
Avg.
LLaVA-1.5-7B, Vision token = 144
Vanilla [ 21 ]
58.2
85.9
78.5
31.6
62.0
1511
64.3
66.1
31.9
34.6
100.00%
Pixel-Shuffle [ 8 ]
54.2
87.0
75.6
29.1
60.2
1422
64.5
61.6
31.7
32.7
96.14%
FasterVLM [ 40 ]
57.1
83.5
76.2
32.2
58.0
1440
64.3
62.3
31.3
34.8
97.64%
C-Abstractor [ 5 ]
54.1
85.9
75.7
28.2
60.4
1392
63.3
62.3
31.1
33.4
95.48%
MQT-LLaVA [ 13 ]
52.6
86.2
76.8
29.8
61.4
1445
64.4
62.0
31.9
34.8
97.22%
Table 1: Performance comparison with pre-LLM compression methods on LLaVA-1.5-7B. The best and second-best results are highlighted in bold and underlined .
Method
Vision
VQA T
POPE
VQA v2
GQA
MMB
MMB-
OCR
Avg.
Token
CN
Bench
LLaVA-1.5-7B
576
58.2
85.9
78.5
62.0
64.3
58.3
297.0
100.00%
FastV [ 6 ]
192
52.5
64.8
67.1
52.7
61.2
57.0
291
89.58%
HiRED [ 2 ]
192
47.4
82.8
74.9
58.7
62.8
54.7
190
89.06%
FitPrune [ 35 ]
192
57.4
83.4
-
60.4
63.3
56.4
-
-
LLaVA-PruMerge [ 28 ]
192
54.3
71.3
70.6
54.3
59.6
52.9
253
88.92%
Table 2: Performance comparison with intra-LLM visual token compression methods on LLaVA-1.5-7B.
Method
Type
GU
SC
RC
SE
LD
Avg.
LLaVA-1.5-7B [ 21 ]
-
56.1
47.4
65.9
72.0
60.9
61.0
MQT-LLaVA [ 13 ]
A
54.7
46.1
64.3
74.7
54.9
59.5
Pixel-Shuffle [ 8 ]
B
53.6
47.0
61.2
70.0
55.3
57.9
C-Abstract [ 5 ]
B
53.2
47.8
59.6
70.3
55.2
57.7
TokenPacker [ 18 ]
B
55.8
47.8
64.7
74.0
57.5
60.5
LDP-V2 [ 9 ]
B
56.8
48.3
63.9
74.3
56.4
60.5
Table 3: Performance comparison on VTCBench. The results are reported across five dimensions: GU (Global Understanding), SC (Spatial and Counting), RC (Reasoning and Common Sense), SE (Style and Emotion), and LD (Local Details). A, B, and C indicate whether the method is based on resampler, pooling, or pruning, respectively.
Method
GU
SN
RC
SE
LD
Avg.
LLaVA-1.5-7B
56.1
47.4
65.9
72.0
60.9
61.0
A
54.7
46.1
64.3
74.7
54.9
59.5
B
56.5
47.8
61.2
71.7
56.2
59.2
C
55.8
47.4
61.6
74.3
60.5
60.3
A+B
56.8
47.8
63.1
72.3
57.6
60.1
A+C
57.2
47.8
62.7
73.0
59.2
60.4
Table 4: Ablation study on different projector combinations across VTCBench. The results are reported across five dimensions: GU (Global Understanding), SC (Spatial and Counting), RC (Reasoning and Common Sense), SE (Style and Emotion), and LD (Local Details). A, B, and C denote the three compression methods: resampler, pooling, and pruning, respectively.
Tokens
VQA T
VQA v2
GQA
MMB
POPE
MM-Vet
Avg.(%)
576
58.2
78.5
62.0
64.3
85.9
31.6
100.00
144
57.1
78.1
62.0
64.8
87.7
33.3
99.49
64
56.0
76.4
60.2
64.6
87.2
30.8
96.91
36
54.2
74.1
58.4
62.3
86.8
28.7
93.67
16
51.0
73.2
58.2
61.3
86.5
26.3
92.34
4
51.1
70.8
56.7
59.6
82.9
26.4
89.03
Table 5: Ablation on the number of visual tokens in QMoP.
Tokens
TFLOPs
KVcache
Train
Inference
Performance
time
time
576
3.82
302.0M
102.3h
3h05m
100%
144
0.94
75.5M
75.6h
2h29m
100.73%
64
0.42
33.6M
65.5h
2h20m
97.97%
36
0.23
18.9M
61.3h
2h20m
94.97%
16
0.10
8.4M
58.0h
2h18m
92.78%
Table 6: Complexity comparison between QMoP and baseline (LLaVA-1.5-7B).
Method
TokenPacker
LDP-V2
QGR
QMoP
FLOPs
10.73G
12.1G
3.57G
7.29G+3.57G
Table 7: Efficiency breakdown of the projector overhead.
Figure 12
Figure 5: Qualitative comparison between LLaVA-1.5-7B and our proposed QMoP.