Multimodal large language models (MLLMs) encode a high-resolution image into thousands of visual tokens, which dominate the cost of the prefill stage. Visual token reduction removes most of these tokens, but two problems remain at extreme budgets. The vision encoder still processes every patch of the image, and the few surviving tokens often miss the small regions that hold the answer. In this paper, we propose Glance-ETD, a two-stage token reduction framework guided by a lightweight question-conditioned scorer. In the first stage, a glance encoder views the image at low resolution and re-encodes only the question-relevant regions at native resolution. In the second stage, Evidence-Tempered Diversity (ETD) selects the final tokens by combining diversity with the evidence predicted by the scorer, so that the kept tokens cover the scene and retain the answer. A single-rollout self-distillation recipe trains the MLLM with the glance encoder. With only 5% of the visual tokens, Glance-ETD retains 91.6% of the full-token performance of Qwen3.5-4B across fourteen benchmarks and outperforms the strongest prior method, LT-OPD, by 17.3 points and the same training recipe with DivPrune by 6.0. It also speeds up the time to first token on 4K images by 3.1x and generalizes to three other MLLMs and to high-resolution document benchmarks, with the largest gains on text-rich tasks.
Figures & tables
Figure 1: Retention across visual-token budgets on Qwen3.5-4B: the mean over nine benchmarks 2 2 2 V ∗ [ 60 ] , HR-Bench 4K [ 55 ] , GQA [ 21 ] , MMMU [ 65 ] , MMBench (circular) [ 33 ] , MME [ 15 ] , POPE [ 29 ] , TextVQA [ 48 ] and OCRBench [ 34 ] . of each score divided by our non-thinking full-token score, with the same number of tokens per image. Solid lines are trained models 2 2 2 V ∗ [ 60 ] , HR-Bench 4K [ 55 ] , GQA [ 21 ] , MMMU [ 65 ] , MMBench (circular) [ 33 ] , MME [ 15 ] , POPE [ 29 ] , TextVQA [ 48 ] and OCRBench [ 34 ] . ; dashed lines are training-free and run on the released model, as in their papers. At 5% of the tokens, Glance-ETD keeps 93.3% (the mean of three training seeds; the other budgets use one seed), 17.5 points above DivPrune.
Figure 2: Two problems at a strict 5% visual-token budget on Qwen3.5-4B. (a) Time to first token on HR-Bench 4K for the full-token model, pruning after the ViT, and Glance-ETD ( Tab. S14 ). Each bar is split into vision encoding, selection, LLM prefill and CPU preprocessing. Pruning after the ViT still spends 2.64 s in the encoder; the glance encoder cuts that stage to 0.57 s. (b) One TextVQA image at the same budget: outlines are kept tokens and the dotted box is the answer. DivPrune keeps 1 of 10 text tokens, relevance top- K keeps all 10 but little context, and ETD keeps 4 plus context. (c) On 4,123 TextVQA validation questions with a model self-distilled at this budget, bars are the share of tokens overlapping the answer OCR box and diamonds the share of questions keeping at least one. The dashed line is the 5% budget.
Figure 3: Glance-ETD. (a) Training: the student (glance encoder and LLM) selects with DivPrune, rolls out, and is distilled token by token towards a frozen full-token teacher; the scorer fϕ is trained separately on box targets with the MLLM frozen. (b) Inference: Stage 1 ( Sec. 3.2 ) encodes the image at a quarter of the patches, the scorer fϕ picks the 25% of the coarse cells with the highest question relevance, and only these are re-encoded at native resolution, giving a grid X with all N native positions. Stage 2 ( Sec. 3.3 ) maps X and the question to an evidence map p , and ETD greedily selects K tokens of X by di(Npi)0.25 (shown for q1 , on an illustrative evidence map).
Table 1: Comparison with token-reduction methods on Qwen3.5-4B and Gemma 4 E4B-it at a strict 5% budget, where every method keeps the same number K of visual tokens per image. Methods are grouped into training-free methods, learned modules trained with the MLLM frozen (ETD trains only its scorer), and methods that adapt the MLLM. The two self-distillation rows share the same training recipe and differ only in the token selector; Self-distillation + ETD is our method with the dense encoder. Ret. is the score on each benchmark relative to the same model’s full-token score, averaged over the fourteen benchmarks. Self-distilled rows report the mean ± std over three training seeds ( Tab. S1 ). ‡ GQA answers are lower-cased before scoring.
Table 3: Ablation of the design choices in Glance-ETD on Qwen3.5-4B and Gemma 4 E4B-it at a strict 5% budget. Each row changes one choice of the full pipeline: Stage 1 rows change which glance cells are re-encoded, and Stage 2 rows change how tokens are selected. Ret. is the retention over the nine LT-OPD benchmarks (six of which are shown), and Δ is its change from the full pipeline. All rows use training seed 0; three-seed means of seven Qwen rows keep the same order ( Sec. D.2 ).
10 × LR
1 × LR
No train
Glance
–
–
✓
✓
–
–
✓
✓
–
–
ETD
–
✓
–
✓
–
✓
–
✓
–
✓
V ∗
75.0
79.1
74.9
83.6
73.8
76.4
73.1
82.0
70.2
78.0
HR-4K
77.1
77.2
76.8
78.5
77.2
77.2
77.1
78.3
73.4
75.8
GQA
54.2
55.0
54.1
54.7
52.3
53.3
52.0
52.5
45.3
47.2
MMMU
48.3
48.8
50.4
49.6
48.9
48.9
48.3
48.3
46.3
46.1
Table 2: Component ablation on Qwen3.5-4B at a strict 5% budget. Glance: glance encoder, otherwise dense. ETD: evidence-tempered selection, otherwise DivPrune. The 10 × and 1 × LR groups self-distil the MLLM with that ViT learning rate (1 × is the LT-OPD recipe); No train is the released model. Ret.: retention over the nine LT-OPD benchmarks, mean ± std over three seeds for trained models. TFLOPs: ViT + LLM prefill on HR-Bench 4K. Shaded: Glance-ETD.
Figure 4: Efficiency at strict 5%, measured on one A800 in bf16. (a) Retention over the nine benchmarks against TTFT speed-up on HR-Bench 4K; the dashed line is the Pareto front. (b) Peak memory on HR-Bench 4K at batch 1, 4 and 8. (c) TTFT and (d) end-to-end speed-up at batch 1, on three benchmarks per model ordered by image size. Tab. S15 covers every benchmark of both models, and Tabs. S14 and S2 the rest of the measurements.
Figure 5: Retention over nine benchmarks at strict 5% on four MLLMs. SD is self-distillation with the dense encoder; Tab. S13 gives the numbers.
Method
Seed
GQA
MMMU
MMB
MME
POPE
V ∗
HR-4K
HR-8K
MME-RW
HRScene
TextVQA
OCRB
DocVQA
InfoVQA
Ret. All
Self-distillation + DivPrune
0
54.0
47.9
73.9
1975
86.3
74.9
77.0
73.5
45.9
56.2
70.5
481
68.4
36.0
85.7
1
54.1
48.3
73.2
2033
85.3
74.9
76.6
72.8
45.2
54.9
70.0
500
67.2
35.6
85.5
2
54.4
48.7
73.5
1957
86.1
75.4
77.8
73.0
45.3
54.8
70.6
486
66.0
35.6
85.4
Glance-ETD
0
53.8
49.6
75.5
1955
88.2
82.7
78.9
75.0
48.0
57.0
76.8
583
82.8
48.7
91.2
1
55.0
50.1
75.2
1998
88.1
84.3
78.9
76.0
50.0
55.0
75.5
603
82.4
47.5
91.8
2
55.3
49.1
74.9
2001
88.4
83.8
77.9
76.6
48.3
58.1
76.8
584
82.5
48.0
91.7
Table S1: Three training seeds at the strict 5% budget of Tab. 1 . The main table reports the mean over the three seeds; the best seed (Ret. All) is in bold.
Method
General
High-res.
Text
All
DivPrune
81.7
87.1
38.4
71.3
ETD (ours)
84.6
93.2
64.9
82.0
LT-OPD
85.7
85.4
46.2
74.3
Self-distillation + DivPrune
92.5
94.8
65.3
85.6
Self-distillation + ETD (ours)
94.0
96.2
79.0
90.5
Glance-ETD (ours)
94.1
99.8
78.1
91.6
Table S2 : Retention (%) by benchmark group of Tab. 1 at strict 5% on Qwen3.5-4B. General: GQA, MMMU, MMBench, MME, POPE. High-res.: V ∗ , HR-Bench 4K / 8K, MME-RealWorld, HRScene. Text: TextVQA, OCRBench, DocVQA, InfoVQA. Trained rows are three-seed means.
General
Fine-grained high resolution
Text and document
Ret. (%)
Method
GQA
MMMU
MMB
MME
POPE
V ∗
HR-4K
HR-8K
MME-RW
HRScene
TextVQA
OCRB
DocVQA
InfoVQA
All
LT-OPD variants
Qwen3.5-4B
Full tokens (100%)
60.0
51.3
79.0
2218
89.9
80.6
78.0
72.6
48.9
62.5
80.5
848
94.6
79.4
100.0
LT-OPD [ 26 ] , released (with merge)
52.8
49.8
73.5
1895
85.2
73.3
71.5
67.8
44.4
54.7
63.3
419
48.9
27.4
80.5
LT-OPD w/o merge
47.4
48.6
66.4
1771
81.9
67.0
69.6
65.1
40.5
51.4
55.8
341
40.6
25.6
74.3
Glance-ETD (ours)
54.7
49.6
75.2
1985
88.2
83.6
78.5
75.9
48.8
56.7
76.3
590
82.6
48.1
91.6 ± 0.3
Table S3 : LT-OPD with and without its learnable token merge on Qwen3.5-4B at a strict 5% budget (CDPruner selection, as trained). The released checkpoint keeps the merge; w/o merge is the LT-OPD-14K recipe retrained with the merge turned off, the LT-OPD row of Tab. 1 and Footnote 2 . Ret. as in Tab. 1 .
Rollouts per prompt, answer cap
Training time
Ret. (%)
8, 1024 (LT-OPD default)
17.4 h
90.1
2, 1024
8.7 h
91.1
1, 1024
7.3 h
90.5
1, 256 (ours)
4.7 h
90.7
1, 128
4.3 h
90.3
Table S4: Training cost of the self-distillation recipe (LT-OPD-14K, 175 steps, 8 A800 GPUs; CDPruner selection at the floor-32 budget of the LT-OPD release). Retention here uses a full-token reference with MMMU 52.4 instead of 51.3, which lowers every value by about 0.2 points relative to the main tables; the comparison within the table is unaffected.
Method
V ∗
HR-4K
GQA
MMMU
MMB
MME
POPE
TextVQA
OCRB
Ret. (%)
DivPrune, strict 5%
Dense ViT
73.8
76.2
52.4
50.1
71.0
1860
85.3
64.9
384
85.4
Dense ViT, 10 × LR
74.9
77.0
54.0
47.9
73.9
1975
86.3
70.5
481
88.6
Low-res (1/4 patches)
70.2
75.1
53.5
46.9
70.4
1942
85.3
70.5
504
86.9
Glance 25%, auxiliary loss
73.3
77.4
53.6
48.1
72.6
1883
84.6
70.3
475
87.5
Glance 25% + window
70.2
72.5
53.7
50.4
73.2
1955
85.7
70.3
501
87.8
Table S5: Vision-encoder variants, all trained with the same LT-OPD recipe and 10 × ViT learning rate except the first row. Low-res, window and glance variants except the last row add the auxiliary cosine loss ( Sec. C.2 ). Seed 0, selection at inference only.
Method
V ∗
HR-4K
GQA
MMMU
MMB
MME
POPE
TextVQA
OCRB
Ret. (%)
Strict 5%
DivPrune ( γ=0 )
73.8
76.2
52.4
50.1
71.0
1860
85.3
64.9
384
85.4
Split: top- K/2 by p + DivPrune
78.0
75.9
52.4
50.0
70.9
1844
85.8
71.6
455
87.7
Split, scorer fine-tuned with GRPO
75.9
77.6
52.5
50.3
72.4
1887
86.1
71.7
481
88.6
Tilt set by the text confidence, di(Npi)t
78.0
77.6
53.2
48.7
71.8
1842
87.7
73.3
486
88.8
ETD, di(Npi)0.25
78.5
77.5
53.2
49.3
72.4
1842
87.6
74.3
510
89.6
Table S6: Selection rule ablation on the LT-OPD + DivPrune model, identical K per image. Top: strict 5%. Bottom: max(32,⌈0.05N⌉) .
Scorer
Data
Boxes
Question
Top- K
DPP- r
ETD
Ours
422K
✓
✓
77.9
82.5
83.8
14K box records
14K
✓
✓
73.6
81.5
81.1
No question
422K
✓
–
73.9
80.1
81.3
No boxes (LearnPruner)
14K
–
✓
57.8
74.1
75.8
Table S7 : Scorer supervision and selector with the MLLM frozen, strict 5%, retention over the nine LT-OPD benchmarks (DivPrune: 75.8%). All scorers have the same 1.9M-parameter architecture; 14K = as many box records as LT-OPD-14K; LearnPruner = its objective on LT-OPD-14K.
Stage-1 cells
Rfine
Ret.
HR-8K
DocVQA
InfoVQA
MME-RW
HRScene
H4
TTFT (s)
Top- p (ours)
88 ‡
93.3
75.9
82.6
48.1
48.8
56.7
59.0
1.28
Diversity (as trained)
32
92.0
70.3
79.5
44.8
46.0
55.6
56.5
1.39
Random
26
91.1
67.9
80.2
46.3
44.8
55.4
56.7
1.31
Query-aware
73
93.0
74.9
81.4
46.6
48.0
56.8
58.2
1.42
Half top- p
79
92.9
74.4
81.8
47.1
48.0
56.4
58.3
1.45
Table S8: Stage-1 cell rules at inference on the glance model at strict 5% (mean of three training seeds; Stage 2 is ETD in every row). Rfine : share of answer-box cells of the 345 glanced TextVQA questions that are re-encoded at native resolution ( ‡ cosine-regularised encoder, seed 0). HR-8K of random: seed 2 only. Ret.: nine LT-OPD benchmarks. H4: mean of DocVQA, InfoVQA, MME-RealWorld and HRScene. TTFT: HR-Bench 4K, batch 1, one A800 (full tokens 4.15 s); timed with the cosine-regularised weights, which do not change the compute; half top- p was timed in a second run and scaled by its top- p time.
Method
V ∗
HR-4K
GQA
MMMU
MMB
MME
POPE
TextVQA
OCRB
Ret. (%)
Full tokens
80.6
78.0
60.0
51.3
79.0
2218
89.9
80.5
848
100.0
DivPrune [ 2 ] , training-free
70.7
73.9
53.4
48.8
75.5
1925
86.6
49.3
431
84.1
CDPruner [ 68 ] , training-free
53.4
62.4
40.8
48.2
74.2
1862
81.9
43.5
398
75.4
LT-OPD [ 26 ] (arXiv’26.09)
67.0
69.6
51.7
49.4
77.8
2073
85.9
56.1
533
86.1
LT-OPD + DivPrune (ours, reproduced)
73.8
76.9
57.5
51.1
80.8
2024
89.5
65.4
589
92.2
Ours and its dense-encoder variants
Table S9: Operating point of the LT-OPD release, K=max(32,⌈0.05N⌉) per image ( ∑K/∑N=7.1% ). Rows above the line are baselines and LT-OPD-recipe models with DivPrune; rows below are ours.
1%
2%
3.5%
5%
7.5%
10%
15%
25%
Untrained
ETD
66.3
75.2
81.0
83.8
87.3
90.1
93.0
95.7
DivPrune
56.0
64.6
71.5
75.8
80.9
83.7
87.6
92.0
DART
49.2
55.5
62.4
67.7
73.2
77.2
84.4
91.0
CDPruner
51.3
58.3
63.7
67.6
71.5
74.3
78.6
84.4
PruneSID
51.7
59.2
65.1
68.7
77.0
80.5
81.7
86.0
Table S10 : Retention (%) versus strict budget ∑K/∑N .
Allocation
strict 5%
floor-32 budget (7.1%)
Same ratio for every image
89.6
95.9
K∝N0.45
94.6
96.3
K∝N0.45⋅2[open]
95.1
97.3
Table S11: Global budget allocation on the LT-OPD model with ETD. The power of N and the open-prompt factor are fitted on training questions only; the scale is set to match the reference ∑K/∑N over the benchmarks shown below.
Method
V ∗
HR-4K
GQA
MMMU
MMB
MME
POPE
TextVQA
OCRB
Ret. (%)
LT-OPD + DivPrune model
DivPrune
73.8
76.2
52.4
50.1
71.0
1860
85.3
64.9
384
85.4
ETD
78.5
77.5
53.2
49.3
72.4
1842
87.6
74.3
510
89.6
ETD, wrong question
77.5
78.0
51.3
49.7
71.4
1871
87.0
68.9
441
87.5
ETD complement
72.3
77.4
51.9
49.4
72.5
1907
85.7
61.6
393
85.3
Blind (0.1%)
51.8
59.6
35.4
46.6
46.1
1314
61.4
10.9
59
55.2
Table S12: Does ETD keep the evidence? Strict 5%. Wrong question : ETD scores tokens with the question of another sample, the model then answers the true question. Complement : ETD-selected tokens are removed from the pool before selecting. Blind : 0.1% of tokens.
Figure S1 : Retention of Gemma 4 E4B-it against its own full-token scores. ETD and DivPrune pass the same number of visual tokens per image. Every marker averages all nine benchmarks.
Method
Qwen3.5 4B
Gemma 4 E4B-it
GLM-4.6V Flash
InternVL3.5 8B
DivPrune
75.8
74.8
74.7
69.4
ETD
83.8
84.8
84.3
82.4
Self-distillation + DivPrune
88.8
87.2
83.5
74.2
Self-distillation + ETD
92.5
91.3
89.3
86.3
Glance-ETD
93.3
91.1
88.2
83.8
Table S13 : Retention over the nine LT-OPD benchmarks at strict 5% on four MLLMs ( Fig. 5 plots GLM and InternVL). Each model is compared with its own full-token scores. Qwen and Gemma trained rows are means of three seeds; GLM and InternVL use one seed.
Figure S2 : Further efficiency views (strict 5%, Qwen3.5-4B). (a)(b) Task-matched trade-off: HR-Bench 4K and V ∗ accuracy against TTFT speed-up on the same benchmark (markers as in Fig. 4 a). (c) Retention against HR-Bench 4K prefill TFLOPs as the budget goes from 1% to 25% ( Tab. S10 ). (d) Prefill FLOPs per question by stage (F full, D dense + ETD, G Glance-ETD). (e) HR-Bench 4K speed-up of each stage (ViT, LLM prefill, TTFT, decoding, end to end). (f) Measured TTFT speed-up against the FLOPs ratio (hatched) on HR-Bench 8K resized to 2K–8K. (g) Throughput speed-up at batch 1 and 8 ( Tab. S18 ). (h) Peak memory at batch 8 on resized HR-Bench 8K. (i) HR-Bench 4K latency by stage at batch 1 for all five configurations (end-to-end ms at the bar ends). (j)(k) TTFT and end-to-end speed-up against the mean number of visual tokens, on the benchmarks where Glance-ETD exceeds 1.15 × . (l) Decode cache per request: KV of the eight full-attention layers plus the constant Gated-DeltaNet state. (a)–(c) and (e)–(i) use the run of Tab. S14 for full tokens and the dense and glance encoders; (d) and (j)–(l) use a 4-GPU run that covers all 12 benchmarks, with speed-ups taken against its own full-token run. The resized sizes in (f)(h) and the V ∗ bars in (g) were measured only in the batched-ViT run described in the protocol below.
Full
Dense + ETD
Glance-ETD
TTFT
4035
3355 (1.20 × )
1290 ( 3.13 × )
V ∗ TTFT
539
470 (1.15 × )
366 (1.47 × )
End-to-end
4329
3658 (1.18 × )
1588 (2.73 × )
TFLOPs
366 + 116
366 + 6
59 + 6
Batch-8 q/s
0.276
0.312 (1.13 × )
0.698 (2.53 × )
Mem. b1/b8 (GB)
13.0 / 33.9
13.3 / 13.3
12.1 / 12.1
Table S14: Efficiency on one A800-SXM4-80GB (bf16, median over 30 questions, configurations run one after another). HR-4K unless noted; batch 1 unless noted. Times are milliseconds. TFLOPs are ViT + LLM, attention included. Speed-ups over full tokens are in parentheses. Peak memory is batch 1 / batch 8. Glance-ETD is the final model with scorer-chosen cells; the HR-4K and batch rows come from the run that timed both, the V ∗ row from its companion run with the same protocol.
Qwen3.5-4B
Gemma 4 E4B
Dense + ETD
Glance-ETD
ETD
Glance-ETD
Benchmark
TTFT
E2E
TTFT
E2E
TTFT
E2E
TTFT
E2E
HR-4K
1.20
1.18
3.13
2.73
1.05
1.04
1.13
1.11
HR-8K
1.18
1.16
2.65
2.40
1.26
1.03
1.15
0.93
HRScene
1.47
1.01
1.92
1.36
1.05
0.82
1.35
0.96
DocVQA
1.20
1.15
1.57
1.30
1.05
1.24
1.37
1.36
Table S15 : Prefill (TTFT) and end-to-end (E2E) speed-up over full tokens on every benchmark with a latency measurement, strict 5% (batch 1, median over 30 questions, one A800). Qwen3.5-4B: same runs as Tab. S14 ; Gemma 4 E4B: trained Glance-ETD model ( Sec. E.5 ), about 1.1k visual tokens per image, configurations interleaved per question. Fig. 4 c,d shows, per panel, the three benchmarks with the largest Glance-ETD speed-up of each model.
HR-Bench 4K
V ∗
Method
Ret.
TTFT
Gain
Sel.
Gain
Mem.
Full tokens
100.0
4099
–
0
–
13.0
DivPrune
71.3
3430
1.19 ×
141
1.18 ×
13.2
DART
62.8
3469
1.18 ×
266
1.14 ×
13.1
CDPruner
61.8
3776
1.09 ×
592
1.05 ×
13.0
PruneSID
65.2
3948
1.04 ×
769
1.09 ×
13.3
Table S16 : Latency of the Tab. 1 methods on Qwen3.5-4B at strict 5% (one A800, bf16, batch 1, median over 30 questions, configurations run one after another; speed-ups over full tokens of the same run). Gain is the speed-up of TTFT over full tokens. TTFT and Sel. (selection) in ms, Mem. is HR-Bench 4K peak memory in GB. Sel. is the selection time on HR-Bench 4K (our implementations; DivPrune and ETD use a CUDA-graph greedy). VisionTrim recomputes a ViT attention map, which the fused attention kernel does not return (1.5 s of extra ViT time); ZOO-Prune spends 1.35 s on 64 zeroth-order probes of the projector. Ret. is Ret. All of Tab. 1 (fourteen benchmarks). The Glance-ETD row was timed with diversity-chosen glance cells; choosing them with the scorer adds one scorer call of about 2 ms ( Sec. F ).
Long side
2K
4K
8K
Visual tokens N
2,688
10,752
16,224
Full TTFT p50/p95
385 / 532
2283 / – ‡
4511 / 4719
Dense + ETD TTFT
331
1913
3845
Glance-ETD p50/p95
230 / 384
1015 / 1801
1797 / 1855
ViT/Sel./LLM/CPU
71/20/60/78
402/127/91/392
678/161/91/865
Speed-up
1.67 ×
2.25 ×
2.51 ×
Table S17 : Image size. HR-Bench 8K images resized to a long side of 2K–8K pixels (24 questions, batch 1). Glance-ETD TTFT is split into ViT, scorer plus selection (Sel.), LLM prefill and CPU; speed-up is full / Glance-ETD at the median. Memory is peak allocated GPU memory of that configuration on the single A800 used for the timing run. Times are in ms.
In multimodal large language models (MLLMs), inference cost is largely dominated by the visual token prefix rather than the language backbone, making token reduction a key factor for improving efficiency. Existing approaches typically assign independent importance scores to visual tokens and retain a fixed number of top-ranked tokens, implicitly assuming token independence and a uniform compression ratio across inputs. In this work, we reformulate visual token pruning as a sequential decision-making process. Specifically, we introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop via a learned termination action. This enables joint optimization of both the selected subset and its size. To enable end-to-end training under standard language modeling objectives, we design a differentiable relaxation based on a variance-preserving noise interpolation scheme, allowing gradients to propagate through the discrete selection process. Extensive experiments on LLaVA-v1.5-7B and Qwen2.5-VL-7B demonstrate that our approach consistently outperforms fixed-ratio baselines across different compression levels. Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.
Recent multimodal large language models are computationally expensive because Transformers must process a large number of visual tokens. We present ReDiPrune, a training-free token pruning method applied before the vision-language projector, where visual features remain rich and discriminative. Unlike post-projection pruning methods that operate on compressed representations, ReDiPrune selects informative tokens directly from vision encoder outputs, preserving fine-grained spatial and semantic cues. Each token is scored by a lightweight rule that jointly consider text-conditioned relevance and max-min diversity, ensuring the selected tokens are both query-relevant and non-redundant. ReDiPrune is fully plug-and-play, requiring no retraining or architectural modifications, and can be seamlessly inserted between the encoder and projector. Across four video and five image benchmarks, it consistently improves the accuracy-efficiency trade-off. For example, on EgoSchema with LLaVA-NeXT-Video-7B, retaining only 15% of visual tokens yields a +2.0% absolute accuracy gain while reducing computation by more than 6× in TFLOPs. Code is available at https://github.com/UA-CVML/ReDiPrune.
Visual token reduction is an effective way to accelerate multimodal large language models (MLLMs), but performance deteriorates rapidly under extremely low token budgets. Existing work has explored both visual-token selection and training-based adaptation to reduced visual inputs. We take a step further by asking how a heavily compressed MLLM should learn from the states induced by its own generations. This setting naturally calls for on-policy self-distillation: a heavily compressed model is supervised on the states induced by its own generations, while its full-token counterpart serves as an information-rich teacher. Based on this insight, we propose LT-OPD, a training framework for extreme visual-token reduction. The student rolls out responses with only a small fraction of visual tokens, and a frozen full-token copy of the same MLLM provides distributional supervision along these student-generated trajectories. To stabilize on-policy learning when visual evidence is severely limited, we further introduce a budget-level curriculum that progressively decreases the token budget during training. Across nine benchmarks on Qwen3.5-4B, LT-OPD raises average retained performance under 5% visual-token retention from 68.6% to 82.3%, outperforming training-free, training-based, and reinforcement-learning baselines at the same budget. The gains transfer consistently to Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. LT-OPD also reduces KV-cache usage by 85.2% and prefill FLOPs by 85.4% without additional inference overhead, demonstrating that on-policy learning can substantially recover capabilities lost to extreme visual-token reduction.
Junxian Li, Ruixuan Yang, Tianao Zhang +3
Shanghai Jiao Tong University · Xi’an Jiaotong University · University of Cambridge +1