Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction

Junxian Li; Ruixuan Yang; Tianao Zhang; Tiange Xu; Weisheng Dong; Yulun Zhang

cs.CVSep 26, 2026

Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction

Authors: Junxian Li, Ruixuan Yang, Tianao Zhang, Tiange Xu, Weisheng Dong, Yulun Zhang

Organizations: Shanghai Jiao Tong University · Xi’an Jiaotong University · University of Cambridge · Xidian University

Abstract

Visual token reduction is an effective way to accelerate multimodal large language models (MLLMs), but performance deteriorates rapidly under extremely low token budgets. Existing work has explored both visual-token selection and training-based adaptation to reduced visual inputs. We take a step further by asking how a heavily compressed MLLM should learn from the states induced by its own generations. This setting naturally calls for on-policy self-distillation: a heavily compressed model is supervised on the states induced by its own generations, while its full-token counterpart serves as an information-rich teacher. Based on this insight, we propose LT-OPD, a training framework for extreme visual-token reduction. The student rolls out responses with only a small fraction of visual tokens, and a frozen full-token copy of the same MLLM provides distributional supervision along these student-generated trajectories. To stabilize on-policy learning when visual evidence is severely limited, we further introduce a budget-level curriculum that progressively decreases the token budget during training. Across nine benchmarks on Qwen3.5-4B, LT-OPD raises average retained performance under 5% visual-token retention from 68.6% to 82.3%, outperforming training-free, training-based, and reinforcement-learning baselines at the same budget. The gains transfer consistently to Qwen3.5-9B, GLM-4.6V-9B, and LLaVA-OV-1.5-4B. LT-OPD also reduces KV-cache usage by 85.2% and prefill FLOPs by 85.4% without additional inference overhead, demonstrating that on-policy learning can substantially recover capabilities lost to extreme visual-token reduction.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation and performances of LT-OPD. Left: different visual token reduction methods all fail to perform well under an extremely low budget. Middle: Under this scenario, predictive distribution mismatch caused by fixed-prefix methods can lead to the wrong answer. LT-OPD, however, leverages on-policy learning to mitigate the problem. “dist.” means distribution and “student prefix” means prefix generated by students. Right: Compared with HiPrune (5%, 10%, 15%, 20%), and EPIC (5%), our LT-OPD reaches the best trade-off.

Method	Training Strategy	Prefix / State Source	On-policy
MQT-LLaVA	Randomly samples the number of query tokens and optimizes the language-modeling objective	Fixed training	No
LLaVA-Mini	Trains compression and modality pre-fusion modules through end-to-end instruction tuning	Fixed training	No
EPIC	Progressive token/layer consistency distillation with SFT and KL supervision	Fixed training	No
LearnPruner	Freezes the base VLM and trains the pruning module with next-token prediction and pruning losses	Fixed training	No
LT-OPD	Rolls out a low-token student and distills a frozen full-token self-teacher with a budget curriculum	Student-generated	Yes

Table 1: Comparison of representative training-based visual-token reduction methods. “Fixed prefix” means that generation states are determined by static training responses rather than rollouts of the current compressed model.

Figure 2: Framework of LT-OPD, consisting of two parts. First, an on-policy self-distillation algorithm is proposed to train a low-token student, with a full-token frozen self-teacher. “dist.” means distribution. Second, we design a budget-level curriculum learning process for stable training.

Figure 3: Visual token keep ratio changes.

Method	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TVQA	OCRB	Avg. Retain Ratio (%)
Full Visual-Token Models (100% Retention)
Mimo-VL-RL	83.3	73.5	55.9	66.7	84.4	1,814.7	88.6	76.8	866	-
MiniCPM-V-4.5	70.7	69.6	61.0	67.7	84.2	2,360.6	87.9	80.7	890	-
Qwen3-VL-Instruct-8B	84.8	79.6	61.6	69.6	84.5	2,387.4	89.0	81.7	819	-
Qwen3.5-4B ( Upper Bound )	84.3	84.4	60.7	77.6	86.4	2,300.8	89.3	81.2	850	100%
Training-Free

Table 2: Main results of different visual token reduction methods on Qwen3.5-4B . We bold the best and underline the second best scores under 5% budget . Orange lines are our results. “Avg. Retain Ratio” means average retain ratio compared to upper bound.

Method	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TextVQA	OCRB	Avg.
Qwen3.5-4B (5% visual token)	52.9	68.4	50.4	45.7	74.0	1,531.8	70.2	43.2	404	68.6%
+ LT-OPD	74.9	73.0	54.8	50.0	78.9	2,077.0	86.3	56.0	535	82.3%
Qwen3.5-9B (5% visual token)	54.5	59.6	49.4	51.7	77.7	1,809.4	82.3	46.6	430	71.1%
+ LT-OPD	70.7	68.1	53.6	52.9	80.5	2,147.9	86.2	61.5	539	81.4%

Table 3: Different model scales. We apply LT-OPD on 4B and 9B version. “Avg.” means average retain ratio compared to upper bound (4B or 9B full), the same as Table 2 .

Method	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TextVQA	OCRB	Avg.
Qwen3.5-4B (5% visual token)	52.9	68.4	50.4	45.7	74.0	1,531.8	70.2	43.2	404	68.6%
+ LT-OPD	74.9	73.0	54.8	50.0	78.9	2,077.0	86.3	56.0	535	82.3%
GLM-4.6V-9B + (5% visual token)	45.0	60.5	46.8	42.3	69.2	1,646.0	76.3	56.1	346	73.8%
+ LT-OPD (5% visual token)	51.3	63.4	53.8	48.3	83.4	2,146.3	83.3	64.8	497	86.3%
LlaVA-OV-1.5-4B + (5% visual token)	64.9	51.9	50.9	47.8	64.6	1,532.1	81.9	38.6	87	71.8%
+ LT-OPD (5% visual token)	64.9	54.0	55.9	49.9	78.5	1,940.5	87.4	46.8	430	84.1%

Table 4: Performance of LT-OPD on different kinds of MLLMs. “Avg.” means average retain ratio compared to upper bound (full version of models), the same as Table 2 .

Method	KV Cache (MB) $\downarrow$		FLOPs Ratio $\downarrow$		Prefill FLOPs (T) $\downarrow$		End-to-End Latency (ms/sample) $\downarrow$
	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$	Value	Speedup $\uparrow$
Qwen3.5-4B (full visual token)	26.59	–	1.000	–	6.002	–	1331.510	$1.000\times$
+ HiPrune (5% visual token)	3.93	85.23%	0.146	85.38%	0.878	85.38%	1308.029	$1.018\times$
+ EPIC (5% visual token)	3.93	85.23%	0.200	80.03%	1.198	80.03%	1296.922	$1.027\times$
+ LT-OPD (5% visual token)	3.93	85.23%	0.146	85.38%	0.878	85.38%	1293.691	$1.029\times$

Table 5: Efficiency of LT-OPD.

\Delta

means comparing with raw Qwen3.5-4B. On the 5% budget, LT-OPD can reduce prefilling cost a lot. Details are in Appendix E .

Method	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TextVQA	OCRB	Avg.
Qwen3.5-4B (5% visual token)	52.9	68.4	50.4	45.7	74.0	1,531.8	70.2	43.2	404	68.6%
+ GRPO	56.5	66.8	53.9	49.0	77.1	1,958.1	82.7	49.4	479	75.8%
+ GSPO	58.6	63.9	53.5	49.9	78.3	2,072.3	85.7	50.6	467	76.8%
+ DAPO	60.7	67.9	54.4	49.6	78.0	2,037.7	85.8	52.1	497	78.2%
+ LT-OPD	74.9	73.0	54.8	50.0	78.9	2,077.0	86.3	56.0	535	82.3%

Table 6: Compared with advanced RL algorithms. “Avg.” means average retain ratio compared to upper bound, the same as Table 2 .

Teacher	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TextVQA	OCRB	Avg.
Initial Policy	74.9	73.0	54.8	50.0	78.9	2,077.0	86.3	56.0	535	82.3%
Trust-Region Regularization	74.3	72.1	52.9	49.0	79.1	2,115.0	86.4	55.4	518	81.5%
EMA	75.4	73.4	54.1	50.0	77.6	2,026.6	84.9	55.7	511	81.3%

Table 7: Different teacher regularization strategies on Qwen3.5-4B. “Avg.” means average retain ratio compared to upper bound, the same as Table 2

Figure 4: Recovery at 5% visual-token retention. We report the scores of LT-OPD and compared RL algorithms. LT-OPD achieves the best recovery performances.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source / task bucket	Questions
OneThinker: multiple choice	3,000
OneThinker: mathematics	1,500
OneThinker: numerical answer	900
OneThinker: OCR	900
OneThinker: regression	900
PixMo Ask Model Anything	3,000

Appendix

Table 8: Composition of the 14,000-question training set.

Item	Value
Training questions / epochs / optimizer updates	14,000 / 1 / 175
Questions per update / rollouts per question	80 / 8
Optimizer	AdamW
Language model / native merger learning rate	$1\times 10^{-6}$ / $1\times 10^{-6}$
Vision encoder learning rate	$2\times 10^{-7}$
Adam betas / epsilon	$(0.9,0.999)$ / $10^{-8}$

Appendix

Table 9: Training configuration of the 14K curriculum model. Learning rates are the values after warmup.

Benchmark	Split / setting	Rows	New tokens
V ∗ Bench	test	191	16
HR-Bench 4K	4K	800	1024
GQA	Testdev balanced	12,578	16
MMMU	validation	900	128
MMBench EN Circular	EN dev (circular)	4,329	1024
POPE	Three COCO subsets	9,000	128

Appendix

Table 10: Evaluated splits, row counts, and maximum new tokens. All tasks also use the shared closing-answer-tag stop.

Quantity	Definition
LLM prefill FLOPs	Analytical cost of language-model prefill and the final-token output head.
Whole-prefill FLOPs	Analytical cost of vision encoding, selection, and language-model prefill.
Full-attention KV	Key/value tensors in the eight full-attention layers after prefill, including text context.
Total decode state	Full-attention KV plus convolutional and recurrent states in the 24 linear-attention layers.
Replay request time	Elapsed time from prepared CPU tensors to decoded text under the fixed reference replay.

Appendix

Table 11: Resource quantities used in the efficiency evaluation.

Method	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TextVQA	OCRB	Avg.
Qwen3.5-4B ( Upper Bound )	84.3	84.4	60.7	77.6	86.4	2,300.8	89.3	81.2	850	100%
Forward KL ( $\mathrm{KL}(p_{T}\\|p_{S})$ )	70.2	68.8	59.5	52.2	82.2	1,568.2	89.1	62.5	599	82.3%
Reverse KL ( $\mathrm{KL}(p_{S}\\|p_{T})$ )	70.2	71.3	59.6	51.4	82.0	1,600.4	89.1	63.6	599	82.8%
JSD ( $\beta=0.5$ )	74.9	73.0	54.8	50.0	78.9	2,077.0	86.3	56.0	535	82.3%

Appendix

Table 12: Training-objective results. Full-token and JSD results are retained from the main text. Avg. is the same as Table 2 in main text.

Method	V ∗ Bench	HR-4K	GQA	MMMU	MMB	MME	POPE	TextVQA	OCRB	Avg.
LT-OPD	74.9	73.0	54.8	50.0	78.9	2,077.0	86.3	56.0	535	82.3%
w/o final holding	72.3	69.1	54.8	49.8	77.7	2,025.0	86.4	55.2	524	80.7%
w/o initial holding	68.1	69.7	54.8	49.7	78.2	2,049.3	86.3	56.0	531	80.6%

Appendix

Table 13: Effect of removing the two constant stages of the visual-budget schedule. Row LT-OPD is the full-schedule model from Table 12 . Avg. is the same as Table 2 in main text.

Method	MuirBench	MVBench	Avg.
Qwen3.5-4B ( Upper Bound )	65.7	71.8	100%
CDPruner	39.1	51.6	65.7%
EPIC	44.0	52.2	69.8%
LT-OPD	52.0	55.6	78.3%

Appendix

Table 14: Performance on multi-image benchmarks. Two baselines keep 5% visual tokens. Avg. is the same as Table 2 in main text.

Figure 5: V ∗ Bench item 18. A plastic stool appears among boats and produce. HiPrune and EPIC select bamboo and wood, respectively; LT-OPD correctly selects plastic.

Figure 6: V ∗ Bench item 29. A red cap occupies a small part of the poolside scene. Both baselines select white; LT-OPD correctly selects red.

Figure 7: OCRBench item 512 (DocVQA). LT-OPD correctly returns the letter’s header date, January 9, 1961. EPIC returns the meeting date mentioned in the body, December 1, 1960; HiPrune returns a different date.

Explore similar work

Jun 23, 2026cs.CV

Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction

Visual token reduction has emerged as an effective strategy for accelerating Multimodal Large Language Models (MLLMs). Many existing methods prune tokens by ranking text-visual attention scores. However, we show that attention is often dominated by a model-induced prior: even without textual instruction, MLLMs tend to focus on certain task-agnostic regions. Consequently, the attention scores of instruction-conditioned tokens are suppressed, increasing the risk that these tokens are discarded during pruning. To address this issue, we propose Prior-Corrected Token Reduction (PriorTR), a training-free token reduction method that explicitly separates task-conditioned attention from the model-induced prior. PriorTR estimates the attention map of the prior, and contrasts it with the task-conditioned attention distribution to measure the additional usable information contributed by each visual token. Importantly, PriorTR computes both the model-induced prior and the task-conditioned posterior within a single forward pass by introducing a null token that serves as an instruction-agnostic probe in the attention block. This design avoids duplicated propagation. Extensive experiments across multiple multimodal benchmarks and MLLMs demonstrate that PriorTR consistently improves the trade-off between accuracy and efficiency over strong training-free baselines, particularly under aggressive token budgets.

Zengjie Chen, Yuxiang Cai, Jingcai Guo +3

School of Software Technology, Zhejiang University, Ningbo, China · Zhejiang Key Laboratory of Digital-Intelligence Service Technology, China · Hong Kong Polytechnic University, Hong Kong SAR +1

Visual Token Pruning Multimodal Large Language Models

Ask PDF

Jun 30, 2026cs.CV

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Yuhao Wang, Mu Qiao, Haiwen Diao +5

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1

Visual Token Pruning Multimodal Large Language Models Rectification

Ask PDF

Jun 14, 2026cs.CV

Stepwise Token Selection for Efficient Multimodal Large Language Models

In multimodal large language models (MLLMs), inference cost is largely dominated by the visual token prefix rather than the language backbone, making token reduction a key factor for improving efficiency. Existing approaches typically assign independent importance scores to visual tokens and retain a fixed number of top-ranked tokens, implicitly assuming token independence and a uniform compression ratio across inputs. In this work, we reformulate visual token pruning as a sequential decision-making process. Specifically, we introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop via a learned termination action. This enables joint optimization of both the selected subset and its size. To enable end-to-end training under standard language modeling objectives, we design a differentiable relaxation based on a variance-preserving noise interpolation scheme, allowing gradients to propagate through the discrete selection process. Extensive experiments on LLaVA-v1.5-7B and Qwen2.5-VL-7B demonstrate that our approach consistently outperforms fixed-ratio baselines across different compression levels. Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.

Landi He, Shawn Young, Lijian Xu

Shenzhen University of Advanced Technology

Visual Token Pruning Multimodal Large Language Models Inference Cost Importance

Ask PDF