AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

Haozhe Qi; Kevin Qu; Mahdi Rad; Rui Wang; Alexander Mathis; Marc Pollefeys

cs.CVMar 30, 2026

AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

Authors: Haozhe Qi, Kevin Qu, Mahdi Rad, Rui Wang, Alexander Mathis, Marc Pollefeys

Organizations: Microsoft Spatial AI Lab · EPFL · ETH Zurich

Abstract

Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token

Figures & tables

Figure 1: We propose AdaptToken, a flexible and efficient token selection strategy for long video understanding. Compared with state-of-the-art frame/token selection methods on several challenging long-video benchmarks, AdaptToken consistently delivers improved performance.

Figure 2: Overall pipeline of AdaptToken. AdaptToken processes long videos by dividing them into frame groups and selecting informative tokens within each group based on group relevance estimated from response entropy. It progressively gathers evidence across groups and stops processing once sufficient information has been collected.

Figure 3: Needle-in-a-Haystack experiments based on InternVL2.5 8B. Response entropy distributions for correct vs. incorrect predictions under varying numbers of input frames, with and without needles.

Figure 4: Real-data entropy experiments on based on InternVL2.5 8B. Response-entropy distributions for correct vs. incorrect predictions on real-world benchmarks (VideoMME and MLVU).

Figure 5: Visualization of AdaptToken token selection. Two frame groups are presented side by side. For each clip, we first estimate intra-group token relevance via cross-modal attention (heatmaps in the second row), and group-level relevance via response entropy, which measures the model’s answer confidence. Based on these signals, we perform global-aware token selection, adaptively allocating a larger token budget to groups that are more relevant to the text prompt (colored masks in the third row). The resulting token set is compact yet information-dense, improving both accuracy and inference efficiency for long-video understanding.

Model	LLM Size	VideoMME	MLVU	LongVideoBench	LVBench
GPT-4o [ 27 ]	-	71.9	64.6	66.7	27.0
Gemini-1.5-Pro [ 35 ]	-	73.2	-	64.0	65.7
Gemini-2.5-Pro [ 8 ]	-	84.3	-	-	78.7
mPLUG-Owl3 [ 49 ]	7B	59.3	63.7	52.1	-
NVILA [ 24 ]	8B	64.2	70.1	57.7	-
ByteVideoLLM [ 36 ]	14B	64.6	70.1	70.1	-

Table 1: Comprehensive evaluation across long-video benchmarks. Results are grouped by model family. Best performance within each block is shown in bold. The first two blocks report representative closed-source and open-source MLLMs as reference points. Subsequent blocks list each base MLLM followed by the corresponding integrated methods.

Methods	AdaptToken-Lite		AdaptToken
Methods	Acc.	Time (s)	Acc.	Time (s)
VideoMME	69.8	8.6	70.5	17.8
LongVideoBench	65.1	11.0	65.2	18.2
MLVU	76.3	10.1	76.8	21.5
LVBench	53.3	19.3	54.8	32.8

Table 2: Accuracy and inference-time comparison between AdaptToken and AdaptToken-Lite. Qwen2.5-VL 7B is used as the base MLLM. AdaptToken-Lite achieves comparable accuracy while reducing average inference time by approximately 50%.

Method	Max frames	VideoMME	MLVU
a. Base InternVL2.5	64	64.2	68.9
b. Base InternVL2.5	256	64.3	67.2
c. b+Group TS	256	66.9	70.1
d. c+Scale up frames	1024	67.1	70.9
e. d+Global TS	1024	68.0	73.6
f. e+Global TR	1024	68.3	74.1

Table 3: Additive ablation study of AdaptToken components. TS denotes token selection and TR denotes token removal. InternVL2.5 8B is used as the base MLLM. The final configuration (f) corresponds to the full AdaptToken.

Max frames	VideoMME	MLVU	LongVideoBench	LVBench
4096	70.5	76.8	65.2	54.8
8192	70.2	77.1	65.5	54.9
10000	70.1	77.1	65.8	55.6

Table 4: AdaptToken with more input frames. Qwen2.5-VL 7B is used as the base MLLM. AdaptToken maintains strong performance as the number of input frames increases, demonstrating robustness to extremely long video inputs (up to 10K frames).

Method	VideoMME	MLVU
Response KL-divergence	70.0	76.5
Response confidence	70.1	76.7
Response entropy	70.5	76.8

Table 5: Comparison of certainty measures. AdaptToken achieves comparable performance when paired with different self-certainty scores. Qwen2.5-VL 7B is used as the base MLLM.

Method	VideoMME	MLVU	LongVideoBench
Base InternVL2.5	64.2	68.9	59.5
Majority voting	64.6	70.9	60.0
Borda voting	64.9	71.1	60.7
Weighted voting	65.3	71.6	61.6
AdaptToken (Ours)	68.3	74.1	63.7

Table 6: Comparison to different voting methods. AdaptToken aggregates tokens from different frame groups in a global-aware manner and obtains better performance over the voting methods (i.e., majority voting, borda voting and weighted voting).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Entropy distributions across benchmarks. Qwen2.5-VL is used as the base MLLM.

Methods	AdaptToken-Lite			AdaptToken
Methods	Acc.	Groups	Time (s)	Acc.	Groups	Time (s)
VideoMME	68.1	6.9	8.4	68.3	11.0	13.0
LongVideoBench	63.8	7.6	9.2	63.7	11.1	13.0
MLVU	74.4	7.4	9.3	74.1	14.2	16.3
LVBench	51.3	11.3	12.3	52.1	16.2	18.8

Appendix

Table 7: Accuracy and inference-time comparison between AdaptToken and AdaptToken-Lite. InternVL2.5 8B is used as the base MLLM. AdaptToken-Lite achieves comparable or better accuracy while reducing inference time.

Method	MLVU
Removing 0.05 $B$	76.1
Removing 0.1 $B$	76.8
Removing 0.2 $B$	76.9
Removing 0.3 $B$	76.7

Appendix

Table 8: Ablation on the token-removal rate. AdaptToken is robust to different removal rates.

Entropy threshold	0.6	0.7	0.7	0.75	0.75	0.75	0.8	0.8
Group threshold	1	1	2	1	2	3	3	4
MLVU	76.2	75.9	76.2	75.9	76.0	76.3	76.0	76.3

Appendix

Table 9: Ablation on early-stopping hyperparameters.

Explore similar work

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Sep 15, 2026Haoyu Guo, Yuan Feng, Junlin Lv +3Efficient VLM Inference Long-Video Understanding
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1
VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning
Mar 26, 2026Zhe Gao, Shiyu Shen, Taifeng Chai +7Tool Use in VLMs Efficient VLM Inference
Nanjing University, Nanjing, China · Nankai University, Tianjin, China · East China Normal University, Shanghai, China +3
One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Apr 15, 2026Zheyu Zhang, Ziqi Pang, Shixing Chen +3Long-Video Understanding Video Token Compression
University of Illinois Urbana-Champaign · Amazon Prime Video

Sep 15, 2026J/K move · Enter open · S save

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

Haoyu Guo, Yuan Feng, Junlin Lv, Mingjun Xiao+2

School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China+1

Efficient VLM Inference Long-Video Understanding Visual Token Compression

Open paper AskPDF arXiv

Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates context windows and incurs prohibitive costs. Current solutions predominantly rely on auxiliary models for token reduction but face a fundamental dilemma: lightweight encoder-driven approaches often overlook critical semantic information, whereas heavyweight MLLM-driven reduction negates the efficiency gains. {In this work, we identify a more fundamental inefficiency underlying this dilemma: while fine-grained visual details are essential for detailed understanding, they are largely redundant for the preliminary task of selecting semantically relevant regions. } Motivated by this, we introduce VideoMM, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity. Specifically, our framework {decouples selection from reasoning} by executing semantic filtering on a cost-effective Macro Proxy (derived from downscaled frames), and projecting the selected regions onto high-fidelity Micro Tokens for detailed understanding only when necessary. Extensive evaluations show that VideoMM significantly outperforms existing solutions. It achieves a 6.13

\times

speedup and a 7.4% accuracy gain over full-context baselines on LongVideoBench, and further accelerates inference by 2.73

\times

over current leading methods, establishing a highly scalable paradigm for long-video understanding. Our code is available at: https://github.com/adfh917k/VideoMM.

Abstract

Explore similar work

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding