cs.CVNov 11, 2026

DiFiT: Distribution-Fidelity-guided Visual Token Pruning for Multimodal Large Language Models

Authors: Jingyuan Zhang, Nihao Niu, Xinyang Lv, Beibei Han, Xiaoyun Wang, Yanning Hou, Ke Xu

Organizations: School of Artificial Intelligence, Anhui University, Hefei 230601, China · College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei 230601, China

Abstract

Visual token pruning reduces the inference cost of multimodal large language models, but answer accuracy only partly reflects its effect. Pruning can change the probabilities of alternative response tokens even when greedy answers remain correct and unchanged. We compare full and pruned response distributions under shared reference prefixes. Average predictive drift grows with compression on fixed correctly answered samples. Separate analyses associate larger drift with more confidence-decision disagreements and sampling failures. We propose DiFiT (Distribution-Fidelity-guided Token pruning) to learn discrete selection from reference answers and full-token response distributions. Dual-View Distillation matches probabilities across the vocabulary, while answer supervision supports task quality. A learned scorer and differentiable Top-KK learn which visual evidence to retain. A multi-budget mask objective supports different retention ratios through one shared ranking over a frozen model. On Qwen2.5-VL-7B, DiFiT reduces sampling-induced negative flips by 30.3% relative to FastV at 10% retention on four-task greedy-correct controls. It preserves 98.74% of baseline performance on MME at 30% retention.

Figures & tables

Explore similar work

CardsList
  1. Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs

    Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM EvaluationConfidence Estimation for VLMs

  2. S2^2Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

    Sep 1, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM InferenceMultimodal Large Language Models

  3. StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

    Sep 12, 2026Hansen Zhang, Landi He, Mingde Yao +1Efficient VLM InferenceVisual Token Pruning