cs.CVNov 16, 2026

How Reliable Is Visual Token Selection? Rethinking Pruning in Multimodal LLMs

Authors: Chaehee Song, Seokil Ham, KiHyun Seol, Jihyun Lee, Yubin Choi, Changick Kim

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Visual token pruning improves the efficiency of multimodal large language models (MLLMs) by removing redundant visual tokens, yet the reliability of existing selection signals has not been systematically examined. We analyze token selection under semantics-preserving perturbations and across representative pruning criteria, revealing two limitations. First, attention-based importance shifts non-uniformly under perturbations, so magnitude alone cannot distinguish stable evidence from unstable saliency (local instability). Second, attention- and diversity-based methods select substantially different subsets despite comparable performance, showing that useful evidence is not confined to a unique subset (global inconsistency). These findings motivate reliable token-set construction. We instantiate this perspective in CoRePruner, a training-free framework combining instruction-aware diversity with reliability-aware attention. Across image benchmarks, multiple MLLM architectures, and token budgets, CoRePruner improves performance, especially under aggressive pruning, and remains competitive on video benchmarks. Our results establish reliability as a central consideration in visual token selection.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

    Jun 25, 2026Tinghao Wang, Yichen Guo, Rui Huang +9Efficient VLM InferenceMultimodal Large Language Models

  2. Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs

    Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM EvaluationConfidence Estimation for VLMs

  3. From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models

    Sep 22, 2026Shengli He, Yongchao Liang, Roumeng He +5Visual Token PruningMultimodal Token Compression