Overview

Token pruning studies how to shorten sequences processed by transformers while retaining information needed for the task, from visual and audio inputs to long-context and diffusion language models. Methods rank tokens using attention, query relevance, entropy, representation changes across layers, or temporal and spatial structure; they then drop, merge, cache, or selectively recompute tokens, often with adaptive budgets. Work evaluates these approaches on image and video understanding, speech, generation, reasoning, and embodied control, using measures such as accuracy, fidelity, latency, throughput, and memory. The central tension is that aggressive pruning can discard evidence needed later, while scoring and selection overhead may reduce real speedups; reliably predicting which tokens remain important across tasks and layers is still an open challenge.

Latest papers

CardsList

57 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.

  1. Exploiting Biases in Attention for Transformers

    Nov 3, 2026Charles Herr, Gabriele Spadaro, Enzo TartaglioneTransformer AttentionAttention Mechanisms

  2. 3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement

    Oct 27, 2026Yongqi Shan, Lu Zhang, Jiazuo Yu +23D Scene Understanding3D VLMs

  3. TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

    Oct 19, 2026Liana Patel, Rishabh Ranjan, Ion Stoica +2Token PruningLLM Inference Acceleration

  4. AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models

    Oct 19, 2026Marwan Farag, Bin YangToken PruningAudio-Language Models

  5. SATOR: Backpropagation-Free 3D Test-Time Adaptation via Sample-Adaptive Token Routing

    Oct 15, 2026Inseok Kong, Geunyoung Jung, Jiyoung JungPoint Cloud Corruption RobustnessTest-Time Adaptation

  6. AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

    Oct 9, 2026Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAudio Token CompressionAudio-Language Models

  7. Sparse Planning in Visual World Models via Cost Gradients

    Oct 7, 2026Yingchen Xu, Edward GrefenstetteWorld Model-Based PlanningToken Pruning

  8. DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

    Oct 6, 2026Shuo Yang, Changbai Li, Linlin Yang +6Multimodal Large Language ModelsToken Pruning

  9. Audio Token Attention Is Predictable Before the Language Model Runs

    Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token CompressionAudio Understanding

  10. FastE: Readout-Triggered Token Compression for LLM Embedding Inference

    Sep 8, 2026Jinsong Shu, Jinyong Wen, Baokun Wang +4Token PruningLLM Inference Acceleration

  11. Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrieval

    Sep 7, 2026PS Rishi, Rajeev Ranjan Dwivedi, Vinod K KurmiLate Interaction RetrievalToken Pruning

  12. MaST: Motion-aware Sparse Pipeline for Lightweight Object Tracking

    Aug 25, 2026Qingmao Wei, Fagui Liu, Dengke Zhang +2Visual Object TrackingEfficient ViTs

  13. Putting Registers to Work: Task Registers for Token Pruning in Vision Transformers

    Aug 11, 2026Hongsen Cao, Mona Jaber, Shanxin Yuan +1Token PruningEfficient ViTs

  14. Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs

    Aug 9, 2026Kyeongyoon Lee, Hongyeob Kim, Youngeun Kim +1Efficient Multimodal InferenceAudio Token Compression

  15. DiffPrune: differentiable information throttling for token pruning in vision-language models

    Aug 3, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language ModelsEfficient VLM Inference

  16. When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents

    Aug 2, 2026Shuaijun Liu, Feiyang You, Xingwei Chen +1Large Language Model-Based Robot PlanningToken Pruning

  17. Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

    Aug 1, 2026Kenny ShaoToken PruningSubword Tokenization

  18. WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

    Jul 30, 2026Haozhe Hu, Hao Wu, Peiran Yin +3LLM PruningToken Pruning

  19. Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

    Jul 30, 2026Jie Ma, Zhike Qiu, Jie Gao +4Efficient Multimodal InferenceMultimodal Large Language Models

  20. Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

    Jun 23, 2026Bin Chen, Yuxiang Cai, Yadan Luo +3Efficient Multimodal InferenceMultimodal Large Language Models

  21. Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping

    Jun 22, 2026Rishubh Parihar, Ayush Raina, R. Venkatesh Babu +1Geometry-Conditioned Image GenerationConditional Image Generation

  22. Entropy Gate: Entropy Quenching for Near-Lossless Token Compression in LLM Pipelines

    Jun 2, 2026Justice Owusu Agyemang, Jerry John Kponyo, Kwame Opuni-Boachie Obour Agyekum +3LLM CompressionToken Pruning

  23. AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference

    May 28, 2026Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan KandemirLLM PruningVision-Language Models

  24. Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models

    May 27, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language ModelsEfficient VLM Inference

  25. Stage-adaptive Token Selection for Efficient Omni-modal LLMs

    May 19, 2026Zijie Xin, Jie Yang, Ruixiang Zhao +4Efficient Multimodal InferenceMultimodal Large Language Models