Token Pruning

Follow this topic Ask about this topic

Token pruning studies how to shorten sequences processed by transformers while retaining information needed for the task, from visual and audio inputs to long-context and diffusion language models. Methods rank tokens using attention, query relevance, entropy, representation changes across layers, or temporal and spatial structure; they then drop, merge, cache, or selectively recompute tokens, often with adaptive budgets. Work evaluates these approaches on image and video understanding, speech, generation, reasoning, and embodied control, using measures such as accuracy, fidelity, latency, throughput, and memory. The central tension is that aggressive pruning can discard evidence needed later, while scoring and selection overhead may reduce real speedups; reliably predicting which tokens remain important across tasks and layers is still an open challenge.

Latest papers

54 papers10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
Oct 19, 2026Liana Patel, Rishabh Ranjan, Ion Stoica +2Token Pruning LLM Inference Acceleration
Stanford University · UC Berkeley
AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models
Oct 19, 2026Marwan Farag, Bin YangToken Pruning Audio-Language Models
Institute of Signal Processing and System Theory (ISS), University of Stuttgart, Stuttgart, Germany
SATOR: Backpropagation-Free 3D Test-Time Adaptation via Sample-Adaptive Token Routing
Oct 15, 2026Inseok Kong, Geunyoung Jung, Jiyoung JungTest-Time Adaptation Token Pruning
Department of Geo Informatics, University of Seoul, South Korea · Department of Artificial Intelligence, University of Seoul, South Korea
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Oct 9, 2026Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAudio Token Compression Audio-Language Models
Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands
Sparse Planning in Visual World Models via Cost Gradients
Oct 7, 2026Yingchen Xu, Edward GrefenstetteWorld Model-Based Planning Token Pruning
University College London
EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation
Oct 7, 2026Inbasekaran SToken Pruning Transformer Pruning
SRM Institute of Science and Technology, India
DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
Oct 6, 2026Shuo Yang, Changbai Li, Linlin Yang +6Multimodal Large Language Models Token Pruning
Beihang University · Communication University of China · National University of Singapore
Later Is Better: Token Reduction for ViTs Under Distribution Shift
Oct 6, 2026Hyeongheon Cha, Hyungjun Yoon, Sung-Ju LeeDistribution Shift Robustness Token Pruning
KAIST
Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination
Sep 30, 2026Xinling Xie, Haodong Wang, Jiazhi Mi +8Efficient Inference for World Action Models Token Pruning
NJU · HKUST · HIT +1
Audio Token Attention Is Predictable Before the Language Model Runs
Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token Compression Audio Understanding
The University of Texas at Austin
FastE: Readout-Triggered Token Compression for LLM Embedding Inference
Sep 8, 2026Jinsong Shu, Jinyong Wen, Baokun Wang +4Token Pruning LLM Inference Acceleration
Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Ant Group +1
Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrieval
Sep 7, 2026PS Rishi, Rajeev Ranjan Dwivedi, Vinod K KurmiLate Interaction Retrieval Token Pruning
Indian Institute of Science Education and Research Bhopal (IISER Bhopal), India
MaST: Motion-aware Sparse Pipeline for Lightweight Object Tracking
Aug 25, 2026Qingmao Wei, Fagui Liu, Dengke Zhang +2Visual Object Tracking Efficient ViTs
South China University of Technology, Guangzhou, China
Putting Registers to Work: Task Registers for Token Pruning in Vision Transformers
Aug 11, 2026Hongsen Cao, Mona Jaber, Shanxin Yuan +1Token Pruning Efficient ViTs
School of Electronic Engineering and Computer Science Queen Mary University of London London, United Kingdom
Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs
Aug 9, 2026Kyeongyoon Lee, Hongyeob Kim, Youngeun Kim +1Efficient Multimodal Inference Audio Token Compression
Sungkyunkwan University · AWS AI Labs
DiffPrune: differentiable information throttling for token pruning in vision-language models
Aug 3, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language Models Efficient VLM Inference
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents
Aug 2, 2026Shuaijun Liu, Feiyang You, Xingwei Chen +1Large Language Model-Based Robot Planning Token Pruning
IoT, Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China.
Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding
Aug 1, 2026Kenny ShaoToken Pruning Subword Tokenization
Department of Computer Science Florida International University
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Jul 30, 2026Haozhe Hu, Hao Wu, Peiran Yin +3LLM Pruning Token Pruning
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Munich Center for Machine Learning, LMU Munich
Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
Jul 30, 2026Jie Ma, Zhike Qiu, Jie Gao +4Efficient Multimodal Inference Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · School of Information Engineering, Xiamen Ocean Vocational College · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Sino-Russian Research Center for Digital Economy
Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models
Jul 26, 2026Yiming Zhong, Chang Nie, Caifeng ShanEfficient Multimodal Inference Multimodal Large Language Models
1Nanjing University
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
Jun 23, 2026Bin Chen, Yuxiang Cai, Yadan Luo +3Efficient Multimodal Inference Multimodal Large Language Models
School of Software Technology, Zhejiang University, Ningbo, China · Zhejiang Key Laboratory of Digital-Intelligence Service Technology, China · The University of Queensland, St Lucia, QLD, Australia +2
Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping
Jun 22, 2026Rishubh Parihar, Ayush Raina, R. Venkatesh Babu +1Geometry-Conditioned Image Generation Conditional Image Generation
1IISc Bangalore · 2Tel Aviv University
Entropy Gate: Entropy Quenching for Near-Lossless Token Compression in LLM Pipelines
Jun 2, 2026Justice Owusu Agyemang, Jerry John Kponyo, Kwame Opuni-Boachie Obour Agyekum +3LLM Compression Token Pruning
1VIA Cybersecurity Lab, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana · 2Quantum and Assistive Technologies Lab, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana
AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
May 28, 2026Yilin Feng, Ahmed Burak Gulhan, Mahmut Taylan KandemirLLM Pruning Vision-Language Models
The Pennsylvania State University
Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models
May 27, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language Models Efficient VLM Inference
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
Stage-adaptive Token Selection for Efficient Omni-modal LLMs
May 19, 2026Zijie Xin, Jie Yang, Ruixiang Zhao +4Efficient Multimodal Inference Multimodal Large Language Models
Renmin University of China · WeChat Vision, Tencent Inc.
OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models
May 18, 2026Morunliu Yang, Ruotao Xu, Le Li +6Efficient Multimodal Inference Multimodal Large Language Models
Soochow University · Peking University
Temporal Aware Pruning for Efficient Diffusion-based Video Generation
May 18, 2026Sheng Li, Yang Sui, Junhao Ran +3Video Diffusion Models Token Pruning
University of Pittsburgh · Rice University · Rutgers University +1
OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance
May 14, 2026Yeo Jeong Park, Hyemi Jang, Minseo Choi +3Efficient Multimodal Inference Multimodal Large Language Models
Samsung Research

Oct 19, 2026J/K move · Enter open · S save

TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

Liana Patel, Rishabh Ranjan, Ion Stoica, Matei Zaharia+1

Stanford University · UC Berkeley

Token Pruning LLM Inference Acceleration Long-Context Language Model Inference Transformer Pruning LLM Context Compression

Open paper AskPDF arXiv

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context inputs. We introduce TokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnable token pruning over long-context inputs for fast and memory-efficient prefill. To enable end-to-end learning despite the non-differentiable top-K operator for token pruning, we introduce token-gated attention, a mechanism inspired by Mixture-of-Experts that re-weights attention logits using token importance scores. Across diverse tasks, TokenGate achieves

1.4

11.3\times

faster time-to-first token (TTFT) and

1.5

3.4\times

lower peak memory at no accuracy degradation for prefill in transformers. Compared to the best-performing prior methods for token-pruning, TokenGate gives up to

6.4\times

faster prefill and

2\times

lower peak memory. TokenGate natively extends both standard transformers (e.g., Qwen3) and hybrid-attention models (e.g., Qwen3.5) with simple lightweight adaptation comparable to standard parameter-efficient fine-tuning. We make our code available at: https://github.com/tokengate-models/TokenGate.