Token Pruning
Overview
Token pruning studies how to shorten sequences processed by transformers while retaining information needed for the task, from visual and audio inputs to long-context and diffusion language models. Methods rank tokens using attention, query relevance, entropy, representation changes across layers, or temporal and spatial structure; they then drop, merge, cache, or selectively recompute tokens, often with adaptive budgets. Work evaluates these approaches on image and video understanding, speech, generation, reasoning, and embodied control, using measures such as accuracy, fidelity, latency, throughput, and memory. The central tension is that aggressive pruning can discard evidence needed later, while scoring and selection overhead may reduce real speedups; reliably predicting which tokens remain important across tasks and layers is still an open challenge.
Latest papers
57 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.
Temporal Aware Pruning for Efficient Diffusion-based Video Generation
OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance
SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection
EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs
Uncertainty-Aware Token Importance Estimation in Spiking Transformers
Latent Inter-Frame Pruning: A Training-Free Method Bridging Traditional Video Compression and Modern Diffusion Transformers for Efficient Generation
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
CATP: Confidence-Aware Token Pruning for Camouflaged Object Detection
Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuning
ToC3D for efficient multi-view ViT-based 3D object detection employs ego-motion-based relevant token selection. However, there are two key limitations: (1) The fixed layer-individual token selection ratios limit computational efficiency during both training and inference. (2) Full end-to-end retraining of the ViT backbone is required for the multi-view 3D object detection method. In this work, we propose an image token compensator combined with a token selection for ViT backbones to accelerate multi-view 3D object detection. Unlike ToC3D, our approach enables dynamic layer-wise token selection within the ViT backbone. Furthermore, we introduce a parameter-efficient fine-tuning strategy, which trains only the proposed modules, thereby reducing the number of fine-tuned parameters from more than million (M) to only M. Experiments on the large-scale NuScenes dataset across three multi-view 3D object detection approaches demonstrate that our proposed method decreases computational complexity (GFLOPs) by ... , inference latency (on an NVIDIA-GV100 GPU) by ... , while still improving mean average precision by ... absolute and NuScenes detection score by ... absolute compared to so-far SOTA ToC3D.