Transformer Pruning

Follow this topic Ask about this topic

Latest papers

42 papers13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Towards Real-Time Full-Waveform LiDAR Transformers via Intensity-Guided Token Reduction and Physics-Aware Augmentation
Nov 5, 2026Kotaro Oishi, Kazuma Ikeda, Ryosei Hara +3Autonomous Driving Perception Transformer Pruning
Keio University
Exploiting Biases in Attention for Transformers
Nov 3, 2026Charles Herr, Gabriele Spadaro, Enzo TartaglioneTransformer Attention Attention Mechanisms
LTCI, Télécom Paris, Institut Polytechnique de Paris, France · University of Turin, Italy · CNR-ISASI, Italy
What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models
Oct 31, 2026Piyush Dhoka, Vedant Pandhare, Varun Inamdar +2LLM Pruning Transformer Pruning
Department of Artificial Intelligence, Vishwakarma University, Pune, India
Two-Stage Clipping: Linear-Time Approximate and Exact Mixed-Norm Projections for Transformers
Oct 28, 2026Michel Barlaud, Jean-Paul MarmoratStructured Sparsity Transformer Pruning
I3S Laboratory, CNRS, Universite Côte d’Azur, France · Ecole des Mines, PSL, France.
Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers
Oct 22, 2026Shreya Sajid, Fabrizio Julian Piva, Hiram Rayo Torres Rodriguez +3Audio Token Compression Transformer Pruning
Eindhoven University of Technology, Eindhoven, Netherlands · NXP Semiconductors, Eindhoven, Netherlands
Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers
Oct 22, 2026Chenyu Zhu, Ruoyu Zhao, Zhichao LuNeural Architecture Performance Prediction Neural Architecture Search
Department of Computer Science, City University of Hong Kong
SliMOO: Interpretable Multi-Objective Evolutionary Search for LLM Depth Pruning
Oct 20, 2026Guanchen Li, Yixing Xu, Xuanwu Yin +2LLM Pruning Transformer Pruning
Advanced Micro Devices, Inc. (AMD)
TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
Oct 19, 2026Liana Patel, Rishabh Ranjan, Ion Stoica +2Token Pruning LLM Inference Acceleration
Stanford University · UC Berkeley
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Oct 16, 2026Rida Assalouh, Tom Jacobs, Rebekka Burkholz +2Muon Optimizer Structured Sparsity
Inria, Paris, France · ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany +3
Your Transformer Is Secretly Elastic
Oct 14, 2026Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts +1Elastic Neural Networks Transformer Pruning
University of Amsterdam · TNO, Intelligent Imaging
Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
Oct 13, 2026Mehdi Makni, Natalia Ponomareva, Rahul MazumderLLM Pruning Neural Network Pruning
Operations Research Center, Massachusetts Institute of Technology · Google Research
Retraining-Free Depth Pruning of Vision Transformers: A Cross Domain Study of Layer Redundancy
Oct 10, 2026Alessandro Viespoli, Loris NanniTransformer Pruning Efficient ViTs
University of Padua
Tabular Foundation Models Are Effectively Shallow
Oct 9, 2026Irene Cannistraci, Julia E. VogtTransformer Pruning Tabular Foundation Models
Department of Computer Science ETH Zurich Switzerland
EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation
Oct 7, 2026Inbasekaran SToken Pruning Transformer Pruning
SRM Institute of Science and Technology, India
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
Oct 6, 2026Davis Wertheimer, Haochen Shen, Ahan Gupta +6KV-Cache Compression Transformer Attention
IBM Research, USA · SSAIL Lab, University of Illinois Urbana-Champaign
QK-Wanda: Coupling Queries and Keys for Unstructured Pruning
Oct 1, 2026Ivan Ilin, Peter RichtárikLLM Pruning Neural Network Pruning
KAUST
DIET: Deletion-response Expert Trimming for Video Diffusion Transformers
Sep 29, 2026Jiachang Zhang, Teng Hu, Bohao Feng +4Mixture-of-Experts Pruning Video Diffusion Models
Xi’an Jiao Tong University, China · Shanghai Jiao Tong University, China · Alibaba Token Hub, Alibaba Group, China +1
Does the VGGT Family Need All Its Layers?
Sep 29, 2026Fengyi Zhang, Holger Caesar, Xiangyu Sun +3Multi-View 3D Reconstruction Transformer Pruning
The University of Queensland · Delft University of Technology · Harbin Institute of Technology
Output-aware Residual Stream Pruning for Large Language Models
Sep 28, 2026Chayne Thrash, Kevin Chen, Soheil KolouriLLM Pruning LLM Compression
Department of Computer Science Vanderbilt University Nashville, TN 37235, USA
Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Sep 23, 2026Rasmus Aagaard, Nicki Skafte DetlefsenSpeech Foundation Models Automatic Speech Recognition
Technical University of Denmark · Laerdal Medical
Forward-Free LLM Depth Pruning via Weight Redundancy
Sep 9, 2026Vincent-Daniel Yun, Woosang LimLLM Pruning LLM Inference Acceleration
University of Southern California, United States · 2Seoul National University, Republic of Korea
TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models
Sep 3, 2026Mehedi Hasan, Ashfak Yeafi, Md Khairul IslamComputational Pathology Pathology Foundation Models
Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · EEE, Khulna University of Engineering & Technology, Khulna, Bangladesh · Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA
XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression
Sep 2, 2026Jundong Hu, Shekar RamachandranTransformer Pruning LLM Compression
PayPal AI
SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs
Aug 25, 2026Ali Bahri, Hang Li, Hongliang Li +1LLM Pruning LLM Compression
Huawei Noah’s Ark Lab, Canada · Huawei Noah’s Ark Lab, Hong Kong SAR, China
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
Aug 25, 2026Peng Liu, Huibing Zeng, Yiqun Zhang +2LLM Pruning Transformer Pruning
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Information and Artificial Intelligence, Yangzhou University, Yangzhou 225000, China
Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression
Aug 11, 2026Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara +1LLM Compression Transformer Pruning
University of Central Florida Orlando, Florida, USA · Old Dominion University Norfolk, Virginia, USA
CausalGate: Causal Importance Distillation for Transformer Module Pruning
Jul 21, 2026Kiran Nair, Smriti Regmi, Rodrigue RizkEfficient Transformer Inference LLM Pruning
USD Artificial Intelligence Research Lab, Department of Computer Science University of South Dakota, Vermillion, SD 57069, USA
CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
Jul 20, 2026Zhiren Gong, Zihao Zeng, Zijie Wang +3LLM Pruning LLM Compression
College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
Jul 8, 2026Yazdan Jamshidi, Alexey ShvetsLLM Pruning LLM Compression
Palo Alto Networks
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
Jun 25, 2026Huzama Ahmad, Cao Viet Hai Nam, Se-Young YunEfficient Transformer Inference Transformer
KAIST

Nov 5, 2026J/K move · Enter open · S save

Towards Real-Time Full-Waveform LiDAR Transformers via Intensity-Guided Token Reduction and Physics-Aware Augmentation

Kotaro Oishi, Kazuma Ikeda, Ryosei Hara, Ryo Yoshida+2

Keio University

Autonomous Driving Perception Transformer Pruning Data Augmentation

Open paper AskPDF arXiv

LiDAR is a critical sensor for autonomous driving and robotics, yet it remains vulnerable to adverse conditions such as fog, rain, transparent objects, and low-reflectance surfaces. Full-waveform LiDAR (FWL) addresses these limitations by capturing the complete return waveform as a histogram, preserving rich reflection characteristics including multi-path echoes and weak signals. While recent Transformer-based approaches have demonstrated promise on FWL data, two fundamental bottlenecks remain: high computational cost from redundant tokens in low-intensity background regions, and limited generalization due to scarce annotated training data. We address both bottlenecks by exploiting the physical structure inherent to FWL data. First, we propose FWL-ToPM, a token pruning and merging mechanism that leverages intensity distributions to selectively reduce tokens while retaining critical waveform content. Second, we introduce FWLAug, a data augmentation framework designed as a regularization method that preserves waveform structure, retaining the Time-of-Flight waveform of each ray and the frustum geometry, to improve generalization to unseen scenes. Evaluated on the Ghost-FWL benchmark, FWL-ToPM achieves over 8 times speedup at real-time throughput alongside a nearly 9-point F1 gain over the prior state of the art, establishing the first real-time FWL Transformer that simultaneously improves accuracy. FWLAug further improves performance on both the standard benchmark and an out-of-distribution test set of unseen scenes. Together, these methods push FWL-based perception toward practical deployment. The code and the OOD dataset are publicly available and can be accessed via the project page: https://keio-csg.github.io/FWL-ToPM