LLM Pruning

LLM: Large Language Model

Overview

Large language model pruning studies how to remove weights, neurons, attention heads, layers, or experts while retaining useful behavior. Methods range from magnitude and activation-based scoring to gradient, curvature, causal, and representation-based criteria, often using calibration data to estimate which components matter; some also recover performance through fine-tuning or distillation. Structured pruning can reduce computation more directly, while unstructured sparsity often needs specialized hardware and kernels to deliver real speedups. A central challenge is that the best pruning decisions depend on the model, task, calibration data, and deployment setting, and preserving benchmark accuracy may not preserve generation quality, fairness, or reliability.

Latest papers

CardsList

191 papers21 papers in the last four weeks, down 19% on the four weeks before. 0.2% of all new papers.

  1. Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference

    Jan 19, 2026Zimeng Wu, Donghao Wang, Chaozhe Jin +2LLM Inference AccelerationLong-Context LLM Inference

  2. Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

    Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM PruningAttention Head Analysis

  3. OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

    Dec 15, 2025Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi +1LLM PruningNeural Network Pruning

  4. Data-Free Pruning of Self-Attention Layers in LLMs

    Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Inference AccelerationLLM Compression

  5. PATCH: Learnable Tile-level Hybrid Sparsity for LLMs

    Sep 27, 2025Younes Hourri, Mohammad Mozaffari, Maryam Mehri DehnaviLLM PruningLLM Compression

  6. A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

    May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model PretrainingLLM Pruning

  7. Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning

    May 13, 2025Ren ZhuangGradient-Based AttributionLLM Pruning

  8. DarwinLM: Evolutionary Structured Pruning of Large Language Models

    Feb 11, 2025Shengkun Tang, Oliver Sieberling, Eldar Kurtic +2LLM PruningStructured Pruning

  9. Subspace Node Pruning

    May 26, 2024Joshua Offergeld, John Chidiac, Marcel van Gerven +1Neural Network PruningLLM Pruning

  10. Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

    Nov 8, 2023Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2LVLM PruningLLM Pruning