LLM Compression

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 92% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 178

All topics

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression
Sep 2, 2026Viacheslav Yusupov, Daria Cherniuk, Evgeny FrolovLLM Compression Fisher Information
1HSE University · 2AXXX
XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression
Sep 2, 2026Jundong Hu, Shekar RamachandranLLM Compression Model Compression
PayPal AI
Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation
Sep 2, 2026Wenhui Chen, Zhifeng Li, Jie Zhou +5LLM Compression Language Model Distillation
1Incept Labs, Houston, TX · 2Titan Holdings, San Francisco, CA
Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
Sep 2, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs
Sep 1, 2026Seungwoo Jung, Dohyeok Kwon, Seungmin Cha +4Mixture-of-Experts Language Models LLM Compression
Department of Computer Science and Engineering, Korea University · Department of Computer Science and Artificial Intelligence, Dongguk University
Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression
Aug 31, 2026Tianyi Zhao, Yinhan He, Wendy Zheng +2LLM Compression Efficient Language Model Reasoning
University of Virginia
TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
Aug 31, 2026Daniel Agyei Asante, Yang LiLLM Compression Long-Context QA
1Iowa State University, United States
Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Aug 31, 2026Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida +1LLM Compression LLM Inference Acceleration
Innopolis University, Innopolis 420500, Russia · Department of Teleinformatics Engineering, Federal University of Cear´a, Fortaleza 60455-760, Brazil · Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland
LaMoC: Loss-Aware Modular Compression for LLMs
Aug 31, 2026Mohanad Odema, Jacob SongModel Compression LLM Compression
LG Electronics USA
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
Aug 30, 2026Hongyu Yu, Yifei ShenLLM Quantization LLM Pruning
Lenovo Research · University of Washington
Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed
Aug 12, 2026Haokun Lin, Kaijie Zhu, Haobo Xu +4LLM Quantization LLM Pruning
Institute of Automation, CAS Beijing, China · Tsinghua University Beijing, China · Harvard Medical School Boston, U.S. +1
Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Merging
Aug 11, 2026Linh Dieu Le, Tong Chen, Shazia Sadiq +3LLM Compression Large Language Model-Based Recommendation
The University of Queensland Brisbane, Australia · Griffith University Brisbane, Australia
LegoLM: Structured Weight Sharing for Large Language Models
Aug 9, 2026Joseph BinghamLLM Quantization LLM Compression
Deparment of Biology, Technion – Israel Institute of Technology
Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs
Aug 9, 2026Mohanad Odema, Gabrielle De Micheli, Dayin Gou +3Model Compression LLM Compression
LG Electronics, North America
Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
Aug 5, 2026Zhengpei Hu, Kai Li, Dapeng Fu +5LLM Compression Long-Context Inference
School of Computer Technology and Application, Qinghai University · Tsinghua University · Ant Group Security and Intelligence Laboratory (SIL)
When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning
Aug 3, 2026Andrew ZhangModel Selection Mixture-of-Experts Pruning
KTH Royal Institute of Technology
Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling
Aug 3, 2026Ravi Satya Durga Prasad YenugulaLLM Compression Continued Pretraining
Independent Researcher
ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
Aug 3, 2026Şuayp Talha Kocabay, Talha Rüzgar Akkuş, Kamer Ali YukselLLM Quantization LLM Compression
Independent Researcher · aiXplain, Inc.
F-WANDA: Fisher-Reweighted Post-Training Pruning for Sustainable Deployment of Large Language Models
Aug 1, 2026Himanshu MishraLLM Pruning LLM Compression
University of British Columbia
Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution
Jul 30, 2026I. Kennedy, T. KennedyModel Compression AI Agent Reliability
Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors
Jul 28, 2026Jianfei Ma, Zhaoxin Feng, Emmanuele Chersoni +1LLM Compression Efficient Language Model Inference
The Hong Kong Polytechnic University
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Jul 24, 2026Hao Wang, Kun Yuan, Wenlin Zhong +4Cross-Tokenizer Knowledge Distillation LLM Compression
University of Chinese Academy of Sciences · KwaiKAT Team · Zhejiang University +1
Progressive Cramming: Reliable Token Compression and What It Reveals
Jul 23, 2026Dmitrii Tarasov, Timofei Lashukov, Elizaveta Goncharova +1LLM Compression Embedding Compression
FusionBrain Lab, AXXX, Moscow, Russia · HSE University, Moscow, Russia · Artificial Intelligence Institute, Innopolis University, Innopolis, Russia
CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
Jul 20, 2026Zhiren Gong, Zihao Zeng, Zijie Wang +3LLM Pruning LLM Compression
College of Computing and Data Science, Nanyang Technological University, Singapore · Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
SALT: Salience-Aware Lexical Trie for Long-Context Compression
Jul 20, 2026Oteo Mamo, Hyunjin Yi, Joydhriti Choudhury +2LLM Compression Long-Context Language Model Inference
Florida State University
A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design
Jul 10, 2026Linhui Xiao, Guiping Cao, Mingyue Guo +6LLM Compression Energy-Efficient ML
1. Pengcheng Laboratory, Shenzhen 518066, China · 2. Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · 3. Peking University, Beijing 100080, China
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Jul 9, 2026David González-Martínez, Shiwei LiuLLM Compression Low-Rank Matrix Decomposition
Max Planck Institute for Intelligent Systems · University of Tübingen · ELLIS Institute Tübingen +1
BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression
Jul 9, 2026Yuantian Shao, Peisong Wang, Zhilei Liu +6LLM Compression Weight-Only Quantization
Nanjing University of Science and Technology, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Huawei, China
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India
Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention
Jul 9, 2026Ryota Kobayashi, Tsubasa Hirakawa, Takayoshi Yamashita +4LLM Pruning LLM Compression
1Chubu University · 2Panasonic Holdings Corporation

Sep 2, 2026, cs.LGJ/K move · Enter open · S save

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

Viacheslav Yusupov, Daria Cherniuk, Evgeny Frolov

1HSE University · 2AXXX

LLM Compression Fisher Information Neural Network Pruning

Open paper AskPDF arXiv

In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analysis for billion-parameter networks where full computation is infeasible. Our approach reveals consistent vulnerability patterns: value projection layers exhibit the highest sensitivity and strongest cross-layer correlations across multiple model families, while other components exhibit architecture-specific behaviors. Through extensive experiments on quantization, sparsification, inter-layer corruption, and post-corruption fine-tuning, we demonstrate that our approximation strongly correlates with both performance degradation and recovery. Our framework provides a practical, theoretically grounded tool for identifying fragile components in large models, opening new avenues for guided compression and optimization strategies, such as mixed-precision allocation, layer-wise sparsity, and adaptive low-rank decomposition across layers and even individual weight groups.

Momentum

Related topics

Latest papers 178