Gradient Compression

Follow this topic Ask about this topic

Gradient compression studies how to shrink the updates workers or clients exchange during training without losing too much accuracy or slowing convergence. Common approaches include quantization such as sign and FP8, sparsification such as Top-k, low-rank projections, and error feedback or variance reduction to control compression error. These methods are tested in distributed large language model training and federated learning, with benchmarks including CIFAR-10, MNIST, and language-model pretraining. A central challenge is balancing communication savings against compression error and added computation, while theoretical convergence guarantees do not always predict practical performance.

Latest papers

14 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection
Sep 29, 2026Jaeseung Heo, J Rosser, Dongwoo KimGradient Compression Influence Functions
POSTECH · University of Oxford
Communication-Efficient Agnostic Federated Learning via Faster Convergence and Compression
Sep 29, 2026Haomin Bai, Junyan Sun, Sifan Yang +2Error Feedback Online Federated Learning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China
Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback
Jul 31, 2026Maria Smirnova, Alexey KravatskiySign-Based Optimization Communication-Efficient Distributed Training
MIRIAI Moscow, Russia
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jul 8, 2026Jieying Wang, Zizhong Wang, Fangru Linghu +3Language Model Pretraining Communication-Efficient Distributed Training
Department of Computer Science, Rutgers University, Piscataway, NJ, USA · Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, USA
MUFFLe: Efficient Model Update Compression via Generalized Deduplication for Federated Learning
Jun 12, 2026Xiaobo Zhao, Daniel E. LucaniGradient Compression Federated Learning
DIGIT, Department of Electrical and Computer Engineering, Aarhus University
A Tight Theory of Error Feedback Algorithms in Distributed Optimization
May 29, 2026Daniel Berg Thomsen, Adrien Taylor, Aymeric DieuleveutCommunication-Efficient Distributed Training Distributed Optimization
Inria, D.I. ENS, CNRS, PSL Research University, Paris, France · CMAP, École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France
Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven
May 7, 2026Ran Ben-Basat, William Kuszmaul, Michael Mitzenmacher +2Quantization Rotation-Based Quantization
UCL and Broadcom · Carnegie Mellon University · Harvard University +2
SignMuon: Communication-Efficient Distributed Muon Optimization
May 4, 2026Neel Mishra, Jichao Jiang, Kushagara Trivedi +3Sign-Based Optimization Communication-Efficient Distributed Training
Microsoft · IIIT Hyderabad
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection
Apr 27, 2026Yutong He, Zhengyang Huang, Jiahe Geng +1Communication-Efficient Distributed Training Gradient Compression
Peking University · Beihang University
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
Apr 16, 2026Adrian Edin, Michel Kieffer, Mikael Johansson +1Communication-Efficient Distributed Training Gradient Compression
Department of Electrical Engineering (ISY), Linköping University, SE-581 83 Linköping, Sweden · CentraleSupélec, CNRS, L2S, Université Paris-Saclay, Gif-sur-Yvette, FR-91192, France · Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden
Communication-Efficient Gluon in Federated Learning
Apr 12, 2026Xun Qian, Alexander Gaponov, Grigory Malinovsky +1Communication-Efficient Distributed Training Federated Learning
Center of Excellence for Generative AI, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia.
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
Feb 9, 2026Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +1Communication-Efficient Distributed Training High-Performance Computing
University College London · VMware Research by Broadcom · Harvard University +1
EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training
Nov 13, 2025Qingao Yi, Jiaang Duan, Jun Zhang +5Communication-Efficient Distributed Training Gradient Compression
University of Shanghai for Science and Technology, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Zhejiang University, Hangzhou, China

Sep 29, 2026J/K move · Enter open · S save

Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection

Jaeseung Heo, J Rosser, Dongwoo Kim

POSTECH · University of Oxford

Gradient Compression Influence Functions Gradient Quantization Training Data Attribution

Open paper AskPDF arXiv

Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example.