Gradient Quantization

Follow this topic Ask about this topic

Latest papers

8 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection
Sep 29, 2026Jaeseung Heo, J Rosser, Dongwoo KimGradient Compression Influence Functions
POSTECH · University of Oxford
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jul 8, 2026Jieying Wang, Zizhong Wang, Fangru Linghu +3Language Model Pretraining Communication-Efficient Distributed Training
Department of Computer Science, Rutgers University, Piscataway, NJ, USA · Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, USA
MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
May 20, 2026Zhong Wang, Zukang Xu, Xing Hu +1Mixed-Precision Quantization Vision-Language Models
Bauman Moscow State Technical University · Houmo AI
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
Apr 28, 2026Haoran Chen, Wentao WangSign-Based Optimization Stochastic Optimization Convergence
´Ecole polytechnique Palaiseau, France
Quantization Impact on the Accuracy and Communication Efficiency Trade-off in Federated Learning for Aerospace Predictive Maintenance
Apr 9, 2026Abdelkarim LoukiliGradient Quantization Federated Learning
OptINC: Optical In-Network-Computing for Scalable Distributed Learning
Mar 30, 2026Sijie Fei, Grace Li Zhang, Bing Li +1Gradient Quantization Communication-Efficient Distributed Training
Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany
On the Gradient Complexity of Private Optimization with Private Oracles
Nov 17, 2025Michael Menart, Aleksandar NikolovQuery Complexity Oracle Complexity
Department of Computer Science, University of Toronto, Vector Institute · Department of Computer Science, University of Toronto

Sep 29, 2026J/K move · Enter open · S save

Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection

Jaeseung Heo, J Rosser, Dongwoo Kim

POSTECH · University of Oxford

Gradient Compression Influence Functions Gradient Quantization Training Data Attribution

Open paper AskPDF arXiv

Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example.