Kronecker-Factored Approximation

Follow this topic Ask about this topic

Latest papers

14 papers2 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Spectrum Is Not Enough: A Random-Matrix Account of Data-Free Preconditioning for Differentially Private Learning
Oct 15, 2026Shilin Zhang, Kangning WangKronecker-Factored Approximation Differential Privacy
School of Electrical and Control Engineering, North China University of Technology, Beijing, China
Does Financial Speech Need Continual Learning Regularization? A Study of EWC and mini-KFAC on Earnings Call ASR
Oct 9, 2026Thai Quoc On, Homayoon BeigiKronecker-Factored Approximation Elastic Weight Consolidation
Department of Computer Science Columbia University, NYC, NY, USA · Mechanical Engineering Dept. and Electrical Engineering Dept. Columbia University, NYC, NY, USA · and Recognition Technologies, Inc. South Salem, NY, USA
Distilling Diffusion Score Discrepancy for Efficient Training Data Attribution
Sep 30, 2026Shixuan Liu, Joan Serrà, Kin Wai Cheuk +6Gradient-Based Attribution Kronecker-Factored Approximation
University of Illinois at Urbana-Champaign · Sony AI · University of Texas at Austin +1
Traceprop: Training Data Attribution in a Single Pass
Sep 26, 2026Amit NautiyalGradient-Based Attribution Kronecker-Factored Approximation
Independent Researcher
On the second-order optimization for spiking neural networks
Sep 24, 2026Ngoc Phu Doan, Ihsen AlouaniNeural Network Optimization Second-Order Optimization
Centre for Secure Information Technologies Queen’s University Belfast
WaterKron and FlipFlop Hessian: Information-Theoretically Grounded Quantization with Kronecker-factored Hessians
Sep 13, 2026Johann Birnick, Rayan SaabNeural Network Quantization Post-Training Quantization
Department of Mathematics University of California San Diego La Jolla, CA 92093, USA · Department of Mathematics and HDSI University of California San Diego La Jolla, CA 92093, USA
BaKron: Efficient Quantization with Kronecker-Factored Hessians
Aug 6, 2026Johann Birnick, Rayan SaabPost-Training Quantization Neural Network Quantization
Department of Mathematics · University of California San Diego · La Jolla, California, United States +1
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
May 30, 2026Bing Liu, Wenjie Zhou, Chengcheng ZhaoSecond-Order Optimization Adaptive Gradient Methods
College of Control Science and Engineering, Zhejiang University · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences
DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning
May 13, 2026Marc Molina Van den Bosch, Riccardo Taiello, Albert Sund Aillet +3Differentially Private Stochastic Gradient Descent Kronecker-Factored Approximation
CERN, Geneva, Switzerland · BCN MedTech, Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain · ICREA, Barcelona, Spain
Efficient Bilevel Optimization with KFAC-Based Hypergradients
Mar 31, 2026Disen Liao, Felix Dangel, Yaoliang YuKronecker-Factored Approximation Bilevel Optimization
University of Waterloo · Vector Institute
Natural Gradient Descent for Online Continual Learning
Mar 21, 2026Joe Khawand, David ColliauxKronecker-Factored Approximation Online Continual Learning
Ecole Polytechnique & Télécom Paris Paris, France · Sony Computer Science Laboratories Paris, France
Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
Feb 19, 2026Angelo Porrello, Pietro Buzzega, Felix Dangel +4Kronecker-Factored Approximation Task Arithmetic
University of Modena and Reggio Emilia, Italy · Vector Institute, Toronto
Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
Feb 18, 2026Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov +3Proximal Optimization Kronecker-Factored Approximation
MBZUAI, Abu Dhabi, UAE · Yandex School of Data Analysis, Moscow, Russia · SB AI Lab, Moscow, Russia +2
Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models
May 23, 2025Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva +2Model Compression LLM Compression
1Fusion Brain · 2MSU · 3AXXX +2

Oct 15, 2026J/K move · Enter open · S save

Spectrum Is Not Enough: A Random-Matrix Account of Data-Free Preconditioning for Differentially Private Learning

Shilin Zhang, Kangning Wang

School of Electrical and Control Engineering, North China University of Technology, Beijing, China

Kronecker-Factored Approximation Differential Privacy

Open paper AskPDF arXiv

At a fixed privacy budget, curvature is exactly the object the budget makes unavailable. A recent answer estimates it from synthetic probes, at neither privacy cost nor distribution shift, and validates a probe by the eigenvalue decay it reproduces. That test reads the spectrum, not the basis, and what a probe reconstructs is unknown. This paper gives a random-matrix account of data-free preconditioning, organized by one dichotomy. Of the two Kronecker factors, the error-signal factor

G

and the architecture-determined bulk of the activation factor

A

are reconstructible without data up to an explicit error-energy bound; only the data-induced spikes of

A

are not. We verify that bound with zero violations along a full training trajectory, though it stays one to nearly three orders of magnitude above the gap it bounds. We locate the failure boundary at a Baik-Ben Arous-Péché threshold: a bulk-only probe suffers an exact damage

s^2/(1-s^2)

from a spike of strength

s^2

, invisible once its amplitude exceeds the scaled-bulk threshold

( \sqrt {c}/(1+ \sqrt {c}))^{1/2}

c = d/n

. The test in use is basis-blind: rotating only the probe eigenbasis at a near-exact matched spectrum inflates the residual by

2.3 \times

89.8 \times

, and a closed-form volume law fixes the within-spectrum ordering of the critical volume while a two-oracle construction proves that no functional of the probe spectrum fixes its magnitude. Over forty seeds on three modalities, per-layer tuning does not transfer to accuracy, no data-free arm beats the flat probe, public-data curvature does on both modalities tested, and the ordering inverts on PathMNIST under real DP noise. The account yields a probe-design protocol and falsifiable predictions for the regimes where data-free curvature can be trusted.