Calibrated Target Noise Recovers Curvature from the Gradients

Arash Jamshidi; Katsiaryna Haitsiukevich; Aristides Gionis; Kai Puolamäki

cs.LGNov 5, 2026

Calibrated Target Noise Recovers Curvature from the Gradients

Authors: Arash Jamshidi, Katsiaryna Haitsiukevich, Aristides Gionis, Kai Puolamäki

Organizations: Department of Computer Science, University of Helsinki, Helsinki, Finland · Division of Theoretical Computer Science, KTH Royal Institute of Technology, Digital Futures, Stockholm, Sweden · Department of Computer Science, Institute for Atmospheric and Earth System Research, University of Helsinki, Helsinki, Finland

Abstract

Estimating curvature information from gradients alone is a fundamental problem in machine learning. The challenge becomes particularly acute in settings where only aggregate mini-batch gradients are available, yet curvature information is still required for applications such as preconditioning in optimisation, sampling, and early stopping. Perhaps surprisingly, we show that a simple target-perturbation scheme suffices to recover such curvature information from gradients alone. Our method injects zero-mean noise into the targets, with variance calibrated to the local output curvature. The covariance of the resulting noisy batch gradients then recovers practical curvature matrices, without requiring per-sample gradients, second-order derivatives, or other internal network quantities. For a broad class of non-linear models, our scheme recovers the Generalised Gauss--Newton (GGN) matrix; for feed-forward ReLU networks, it additionally recovers the diagonal blocks of the population Hessian. We demonstrate that the resulting estimator is simple, lightweight, provably optimal, and easy to integrate into existing training pipelines. Experiments demonstrate accurate curvature recovery, faster optimisation when the estimator is used for preconditioning, and the utility of the estimated GGN for early stopping.

Figures & tables

Centered estimator	Swamping estimator

1: Input: parameter $w$ , noise multiplier $\beta$ .	1: Input: parameter $w$ , noise multiplier $\beta$ , center $\mu_{t}$ .
2: Draw a mini-batch $\{(x_{i},y_{i})\}_{i=1}^{n}$ .	2: Draw a mini-batch $\{(x_{i},y_{i})\}_{i=1}^{n}$ .
3: Compute $\eta_{i}=f_{w}(x_{i})$ , $\forall i\in[n]$ .	3: Compute $\eta_{i}=f_{w}(x_{i})$ , $\forall i\in[n]$ .
4: Set $v_{i}=nA^{\prime\prime}(\eta_{i})$ , $\forall i\in[n]$ .	4: Set $v_{i}=nA^{\prime\prime}(\eta_{i})$ , $\forall i\in[n]$ .
5: Sample $\xi_{i}\sim\mathrm{Unif}\{+\sqrt{v_{i}},-\sqrt{v_{i}}\}$ , $\forall i\in[n]$ .	5: Sample $\xi_{i}\sim\mathrm{Unif}\{+\sqrt{v_{i}},-\sqrt{v_{i}}\}$ , $\forall i\in[n]$ .

Algorithm 1 Centred and swamping noisy-gradient curvature (CNGC) estimators.

Figure 1: Relative operator norm between GGN and its estimates showing the effects of centring ( left ), swamping ( middle ) and the added noise distribution ( right ) on curvature recovery. Both centring and swamping with a large noise multiplier make the estimates robust. Rademacher noise distribution yields a more accurate estimator with lower variance.

Figure 2: Comparison of GGN estimators with an MLP for a classification task ( left ) and a CNN for a regression task ( middle ) by the relative operator norm of the difference of the full GGN matrix and its estimators. The wall-clock time for calculation of the estimators for the CNN model ( right ). The proposed CNGC estimator performs consistently across classification and regression tasks and is computationally cheaper than the full GGN calculation.

Figure 3: The effect of calibrated noisy gradient covariance as a preconditioner (blue lines) for a classification task (telescope dataset) with an MLP, a time series classification (occupancy detection dataset) and regression (air quality dataset) with a 1D CNN and a transformer encoder.

Optimiser	MLP Telescope	CNN Occupancy	CNN Air	Transformer Occupancy	Transformer Air
GD	$0.22\pm 0.00$	$0.42\pm 0.01$	$0.18\pm 0.00$	$3.85\pm 0.03$	$1.81\pm 0.02$
Adam	$0.25\pm 0.00$	$0.46\pm 0.01$	$0.20\pm 0.00$	$4.05\pm 0.06$	$1.89\pm 0.02$
Sophia	$0.26\pm 0.00$	$0.49\pm 0.01$	$0.22\pm 0.00$	$4.42\pm 0.06$	$2.05\pm 0.02$
SophiaC (ours)	$0.26\pm 0.00$	$0.50\pm 0.00$	$0.22\pm 0.00$	$4.41\pm 0.04$	$2.06\pm 0.02$
GD, PC (ours)	$0.35\pm 0.01$	$0.66\pm 0.01$	$0.29\pm 0.00$	$4.90\pm 0.05$	$2.29\pm 0.02$

Table 1: Wall-clock time per training epoch (seconds). The results are mean

\pm

one standard deviation over 10 random seeds.

Optimiser	MLP Telescope	CNN Occupancy	CNN Air	Transformer Occupancy	Transformer Air
Test-loss threshold	$0.3402$	$0.1701$	$0.4804$	$0.1767$	$0.2526$
GD	$22.29\pm 0.23^{\dagger}$	$8.36\pm 0.11^{\dagger}$	$18.22\pm 0.35^{\dagger}$	$77.08\pm 0.66^{\dagger}$	$181.03\pm 2.00^{\dagger}$
Adam	$16.02\pm 0.17$	$\mathit{3.81\pm 0.05}$	$20.34\pm 0.22^{\dagger}$	$44.71\pm 0.66$	$169.94\pm 2.04$
Sophia	$7.62\pm 0.09$	$4.10\pm 0.04$	$16.78\pm 0.28$	$44.37\pm 0.56$	$96.71\pm 0.76$
SophiaC (ours)	$\mathit{7.42\pm 0.11}$	$4.12\pm 0.05$	$\mathit{12.00\pm 0.14}$	$\mathit{39.90\pm 0.38}$	$\mathit{86.46\pm 0.92}$
GD, PC (ours)	$\mathbf{1.65\pm 0.02}$	$\mathbf{0.96\pm 0.02}$	$\mathbf{2.60\pm 0.02}$	$\mathbf{10.01\pm 0.11}$	$\mathbf{16.32\pm 0.16}$

Table 2: Wall-clock time (seconds) to reach within 10% of the best test loss achieved in each experiment. The corresponding thresholds are listed below.

\dagger

indicates that the threshold was not reached within the training budget; the full elapsed training time is reported instead. Bold and italic indicate the fastest and second-fastest methods, respectively. The results are mean

\pm

one standard deviation over 10 random seeds.

Figure 4: The effect of calibrated noisy gradient covariance for early stopping (red vertical lines) for the same tasks as in the optimisation experiment compared to the baseline.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

1:	Input: initial parameter $w_{0}$ , estimator type, learning-rate rule $\alpha(\cdot)$ , layer blocks $u_{j}=\mathrm{vec}(W_{j})$ , $j\in[m]$ .
2:	Initialize $P_{j,0}=I$ for all layers $j\in[m]$ , and set $\alpha_{0}=10^{-4}$ , $\tau_{0}=1$ , $\omega=0.9$ .
3:	for epoch $e=0,1,\ldots,E-1$ do
4:	Initialize $\overline{G}_{j,e}=0$ for all layers $j\in[m]$ .
5:	for batch $t=0,1,\ldots,T_{e}-1$ do
6:	Compute $g_{t}=\nabla_{w}\sum_{i}\ell(f_{w_{t}}(x_{i}),y_{i})/n$ .
7:	Call Algorithm 1 with the chosen estimator type to obtain block samples $s_{j,t}$ , $j\in[m]$ .

Appendix

Algorithm 2 Block-wise preconditioned gradient descent with noisy-gradient curvature estimates (GD, PC).

Figure 5: Illustrative example comparing CNGC and GGN with the true Hessian for a small convolutional network with 12 parameters. Top: regression, bottom: classification.

Figure 6: Relative operator norm between the spectrum of the true Hessian and the curvature estimator with centring and swamping with noise multiplier equal to 1 and to 10. Both centring and swamping with noise, with a multiplier equal to 10, capture the eigenspectrum of the true Hessian.

Panel of Figure 7 — Figure 7: Top: Comparison of GGN estimators with an MLP for a classification task ( left ) and a CNN for a regression task ( middle ) by the relative operator norm of the difference of the eigenvalue spectrum of the full GGN matrix and its estimates. The wall-clock time for the calculations of GGN estimates for the MLP model ( right ). Bottom: Comparison of GGN estimators with the true Hessian for a CNN on a regression task. The relative operator norm of the difference between (positive) Hessian eigenvalues and the eigenvalues of the estimates ( left ), the relative operator norm of the difference between full matrices of the estimates and the Hessian ( right ).

Figure 8: The effect of calibrated noisy gradient covariance as a preconditioner for logistic regression and MLPs on classification tasks for telescope, bank marketing, and adult income datasets.

Optimiser	MLP Telescope	CNN Occupancy	CNN Air	Transformer Occupancy	Transformer Air
GD, PC (ours)	$0.3005\pm 0.0032$	$0.1080\pm 0.0065$	$0.4053\pm 0.0200$	$0.1136\pm 0.0573$	$0.1456\pm 0.0096$
Exact GGN blocks (oracle)	$0.2981\pm 0.0040$	$0.1082\pm 0.0130$	$0.3887\pm 0.0214$	$0.0918\pm 0.0058$	$0.1384\pm 0.0118$

Appendix

Table 3: Best test loss achieved using CNGC and exact GGN blocks. The results are mean

\pm

one standard deviation over 10 random seeds.

Optimiser	MLP Telescope	CNN Occupancy	CNN Air	Transformer Occupancy	Transformer Air
GD, PC (ours)	$4.50\pm 0.50$	$1.10\pm 0.30$	$11.00\pm 3.82$	$3.70\pm 5.46^{\dagger}$	$7.50\pm 2.29$
Exact GGN blocks (oracle)	$3.40\pm 0.80$	$1.10\pm 0.30$	$3.70\pm 1.10$	$2.40\pm 1.62$	$7.20\pm 3.16$

Appendix

Table 4: Number of training epochs required to reach within 10% of the best test loss for each optimiser and model–dataset combination.

\dagger

marks an entry for which the threshold was not reached within the training budget for some of the seeds. The results are mean

\pm

one standard deviation over 10 random seeds.

Stopping criterion	MLP Telescope	CNN Occupancy	CNN Air	Transformer Occupancy	Transformer Air
CNGC (ours)	$0.3075\pm 0.0034$	$0.1279\pm 0.0127$	$0.4647\pm 0.0234$	$0.1223\pm 0.0580$	$0.1843\pm 0.0271$
GradStop	$0.3056\pm 0.0039$	$0.1350\pm 0.0196$	$0.4799\pm 0.0419$	$0.1227\pm 0.0610$	$0.2411\pm 0.1246$
Validation loss	$0.3061\pm 0.0039$	$0.1273\pm 0.0197$	$0.5223\pm 0.2019$	$0.1078\pm 0.0137$	$0.1646\pm 0.0096$

Appendix

Table 5: Test loss obtained with each early-stopping criterion using patience 3. The results are mean

\pm

one standard deviation over 10 random seeds.

Figure 9: The calculated stopping points on the occupancy detection dataset with GradStop and the proposed criterion with CNGC.

Curvature-Aware Optimization

See all 27 papers

Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
Jun 29, 2026Muhammad Hamza, Ayush GoelLearning Rate Scheduling Stochastic Optimization Convergence
Indian Institute of Technology Kharagpur · Indian Institute of Technology Kharagpur (IIT KGP)
Exploiting weight-space symmetries for approximating curvature
May 30, 2026Artem Artemev, Rui Xia, Benjamin M. Boyd +4Neural Network Optimization Second-Order Optimization
MediaTek, Cambridge, UK · University of Cambridge, Cambridge, UK · Vector Institute, Toronto, Canada
Exact information accounting for SGD methods
Sep 30, 2026Akshay BalsubramaniNeural Network Generalization Stochastic Optimization Convergence

Jun 29, 2026J/K move · Enter open · S save

Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

Muhammad Hamza, Ayush Goel

Indian Institute of Technology Kharagpur · Indian Institute of Technology Kharagpur (IIT KGP)

Learning Rate Scheduling Stochastic Optimization Convergence Stochastic Gradient Descent Curvature-Aware Optimization

Open paper AskPDF arXiv

The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there. We introduce Curvature-Weighted Gradient Diversity (CWGD), a geometry-aware measure that weights per-sample gradient diversity by the inverse square root of the Hessian, providing a tighter proxy for the effective optimization noise. For strongly convex quadratic objectives with diagonal Hessians and isotropic noise, we prove that a CWGD-modulated cosine learning-rate schedule can reduce the asymptotic optimization error floor by up to a factor of two compared with standard cosine annealing. We implement this idea as CWGD-Cosine using a Hutchinson-based diagonal Hessian estimator that is exact for quadratic objectives. Across a range of condition numbers, batch sizes, and noise structures, CWGD-Cosine consistently achieves approximately 20% lower final optimization error than standard cosine annealing while incurring negligible overhead in the quadratic setting. We also identify and correct a degenerate curvature estimator, analyze the robustness of the proposed estimator, and explicitly discuss the limitations of the method, including Hessian staleness in non-convex optimization. These results establish CWGD as a principled geometry-aware measure of optimization noise and motivate future extensions to more general learning problems.

Abstract

Curvature-Aware Optimization

Explore similar work

Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

Exploiting weight-space symmetries for approximating curvature

Exact information accounting for SGD methods