cs.LGOct 31, 2026

Profile-aware assessment of block-diagonal Hessian approximations in neural networks

Authors: Maxim Bolshim, Alexander Kugaevskikh

Organizations: ITMO University, Saint Petersburg, Russia · ITMO University, 49 Kronverksky Prospekt, Building A, Saint Petersburg, 197101, Russia

Abstract

Evaluating block-diagonal curvature models in neural networks involves two questions: how much curvature they omit, and what a specific correction changes at a finite step. We measure the squared off-block fraction of the parameter Hessian against references for group sizes and the observed within-group curvature profile. In CIFAR-100 ResNet-18 and a skip-free counterpart, similar initial discarded fractions accompany different profiles. At saved trained ResNet-18 weights, changing from stage to layer groups moves estimated off-block mass from below to above the profile reference. An approximate assessment of probe uncertainty conditional on fixed weights and inputs resolves this crossing; some skip-free comparisons remain unresolved. Separately, in nine saved ResNet-50/Stanford Cars states from three training seeds, we compare candidates formed from off-group Hessian action, the remaining model residual and their combination. For each state, we construct two full-Hessian-informed maps on separate geometry draws. Within each draw, all three inputs share one map. At unit scale off-group input alone raises evaluated minibatch loss relative to the base preconditioner candidate in 17 of 18 state/draw comparisons, yet the combined candidate has lower loss than the remainder-only candidate in nine of those cases; preferences vary with scale and correction-norm matching. The profiles help assess block models and alternative groupings, while same-state candidate comparisons test a particular correction without equating structural mass with training gain.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 13, 2026

Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

Automatic differentiation computes loss Hessians, while interpreting inter-layer interactions requires an explicit choice of graph-indexed blocks. This paper presents an analytical formalism that decomposes the full Hessian into blocks indexed by the DAG of an architecture. The standard decomposition H=HGN+HTH = H^{GN} + H^T separates the Gauss-Newton component (positive semidefinite for a convex output loss) from the tensor component (residual curvature). If every node map is locally affine in its parent activations, the tensor component of the input Hessian vanishes on each smooth cell (Hv,wT=0H^{T}_{v,w}=0, Hv,wf=Hv,wGNH^f_{v,w}=H^{GN}_{v,w}); ReLU activations alone do not suffice. The full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we use diagnostic metrics (inter-layer resonance R\mathcal{R}, geometric coupling C\mathcal{C}, stable rank D\mathcal{D}, GN-Gap) with stochastic HVP-based estimates whose cost depends on graph evaluation and the number of probes. The theoretical analysis gives conditional bounds on curvature decay and residual-chain transport; empirical examples span fully connected MLPs (Exp. 1-5) and convolutional architectures (ResNet-18, ∼11{\sim}11M parameters, Exp. 6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian ∇θ2L(θ)∈Rp×p\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}. Archival correction. The original general novelty claims are no longer maintained: general DAG-Hessian calculus has prior precedents. This is a corrected archival version.
May 25, 2026

Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training

The loss and the norm of its gradient separate the healthy and the pathological regimes of neural-network training only weakly, whilst the curvature of the empirical risk differs qualitatively between them but is inaccessible explicitly at parameter counts P∼106−108P\sim 10^{6}-10^{8}. We present a stochastic estimator of the trace of the diagonal blocks of the Hessian matrix of the empirical risk of a neural network. The procedure combines the Hutchinson stochastic trace estimator with a single Hessian-vector product over the whole parameter vector and recovers unbiased estimates of every per-layer trace in one backward pass through the computational graph. We show that correctness under weight sharing requires the layer-wise Hessian to be assembled before the second differentiation: unrolling shared weights into independent coordinates introduces a systematic bias whose sign and magnitude are governed by the cross-instance blocks of the unrolled Hessian. A closed-form expression for the variance of the estimator at a fixed Hessian is derived, together with a decomposition of the total variance under the mini-batch sampling distribution. This decomposition yields a critical probe count K⋆K^{\star} that balances the two sources of randomness and supports the practical recommendation K∈[5,10]K\in[5,10] in the on-line monitoring regime. The estimator is applied to the detection of the label-memorisation regime of ResNet-18, ResNet-34, and VGG-11 on CIFAR-10 and CIFAR-100, where a calibrated cumulative-sum decision rule attains an empirical detection power of 179/180179/180 at a false-alarm rate of 16/12016/120.
Sep 28, 2026

Reducing the Adaptation Gap Through Reachable Fisher Geometry

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are often used instead. We study what the trace reveals and what it loses through the reachable Fisher: each subgroup's full-model Fisher pulled back through the adapter Jacobian. Under likelihood losses, it represents the Gauss-Newton curvature accessible to the adapter, and its trace can be computed from score-gradient norms without forming the full matrix. Under matched subgroup gradients, a positive-definite reachable-Fisher difference, with a margin exceeding the Hessian-Fisher defect, implies that every sufficiently small nonzero model-changing update increases the signed gap. In contrast, the restricted operator norm determines worst-case quadratic change, while a matrix-free Frobenius discrepancy bounds its reachable-Fisher component. Trace alone cannot certify definiteness or control matrix mismatch. Equal traces rule out a positive-definite difference but can still hide large operator discrepancies. Across 306 single-seed models, higher trace accompanies greater subgroup difficulty in 75.7 percent of 1,218 eligible evaluations, while trace matching reduces the best-worst subgroup gap in all 30 dataset-encoder-adapter combinations. However, held-out audits show that operator discrepancy decreases in 23 of 30 combinations, while the unbiased squared-Frobenius statistic decreases in only 16 of 30. Fisher trace is therefore a scalable diagnostic and training heuristic, but not a certificate of local gap behavior or matrix alignment.