cs.LGOct 31, 2026

Profile-aware assessment of block-diagonal Hessian approximations in neural networks

Authors: Maxim Bolshim, Alexander Kugaevskikh

Organizations: ITMO University, Saint Petersburg, Russia · ITMO University, 49 Kronverksky Prospekt, Building A, Saint Petersburg, 197101, Russia

Abstract

Evaluating block-diagonal curvature models in neural networks involves two questions: how much curvature they omit, and what a specific correction changes at a finite step. We measure the squared off-block fraction of the parameter Hessian against references for group sizes and the observed within-group curvature profile. In CIFAR-100 ResNet-18 and a skip-free counterpart, similar initial discarded fractions accompany different profiles. At saved trained ResNet-18 weights, changing from stage to layer groups moves estimated off-block mass from below to above the profile reference. An approximate assessment of probe uncertainty conditional on fixed weights and inputs resolves this crossing; some skip-free comparisons remain unresolved. Separately, in nine saved ResNet-50/Stanford Cars states from three training seeds, we compare candidates formed from off-group Hessian action, the remaining model residual and their combination. For each state, we construct two full-Hessian-informed maps on separate geometry draws. Within each draw, all three inputs share one map. At unit scale off-group input alone raises evaluated minibatch loss relative to the base preconditioner candidate in 17 of 18 state/draw comparisons, yet the combined candidate has lower loss than the remainder-only candidate in nine of those cases; preferences vary with scale and correction-norm matching. The profiles help assess block models and alternative groupings, while same-state candidate comparisons test a particular correction without equating structural mass with training gain.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

    Apr 13, 2026Maxim Bolshim, Alexander KugaevskikhNeural Network OptimizationMultilayer Perceptrons

  2. Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training

    May 25, 2026Maxim Bolshim, Alexander KugaevskikhNeural Network MemorizationStochastic Approximation

  3. Reducing the Adaptation Gap Through Reachable Fisher Geometry

    Sep 28, 2026Wasif Jalal, Sachin Deb, Asif SalekinParameter-Efficient Fine-TuningGroup Fairness