cs.LGOct 6, 2025

Closed-Form Last Layer Optimization

Authors: Alexandre Galashov, Nathaël Da Costa, Liyuan Xu, Philipp Hennig, Arthur Gretton

Organizations: Google DeepMind Gatsby Unit, University College London · Tübingen AI Center, University of Tübingen · Secondmind†

Abstract

Neural networks are typically optimized with variants of stochastic gradient descent. Under a squared loss, however, the optimal solution to the linear last layer weights is known in closed-form. We propose to leverage this during optimization, treating the last layer as a function of the backbone parameters, and optimizing solely for these parameters. We show this is equivalent to alternating between gradient descent steps on the backbone and closed-form updates on the last layer. We adapt the method for the setting of stochastic gradient descent, by trading off the loss on the current batch against the accumulated information from previous batches. We provide theoretical analyses showing convergence of the method to an optimal solution in the neural tangent kernel regime, as well as quantifying the gains compared to standard SGD in a one-step analysis. Finally, we demonstrate the effectiveness of our approach compared with SGD and Adam on a squared loss in several regression tasks, including neural operators and causal inference.

Figures & tables

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

    Apr 1, 2026Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1Deep Learning OptimizationNatural Gradient Methods

  2. On the Principles Behind Neural Network Optimizers

    Aug 17, 2026Yushun ZhangNeural Network OptimizationAdaptive Gradient Methods