Neural networks are typically optimized with variants of stochastic gradient descent. Under a squared loss, however, the optimal solution to the linear last layer weights is known in closed-form. We propose to leverage this during optimization, treating the last layer as a function of the backbone parameters, and optimizing solely for these parameters. We show this is equivalent to alternating between gradient descent steps on the backbone and closed-form updates on the last layer. We adapt the method for the setting of stochastic gradient descent, by trading off the loss on the current batch against the accumulated information from previous batches. We provide theoretical analyses showing convergence of the method to an optimal solution in the neural tangent kernel regime, as well as quantifying the gains compared to standard SGD in a one-step analysis. Finally, we demonstrate the effectiveness of our approach compared with SGD and Adam on a squared loss in several regression tasks, including neural operators and causal inference.
Figures & tables
Figure 1: The squared loss landscape of a two-parameter neural network f(x) = W ReLU ( θ x) with three random training data points. Dark / light regions correspond to values of high / low loss respectively. We plot in blue the optimal last layer parameter W⋆(θ) as a function of the backbone parameter θ . We propose to optimize along the blue curve, rather than in two-dimensional space.
Figure 2: QM9 results . X-axis is the number of iterations, Y-axis is a test set mean squared error (MSE), columns represents different batch sizes, rows represent different backbone optimizers. Different colors indicate different methods.
Figure 3: QM9, impact of λ . X-axis is λ , Y-axis is the final test mean squared error. First column shows results with the SGD optimizer, while second column shows results with the Adam optimizer.
Figure 4: FNO results . X-axis is the number of iterations, Y-axis is a test set mean squared error (MSE), columns represents different batch sizes. Different colors indicate different methods. We use a rolling average with window size 5 to smooth the curves.
Figure 5: DFIV results . X-axis is the number of iterations, Y-axis is a test set MSE. Each column corresponds to a different batch size. Different colors indicate different methods. Solid lines use the last layer re-estimated on the entire training set, while dashed lines use current last layer estimates. We use a rolling average with window size 5 to smooth the curves.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure G.1: CIFAR-100 results. . X-axis is the number of iterations, Y-axis is a test set accuracy. Each column corresponds to a different batch size. Different colors indicate different methods.
Figure G.2: Dependence on hyperparameters on CIFAR-100 . Top: SGD performance. Bottom: Adam performance. X-axis is the number of iterations, Y-axis is a test set accuracy. Left , ablation over λ for ‘ ℓ2 c.f. proximal ( λ) ”. Center , ablation over β for “ ℓ2 c.f. ridge ( β) ”. Right , the best learning rate per method.
Batch size
Cross entropy max probs
Our method max probs
32
0.99
0.63
128
0.99
0.62
1024
0.99
0.73
4096
0.99
0.76
Appendix
Table G.1: Comparison of maximum probabilities
Method
B=128
B=1024
Cross Entropy
45.0
46.2
ℓ2 c.f. proximal ( λ )
47.4
50.4
Appendix
Table G.2: ViT-S on CIFAR-100 . Test accuracy (%) at the end of training, averaged over 3 seeds, with an SGD backbone.
Method
Test accuracy
Cross Entropy
0.662
ℓ2 c.f. proximal ( λ )
0.686
Cross Entropy, last layer via 1 -step L-BFGS
0.667
Cross Entropy, last layer via 2 -step L-BFGS
0.678
Cross Entropy, last layer via 5 -step L-BFGS
0.679
Appendix
Table G.3: Approximating the last layer for cross entropy on CIFAR-100 . The closed-form update is replaced by K steps of L-BFGS on the cross-entropy loss for fixed backbone features.
Figure G.3: ImageNet results . X-axis is the number of iterations, Y-axis is a test set accuracy. Each column corresponds to a different batch size. Different colors indicate different methods.
Figure G.4: QM9, impact of initialization . X-axis is the number of iterations, Y-axis is a test set mean squared error (MSE), columns represents different batch sizes, rows represent different backbone optimizers. Different colors indicate different ways to initialize the last layer.
Figure G.5: CIFAR-100, impact of initialization . X-axis is the number of iterations, Y-axis is a test set accuracy. Each column corresponds to a different batch size. Different colors indicate different methods. We use SGD optimizer.
Figure G.6: CIFAR-100, Whether to use a bias . X-axis is the number of iterations, Y-axis is a test set accuracy. Each column corresponds to a different batch size. Different colors indicate different methods.
Figure G.7: QM9 algorithms comparison . X-axis is the number of iterations, Y-axis is a test set mean squared error (MSE), columns represents different batch sizes, rows represent different backbone optimizers. Different colors indicate different algorithms. We use SGD optimizer.
Figure G.8: Comparison of Algorithm 1 and Algorithm 2 . X-axis is the number of iterations, Y-axis is a test set accuracy. A column indicates a batch size while a color represents an algorithm.
Window
0–1k
5k–6k
final 1k
Grad-norm ratio
0.82
1.03
1.12
Appendix
Table G.4: Backbone gradient-norm ratio on QM9 (“ ℓ2 c.f. proximal ( λ) ” / “ ℓ2 loss ”), batch size 128 , SGD with learning rate 0.005 , averaged over three windows of training steps.
ℓ2 c.f. proximal
Optimizer
ℓ2 loss
fixed λ
adaptive λt
SGD
1.05
0.84
0.84
Adam
0.811
0.818
0.795
Appendix
Table G.5: Adaptive proximal coefficient on QM9 . Test MSE at batch size 128 , averaged over 3 random seeds.
Method
Test MSE
ℓ2 loss
12.6
ℓ2 c.f. proximal ( λ )
0.8
Appendix
Table G.6: L-BFGS backbone optimizer on QM9 . Test MSE at batch size 128 , averaged over 3 seeds.
Ablation
Hidden dims
ℓ2 loss
ℓ2 c.f. proximal ( λ )
Width
[64]×3
1.19
1.07
Width
[256]×3
1.08
0.87
Width
[1024]×3
1.08
0.82
Width
[4096]×3
1.43
0.80
Depth
[256]×2
1.74
0.93
Depth
[256]×3
1.11
0.88
Appendix
Table G.7: MLP size ablations on QM9 . Test MSE at batch size 128 , averaged over 3 seeds. Hidden dimensions list the widths of the hidden layers.
Batch Size
Last Layer Dim
SPS (Ours)
SPS (CE)
rSPS
Time (Ours)
Time (CE)
rTime
32
128
89.98
81.02
1.11
2164.20
2348.11
0.92
32
256
87.35
78.97
1.11
2214.96
2400.73
0.92
32
512
78.48
79.86
0.98
2413.57
2374.19
1.02
32
1024
70.23
81.89
0.86
2633.73
2325.33
1.13
32
2048
51.47
79.66
0.65
3426.96
2383.31
1.44
32
4096
30.41
82.44
0.37
5488.86
2318.55
2.37
Appendix
Table H.1: Wall clock time comparisons on ResNet-18 with A100 GPU.
Batch Size
Last Layer Dim
SPS (Ours)
SPS (CE)
rSPS
Time (Ours)
Time (CE)
rTime
32
128
36.96
42.94
0.86
4735.09
2855.43
1.66
32
256
41.18
42.84
0.96
4240.65
4056.31
1.05
32
512
35.85
39.59
0.91
4903.87
3496.42
1.40
32
1024
35.22
43.28
0.81
4903.89
4002.80
1.23
32
2048
30.85
41.52
0.74
5482.36
4173.49
1.31
32
4096
21.68
41.97
0.52
7605.07
4133.77
1.84
Appendix
Table H.2: Wall clock time comparisons on ResNet-50 with A100 GPU.
Figure I.1: Scalability summary on QM9 (A100 GPU). (a) Relative step-time overhead vs. depth L ( H=2048,B=2048 ): Algorithm 2 plateaus at +33.3% , whereas Algorithm 1 decays as O(1/L) to +3.6% at L=32,d=256 . (b) Wide backbones ( H∈[2048,8192] ) with a d=256 bottleneck achieve <5% overhead with Algorithm 1 . (c) Scaling the output dimension o by 4096× ( o=1→4096 ) changes the overhead of Algorithm 1 by only +1.3% at L=32 . (d) Peak GPU memory of Algorithm 1 is nearly identical to “ ℓ2 loss ” ( <1.5% overhead).
Figure I.2: Detailed scaling curves on A100 GPU. (a) Log-scale overhead vs. depth L across d∈{256,512,1024,2048,4096} . (b) Bottleneck regime ( d=256 ) across hidden widths H∈{2048,4096,8192} . (c) Scaling with output dimension o∈{1,12,128,1024,4096} .
Regime
H
d
L
Params
ℓ2 loss
Algorithm 2 (rTime)
Algorithm 1 (rTime)
Overhead ( Algorithm 1 )
Memory (GiB)
Depth L ( B=2048 , o=12 )
2048
256
4
14.0M
2.72 ms
4.07 ms (1.49 × )
3.39 ms (1.25 × )
+24.5%
0.81 → 0.87
2048
256
8
30.8M
5.29 ms
7.43 ms (1.40 × )
6.06 ms (1.15 × )
+14.5%
1.44 → 1.50
2048
256
16
64.4M
10.66 ms
14.20 ms (1.33 × )
11.46 ms (1.08 × )
+7.6%
2.69 → 2.81
2048
256
32
131.5M
21.35 ms
27.76 ms (1.30 × )
22.12 ms (1.04 × )
+3.6%
5.33 → 5.46
2048
512
32
132.0M
21.41 ms
28.73 ms (1.34 × )
23.06 ms (1.08 × )
+7.7%
5.35 → 5.44
2048
1024
32
133.1M
21.60 ms
30.96 ms (1.43 × )
25.15 ms (1.16 × )
+16.4%
5.37 → 5.53
Appendix
Table I.1: Wall-clock step time, rTime, and peak GPU memory on QM9 (A100 GPU). Comparison of “ ℓ2 loss ” (SGD), Algorithm 2 , and Algorithm 1 .
Figure I.3: Shallow-backbone ( L=3 ) rTime heatmaps across (H,d) at B=128 and B=2048 . For d≤512 , rTime remains near 1.0× .
Figure I.4: End-to-end QM9 training ablations ( H=256,L=3,B=128 ). (a) Test MSE vs. d . (b) Training step time vs. d . (c) Peak GPU memory vs. d .
Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford +2