A Systematic Analysis of a Small Optimal Subspace for Physics-Informed Neural Networks: -DAS
Organizations: Division of Applied Mathematics, Brown University, Providence, RI 02912, USA
Abstract
The search for a low-dimensional subspace that governs neural network training has been a persistent goal in optimization. Yet subspaces defined by Hessian eigenvalue magnitude, such as the dominant/bulk decomposition underlying Dom-SGD and Bulk-SGD, have repeatedly failed to capture where optimization takes place, and adaptive Krylov-based alternatives built on energy-fraction thresholds show promise but still struggle at lower eigenmodes. We argue this failure is structural: the directions of highest curvature are not, in general, the directions along which training moves. We introduce the dominant aligned subspace (DAS), constructed from a full eigendecomposition of the Hessian but selected by gradient alignment rather than eigenvalue magnitude, as a tool for understanding optimization dynamics and a foundation for designing methods that exploit them. To expose this structure under the hardest conditions, we use physics-informed neural networks (PINNs) as a testbed: differential operators from the parabolic, elliptic, and hyperbolic classes induce severely ill-conditioned, anisotropic loss landscapes. Across Poisson, heat, wave, and viscous Burgers benchmarks DAS-restricted training with a line search matches full-space training even where the projected Hessian remains ill-conditioned, showing empirically that convergence does not require all Hessian eigendirections but only a low-dimensional, gradient-aligned subset. We extend the analysis to Gauss-Newton within the DAS and discuss what this structure implies for future subspace-aware optimizers.
Figures & tables
| Benchmark | PDE | Exact solution | Int. | Bdry | Init | |
|---|---|---|---|---|---|---|
| Poisson-2D | 1981 | 400 | 100 | — | ||
| heat-1D | 1981 | 80 | 24 | 24 | ||
| wave-1D | 1981 | 100 | 24 | 24 | ||
| Burgers-1D | Cole–Hopf | 6391 | 1500 | — | — |
| Rel. | Rel. | Rel. | Rel. | Max | BC | IC | Res. | Res. | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | err. | RMSE | err. | err. | RMS | max | ||||
| Proj BT | 0.1920 | 0.4844 | 0.1811 | 0.3388 | 0.4844 | 0.0961 | 0.4844 | 0.0811 | 0.0312 | 0.1365 |
| Proj Raw | 0.3094 | 0.4842 | 0.3040 | 0.4145 | 0.4842 | 0.1548 | 0.4842 | 0.1547 | 0.0338 | 0.1084 |
| Proj Wolfe | 0.1807 | 0.4615 | 0.1698 | 0.3196 | 0.4615 | 0.0904 | 0.4615 | 0.0747 | 0.0298 | 0.1165 |
| Proj BT 99 | 0.1696 | 0.4356 | 0.1588 | 0.3001 | 0.4356 | 0.0848 | 0.4356 | 0.0696 | 0.0277 | 0.1210 |
| Proj Raw 99 | 0.3068 | 0.4498 | 0.3003 | 0.3968 | 0.4498 | 0.1535 | 0.4498 | 0.1409 | 0.0455 | 0.1525 |
| Training | Test | Relative | Iterations to | |||||
|---|---|---|---|---|---|---|---|---|
| Method | Global. | median | mean | loss | loss | error | loss | |
| Full-space | BT | — | — | — | 70 | |||
| Full-space | Wolfe | — | — | — | 50 | |||
| Full-space | LM | — | — | — | 330 | |||
| DAS | BT | 0.999 | 10 | 18.5 | 2580 | |||
| DAS | Wolfe | 0.999 | 10 | 18.3 | 2270 | |||
| Directions saved | Matrix | Dense | |||||
| Global. | Mean | mean | median | worst | entries saved | solve saved | |
| BT | 0.999 | 18.5 | 96.3% | 98.0% | 77.6% | 99.77% | 99.980% |
| Wolfe | 0.999 | 18.3 | 96.4% | 98.0% | 77.6% | 99.72% | 99.963% |
| LM | 0.999 | 21.2 | 95.8% | 97.4% | 75.2% | 99.58% | 99.928% |
| BT | 0.9999 | 17.8 | 96.4% | 97.6% | 74.4% | 99.70% | 99.949% |
| Wolfe | 0.9999 | 36.9 | 92.6% | 96.8% | 74.6% | 98.76% | 99.719% |
| Step rule | Full space | projB | projF |
|---|---|---|---|
| Final full-batch training loss | |||
| Fixed step | |||
| Backtracking | |||
| Wolfe | |||
| Final relative error | |||
| Fixed step | |||
| Training | Test | Relative | Iterations to loss | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | Selection | Global. | median | mean | loss | loss | error | |||
| Full-space | — | BT | — | — | — | 160 | 790 | |||
| Full-space | — | Wolfe | — | — | — | 260 | 790 | |||
| Full-space | — | LM | — | — | — | 330 | 1680 | |||
| DAS | batch | BT | 0.999 | 8 | 10.6 | 2650 | — | |||
| DAS | batch | Wolfe | 0.999 | 10 | 13.6 | 1780 | — | |||
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.