cs.LGNov 18, 2026

A Systematic Analysis of a Small Optimal Subspace for Physics-Informed Neural Networks: nn-DAS ⊂Rp\subset \mathbb{R}^p

Authors: Ahmad AliAhmad, Khemraj Shukla

Organizations: Division of Applied Mathematics, Brown University, Providence, RI 02912, USA

Abstract

The search for a low-dimensional subspace that governs neural network training has been a persistent goal in optimization. Yet subspaces defined by Hessian eigenvalue magnitude, such as the dominant/bulk decomposition underlying Dom-SGD and Bulk-SGD, have repeatedly failed to capture where optimization takes place, and adaptive Krylov-based alternatives built on energy-fraction thresholds show promise but still struggle at lower eigenmodes. We argue this failure is structural: the directions of highest curvature are not, in general, the directions along which training moves. We introduce the dominant aligned subspace (DAS), constructed from a full eigendecomposition of the Hessian but selected by gradient alignment rather than eigenvalue magnitude, as a tool for understanding optimization dynamics and a foundation for designing methods that exploit them. To expose this structure under the hardest conditions, we use physics-informed neural networks (PINNs) as a testbed: differential operators from the parabolic, elliptic, and hyperbolic classes induce severely ill-conditioned, anisotropic loss landscapes. Across Poisson, heat, wave, and viscous Burgers benchmarks DAS-restricted training with a line search matches full-space training even where the projected Hessian remains ill-conditioned, showing empirically that convergence does not require all Hessian eigendirections but only a low-dimensional, gradient-aligned subset. We extend the analysis to Gauss-Newton within the DAS and discuss what this structure implies for future subspace-aware optimizers.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks

    Apr 16, 2026Kang An, Chenhao Si, Shiqian Ma +1Neural Network OptimizationSecond-Order Optimization

  2. An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

    Jul 2, 2026Joseph Webb, Sadok Jerad, Coralia CartisGauss-Newton OptimizationPDE Solving

  3. Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks

    Apr 6, 2026Anas Jnini, Elham Kiyani, Khemraj Shukla +5Physics-Informed MLNeural PDE Solvers