math.NANov 9, 2026

Block-Sparse Gauss-Newton Optimization of Finite Basis Physics-Informed Neural Networks: Sparse Automatic Differentiation Approach

Authors: Denis Korolev, Adrian Hill, Jeeeun Lee

Organizations: Weierstrass Institute for Applied Analysis and Stochastics, Anton-Wilhelm-Amo-Str. 39, Berlin, 10117, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Franklinstr. 28/29, Berlin, 10587, Germany · Machine Learning Group, Technical University of Berlin, Franklinstr. 28/29, Berlin, 10587, Germany · KAIST InnoCORE PRISM-AI Center, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea

Abstract

In this work, we develop an efficient Gauss-Newton optimization approach for finite basis physics-informed neural networks (FBPINNs). The key ingredient is the overlapping domain decomposition architecture of FBPINNs, which localizes the influence of each subnetwork and induces block sparsity in both the residual Jacobian and the corresponding Gauss-Newton matrix. We characterize this structure and exploit it through sparse automatic differentiation, using graph coloring to partition structurally orthogonal Jacobian columns or rows. This reduces differentiation costs and memory requirements while preserving all structurally nonzero derivative information. The resulting compressed Jacobian representations allow us to assemble the Gauss-Newton system directly in block-sparse form, enabling the use of sparse direct solvers and preconditioned iterative methods. In this way, the otherwise expensive second-order optimization problem is reduced to scalable sparse linear algebra. We demonstrate the approach on multiscale PDEs, where accurate approximation requires rich neural network parametrizations and large numbers of collocation points. Numerical experiments show that our block-sparse Gauss-Newton method attains high accuracy and substantially accelerates optimization relative to first-order baselines while remaining computationally tractable for large problem sizes.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Sep 28, 2026

Tensor-Train Compressed Separable PINNs: A Curvature-Aware Optimization Framework for Parametric PDEs in High Dimensions

In this work, we develop a second-order optimization framework for physics-informed neural networks (PINNs) applied to high-dimensional parametric partial differential equations (PDEs). The framework is built on the Gauss--Newton pullback metric, which provides an operator-informed notion of curvature in parameter space and connects the method to the broader family of natural gradient schemes. We show that, for coordinate-separable neural architectures and linear differential operators (or linearized operators in the nonlinear case) admitting a finite separable representation, the residual Jacobian inherits a structured separable factorization. This yields an exact compressed formulation of the Gauss--Newton step in a reduced space, without assembling the full residual Jacobian on the exponentially large tensor-product collocation grid. The dimension of the reduced space (the effective compressed dimension) is determined by the local collocation grid sizes, the separable operator structure, and the contraction pattern of the architecture, thereby avoiding dependence on the full tensor-product grid size and replacing dense linear algebra in parameter space by a substantially smaller structured problem. Within our framework, we investigate canonical polyadic and tensor-train parametrizations and derive their full algebraic characterization relevant to the Gauss--Newton method, including the structure of the residual Jacobian, the resulting compressed system, and its effective compressed dimension. Numerical experiments on high-dimensional PDEs, including parametric problems, demonstrate the high efficiency of the proposed compressed Gauss--Newton method, which achieves substantially lower errors than tensor-compressed first-order baselines with orders of magnitude fewer iterations and only a fraction of the computing time.
Jul 2, 2026

An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as a promising route to solve partial differential equations, yet they have struggled to reach the precision of classical solvers. The obstacle is increasingly understood to be one of optimisation, owing to the severely ill-conditioned loss landscape. We present DSGNAR\textbf{DSGNAR}: Doubly-Sketched Gauss-Newton with Adaptive Ratio, a scalable second-order optimisation framework that confronts this ill-conditioning and, in doing so, obtains unprecedented accuracy and speed. DSGNAR\textbf{DSGNAR} couples a doubly-sketched Gauss-Newton model with a novel strategy that carefully controls both regularisation and step length. Across a suite of problems spanning nonlinear, chaotic, multi-scale, high-dimensional, and Navier-Stokes, the framework greatly improves on the state of the art: able to attain relative ℓ2\ell_2 errors as low as 3×10−163\times10^{-16} in double precision, improve contemporary results by five orders of magnitude on the canonical Burgers' equation, and as much as eight orders on a high-dimensional Poisson problem, while remaining markedly faster. We further show that, in single precision, solutions at the limit of round-off error can be obtained very quickly: Burgers' equation to ℓ2rel=4.75×10−7\ell_2^{\text{rel}} = 4.75 \times 10^{-7} in under ten seconds. The framework is also robust to the choice of architecture, arithmetic precision, and initial hyperparameters. The code is available at https://www.github.com/wephy/physics-informed-neural-networks
Aug 29, 2026

SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning

Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve local conditioning but require expensive optimizer state, while Kronecker-factored methods such as SOAP scale to larger networks but rely on periodic basis updates. We introduce \method, which augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry and an adaptive basis update followed by variance-state downscaling. We characterize the directional secant matching induced by the scalar correction and give a bound on variance-state mismatch across basis changes. Across eight PDE benchmarks, \method attains the lowest final residual on six, including Burgers and Boussinesq, while SOAP-family baselines perform better on Gray-Scott and Ginzburg-Landau. On Boussinesq, \method reaches a residual of 10−510^{-5} in 4.1 hours with 9.2 GB peak VRAM, while Adam does not reach this target within 14 hours. Three-seed L2L^2 and H1H^1 errors on four representative PDEs support the link between lower residuals and improved solution accuracy. These results position \method as a scalable option for stiff, high-accuracy physics-informed training, rather than a uniform replacement for existing optimizers.