Block-Sparse Gauss-Newton Optimization of Finite Basis Physics-Informed Neural Networks: Sparse Automatic Differentiation Approach
Organizations: Weierstrass Institute for Applied Analysis and Stochastics, Anton-Wilhelm-Amo-Str. 39, Berlin, 10117, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Franklinstr. 28/29, Berlin, 10587, Germany · Machine Learning Group, Technical University of Berlin, Franklinstr. 28/29, Berlin, 10587, Germany · KAIST InnoCORE PRISM-AI Center, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea
Abstract
In this work, we develop an efficient Gauss-Newton optimization approach for finite basis physics-informed neural networks (FBPINNs). The key ingredient is the overlapping domain decomposition architecture of FBPINNs, which localizes the influence of each subnetwork and induces block sparsity in both the residual Jacobian and the corresponding Gauss-Newton matrix. We characterize this structure and exploit it through sparse automatic differentiation, using graph coloring to partition structurally orthogonal Jacobian columns or rows. This reduces differentiation costs and memory requirements while preserving all structurally nonzero derivative information. The resulting compressed Jacobian representations allow us to assemble the Gauss-Newton system directly in block-sparse form, enabling the use of sparse direct solvers and preconditioned iterative methods. In this way, the otherwise expensive second-order optimization problem is reduced to scalable sparse linear algebra. We demonstrate the approach on multiscale PDEs, where accurate approximation requires rich neural network parametrizations and large numbers of collocation points. Numerical experiments show that our block-sparse Gauss-Newton method attains high accuracy and substantially accelerates optimization relative to first-order baselines while remaining computationally tractable for large problem sizes.
Figures & tables
| Dimension | Partition | Overlap ratio | Stored GN blocks | 1-layer net | 2-layer net | |
|---|---|---|---|---|---|---|
| GB | GB | |||||
| GB | GB | |||||
| GB | GB | |||||
| GB | GB | |||||
| GB | TB | |||||
| GB | TB |
| Overlap ratio | |||||
| Method | Quantity | ||||
| Gauss–Newton (direct) | Rel. error | ||||
| Norm. test loss | |||||
| Time [s] | |||||
| PCG (no preconditioner) | Rel. error | ||||
| Common stages | Linear solve | ||||||
|---|---|---|---|---|---|---|---|
| Sparse AD | Assembly | Direct | None | Block | Cross | King | |
| 1.860 | 0.018 | 0.711 | 0.522 | 0.685 | 2.233 | 4.787 | |
| 0.583 | 0.012 | 0.218 | 0.114 | 0.181 | 0.687 | 1.271 | |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Gauss-Newton Optimization
Gauss–Newton optimization uses residual Jacobians to build tractable curvature-aware updates for nonlinear least-squares problems and machine learning.
See all 15 papers