Tensor-Train Compressed Separable PINNs: A Curvature-Aware Optimization Framework for Parametric PDEs in High Dimensions

Denis Korolev; Martin Eigel

math.OCSep 28, 2026

Tensor-Train Compressed Separable PINNs: A Curvature-Aware Optimization Framework for Parametric PDEs in High Dimensions

Authors: Denis Korolev, Martin Eigel

Organizations: Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Anton-Wilhelm-Amo-Straße 39, 10117 Berlin, Germany

Abstract

In this work, we develop a second-order optimization framework for physics-informed neural networks (PINNs) applied to high-dimensional parametric partial differential equations (PDEs). The framework is built on the Gauss--Newton pullback metric, which provides an operator-informed notion of curvature in parameter space and connects the method to the broader family of natural gradient schemes. We show that, for coordinate-separable neural architectures and linear differential operators (or linearized operators in the nonlinear case) admitting a finite separable representation, the residual Jacobian inherits a structured separable factorization. This yields an exact compressed formulation of the Gauss--Newton step in a reduced space, without assembling the full residual Jacobian on the exponentially large tensor-product collocation grid. The dimension of the reduced space (the effective compressed dimension) is determined by the local collocation grid sizes, the separable operator structure, and the contraction pattern of the architecture, thereby avoiding dependence on the full tensor-product grid size and replacing dense linear algebra in parameter space by a substantially smaller structured problem. Within our framework, we investigate canonical polyadic and tensor-train parametrizations and derive their full algebraic characterization relevant to the Gauss--Newton method, including the structure of the residual Jacobian, the resulting compressed system, and its effective compressed dimension. Numerical experiments on high-dimensional PDEs, including parametric problems, demonstrate the high efficiency of the proposed compressed Gauss--Newton method, which achieves substantially lower errors than tensor-compressed first-order baselines with orders of magnitude fewer iterations and only a fraction of the computing time.

Figures & tables

Panel of Figure 1 — Figure 1: Continuous Gauss–Newton geometry (top row in the diagram) and its collocation realization (bottom row) obtained by collocation evaluation $\mathrm{ev}_{\mathcal{X}_{\mathcal{D}}}$ and equal-weight quadrature on the tensor grid $\mathcal{X}_{\mathcal{D}}$ of cardinality $\mathcal{N}_{{}_{\mathcal{D}}}$ .

Panel of Figure 2 — Figure 2: Functional setting of the structured PDE formulation. The PDE operator $\mathcal{A}$ is defined on $U$ , while its restriction to the densely embedded tensor-product space $H$ admits a separable representation; the diagram commutes under the embedding $H\hookrightarrow U$ .

Panel of Figure 3 — Figure 3: Structure exploited by the compressed Gauss–Newton step. The residual Jacobian admits a coordinate-wise factorization in which the local parameter derivatives are collected in the block-diagonal factor $\mathsf{C}$ , while $\Psi$ contains the remaining factor evaluations and is never assembled explicitly.

Panel of Figure 4 — Figure 4: Collocation grids for $d_{x}=d_{\xi}=1$ , $z=(x_{1},\xi_{1})$ . Left: tensor-product grid $\mathcal{X}_{\mathcal{D}}=\mathcal{X}_{1}\times\mathcal{X}_{2}$ . Right: lateral boundary grids $\mathcal{X}_{\Gamma_{1}^{\pm}}=\{z_{1}^{\pm}\}\times\mathcal{X}_{2}$ . No boundary conditions are imposed on $\partial\Xi$ (dashed).

Panel of Figure 5 — Figure 5: Compressed CP Gramian with hierarchical structure: every block is built from the univariate cross-Grams $\mathsf{H}_{m}^{(\beta,\gamma)}\in\mathbb{R}^{R\times R}$ , $m=1,\dots,d$ , through leave-one-out and leave-two-out Hadamard products.

Panel of Figure 6 — Figure 6: Tensor train diagram for $\mathsf{G}_{1}(i_{1})\cdots\mathsf{G}_{d}(i_{d})$ . Each box is a core $\mathsf{G}_{m}\in\mathbb{R}^{r_{m-1}\times n_{m}\times r_{m}}$ ; the horizontal lines are the TT bonds $\alpha_{1},\dots,\alpha_{d-1}$ , and the downward legs are the collocation indices $i_{1},\dots,i_{d}$ .

Panel of Figure 7 — Figure 7: Double-layer contraction diagram for the off-diagonal Gramian block

Panel of Figure 8 — Figure 8: Double-layer contraction diagram for the compressed residual

Figure 9: High-dimensional Poisson problem: CP and TT compressed Gauss–Newton convergence for canonical rank $R=4$ and TT rank $r=2$

Figure 10: High-dimensional Poisson problem: CP and TT compressed Gauss–Newton convergence for canonical rank $R=9$ and TT rank $r=3$ .

Figure 11: Heat equation: (a) compressed Gauss–Newton ( $500$ iterations) and (b) TT-Adam ( $50\,000$ iterations). Solid curves show the median over ten seeds, shaded regions the interquartile range between the 25th and 75th percentiles. From left to right: loss, relative $L^{2}$ error, and relative $L^{2}$ error versus wall-clock time; the first two are plotted against the iteration count.

Figure 12: Darcy problem: compressed Gauss–Newton for (a) the affine coefficient and (b) the quadratic-interaction coefficient. Solid curves show the median over ten seeds, shaded regions the interquartile range between the 25th and 75th percentiles. From left to right: loss, relative $H^{1}$ error, and relative $H^{1}$ error versus wall-clock time; the first two are plotted against the Gauss–Newton iteration count.

Figure 13: Affine coefficient: Gauss–Newton TT-PINN prediction (top), reference solution (middle) and pointwise absolute error (bottom) on the slice $x_{3}=0.5$ , for three random parameters $\xi^{(k)}\sim\operatorname{Unif}(-1,1)^{d_{\xi}}$ .

Figure 14: Quadratic-interaction coefficient: Gauss–Newton TT-PINN prediction (top), reference solution (middle) and pointwise absolute error (bottom) on the slice $x_{3}=0.35$ , for three random parameters $\xi^{(k)}\sim\operatorname{Unif}(-1,1)^{d_{\xi}}$ .

Figure 15: Darcy problem: TT-Adam baseline for (a) the affine coefficient and (b) the quadratic-interaction coefficient. Solid curves show the median over ten seeds, shaded regions the interquartile range between the 25th and 75th percentiles. From left to right: loss, relative $H^{1}$ error, and relative $H^{1}$ error versus wall-clock time; the first two are plotted against the Adam iteration count.

Panel of Figure 16 — Figure 16: Double-layer contraction diagram for an off-diagonal boundary Gramian block with $s<b<t$ .

Neural PDE Solvers

Neural PDE solvers use trainable models to approximate solutions or solution operators for partial differential equations, often with physical constraints.

See all 271 papers

Sep 17, 2026

Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers

Physics-informed neural networks and finite element methods provide two different paradigms for the numerical approximation of partial differential equations: the former are commonly trained by minimizing pointwise strong residuals, whereas the latter are naturally built from weak variational formulations and the finite-dimensional systems obtained after discretization. In this work, we introduce a common framework based on the discretization of functional Gauss--Newton problems by finite families of linear measurements. We show that, through an appropriate duality pairing, the linear measurements can be represented by test functions. The resulting Gauss--Newton system is then precisely a Petrov--Galerkin discretization of the linearized functional problem. This perspective recovers pointwise collocation and natural-gradient constructions as particular cases, while making the choice of test functions an explicit algorithmic design choice. We specialize this framework to elliptic problems, where it naturally leads to weak residual formulations and to a hybrid finite element--neural construction acting on complementary approximation spaces. Numerical experiments support the proposed framework and demonstrate the effectiveness of weak Gauss--Newton formulations and hybrid finite element--neural approximations.

Nilo Schwencke, Roland Maier

Gauss-Newton Optimization Neural PDE Solvers PDE Solving Second-Order Optimization Finite Element Method

Ask PDF

Jul 2, 2026

An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as a promising route to solve partial differential equations, yet they have struggled to reach the precision of classical solvers. The obstacle is increasingly understood to be one of optimisation, owing to the severely ill-conditioned loss landscape. We present

\textbf{DSGNAR}

: Doubly-Sketched Gauss-Newton with Adaptive Ratio, a scalable second-order optimisation framework that confronts this ill-conditioning and, in doing so, obtains unprecedented accuracy and speed.

\textbf{DSGNAR}

couples a doubly-sketched Gauss-Newton model with a novel strategy that carefully controls both regularisation and step length. Across a suite of problems spanning nonlinear, chaotic, multi-scale, high-dimensional, and Navier-Stokes, the framework greatly improves on the state of the art: able to attain relative

\ell_2

errors as low as

3\times10^{-16}

in double precision, improve contemporary results by five orders of magnitude on the canonical Burgers' equation, and as much as eight orders on a high-dimensional Poisson problem, while remaining markedly faster. We further show that, in single precision, solutions at the limit of round-off error can be obtained very quickly: Burgers' equation to

\ell_2^{\text{rel}} = 4.75 \times 10^{-7}

in under ten seconds. The framework is also robust to the choice of architecture, arithmetic precision, and initial hyperparameters. The code is available at https://www.github.com/wephy/physics-informed-neural-networks

Joseph Webb, Sadok Jerad, Coralia Cartis

Mathematical Institute, University of Oxford, Oxford, United Kingdom

Gauss-Newton Optimization PDE Solving Second-Order Optimization Physics-Informed ML

Ask PDF

Apr 6, 2026

Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks

Efficient and robust optimization is essential for neural networks, enabling scientific machine learning models to converge rapidly to very high accuracy -- faithfully capturing complex physical behavior governed by differential equations. In this work, we present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs) for challenging partial (PDEs) and ordinary differential equations (ODEs). Specifically, we provide efficient implementations of the Natural Gradient (NG) optimizer, Self-Scaling BFGS and Broyden optimizers, and demonstrate their performance on problems including the Helmholtz equation, Stokes flow, inviscid Burgers equation, Euler equations for high-speed flows, and stiff ODEs arising in pharmacokinetics and pharmacodynamics. Beyond optimizer development, we also propose new PINN-based methods for solving the inviscid Burgers and Euler equations, and compare the resulting solutions against high-order numerical methods to provide a rigorous and fair assessment. Finally, we address the challenge of scaling these quasi-Newton optimizers for batched training, enabling efficient and scalable solutions for large data-driven problems.

Anas Jnini, Elham Kiyani, Khemraj Shukla +5

Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Division of Applied Mathematics, Brown University, Providence, RI, USA · Departament de Física, Universitat d’Alacant, Spain +3

Physics-Informed ML Neural PDE Solvers Quasi-Newton Methods Natural Gradient Methods

Ask PDF

Abstract

Neural PDE Solvers

Explore similar work

Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers

An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks