Tensor-Train Compressed Separable PINNs: A Curvature-Aware Optimization Framework for Parametric PDEs in High Dimensions

Denis Korolev; Martin Eigel

math.OCSep 28, 2026

Tensor-Train Compressed Separable PINNs: A Curvature-Aware Optimization Framework for Parametric PDEs in High Dimensions

Authors: Denis Korolev, Martin Eigel

Organizations: Weierstrass Institute for Applied Analysis and Stochastics (WIAS), Anton-Wilhelm-Amo-Straße 39, 10117 Berlin, Germany

Abstract

In this work, we develop a second-order optimization framework for physics-informed neural networks (PINNs) applied to high-dimensional parametric partial differential equations (PDEs). The framework is built on the Gauss--Newton pullback metric, which provides an operator-informed notion of curvature in parameter space and connects the method to the broader family of natural gradient schemes. We show that, for coordinate-separable neural architectures and linear differential operators (or linearized operators in the nonlinear case) admitting a finite separable representation, the residual Jacobian inherits a structured separable factorization. This yields an exact compressed formulation of the Gauss--Newton step in a reduced space, without assembling the full residual Jacobian on the exponentially large tensor-product collocation grid. The dimension of the reduced space (the effective compressed dimension) is determined by the local collocation grid sizes, the separable operator structure, and the contraction pattern of the architecture, thereby avoiding dependence on the full tensor-product grid size and replacing dense linear algebra in parameter space by a substantially smaller structured problem. Within our framework, we investigate canonical polyadic and tensor-train parametrizations and derive their full algebraic characterization relevant to the Gauss--Newton method, including the structure of the residual Jacobian, the resulting compressed system, and its effective compressed dimension. Numerical experiments on high-dimensional PDEs, including parametric problems, demonstrate the high efficiency of the proposed compressed Gauss--Newton method, which achieves substantially lower errors than tensor-compressed first-order baselines with orders of magnitude fewer iterations and only a fraction of the computing time.

Figures & tables

Panel of Figure 1 — Figure 1: Continuous Gauss–Newton geometry (top row in the diagram) and its collocation realization (bottom row) obtained by collocation evaluation $\mathrm{ev}_{\mathcal{X}_{\mathcal{D}}}$ and equal-weight quadrature on the tensor grid $\mathcal{X}_{\mathcal{D}}$ of cardinality $\mathcal{N}_{{}_{\mathcal{D}}}$ .

Panel of Figure 2 — Figure 2: Functional setting of the structured PDE formulation. The PDE operator $\mathcal{A}$ is defined on $U$ , while its restriction to the densely embedded tensor-product space $H$ admits a separable representation; the diagram commutes under the embedding $H\hookrightarrow U$ .

Panel of Figure 3 — Figure 3: Structure exploited by the compressed Gauss–Newton step. The residual Jacobian admits a coordinate-wise factorization in which the local parameter derivatives are collected in the block-diagonal factor $\mathsf{C}$ , while $\Psi$ contains the remaining factor evaluations and is never assembled explicitly.

Panel of Figure 4 — Figure 4: Collocation grids for $d_{x}=d_{\xi}=1$ , $z=(x_{1},\xi_{1})$ . Left: tensor-product grid $\mathcal{X}_{\mathcal{D}}=\mathcal{X}_{1}\times\mathcal{X}_{2}$ . Right: lateral boundary grids $\mathcal{X}_{\Gamma_{1}^{\pm}}=\{z_{1}^{\pm}\}\times\mathcal{X}_{2}$ . No boundary conditions are imposed on $\partial\Xi$ (dashed).

Panel of Figure 5 — Figure 5: Compressed CP Gramian with hierarchical structure: every block is built from the univariate cross-Grams $\mathsf{H}_{m}^{(\beta,\gamma)}\in\mathbb{R}^{R\times R}$ , $m=1,\dots,d$ , through leave-one-out and leave-two-out Hadamard products.

Panel of Figure 6 — Figure 6: Tensor train diagram for $\mathsf{G}_{1}(i_{1})\cdots\mathsf{G}_{d}(i_{d})$ . Each box is a core $\mathsf{G}_{m}\in\mathbb{R}^{r_{m-1}\times n_{m}\times r_{m}}$ ; the horizontal lines are the TT bonds $\alpha_{1},\dots,\alpha_{d-1}$ , and the downward legs are the collocation indices $i_{1},\dots,i_{d}$ .

Panel of Figure 7 — Figure 7: Double-layer contraction diagram for the off-diagonal Gramian block

Panel of Figure 8 — Figure 8: Double-layer contraction diagram for the compressed residual

Figure 9: High-dimensional Poisson problem: CP and TT compressed Gauss–Newton convergence for canonical rank $R=4$ and TT rank $r=2$

Figure 10: High-dimensional Poisson problem: CP and TT compressed Gauss–Newton convergence for canonical rank $R=9$ and TT rank $r=3$ .

Figure 11: Heat equation: (a) compressed Gauss–Newton ( $500$ iterations) and (b) TT-Adam ( $50\,000$ iterations). Solid curves show the median over ten seeds, shaded regions the interquartile range between the 25th and 75th percentiles. From left to right: loss, relative $L^{2}$ error, and relative $L^{2}$ error versus wall-clock time; the first two are plotted against the iteration count.

Figure 12: Darcy problem: compressed Gauss–Newton for (a) the affine coefficient and (b) the quadratic-interaction coefficient. Solid curves show the median over ten seeds, shaded regions the interquartile range between the 25th and 75th percentiles. From left to right: loss, relative $H^{1}$ error, and relative $H^{1}$ error versus wall-clock time; the first two are plotted against the Gauss–Newton iteration count.

Figure 13: Affine coefficient: Gauss–Newton TT-PINN prediction (top), reference solution (middle) and pointwise absolute error (bottom) on the slice $x_{3}=0.5$ , for three random parameters $\xi^{(k)}\sim\operatorname{Unif}(-1,1)^{d_{\xi}}$ .

Figure 14: Quadratic-interaction coefficient: Gauss–Newton TT-PINN prediction (top), reference solution (middle) and pointwise absolute error (bottom) on the slice $x_{3}=0.35$ , for three random parameters $\xi^{(k)}\sim\operatorname{Unif}(-1,1)^{d_{\xi}}$ .

Figure 15: Darcy problem: TT-Adam baseline for (a) the affine coefficient and (b) the quadratic-interaction coefficient. Solid curves show the median over ten seeds, shaded regions the interquartile range between the 25th and 75th percentiles. From left to right: loss, relative $H^{1}$ error, and relative $H^{1}$ error versus wall-clock time; the first two are plotted against the Adam iteration count.

Panel of Figure 16 — Figure 16: Double-layer contraction diagram for an off-diagonal boundary Gramian block with $s<b<t$ .

Neural PDE Solvers

Neural PDE solvers use trainable models to approximate solutions or solution operators for partial differential equations, often with physical constraints.

See all 271 papers

Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers
Sep 17, 2026Nilo Schwencke, Roland MaierGauss-Newton Optimization Neural PDE Solvers
An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks
Jul 2, 2026Joseph Webb, Sadok Jerad, Coralia CartisGauss-Newton Optimization PDE Solving
Mathematical Institute, University of Oxford, Oxford, United Kingdom
Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks
Apr 6, 2026Anas Jnini, Elham Kiyani, Khemraj Shukla +5Physics-Informed ML Neural PDE Solvers
Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Division of Applied Mathematics, Brown University, Providence, RI, USA · Departament de Física, Universitat d’Alacant, Spain +3

Sep 17, 2026J/K move · Enter open · S save

Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers

Nilo Schwencke, Roland Maier

Gauss-Newton Optimization Neural PDE Solvers PDE Solving Second-Order Optimization Finite Element Method

Open paper AskPDF arXiv

Physics-informed neural networks and finite element methods provide two different paradigms for the numerical approximation of partial differential equations: the former are commonly trained by minimizing pointwise strong residuals, whereas the latter are naturally built from weak variational formulations and the finite-dimensional systems obtained after discretization. In this work, we introduce a common framework based on the discretization of functional Gauss--Newton problems by finite families of linear measurements. We show that, through an appropriate duality pairing, the linear measurements can be represented by test functions. The resulting Gauss--Newton system is then precisely a Petrov--Galerkin discretization of the linearized functional problem. This perspective recovers pointwise collocation and natural-gradient constructions as particular cases, while making the choice of test functions an explicit algorithmic design choice. We specialize this framework to elliptic problems, where it naturally leads to weak residual formulations and to a hybrid finite element--neural construction acting on complementary approximation spaces. Numerical experiments support the proposed framework and demonstrate the effectiveness of weak Gauss--Newton formulations and hybrid finite element--neural approximations.

Abstract

Neural PDE Solvers

Explore similar work

Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers

An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks