Preconditioned Physics-Informed Neural Operator Training

Shizheng Wen; Siddhartha Mishra; Marius Zeinhofer

cs.LGSep 28, 2026

Preconditioned Physics-Informed Neural Operator Training

Authors: Shizheng Wen, Siddhartha Mishra, Marius Zeinhofer

Organizations: ETH Zürich

Abstract

Neural operators are typically trained in a supervised fashion, which requires a dataset to be generated with a classical solver. Training them physics-informed, i.e., purely from the governing equations, removes this large offline cost and allows fresh samples to be drawn at every optimization step, but has so far been limited to simplified problems and trails supervised training in accuracy. The obstacle is the ill-conditioning of physics-informed losses, which differential operators induce and which worsens as the discretization is refined. We therefore propose a preconditioned residual loss function and show mesh-independent conditioning for elliptic problems and greatly improved conditioning for saddle point problems. Realized through geometric and algebraic multigrid, the construction applies to linear and nonlinear equations, steady or time-dependent, on structured and unstructured meshes, is agnostic to the neural operator architecture, and adds no cost at inference. On the Poisson, Allen-Cahn and stationary Stokes equations, the resulting label-free training matches supervised training and is four to twenty-five times more accurate than previous physics-informed operator learning methods.

Figures & tables

Figure 1: Ill-conditioning in physics-informed learning: The figure compares Adam minimization of a linear model with and without preconditioning for a physics-informed loss originating from a Poisson problem on a $65^{2}$ grid. The left and centre report error in each eigenmode of $A$ along the iteration, with the smoothest mode at the bottom. The right figure compares the relative $L^{2}$ error during training.

PDE	loss	physics-	derivatives	rel. test $L^{2}$ [%]		time
PDE	loss	informed	derivatives	$u$	$p$	[s]
Poisson	$L_{\textup{PLS}}$	yes	FEM	$\mathbf{5.1\pm 0.8}$	—	1.2
	$L_{\textup{data}}$	no	—	$4.7\pm 0.8$	—	1.1
	PINO	yes	FD	$33.1\pm 1.1$	—	1.1
	PI-DeepONet	yes	AD	$21.4\pm 4.7$	—	3.1
Allen–Cahn	$L_{\textup{PLS}}$	yes	FEM	$\mathbf{3.2\pm 0.3}$	—	46

Table 1: We summarize our main results across the three PDE instances tested: Poisson, Allen–Cahn and Stokes equation. We report mean relative

L^{2}

errors on the test set in percent over three seeds with half-range. For Stokes we report velocity and pressure error separately. Poisson and Allen–Cahn use a uniform grid with an FNO; for Stokes we employ an unstructured grid on a domain with a circular hole and use the GAOT operator. The last column is the wall-clock time of one training epoch on a single RTX 4090. Bold marks the most accurate physics-informed approach;

L_{\textup{data}}

requires labelled solutions and is included as the supervised reference.

Figure 2: Preconditioned least-squares training for the Poisson equation. Left: validation relative

L^{2}

vs. epoch as the preconditioner

P_{t}

is swept from

I

A^{-1}

, together with the geometric multigrid preconditioner Right: test error, at the best-validation checkpoint, of the

P_{t}

preconditioned sweeps against the condition number

\kappa(H_{t})

of the preconditioned loss Hessian; the dashed line is the geometric multigrid preconditioner, which has no finite

\kappa(H_{t})

Figure 3: Test error vs. training set size at a fixed optimization budget for finite and infinite datasets.

Panel of Figure 4 — Figure 4: Allen–Cahn. Left: validation space-time relative $L^{2}$ (pooled over the validation set) vs. epoch; mean over three seeds with min–max band, smoothed by a centered 15-epoch moving average. Right: test relative $L^{2}$ per time step of a free roll-out to $T=1$ , mean over test samples of the per-sample error, for the best seed of each loss (selected on validation); PI-DeepONet, whose error stays between $0.7$ and $1$ , is omitted. The dashed line marks the training horizon $t=0.1$ .

Figure 5: Stokes flow past an obstacle. Left: velocity magnitude of one reference solution to illustrate the domain, triangulation and zero Dirichlet (no-slip) boundary conditions. Right: relative

L^{2}

error of the velocity on the validation set vs. epoch. We report the mean over three seeds and shade the min–max band. Preconditioned training separates from supervised training after roughly a hundred epochs and stays below it; the unpreconditioned control

L_{\textup{LS}}

fails to converge and stagnates at around

30\%

error.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Test error of the Poisson problem of Section 3.1 , trained on the

65\times 65

grid and evaluated on different resolutions. Mean over three seeds with min–max band. Both losses transfer in the same way across different grids.

Figure 7: Samples from the Poisson data set, drawn from equation 20 on a $65^{2}$ grid: source and solution at $K=4$ (left pair) and at $K=10$ (right pair). Note the differing magnitudes of $u$ .

Component	Value
Fourier modes.	$16\times 16$
Hidden channels (width)	$64$
Fourier layers	$5$
Lifting / projection channels	$128$ / $128$
Positional embedding	grid coordinates, appended as channels
Input / output channels	$1$ / $1$

Appendix

Table 2: FNO hyperparameters used for all Poisson experiments.

Setting	Value
Optimizer	Adam, $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\varepsilon=10^{-8}$
Weight decay	$0$
Schedule	cosine, annealed from $\eta$ to $\eta/10$ over all epochs
Learning rate $\eta$	$3\cdot 10^{-4}$ for $L_{\textup{data}}$ and $L_{\textup{PLS}}$
	$3\cdot 10^{-3}$ for $L_{\textup{LS}}$ and PINO
	$10^{-4}$ for PI-DeepONet, with boundary penalty weight $\lambda_{\mathrm{bc}}=0.03$

Appendix

Table 3: Optimization hyperparameters for the Poisson experiments. Shared across losses except the learning rate, which is swept per loss as described above.

Figure 8: Roll-out of the Allen–Cahn time-stepping operator, from the initial condition to $T=1$ past the training horizon $t=0.1$ marked by the star. The different approaches are in the rows. Shown is one test sample. Between $t=0.01$ and $t=0.1$ the phases separate and the interfaces form, after which the solution only coarsens. All physics-informed arms except PI-DeepONet track the reference far beyond the training horizon, and the error of $L_{\textup{PLS}}$ decreases along the roll-out, to $0.8\%$ at $T=1$ ; PI-DeepONet never separates the phases.

loss	FNO	GAOT
$L_{\textup{data}}$ (supervised)	$0.59\pm 0.06$	$0.55\pm 0.05$
$L_{\textup{PLS}}$ (preconditioned)	$0.66\pm 0.06$	$0.61\pm 0.07$
$L_{\textup{LS}}$ (bare residual)	$29.13\pm 2.22$	$37.57\pm 2.26$

Appendix

Table 4: The ordering of the losses does not depend on the backbone. Structured Poisson,

64^{2}

K=4

; test relative

L^{2}

in percent, mean

\pm

half-range over seeds

42/43/44

. Learning rate tuning picked

10^{-3}

for both architectures.

Setting	Value
Learning rate $\eta$	$10^{-3}$ for $L_{\textup{PLS}}$ , with Schur weight $\omega_{S}=16$
	$10^{-3}$ for $L_{\textup{data}}$ and $L_{\textup{LS}}$
	$10^{-4}$ for PI-DeepONet, with $w=100$ and $\lambda_{\mathrm{bc}}=0.1$
Collocation (PI-DeepONet)	all $3714$ interior nodes
Model selection	lowest validation mean of the velocity and pressure relative $L^{2}$
Preconditioner	block; velocity half one algebraic V $(2,2)$ cycle

Appendix

Table 5: Hyperparameters for the Stokes experiment; everything not listed is as in Table 3 .

Figure 9: Rows, top to bottom: velocity magnitude; pointwise velocity error; pressure; pointwise pressure error. The leftmost column is the reference velocity, the body force that produced it, and the reference pressure.

L_{\textup{PLS}}

and

L_{\textup{data}}

are visually indistinguishable from the reference in both fields; the unpreconditioned control has lost amplitude in the recirculation left of the obstacle and carries its pressure error in a ring around it; PI-DeepONet captures only the coarsest structure of either field.

	$L_{\textup{LS}}$	$L_{\textup{PLS}}$ , by Schur weight $\omega_{S}$
	no $\mathcal{P}$	$0.5$	$4$	$16$	$32$	$64$	$256$
velocity [%]	$30.51$	$2.58$	$1.63$	$\mathbf{1.69}$	$3.35$	$4.25$	$22.94$
pressure [%]	$10.22$	$1.25$	$0.95$	$\mathbf{0.79}$	$1.40$	$1.62$	$3.93$
$\kappa$	$1.2\cdot 10^{11}$	$3.7\cdot 10^{6}$	$1.0\cdot 10^{6}$	$1.9\cdot 10^{6}$	$3.2\cdot 10^{6}$	$5.8\cdot 10^{6}$	$2.1\cdot 10^{7}$

Appendix

Table 6: Preconditioner ablation on the unstructured Stokes problem: test relative finite element

L^{2}

error, seed

42

, everything else as in Table 1 . The bold column is the selected value; selection is on the validation mean of the two fields, which is why it is

\omega_{S}=16

and not the marginally lower velocity error at

\omega_{S}=4

. The last row is the condition number of the loss Hessian

\kappa(A^{2})

for

L_{\textup{LS}}

and

\kappa(APA)

for

L_{\textup{PLS}}

. It is minimized at

\omega_{S}=4

and follows the error across the sweep.

Explore similar work

On the training of physics-informed neural operators for solving parametric partial differential equations
Jun 4, 2026Nanxi Chen, Chuanjie Cui, Airong Chen +2Neural Network Optimization Gradient Interference
College of Civil Engineering, Tongji University, Shanghai 200092 · Department of Engineering Science, University of Oxford, Oxford OX1 3PJ · Institute for Foundations of Data Science, Yale University, New Haven, CT 06520
Learning Physical Operators using Neural Operators
Feb 26, 2026Vignesh Gopakumar, Ander Gray, Dan Giles +5PDE Surrogate Modeling Physics-Informed Neural Operators
UCL Centre for Artificial Intelligence · UK Atomic Energy Authority · LIX, CNRS, ´Ecole Polytechnique +2
Operator Learning Using Weak Supervision from Walk-on-Spheres
Mar 1, 2026Hrishikesh Viswanath, Hong Chul Nam, Xi Deng +3PDE Operator Learning Physics-Informed ML
Purdue University, USA · ETH Zurich, Switzerland · California Institute of Technology, USA +1

Jun 4, 2026J/K move · Enter open · S save

On the training of physics-informed neural operators for solving parametric partial differential equations

Nanxi Chen, Chuanjie Cui, Airong Chen, Sifan Wang+1

College of Civil Engineering, Tongji University, Shanghai 200092 · Department of Engineering Science, University of Oxford, Oxford OX1 3PJ · Institute for Foundations of Data Science, Yale University, New Haven, CT 06520

Neural Network Optimization Gradient Interference PDE Operator Learning Physics-Informed ML Physics-Informed Neural Operators

Open paper AskPDF arXiv

Physics-informed neural operators (PINOs) aim to learn solution operators for partial differential equations by using the governing physics as supervision, rather than relying solely on paired input-output simulation data. By incorporating physical constraints into the training objective, PINOs combine the cross-instance generalization of neural operators with the data efficiency of physics-informed learning. Despite this promise, how to train PINOs efficiently and robustly remains less well-understood than the training of either data-driven neural operators or physics-informed neural networks (PINNs). To bridge this gap, we examine key components of the PINO training pipeline, including architecture design, optimizer choice, loss balancing, and collocation-point sampling strategy. We study three representative operator backbones, Deep Operator Network (DeepONet), Fourier Neural Operator (FNO), and Continuous Vision Transformer (CViT), across five diverse parametric PDE systems. Our results show that CViT provides consistently strong and stable performance across the considered benchmarks. Beyond architecture, we find that several optimization pathologies previously identified in PINN training naturally arise in PINOs, including gradient conflicts and causal violation. We also find that mitigation algorithms developed for PINNs remain effective in the PINO setting. We further compare physics-informed and data-driven training under different data regimes, revealing that a carefully designed physics-informed training pipeline can match, and in some cases, outperform purely data-driven neural operators. Taken together, these findings provide a systematic empirical understanding of the optimization challenges in PINO training and inform a practical pipeline for efficient and robust physics-informed operator learning. Code and data are available at https://github.com/NanxiiChen/PI-CViT.

Abstract

Explore similar work

On the training of physics-informed neural operators for solving parametric partial differential equations

Learning Physical Operators using Neural Operators

Operator Learning Using Weak Supervision from Walk-on-Spheres