Preconditioned Physics-Informed Neural Operator Training

Shizheng Wen; Siddhartha Mishra; Marius Zeinhofer

cs.LGSep 28, 2026

Preconditioned Physics-Informed Neural Operator Training

Authors: Shizheng Wen, Siddhartha Mishra, Marius Zeinhofer

Organizations: ETH Zürich

Abstract

Neural operators are typically trained in a supervised fashion, which requires a dataset to be generated with a classical solver. Training them physics-informed, i.e., purely from the governing equations, removes this large offline cost and allows fresh samples to be drawn at every optimization step, but has so far been limited to simplified problems and trails supervised training in accuracy. The obstacle is the ill-conditioning of physics-informed losses, which differential operators induce and which worsens as the discretization is refined. We therefore propose a preconditioned residual loss function and show mesh-independent conditioning for elliptic problems and greatly improved conditioning for saddle point problems. Realized through geometric and algebraic multigrid, the construction applies to linear and nonlinear equations, steady or time-dependent, on structured and unstructured meshes, is agnostic to the neural operator architecture, and adds no cost at inference. On the Poisson, Allen-Cahn and stationary Stokes equations, the resulting label-free training matches supervised training and is four to twenty-five times more accurate than previous physics-informed operator learning methods.

Figures & tables

Figure 1: Ill-conditioning in physics-informed learning: The figure compares Adam minimization of a linear model with and without preconditioning for a physics-informed loss originating from a Poisson problem on a $65^{2}$ grid. The left and centre report error in each eigenmode of $A$ along the iteration, with the smoothest mode at the bottom. The right figure compares the relative $L^{2}$ error during training.

PDE	loss	physics-	derivatives	rel. test $L^{2}$ [%]		time
PDE	loss	informed	derivatives	$u$	$p$	[s]
Poisson	$L_{\textup{PLS}}$	yes	FEM	$\mathbf{5.1\pm 0.8}$	—	1.2
	$L_{\textup{data}}$	no	—	$4.7\pm 0.8$	—	1.1
	PINO	yes	FD	$33.1\pm 1.1$	—	1.1
	PI-DeepONet	yes	AD	$21.4\pm 4.7$	—	3.1
Allen–Cahn	$L_{\textup{PLS}}$	yes	FEM	$\mathbf{3.2\pm 0.3}$	—	46

Table 1: We summarize our main results across the three PDE instances tested: Poisson, Allen–Cahn and Stokes equation. We report mean relative

L^{2}

errors on the test set in percent over three seeds with half-range. For Stokes we report velocity and pressure error separately. Poisson and Allen–Cahn use a uniform grid with an FNO; for Stokes we employ an unstructured grid on a domain with a circular hole and use the GAOT operator. The last column is the wall-clock time of one training epoch on a single RTX 4090. Bold marks the most accurate physics-informed approach;

L_{\textup{data}}

requires labelled solutions and is included as the supervised reference.

Figure 2: Preconditioned least-squares training for the Poisson equation. Left: validation relative

L^{2}

vs. epoch as the preconditioner

P_{t}

is swept from

I

A^{-1}

, together with the geometric multigrid preconditioner Right: test error, at the best-validation checkpoint, of the

P_{t}

preconditioned sweeps against the condition number

\kappa(H_{t})

of the preconditioned loss Hessian; the dashed line is the geometric multigrid preconditioner, which has no finite

\kappa(H_{t})

Figure 3: Test error vs. training set size at a fixed optimization budget for finite and infinite datasets.

Panel of Figure 4 — Figure 4: Allen–Cahn. Left: validation space-time relative $L^{2}$ (pooled over the validation set) vs. epoch; mean over three seeds with min–max band, smoothed by a centered 15-epoch moving average. Right: test relative $L^{2}$ per time step of a free roll-out to $T=1$ , mean over test samples of the per-sample error, for the best seed of each loss (selected on validation); PI-DeepONet, whose error stays between $0.7$ and $1$ , is omitted. The dashed line marks the training horizon $t=0.1$ .

Figure 5: Stokes flow past an obstacle. Left: velocity magnitude of one reference solution to illustrate the domain, triangulation and zero Dirichlet (no-slip) boundary conditions. Right: relative

L^{2}

error of the velocity on the validation set vs. epoch. We report the mean over three seeds and shade the min–max band. Preconditioned training separates from supervised training after roughly a hundred epochs and stays below it; the unpreconditioned control

L_{\textup{LS}}

fails to converge and stagnates at around

30\%

error.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Test error of the Poisson problem of Section 3.1 , trained on the

65\times 65

grid and evaluated on different resolutions. Mean over three seeds with min–max band. Both losses transfer in the same way across different grids.

Figure 7: Samples from the Poisson data set, drawn from equation 20 on a $65^{2}$ grid: source and solution at $K=4$ (left pair) and at $K=10$ (right pair). Note the differing magnitudes of $u$ .

Component	Value
Fourier modes.	$16\times 16$
Hidden channels (width)	$64$
Fourier layers	$5$
Lifting / projection channels	$128$ / $128$
Positional embedding	grid coordinates, appended as channels
Input / output channels	$1$ / $1$

Appendix

Table 2: FNO hyperparameters used for all Poisson experiments.

Setting	Value
Optimizer	Adam, $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\varepsilon=10^{-8}$
Weight decay	$0$
Schedule	cosine, annealed from $\eta$ to $\eta/10$ over all epochs
Learning rate $\eta$	$3\cdot 10^{-4}$ for $L_{\textup{data}}$ and $L_{\textup{PLS}}$
	$3\cdot 10^{-3}$ for $L_{\textup{LS}}$ and PINO
	$10^{-4}$ for PI-DeepONet, with boundary penalty weight $\lambda_{\mathrm{bc}}=0.03$

Appendix

Table 3: Optimization hyperparameters for the Poisson experiments. Shared across losses except the learning rate, which is swept per loss as described above.

Figure 8: Roll-out of the Allen–Cahn time-stepping operator, from the initial condition to $T=1$ past the training horizon $t=0.1$ marked by the star. The different approaches are in the rows. Shown is one test sample. Between $t=0.01$ and $t=0.1$ the phases separate and the interfaces form, after which the solution only coarsens. All physics-informed arms except PI-DeepONet track the reference far beyond the training horizon, and the error of $L_{\textup{PLS}}$ decreases along the roll-out, to $0.8\%$ at $T=1$ ; PI-DeepONet never separates the phases.

loss	FNO	GAOT
$L_{\textup{data}}$ (supervised)	$0.59\pm 0.06$	$0.55\pm 0.05$
$L_{\textup{PLS}}$ (preconditioned)	$0.66\pm 0.06$	$0.61\pm 0.07$
$L_{\textup{LS}}$ (bare residual)	$29.13\pm 2.22$	$37.57\pm 2.26$

Appendix

Table 4: The ordering of the losses does not depend on the backbone. Structured Poisson,

64^{2}

K=4

; test relative

L^{2}

in percent, mean

\pm

half-range over seeds

42/43/44

. Learning rate tuning picked

10^{-3}

for both architectures.

Setting	Value
Learning rate $\eta$	$10^{-3}$ for $L_{\textup{PLS}}$ , with Schur weight $\omega_{S}=16$
	$10^{-3}$ for $L_{\textup{data}}$ and $L_{\textup{LS}}$
	$10^{-4}$ for PI-DeepONet, with $w=100$ and $\lambda_{\mathrm{bc}}=0.1$
Collocation (PI-DeepONet)	all $3714$ interior nodes
Model selection	lowest validation mean of the velocity and pressure relative $L^{2}$
Preconditioner	block; velocity half one algebraic V $(2,2)$ cycle

Appendix

Table 5: Hyperparameters for the Stokes experiment; everything not listed is as in Table 3 .

Figure 9: Rows, top to bottom: velocity magnitude; pointwise velocity error; pressure; pointwise pressure error. The leftmost column is the reference velocity, the body force that produced it, and the reference pressure.

L_{\textup{PLS}}

and

L_{\textup{data}}

are visually indistinguishable from the reference in both fields; the unpreconditioned control has lost amplitude in the recirculation left of the obstacle and carries its pressure error in a ring around it; PI-DeepONet captures only the coarsest structure of either field.

	$L_{\textup{LS}}$	$L_{\textup{PLS}}$ , by Schur weight $\omega_{S}$
	no $\mathcal{P}$	$0.5$	$4$	$16$	$32$	$64$	$256$
velocity [%]	$30.51$	$2.58$	$1.63$	$\mathbf{1.69}$	$3.35$	$4.25$	$22.94$
pressure [%]	$10.22$	$1.25$	$0.95$	$\mathbf{0.79}$	$1.40$	$1.62$	$3.93$
$\kappa$	$1.2\cdot 10^{11}$	$3.7\cdot 10^{6}$	$1.0\cdot 10^{6}$	$1.9\cdot 10^{6}$	$3.2\cdot 10^{6}$	$5.8\cdot 10^{6}$	$2.1\cdot 10^{7}$

Appendix

Table 6: Preconditioner ablation on the unstructured Stokes problem: test relative finite element

L^{2}

error, seed

42

, everything else as in Table 1 . The bold column is the selected value; selection is on the validation mean of the two fields, which is why it is

\omega_{S}=16

and not the marginally lower velocity error at

\omega_{S}=4

. The last row is the condition number of the loss Hessian

\kappa(A^{2})

for

L_{\textup{LS}}

and

\kappa(APA)

for

L_{\textup{PLS}}

. It is minimized at

\omega_{S}=4

and follows the error across the sweep.

Explore similar work

Jun 4, 2026

On the training of physics-informed neural operators for solving parametric partial differential equations

Physics-informed neural operators (PINOs) aim to learn solution operators for partial differential equations by using the governing physics as supervision, rather than relying solely on paired input-output simulation data. By incorporating physical constraints into the training objective, PINOs combine the cross-instance generalization of neural operators with the data efficiency of physics-informed learning. Despite this promise, how to train PINOs efficiently and robustly remains less well-understood than the training of either data-driven neural operators or physics-informed neural networks (PINNs). To bridge this gap, we examine key components of the PINO training pipeline, including architecture design, optimizer choice, loss balancing, and collocation-point sampling strategy. We study three representative operator backbones, Deep Operator Network (DeepONet), Fourier Neural Operator (FNO), and Continuous Vision Transformer (CViT), across five diverse parametric PDE systems. Our results show that CViT provides consistently strong and stable performance across the considered benchmarks. Beyond architecture, we find that several optimization pathologies previously identified in PINN training naturally arise in PINOs, including gradient conflicts and causal violation. We also find that mitigation algorithms developed for PINNs remain effective in the PINO setting. We further compare physics-informed and data-driven training under different data regimes, revealing that a carefully designed physics-informed training pipeline can match, and in some cases, outperform purely data-driven neural operators. Taken together, these findings provide a systematic empirical understanding of the optimization challenges in PINO training and inform a practical pipeline for efficient and robust physics-informed operator learning. Code and data are available at https://github.com/NanxiiChen/PI-CViT.

Nanxi Chen, Chuanjie Cui, Airong Chen +2

College of Civil Engineering, Tongji University, Shanghai 200092 · Department of Engineering Science, University of Oxford, Oxford OX1 3PJ · Institute for Foundations of Data Science, Yale University, New Haven, CT 06520

Neural Network Optimization Gradient Interference PDE Operator Learning Physics-Informed ML Physics-Informed Neural Operators

Ask PDF

Feb 26, 2026

Learning Physical Operators using Neural Operators

Neural operators have emerged as promising surrogate models for solving partial differential equations (PDEs), but struggle to generalise beyond training distributions and are often constrained to a fixed temporal discretisation. This work introduces a physics-informed training framework that addresses these limitations by decomposing PDEs using operator splitting methods, training separate neural operators to learn individual non-linear physical operators while approximating linear operators with fixed finite-difference convolutions. This modular mixture-of-experts architecture enables generalisation to novel physical regimes by explicitly encoding the underlying operator structure. We formulate the modelling task as a neural ordinary differential equation (ODE) where these learned operators constitute the right-hand side, enabling continuous-in-time predictions through standard ODE solvers and implicitly enforcing PDE constraints. Demonstrated on incompressible and compressible Navier--Stokes equations, our approach achieves better convergence and superior performance when generalising to unseen physics. The method remains parameter-efficient, enabling temporal extrapolation beyond training horizons, and provides interpretable components whose behaviour can be verified against known physics.

Vignesh Gopakumar, Ander Gray, Dan Giles +5

UCL Centre for Artificial Intelligence · UK Atomic Energy Authority · LIX, CNRS, ´Ecole Polytechnique +2

PDE Surrogate Modeling Physics-Informed Neural Operators Computational Fluid Dynamics Modular Neural Networks

Ask PDF

Mar 1, 2026

Operator Learning Using Weak Supervision from Walk-on-Spheres

Training neural PDE solvers is often bottlenecked by expensive data generation or unstable physics-informed neural network (PINN) involving challenging optimization landscapes due to higher-order derivatives. To tackle this issue, we propose an alternative approach using Monte Carlo approaches to estimate the solution to the PDE as a stochastic process for weak supervision during training. Leveraging the Walk-on-Spheres method, we introduce a learning scheme called Walk-on-Spheres Neural Operator (WoS-NO) which uses weak supervision from WoS to train any given neural operator. We propose to amortize the cost of Monte Carlo walks across the distribution of PDE instances using stochastic representations from the WoS algorithm to generate cheap, noisy, estimates of the PDE solution during training. This is formulated into a data-free physics-informed objective where a neural operator is trained to regress against these weak supervisions, allowing the operator to learn a generalized solution map for an entire family of PDEs. This strategy does not require expensive pre-computed datasets, avoids computing higher-order derivatives for loss functions that are memory-intensive and unstable, and demonstrates zero-shot generalization to novel PDE parameters and domains. Experiments show that for the same number of training steps, our method exhibits up to 8.75

\times

improvement in

L_2

-error compared to standard physics-informed training schemes, up to 6.31

\times

improvement in training speed, and reductions of up to 2.97

\times

in GPU memory consumption. We present the code at https://github.com/neuraloperator/WoS-NO

Hrishikesh Viswanath, Hong Chul Nam, Xi Deng +3

Purdue University, USA · ETH Zurich, Switzerland · California Institute of Technology, USA +1

PDE Operator Learning Physics-Informed ML Neural Operators Weakly Supervised Learning

Ask PDF