Performance of Neural and Polynomial Operator Surrogates

Josephine Westermann; Benno Huber; Thomas O'Leary-Roseberry; Jakob Zech

cs.LGApr 1, 2026

Performance of Neural and Polynomial Operator Surrogates

Authors: Josephine Westermann, Benno Huber, Thomas O'Leary-Roseberry, Jakob Zech

Organizations: Heidelberg University · The Ohio State University

Abstract

We consider the problem of constructing surrogate operators for parameter-to-solution maps arising from parametric partial differential equations, where repeated forward model evaluations are computationally expensive. We present a systematic empirical comparison of neural operator surrogates, including a reduced-basis neural operator trained with $L^2_μ$ and $H^1_μ$ objectives and the Fourier neural operator, against polynomial surrogate methods, specifically a reduced-basis sparse-grid surrogate and a reduced-basis tensor-train surrogate. All methods are evaluated on a linear parametric diffusion problem and a nonlinear parametric hyperelasticity problem, using input fields with algebraically decaying spectral coefficients at varying rates of decay $s$ . To enable fair comparisons, we analyze ensembles of surrogate models generated by varying hyperparameters and compare the resulting Pareto frontiers of cost versus approximation accuracy, decomposing cost into contributions from data generation, setup, and evaluation. Our results show that no single method is universally superior. Polynomial surrogates achieve substantially better data efficiency for smooth input fields ( $s \geq 2$ ), with convergence rates for the sparse-grid surrogate in agreement with theoretical predictions. For rough inputs ( $s \leq 1$ ), the Fourier neural operator displays the fastest convergence rates. Derivative-informed training consistently improves data efficiency over standard $L^2_μ$ training, providing a competitive alternative for rough inputs in the low-data regime when Jacobian information is available at reasonable cost. These findings highlight the importance of matching the surrogate methodology to the regularity of the problem as well as accuracy demands and computational constraints of the application.

Figures & tables

Panel of Figure 1 — Figure 1: Encoder-decoder architecture.

Figure 2: Input samples constructed as in ( 26 ) using a fixed coefficient vector ${\bm{c}}$ and varying $s$ values. As $s$ decreases, the correlation length becomes smaller.

Figure 3: A random parameter field $x({\bm{\xi}})$ (left) and corresponding FE solution $y({\bm{\xi}})$ of the linear elliptic PDE ( 27 ) (right).

Panel of Figure 4 — Figure 4: 2D deformation problem

Figure 5: A random parameter field $x({\bm{\xi}})$ (left) and corresponding FE solution $y({\bm{\xi}})$ of the hyperelasticity problem (right).

Figure 6:

\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{L^{2}_{\mu}-{\text{RB-NO}}})

for the diffusion problem ( 27 ) over network depth and width for different input smoothness

s

. Top row: GELU activation function; bottom row: tanh activation function. Coefficient-space dimensions fixed at

d_{\rm in}=d_{\rm out}=200

, training set size

n=10^{4}

Figure 7: $\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{H^{1}_{\mu}-{\text{RB-NO}}})$ for the diffusion problem ( 27 ) over network depth and width for different input smoothness $s$ . Coefficient-space dimensions fixed at $d_{\rm in}=d_{\rm out}=200$ , training set size $n=10^{4}$ .

Figure 8: $\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{\text{FNO}})$ for the diffusion problem ( 27 ) over the number of channels and Fourier modes for different input smoothness $s$ . Network depth $L=4$ , training set size $n=10^{4}$ .

Figure 9: $\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{\text{RB-SG}})$ for the diffusion problem ( 27 ) in dependence of the interpolation hyperparameters $a$ and $b$ as in ( 20 ) for different operator input smoothness $s$ , with the minimum of the error landscape marked by $\star$ . Fixed $d_{\rm in}=d_{\rm out}=200$ and $n=10^{4}$ .

Figure 10:

\varepsilon_{L^{2}_{\mu}}

for the diffusion problem ( 27 ) in dependence of encoder and decoder dimensions,

d_{\rm in}

and

d_{\rm out}

, for various values of

s

and

n=10^{4}

training samples.

\tilde{\mathcal{G}}_{L^{2}_{\mu}-{\text{RB-NO}}}

uses networks of depth

7

(for

s\in\{0.5,1.0\}

) or depth

9

(for

s\in\{2.0,3.0\}

) and width

200

, while

\tilde{\mathcal{G}}_{H^{1}_{\mu}-{\text{RB-NO}}}

uses networks of depth

7

and width

400

\tilde{\mathcal{G}}_{\text{RB-SG}}

is constructed using hyperparameters

a=0.5

and

b=1.2

. These hyperparameters were selected to lie within regimes of low surrogate error as shown in Figures 6 , 7 and 9 .

Figure 13:

\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{\text{RB-TT}})

for the elliptic PDE ( 27 ) over number of PDE solves

n

and TT surrogate size

N

for

d_{\rm in}=100

and

s=2.0

Figure 14:

\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{\text{RB-TT}})

for the elliptic PDE ( 27 ) over number of PDE solves

n

and TT surrogate size

N

for

d_{\rm in}=100

\theta=1.0

and

s=2.0

. ALSx performs x end-to-end iterations over the TT.

Figure 15:

\varepsilon_{L^{2}_{\mu}}(\tilde{\mathcal{G}}_{\text{RB-TT}})

for the elliptic PDE ( 27 ) over TT surrogate size

N

for

d_{\rm in}=100

and

s=2.0

Model	Parameters
$\tilde{\mathcal{G}}_{L^{2}_{\mu}-{\text{RB-NO}}}$	$d_{\rm in}=d_{\rm out}$ 50, 100, 200, 400, 800 width 200, 400, 800 depth 3, 5, 7
$\tilde{\mathcal{G}}_{H^{1}_{\mu}-{\text{RB-NO}}}$	$d_{\rm in}=d_{\rm out}$ 50, 100, 200, 400, 800 width 200, 400, 800 depth 3, 5, 7
$\tilde{\mathcal{G}}_{\text{FNO}}$	modes 2, 4, 8, 16, 32 width 8, 16, 32, 64
$\tilde{\mathcal{G}}_{\text{RB-SG}}$	$d_{\rm in}=d_{\rm out}$ 50, 100, 200, 400, 800 $a$ 0.2, 0.5, 1.2, 3.0 $b$ 0.2, 0.5, 1.2, 3.0
$\tilde{\mathcal{G}}_{\text{RB-TT}}$	$d_{\rm in}$ 50, 100, 200 ( $s\geq 1)$ 10, 15, 20, 25 ( $s=0.5)$

Table 1: Architectural parameters used throughout the experiments in Section 6.3 .

Figure 18:

\varepsilon_{L^{2}_{\mu}}

-error versus surrogate evaluation time

t_{E}

per sample for the diffusion problem ( 27 ). The top row shows results obtained when evaluating a single sample, while the bottom row shows evaluation time per sample when evaluating the surrogates on a batch of size

n_{\rm batch}=10^{4}

, for different

s

Figure 19: Evaluation cost over batch size for

s=2

and

\varepsilon_{L^{2}_{\mu}}\sim 10^{-3}

for the diffusion problem ( 27 ).

Figure 20:

\varepsilon_{L^{2}_{\mu}}

-convergence over parametric surrogate complexity

N

for different

s

for the diffusion problem ( 27 ).

Figure 21:

\varepsilon_{L^{2}_{\mu}}

-convergence over training/setup time

t_{T}

for different

s

for the diffusion problem ( 27 ).

Explore similar work

Oct 13, 2026

Operator Surrogates

This chapter provides a mathematical introduction to operator surrogates, which are finite-parametric maps approximating complex (typically nonlinear) operators between infinite-dimensional (function) spaces. A typical approach to constructing operator surrogates is the encoder-approximator-decoder paradigm, whereby infinite-dimensional inputs are mapped to a finite-dimensional latent space, processed, and then mapped back to the infinite-dimensional output space. We discuss various methods for creating efficient representation systems, including stable linear frameworks like Banach frames and data-driven techniques such as Principal Component Analysis (PCA), Active Subspaces, and nonlinear autoencoders. Subsequently, we survey key surrogate architectures, contrasting spectral methods, based on polynomial chaos expansions and related technologies, with more recent neural network approaches. We further discuss architectures that do not rely on representation systems, such as Fourier Neural Operators and transformers. We elaborate on their relationship and how they relate to other neural network operator surrogates, particularly those obtained by so-called "algorithm-unrolling". Additionally, we provide an introduction to the theoretical analysis of approximation and generalization errors. We explain how underlying low-dimensional structures, provided for example by holomorphy, are essential to overcome the so-called "curse of dimensionality": this refers to the fact that the complexity of numerical algorithms tends to increase exponentially in the problem dimensionality. We present convergence rate bounds for holomorphic operators that depend on the number of trainable parameters and the regularity of the input and output data. Throughout, we prioritize intuition over detailed technical proofs and provide precise pointers to the literature for detailed arguments.

Christoph Schwab, Josephine Westermann, Jakob Zech

SAM, ETH Zürich, Rämistrasse 101, 8092 Zürich, Switzerland · IWR, Uni Heidelberg, INF 205, 69120 Heidelberg, Germany

Surrogate Modeling Operator Learning Neural Operators Neural Network Approximation Theory

Ask PDF

Jun 16, 2026

Operator Boosting Produces Pareto-Efficient PDE Surrogates

Neural operators are widely used as surrogate solution maps for partial differential equations (PDEs), but full-size models can be costly to store, deploy, and evaluate in many-query scientific workflows. This work introduces Operator Boosting, a stagewise residual-learning framework for constructing compact neural-operator surrogates directly, rather than training a large model and compressing it afterward. Starting from the empirical mean predictor in normalized output coordinates, the method trains a sequence of tiny same-family neural operators on residual fields and incorporates each correction through validation-selected shrinkage. We instantiate the framework with Fourier neural operators (FNOs), DeepONets, and convolutional neural operators (CNOs), and compare boosted tiny stacks against full-size monolithic baselines across one-, two-, and three-dimensional PDE benchmarks from PDEBench, APEBench, and The Well. Across 30 dataset-architecture pairs, 21 show positive mean accuracy gains and 17 have positive confidence intervals, while all boosted stacks reduce trainable parameter count by approximately 72-95%. Best-model comparisons show empirical Pareto improvements on 7 of 10 completed PDE benchmarks, including two-dimensional Navier-Stokes, shallow-water dynamics, Darcy flow, one-dimensional transport and reaction systems, and three-dimensional compressible Navier-Stokes. These results show that Operator Boosting often improves the empirical accuracy-parameter Pareto frontier of neural PDE surrogates, while also exposing PDE- and architecture-dependent regimes where residual boosting fails to offset compression.

Lennon J. Shikhman

Georgia Institute of Technology College of Computing Atlanta, Georgia, USA

PDE Surrogate Modeling Deep Operator Networks Neural Network Compression Neural Operators

Ask PDF

Sep 21, 2026

Cost-Accuracy Trade-offs: Neural Operator vs Classical Numerical Solver

Neural operators are data-driven models that learn mappings from inputs that parameterize partial differential equations, such as spatially varying coefficients, initial conditions, forcing terms, boundary conditions, or geometries, to solution fields or quantities of interest. Once trained, they can serve as surrogates for classical numerical solvers in many-query settings that require repeated evaluations for varying inputs. We address the question of when, and then why, neural operator surrogates outperform classical numerical solvers, in terms of cost for a given accuracy. We focus on the post-training, many-query limit, in which data-acquisition and training costs are treated as fixed and fully amortized. Even in this deliberately favorable regime for neural operators, there are regimes in which classical solvers outperform the surrogate models. We compare the cost-accuracy performance of neural operator surrogates and classical numerical solvers through a reproducible benchmark study comparing neural operators with problem-matched classical solvers on representative problems in computational science and engineering, focusing on prediction error, per-query floating-point cost, and wall-clock runtime. Neural operators are most competitive at low-to-moderate accuracy requirements. Their floating-point cost advantage depends strongly on the problem structure, arising when they avoid temporal or nonlinear iterations or predict a reduced quantity of interest rather than a full solution field. Additional wall-clock speedups result from dense tensor operations that are well suited to modern hardware. As the target accuracy is tightened, achieving the required accuracy with neural operators becomes increasingly challenging, and classical solvers outperform surrogates in this regime; thus classical solvers will remain important for verification and high-accuracy computation.

Daniel Zhengyu Huang, Andrew M. Stuart

Beijing International Center for Mathematical Research, Center for Machine Learning Research, Peking University, Beijing, China · California Institute of Technology, Pasadena, CA

Neural Surrogate Modeling Neural PDE Solvers PDE Solving Neural Operators

Ask PDF