Extending Neural Operators: Robust Handling of Functions Beyond the Training Set

Blaine Quackenbush; Paul J. Atzberger

cs.LGMar 4, 2026

Extending Neural Operators: Robust Handling of Functions Beyond the Training Set

Authors: Blaine Quackenbush, Paul J. Atzberger

Organizations: Department of Mathematics, University of California Santa Barbara (UCSB). · Department of Mathematics, Department of Mechanical Engineering, University of California Santa Barbara (UCSB)

Abstract

We develop a rigorous framework for extending neural operators to handle out-of-distribution input functions. We leverage kernel approximation techniques and provide theory for characterizing the input-output function spaces in terms of Reproducing Kernel Hilbert Spaces (RKHSs). We provide theorems on the requirements for reliable extensions and their predicted approximation accuracy. We also establish formal relationships between specific kernel choices and their corresponding Sobolev Native Spaces. This connection further allows the extended neural operators to reliably capture not only function values but also their derivatives. Our methods are empirically validated through the solution of elliptic partial differential equations (PDEs) involving operators on manifolds having point-cloud representations and handling geometric contributions. We report results on key factors impacting the accuracy and computational performance of the extension approaches.

Figures & tables

Figure 1: Operator Extension Methods. We develop methods for extending conventional neural operators and geometric neural operators to robustly handle inputs beyond the functions in the training data. We leverage properties of kernel approximations. Neural operators process an input function $w(\cdot)$ and geometric neural operators also process information from the geometry $\Phi(\cdot)$ . The operators use a combination of a featurizing lifting operator $\mathcal{P}$ , kernel operator layers, and a projection operator $\mathcal{Q}$ to obtain the output function $u(\cdot)$ , (top) . We obtain representations for input functions in terms of kernels $k_{\sigma}$ , (lower-left) . The kernel constructions are shown for a subset of the points in the sum. The operator layer involves a combination of a kernel integration $\mathcal{K}[v]$ and a linear operation ${W}[v]$ that is passed through a non-linear activation function $\sigma(\cdot)$ , (lower-center) . The kernel representations are then used to construct the output function $\tilde{u}(\cdot)$ , (lower-right) .

Figure 2: Kernels Restricted to Manifolds $k_{\mathcal{M}}$ . We develop theory for kernel approximation based on restricting kernels from the bulk ambient space $k(x,y)$ to a manifold surface $k_{\mathcal{M}}(x,y)$ . Even when the bulk kernel $k$ has nice properties and symmetries, such as being a radial basis function, the restricted kernels $k_{\mathcal{M}}$ may not inherit these attributes. The $k_{\mathcal{M}}$ may no longer be radial symmetric $k_{\mathcal{M}}(x,y)\neq k_{\mathcal{M}}(|x-y|)$ or translation invariant $k_{\mathcal{M}}(x,y)\neq k_{\mathcal{M}}(x,y+\Delta)$ .

Kernel	Radial Function $\bm{\Phi(r)}$	$\bm{\mathcal{H}^{s}(\mathbb{R}^{d})}$	$\bm{\mathcal{H}^{s}(\mathcal{M})}$
Gaussian	$\exp\left(-r^{2}\right)$	N/A	N/A
Matérn Basic, $\nu=\frac{1}{2}$	$\exp\left(-r\right)$	$s=\frac{d+1}{2}$	$s=\frac{3}{2}$
Matérn Linear, $\nu=\frac{3}{2}$	$(1+\sqrt{3}r)e^{-\sqrt{3}r}$	$s=\frac{d+3}{2}$	$s=\frac{5}{2}$
Matérn Quadratic, $\nu=\frac{5}{2}$	$(1+\sqrt{5}r+\frac{5}{3}r^{2})e^{-\sqrt{5}r}$	$s=\frac{d+5}{2}$	$s=\frac{7}{2}$
Wendland, $k=0$ , $l=\lfloor d/2\rfloor+1$	$(1-r)_{+}^{l}$	$s=\frac{d+1}{2}$	$s=\frac{3}{2}$
Wendland, $k=1$ , $l=\lfloor d/2\rfloor+2$	$(1-r)_{+}^{l+1}((l+1)r+1)$	$s=\frac{d+3}{2}$	$s=\frac{5}{2}$

Table 1: Kernels and RKHS Equivalent Native Spaces. For the different kernels

k(x,y)=\Phi(|x-y|)

and RKHS

\mathcal{H}_{k}

generated by

k

, we give the equivalent native spaces. In the cases we consider, these are Sobolev spaces

\mathcal{H}^{s}

, [ 46 ] . For an input

\mathbf{x}\in\mathbb{R}^{d}

, we take

r=\left\lVert\mathbf{x}\right\rVert_{2}

. We also consider

\mathbf{x}\in\mathcal{M}

restricted to a manifold surface

\mathcal{M}\subset\mathbb{R}^{3}

. For kernels obtained by restricting the free-space kernel

k(x,y)

, we also give the corresponding Sobolev space

\mathcal{H}^{s}(\mathcal{M})

Figure 3: Kernels. We show a few different kernels used in our approximations and comparison studies. For the Gaussian kernels, we use $\Phi(\sigma\cdot r)$ with values for $\sigma=\ell_{0}^{-1}$ with $\ell_{0}=0.2,0.3,0.4$ for the cases labeled $A$ , $B$ , $C$ . For the Matérn kernels, the $A$ , $B$ , $C$ cases correspond to the Basic, Linear, and Quadratic kernels with the parameters shown in Table 1 . For the Wendland kernels, we consider the cases with $k=0$ , $k=1$ , $k=2$ with parameters in Table 1 .

Figure 4: Kernel Integration: Methods for Improving Performance. In neural operators a major computational cost during both training and evaluation is to compute the kernel integral operations $\mathcal{K}[v]$ . An edge-conditioned convolution is widely used for this approximation, but scales as $O(N^{2})$ becoming prohibitively expensive as the density of points increases within the support of the kernel, (top) . We develop more computationally efficient alternatives using node-conditioned convolutions based on factoring kernels into a separable form $k(x,y)=k_{1}(x)k_{2}(y)$ . The node-based methods scale as $O(N)$ allowing for approximations using efficient gathering and scattering operations greatly reducing computational costs, (bottom) .

Figure 5: Manifold Shapes and Kernel Restrictions. We show the manifolds used in our comparison studies. We also show how the Wendland kernel with parameter $k=0$ behaves when restricted to the manifold surface to obtain $k_{\mathcal{M}}$ . During training the different kernels are centered at a sampling of locations $\{x_{i}\}$ on the manifold providing a collection of functions $k_{\sigma}(\cdot,x_{i})$ for use during training.

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Manifold $A$
Gaussian	10	2.16e-01	2.18e-01	6.69e-01	7.76e+00
Gaussian	5	2.80e+03	1.85e+04	2.80e+04	4.26e+04
Matérn, $\nu=1/2$	10	1.14e-01	1.04e-01	1.02e-01	1.02e-01
Matérn, $\nu=1/2$	5	1.52e-01	1.53e-01	1.54e-01	1.54e-01
Matérn, $\nu=3/2$	10	6.48e-02	6.29e-02	6.27e-02	6.27e-02

Table 2: Relative

\bm{\mathcal{H}^{1}}

-norm Accuracy. We show results for the accuracy of the extended neural operators when used to obtain solutions of equation 51 . We show in bold all relative errors above

10^{0}

. The Gaussian kernels are found to perform poorly as

N

increases. The extension methods using the Matérn and Wendland kernels are found to perform well as

N

increases.

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Gaussian	10	1.34e+03	6.65e+03	6.32e+05	2.05e+07
Gaussian	5	2.05e+08	3.08e+09	6.84e+09	1.13e+10
Matérn, $\nu=1/2$	10	1.28e+03	1.49e+03	1.59e+03	1.63e+03
Matérn, $\nu=1/2$	5	1.53e+03	1.79e+03	1.91e+03	1.96e+03
Matérn, $\nu=3/2$	10	1.17e+03	1.26e+03	1.30e+03	1.32e+03
Matérn, $\nu=3/2$	5	2.26e+03	2.57e+03	2.77e+03	2.89e+03

Table 3: Coefficient

\bm{\ell^{1}}

-norms and

\bm{C_{2}}

. We show the

\|\alpha\|_{\ell^{1}}

of the coefficients

\{\alpha_{i}\}

in equation 6 when solving equation 27 . Results are averaged over all functions and manifolds. This gives an indication of the size of

C_{2}=\|\alpha\|_{\ell^{1}}

in the accuracy bounds in Theorem 1.2 in equation 16 . We highlight in bold all values larger than

10^{4}

. As the number of sample points increases, we see the Gaussian kernels have especially large values exceeding those of our other kernels by a few orders of magnitude. Relative to the other kernels, this indicates using Gaussians in the extension methods could be inefficient and sensitive to the accuracy of the trained neural operators, see equation 16 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Gaussian	10	1.07e+03	7.45e+05	3.99e+11	6.36e+19
Gaussian	5	1.21e+10	5.36e+18	1.19e+20	1.23e+21
Matérn, $\nu=1/2$	10	1.61e+01	4.33e+01	1.34e+02	3.59e+02
Matérn, $\nu=1/2$	5	1.19e+02	3.36e+02	1.07e+03	2.84e+03
Matérn, $\nu=3/2$	10	3.72e+01	1.77e+02	1.25e+03	6.01e+03
Matérn, $\nu=3/2$	5	8.81e+02	4.82e+03	3.73e+04	1.83e+05

Appendix

Table 4: Condition Numbers for the Gram Matrix. We show results for the Gram Matrix

K_{ij}=k(x_{i},x_{j})

for the Gaussian, Matérn, and Wendland kernels. We highlight in bold conditions numbers larger than

10^{6}

. We find as

N

increases the Gaussian kernels produce particularly ill-conditioned gram matrices which are orders of magnitude larger than the other kernels. The condition numbers are seen to grow much more slowly for the Matérn and Wendland kernels. The larger condition numbers can be seen to correspond to the less accurate cases in Table 3 . These results further highlight importance of the choice of kernel when using the extension methods.

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Manifold $A$
Gaussian	10	1.10e-01	1.11e-01	5.02e-01	5.13e+00
Gaussian	5	1.79e+03	1.25e+04	1.94e+04	2.99e+04
Matérn, $\nu=1/2$	10	6.09e-02	4.50e-02	4.26e-02	4.24e-02
Matérn, $\nu=1/2$	5	5.37e-02	5.45e-02	5.52e-02	5.55e-02
Matérn, $\nu=3/2$	10	2.93e-02	2.81e-02	2.83e-02	2.83e-02

Appendix

Table 5:

\bm{L^{2}}

-norm Relative Errors for

\bm{\tilde{u}}

. We show the accuracy of approximating

\bm{\tilde{u}}

when solving equation 27 . We show in bold errors larger than

1.0

. The accuracy of these results indicates that the main source of error in the Sobolev-norm in Table 2 likely arises from approximation of the gradients. We report these results in Table 6 .

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Manifold $A$
Gaussian	10	2.42e-01	2.44e-01	7.18e-01	8.43e+00
Gaussian	5	3.04e+03	2.01e+04	3.03e+04	4.61e+04
Matérn, $\nu=1/2$	10	1.27e-01	1.16e-01	1.15e-01	1.15e-01
Matérn, $\nu=1/2$	5	1.70e-01	1.72e-01	1.73e-01	1.73e-01
Matérn, $\nu=3/2$	10	7.25e-02	7.05e-02	7.03e-02	7.03e-02

Appendix

Table 6:

\bm{L^{2}}

-norm Relative Errors for

\bm{\nabla\tilde{u}}

. We show the accuracy of approximating

\nabla\tilde{u}

when solving equation 27 . We show in bold errors larger than

1.0

. The accuracy of these results indicates that the gradients are the dominant source of approximation error in the Sobolev-norm in Table 2 .

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 310 papers

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition
Sep 20, 2026Hang-Cheng Dong, Pengcheng ChengOOD Generalization PDE Operator Learning
School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin 150001, China · School of Mathematics, Jilin University, Changchun 130012, China
Into the danger zone: stable extrapolation in high-dimensional function and operator learning
Sep 29, 2026Ben Adcock, Simone Brugiapaglia, Xuemeng WangOOD Generalization Generalization Bounds
Simon Fraser University · Concordia University
Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces
Jun 16, 2026Yahong Yang, Zecheng Zhang, Wei Zhu +2Neural Network Approximation Theory Operator Learning
School of Mathematics, Georgia Institute of Technology, 686 Cherry Street, Atlanta, GA 30332-0160, USA. · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN 46556, USA. · Department of Mathematics, Hong Kong Baptist University, FSC1202, Fong Shu Chuen Building, Hong Kong Baptist University, Kowloon Tong, Hong Kong.

Sep 20, 2026J/K move · Enter open · S save

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition

Hang-Cheng Dong, Pengcheng Cheng

School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin 150001, China · School of Mathematics, Jilin University, Changchun 130012, China

OOD Generalization PDE Operator Learning Neural Operators

Open paper AskPDF arXiv

Neural operators have emerged as powerful surrogates for solving partial differential equations (PDEs), yet their reliability under distribution shift remains a critical barrier to deployment. Existing approaches to out-of-distribution (OOD) generalization in operator learning are largely empirical and black-box: they report aggregate error metrics without explaining why errors arise or when they will grow. We propose a structure-preserving framework that makes OOD generalization predictable and auditable. Our key idea is to parameterize the learned solution operator as a spectral filter

h_θ(λ)

acting on the eigenvalues of the underlying elliptic operator, implemented via Chebyshev polynomial expansions and trained with a weak-form objective. This parameterization admits an exact decomposition of the energy-norm error into two observable components: a model-dependent spectral approximation term and a distribution-dependent spectral weighting term induced by the input. From this decomposition we derive three diagnostics: a conservative in-band supremum

\vareps_{\mathrm{sup}}

, a global RMS proxy

\vareps_{\mathrm{rms}}

, and a sample-dependent effective metric

\vareps_{\mathrm{eff}}(f)

. These diagnostics can be computed without access to ground-truth solutions. Through four controlled experiments, we show that

\vareps_{\mathrm{eff}}(f)\|f\|

consistently predicts energy error under in-distribution, in-band spectral shift, out-of-band tail, and compound shifts, whereas global metrics can be systematically misleading. Our framework shifts OOD assessment of neural operators from black-box benchmarking to operator-structure diagnostics, providing a practical route to auditable scientific machine learning.

Abstract

Neural Operators

Explore similar work

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition

Into the danger zone: stable extrapolation in high-dimensional function and operator learning

Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces