Extending Neural Operators: Robust Handling of Functions Beyond the Training Set

Blaine Quackenbush; Paul J. Atzberger

cs.LGMar 4, 2026

Extending Neural Operators: Robust Handling of Functions Beyond the Training Set

Authors: Blaine Quackenbush, Paul J. Atzberger

Organizations: Department of Mathematics, University of California Santa Barbara (UCSB). · Department of Mathematics, Department of Mechanical Engineering, University of California Santa Barbara (UCSB)

Abstract

We develop a rigorous framework for extending neural operators to handle out-of-distribution input functions. We leverage kernel approximation techniques and provide theory for characterizing the input-output function spaces in terms of Reproducing Kernel Hilbert Spaces (RKHSs). We provide theorems on the requirements for reliable extensions and their predicted approximation accuracy. We also establish formal relationships between specific kernel choices and their corresponding Sobolev Native Spaces. This connection further allows the extended neural operators to reliably capture not only function values but also their derivatives. Our methods are empirically validated through the solution of elliptic partial differential equations (PDEs) involving operators on manifolds having point-cloud representations and handling geometric contributions. We report results on key factors impacting the accuracy and computational performance of the extension approaches.

Figures & tables

Figure 1: Operator Extension Methods. We develop methods for extending conventional neural operators and geometric neural operators to robustly handle inputs beyond the functions in the training data. We leverage properties of kernel approximations. Neural operators process an input function $w(\cdot)$ and geometric neural operators also process information from the geometry $\Phi(\cdot)$ . The operators use a combination of a featurizing lifting operator $\mathcal{P}$ , kernel operator layers, and a projection operator $\mathcal{Q}$ to obtain the output function $u(\cdot)$ , (top) . We obtain representations for input functions in terms of kernels $k_{\sigma}$ , (lower-left) . The kernel constructions are shown for a subset of the points in the sum. The operator layer involves a combination of a kernel integration $\mathcal{K}[v]$ and a linear operation ${W}[v]$ that is passed through a non-linear activation function $\sigma(\cdot)$ , (lower-center) . The kernel representations are then used to construct the output function $\tilde{u}(\cdot)$ , (lower-right) .

Figure 2: Kernels Restricted to Manifolds $k_{\mathcal{M}}$ . We develop theory for kernel approximation based on restricting kernels from the bulk ambient space $k(x,y)$ to a manifold surface $k_{\mathcal{M}}(x,y)$ . Even when the bulk kernel $k$ has nice properties and symmetries, such as being a radial basis function, the restricted kernels $k_{\mathcal{M}}$ may not inherit these attributes. The $k_{\mathcal{M}}$ may no longer be radial symmetric $k_{\mathcal{M}}(x,y)\neq k_{\mathcal{M}}(|x-y|)$ or translation invariant $k_{\mathcal{M}}(x,y)\neq k_{\mathcal{M}}(x,y+\Delta)$ .

Kernel	Radial Function $\bm{\Phi(r)}$	$\bm{\mathcal{H}^{s}(\mathbb{R}^{d})}$	$\bm{\mathcal{H}^{s}(\mathcal{M})}$
Gaussian	$\exp\left(-r^{2}\right)$	N/A	N/A
Matérn Basic, $\nu=\frac{1}{2}$	$\exp\left(-r\right)$	$s=\frac{d+1}{2}$	$s=\frac{3}{2}$
Matérn Linear, $\nu=\frac{3}{2}$	$(1+\sqrt{3}r)e^{-\sqrt{3}r}$	$s=\frac{d+3}{2}$	$s=\frac{5}{2}$
Matérn Quadratic, $\nu=\frac{5}{2}$	$(1+\sqrt{5}r+\frac{5}{3}r^{2})e^{-\sqrt{5}r}$	$s=\frac{d+5}{2}$	$s=\frac{7}{2}$
Wendland, $k=0$ , $l=\lfloor d/2\rfloor+1$	$(1-r)_{+}^{l}$	$s=\frac{d+1}{2}$	$s=\frac{3}{2}$
Wendland, $k=1$ , $l=\lfloor d/2\rfloor+2$	$(1-r)_{+}^{l+1}((l+1)r+1)$	$s=\frac{d+3}{2}$	$s=\frac{5}{2}$

Table 1: Kernels and RKHS Equivalent Native Spaces. For the different kernels

k(x,y)=\Phi(|x-y|)

and RKHS

\mathcal{H}_{k}

generated by

k

, we give the equivalent native spaces. In the cases we consider, these are Sobolev spaces

\mathcal{H}^{s}

, [ 46 ] . For an input

\mathbf{x}\in\mathbb{R}^{d}

, we take

r=\left\lVert\mathbf{x}\right\rVert_{2}

. We also consider

\mathbf{x}\in\mathcal{M}

restricted to a manifold surface

\mathcal{M}\subset\mathbb{R}^{3}

. For kernels obtained by restricting the free-space kernel

k(x,y)

, we also give the corresponding Sobolev space

\mathcal{H}^{s}(\mathcal{M})

Figure 3: Kernels. We show a few different kernels used in our approximations and comparison studies. For the Gaussian kernels, we use $\Phi(\sigma\cdot r)$ with values for $\sigma=\ell_{0}^{-1}$ with $\ell_{0}=0.2,0.3,0.4$ for the cases labeled $A$ , $B$ , $C$ . For the Matérn kernels, the $A$ , $B$ , $C$ cases correspond to the Basic, Linear, and Quadratic kernels with the parameters shown in Table 1 . For the Wendland kernels, we consider the cases with $k=0$ , $k=1$ , $k=2$ with parameters in Table 1 .

Figure 4: Kernel Integration: Methods for Improving Performance. In neural operators a major computational cost during both training and evaluation is to compute the kernel integral operations $\mathcal{K}[v]$ . An edge-conditioned convolution is widely used for this approximation, but scales as $O(N^{2})$ becoming prohibitively expensive as the density of points increases within the support of the kernel, (top) . We develop more computationally efficient alternatives using node-conditioned convolutions based on factoring kernels into a separable form $k(x,y)=k_{1}(x)k_{2}(y)$ . The node-based methods scale as $O(N)$ allowing for approximations using efficient gathering and scattering operations greatly reducing computational costs, (bottom) .

Figure 5: Manifold Shapes and Kernel Restrictions. We show the manifolds used in our comparison studies. We also show how the Wendland kernel with parameter $k=0$ behaves when restricted to the manifold surface to obtain $k_{\mathcal{M}}$ . During training the different kernels are centered at a sampling of locations $\{x_{i}\}$ on the manifold providing a collection of functions $k_{\sigma}(\cdot,x_{i})$ for use during training.

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Manifold $A$
Gaussian	10	2.16e-01	2.18e-01	6.69e-01	7.76e+00
Gaussian	5	2.80e+03	1.85e+04	2.80e+04	4.26e+04
Matérn, $\nu=1/2$	10	1.14e-01	1.04e-01	1.02e-01	1.02e-01
Matérn, $\nu=1/2$	5	1.52e-01	1.53e-01	1.54e-01	1.54e-01
Matérn, $\nu=3/2$	10	6.48e-02	6.29e-02	6.27e-02	6.27e-02

Table 2: Relative

\bm{\mathcal{H}^{1}}

-norm Accuracy. We show results for the accuracy of the extended neural operators when used to obtain solutions of equation 51 . We show in bold all relative errors above

10^{0}

. The Gaussian kernels are found to perform poorly as

N

increases. The extension methods using the Matérn and Wendland kernels are found to perform well as

N

increases.

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Gaussian	10	1.34e+03	6.65e+03	6.32e+05	2.05e+07
Gaussian	5	2.05e+08	3.08e+09	6.84e+09	1.13e+10
Matérn, $\nu=1/2$	10	1.28e+03	1.49e+03	1.59e+03	1.63e+03
Matérn, $\nu=1/2$	5	1.53e+03	1.79e+03	1.91e+03	1.96e+03
Matérn, $\nu=3/2$	10	1.17e+03	1.26e+03	1.30e+03	1.32e+03
Matérn, $\nu=3/2$	5	2.26e+03	2.57e+03	2.77e+03	2.89e+03

Table 3: Coefficient

\bm{\ell^{1}}

-norms and

\bm{C_{2}}

. We show the

\|\alpha\|_{\ell^{1}}

of the coefficients

\{\alpha_{i}\}

in equation 6 when solving equation 27 . Results are averaged over all functions and manifolds. This gives an indication of the size of

C_{2}=\|\alpha\|_{\ell^{1}}

in the accuracy bounds in Theorem 1.2 in equation 16 . We highlight in bold all values larger than

10^{4}

. As the number of sample points increases, we see the Gaussian kernels have especially large values exceeding those of our other kernels by a few orders of magnitude. Relative to the other kernels, this indicates using Gaussians in the extension methods could be inefficient and sensitive to the accuracy of the trained neural operators, see equation 16 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Gaussian	10	1.07e+03	7.45e+05	3.99e+11	6.36e+19
Gaussian	5	1.21e+10	5.36e+18	1.19e+20	1.23e+21
Matérn, $\nu=1/2$	10	1.61e+01	4.33e+01	1.34e+02	3.59e+02
Matérn, $\nu=1/2$	5	1.19e+02	3.36e+02	1.07e+03	2.84e+03
Matérn, $\nu=3/2$	10	3.72e+01	1.77e+02	1.25e+03	6.01e+03
Matérn, $\nu=3/2$	5	8.81e+02	4.82e+03	3.73e+04	1.83e+05

Appendix

Table 4: Condition Numbers for the Gram Matrix. We show results for the Gram Matrix

K_{ij}=k(x_{i},x_{j})

for the Gaussian, Matérn, and Wendland kernels. We highlight in bold conditions numbers larger than

10^{6}

. We find as

N

increases the Gaussian kernels produce particularly ill-conditioned gram matrices which are orders of magnitude larger than the other kernels. The condition numbers are seen to grow much more slowly for the Matérn and Wendland kernels. The larger condition numbers can be seen to correspond to the less accurate cases in Table 3 . These results further highlight importance of the choice of kernel when using the extension methods.

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Manifold $A$
Gaussian	10	1.10e-01	1.11e-01	5.02e-01	5.13e+00
Gaussian	5	1.79e+03	1.25e+04	1.94e+04	2.99e+04
Matérn, $\nu=1/2$	10	6.09e-02	4.50e-02	4.26e-02	4.24e-02
Matérn, $\nu=1/2$	5	5.37e-02	5.45e-02	5.52e-02	5.55e-02
Matérn, $\nu=3/2$	10	2.93e-02	2.81e-02	2.83e-02	2.83e-02

Appendix

Table 5:

\bm{L^{2}}

-norm Relative Errors for

\bm{\tilde{u}}

. We show the accuracy of approximating

\bm{\tilde{u}}

when solving equation 27 . We show in bold errors larger than

1.0

. The accuracy of these results indicates that the main source of error in the Sobolev-norm in Table 2 likely arises from approximation of the gradients. We report these results in Table 6 .

Kernel	Shape $\bm{\sigma}$	$\bm{N=1250}$	$\bm{2500}$	$\bm{5000}$	$\bm{10000}$
Manifold $A$
Gaussian	10	2.42e-01	2.44e-01	7.18e-01	8.43e+00
Gaussian	5	3.04e+03	2.01e+04	3.03e+04	4.61e+04
Matérn, $\nu=1/2$	10	1.27e-01	1.16e-01	1.15e-01	1.15e-01
Matérn, $\nu=1/2$	5	1.70e-01	1.72e-01	1.73e-01	1.73e-01
Matérn, $\nu=3/2$	10	7.25e-02	7.05e-02	7.03e-02	7.03e-02

Appendix

Table 6:

\bm{L^{2}}

-norm Relative Errors for

\bm{\nabla\tilde{u}}

. We show the accuracy of approximating

\nabla\tilde{u}

when solving equation 27 . We show in bold errors larger than

1.0

. The accuracy of these results indicates that the gradients are the dominant source of approximation error in the Sobolev-norm in Table 2 .

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 310 papers

Sep 20, 2026

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition

Neural operators have emerged as powerful surrogates for solving partial differential equations (PDEs), yet their reliability under distribution shift remains a critical barrier to deployment. Existing approaches to out-of-distribution (OOD) generalization in operator learning are largely empirical and black-box: they report aggregate error metrics without explaining why errors arise or when they will grow. We propose a structure-preserving framework that makes OOD generalization predictable and auditable. Our key idea is to parameterize the learned solution operator as a spectral filter

h_θ(λ)

acting on the eigenvalues of the underlying elliptic operator, implemented via Chebyshev polynomial expansions and trained with a weak-form objective. This parameterization admits an exact decomposition of the energy-norm error into two observable components: a model-dependent spectral approximation term and a distribution-dependent spectral weighting term induced by the input. From this decomposition we derive three diagnostics: a conservative in-band supremum

\vareps_{\mathrm{sup}}

, a global RMS proxy

\vareps_{\mathrm{rms}}

, and a sample-dependent effective metric

\vareps_{\mathrm{eff}}(f)

. These diagnostics can be computed without access to ground-truth solutions. Through four controlled experiments, we show that

\vareps_{\mathrm{eff}}(f)\|f\|

consistently predicts energy error under in-distribution, in-band spectral shift, out-of-band tail, and compound shifts, whereas global metrics can be systematically misleading. Our framework shifts OOD assessment of neural operators from black-box benchmarking to operator-structure diagnostics, providing a practical route to auditable scientific machine learning.

Hang-Cheng Dong, Pengcheng Cheng

School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin 150001, China · School of Mathematics, Jilin University, Changchun 130012, China

OOD Generalization PDE Operator Learning Neural Operators

Ask PDF

Sep 29, 2026

Into the danger zone: stable extrapolation in high-dimensional function and operator learning

Out-of-distribution (OOD) generalization is a central challenge in scientific machine learning. We study regression problems in which the test distribution differs from the training distribution and ask: under what assumptions on the target function or operator is stable extrapolation possible, and how far beyond the training domain can one extrapolate? Existing theory controls the test error through additive penalties measuring the discrepancy between the training and test distributions. Such guarantees show robustness to small distribution shifts, but can be very pessimistic in comparison to OOD performance observed empirically. We identify classes of holomorphic functions and operators for which the OOD generalization error converges at algebraic rates even in the presence of large distribution shifts. This phenomenon stems from the increasing smoothness of higher-index coordinates, leading to what we term a `blessing of high dimensionality'. For learning with either polynomials, deep neural networks or deep neural operators, we derive explicit rates for arbitrary test measures supported on suitable domains and quantify how the admissible domain depends on the underlying regularity of the function or operator. Our extrapolation guarantees are independent of the test distribution, depending only on its support. We also present a series of numerical experiments across a range of functions and operators that support the main theoretical findings.

Ben Adcock, Simone Brugiapaglia, Xuemeng Wang

Simon Fraser University · Concordia University

OOD Generalization Generalization Bounds Operator Learning

Ask PDF

Jun 16, 2026

Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces

We develop approximation and generalization error estimates for multi-input neural operators, with the output error measured in Sobolev norms. In contrast to standard operator-learning settings with a single input function, our framework allows multiple input functions defined on possibly different domains, with different dimensions and Sobolev regularities. The derived rates explicitly quantify the contribution of each input space to the final error bound. In particular, in the balanced regime, the approximation and generalization rates are governed by the interaction between the input dimensions, regularities, and Sobolev orders, while the dependence on the model complexity retains a

\log\log/\log

-type structure. Our analysis provides a general theoretical framework for multi-input operator learning, including Sobolev training, and is applicable to operator learning problems arising from partial differential equations and scientific computing.

Yahong Yang, Zecheng Zhang, Wei Zhu +2

School of Mathematics, Georgia Institute of Technology, 686 Cherry Street, Atlanta, GA 30332-0160, USA. · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN 46556, USA. · Department of Mathematics, Hong Kong Baptist University, FSC1202, Fong Shu Chuen Building, Hong Kong Baptist University, Kowloon Tong, Hong Kong.

Neural Network Approximation Theory Operator Learning Neural Operators Generalization Bounds

Ask PDF

Abstract

Neural Operators

Explore similar work

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition

Into the danger zone: stable extrapolation in high-dimensional function and operator learning

Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces