Into the danger zone: stable extrapolation in high-dimensional function and operator learning

Ben Adcock; Simone Brugiapaglia; Xuemeng Wang

math.NASep 29, 2026

Into the danger zone: stable extrapolation in high-dimensional function and operator learning

Authors: Ben Adcock, Simone Brugiapaglia, Xuemeng Wang

Organizations: Simon Fraser University · Concordia University

Abstract

Out-of-distribution (OOD) generalization is a central challenge in scientific machine learning. We study regression problems in which the test distribution differs from the training distribution and ask: under what assumptions on the target function or operator is stable extrapolation possible, and how far beyond the training domain can one extrapolate? Existing theory controls the test error through additive penalties measuring the discrepancy between the training and test distributions. Such guarantees show robustness to small distribution shifts, but can be very pessimistic in comparison to OOD performance observed empirically. We identify classes of holomorphic functions and operators for which the OOD generalization error converges at algebraic rates even in the presence of large distribution shifts. This phenomenon stems from the increasing smoothness of higher-index coordinates, leading to what we term a `blessing of high dimensionality'. For learning with either polynomials, deep neural networks or deep neural operators, we derive explicit rates for arbitrary test measures supported on suitable domains and quantify how the admissible domain depends on the underlying regularity of the function or operator. Our extrapolation guarantees are independent of the test distribution, depending only on its support. We also present a series of numerical experiments across a range of functions and operators that support the main theoretical findings.

Figures & tables

Panel of Figure 1 — Figure 1: For both function (left) and operator (right) learning tasks, the OOD generalization error decays algebraically even though $W_{2}(\varrho,\mu)$ is large. Top row: the squared relative $L^{2}_{\mu}$ -norm error versus number of training samples $m$ for various test distributions $\mu=\mu_{\theta}$ . Here $\theta$ is a parameter that controls how far the test distribution is from the training distribution, which corresponds to the case $\theta=0$ . Bottom row: the distance $W_{2}(\varrho,\mu)$ between the training and distributions $\varrho$ and $\mu$ . This experiment considers the wing weight function (left), learned with a fully-connected DNN, and the solution operator of the Navier–Stokes PDE (right), learned with a so-called Fourier neural operator. See § 6.2 and § 6.3 , respectively, for further details and discussion on these experiments.

Panel of Figure 2 — Figure 2: Error versus number of training samples $m$ for function approximation using polynomial estimators computed by the ALS algorithm. The function is given by ( 26 ) with $\delta_{i}=2i^{2}$ , $\forall i\in[d]$ and $d=8$ (left) or $d=32$ (middle). The training and test distributions are $\varrho=\mathcal{U}([-1,1]^{d})$ and $\mu=\mathcal{U}(\otimes^{d}_{i=1}[-\omega_{i},\omega_{i}])$ , where $\omega_{1}=\cdots=\omega_{d}=\omega$ . The value $q_{\mathrm{thy}}$ is given by ( 29 ) in the case $d=32$ . A dashed line means that $q_{\mathrm{thy}}\leq 0$ , i.e., no convergence. The values $q_{\mathrm{fit}}$ were computed by empirical fitting in the case $d=32$ .

Panel of Figure 3 — Figure 3: The same as Fig. 2 , except with varying side lengths $\omega_{i}=i^{b}$ and the value $q_{\mathrm{thy}}$ given by ( 30 ) in the case $d=32$ once more.

Panel of Figure 4 — Figure 4: Error versus number of training samples $m$ for approximating the circuit (left) and piston (right) functions via tanh DNNs with $L=15$ layers and width $W=150$ .

Panel of Figure 5 — Figure 5: Error versus number of training samples $m$ for the Darcy flow problem with $\nu=\mu^{\mathrm{LN}}_{3,2}$ and $\mu=\mu^{\mathrm{LN}}_{3,\alpha}$ (left), $\mu=\mu^{\mathrm{PC}}_{3,\alpha}$ (middle); $\nu=\mu_{3,1}^{LN}$ and $\mu=\mu^{\mathrm{cookies}}$ (right).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Hidden layers	15
Hidden layer width	150
Activation	$\tanh$
Training samples	$\{10,20,40,80,160,320,640,1280\}$
Test samples	10,000
Batch size	Full batch (all training samples)

Appendix

Table 1: DNN architecture and training parameters for benchmark functions from the Virtual Library of Simulation Experiments .

Parameter	Value
Training resolution	$64\times 64$
Testing resolution	$64\times 64$
Training samples	$\{10,20,40,80,160,320,640,1280\}$
Test samples	200
Input channels	1
Output channels	1

Appendix

Table 2: FNO architecture and training parameters for the Darcy Flow problem.

Figure 1: Representative input-output pairs from the Darcy flow training and test distributions. The top row shows the diffusion coefficients $a$ , and the bottom row shows the corresponding solutions $u$ . The distributions are $\mu^{\mathrm{LN}}_{3,2}$ , $\mu^{\mathrm{LN}}_{3,1}$ , $\mu^{\mathrm{cookies}}$ and $\mu^{\mathrm{PC}}_{3,1}$ (left to right).

Parameter	Value
Training resolution	$64\times 64\times 50$
Testing resolution	$64\times 64\times 50$
Training samples	$[150,300,600,1200,2400,4800]$
Test samples	200
Input channels	4
Output channels	1

Appendix

Table 3: FNO architecture and training parameters for the Navier-Stokes problem.

Explore similar work

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition
Sep 20, 2026Hang-Cheng Dong, Pengcheng ChengOOD Generalization PDE Operator Learning
School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin 150001, China · School of Mathematics, Jilin University, Changchun 130012, China
Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization
May 8, 2026Leonel Aguilar, Jan Nagler, Christoph Hoelscher +1OOD Generalization Inductive Bias
Chair of Cognitive Science, ETH Zürich · Centre for Human and Machine Intelligence, Frankfurt School · Aisot Technologies AG +1
On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation
Sep 17, 2026Menghui Zhou, Gaoshan Bi, Vitaveska Lanfranchi +1OOD Generalization Invariant Representation Learning
School of Computer Science University of Sheffield

Sep 20, 2026, cs.LGJ/K move · Enter open · S save

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition

Hang-Cheng Dong, Pengcheng Cheng

School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin 150001, China · School of Mathematics, Jilin University, Changchun 130012, China

OOD Generalization PDE Operator Learning Neural Operators

Open paper AskPDF arXiv

Neural operators have emerged as powerful surrogates for solving partial differential equations (PDEs), yet their reliability under distribution shift remains a critical barrier to deployment. Existing approaches to out-of-distribution (OOD) generalization in operator learning are largely empirical and black-box: they report aggregate error metrics without explaining why errors arise or when they will grow. We propose a structure-preserving framework that makes OOD generalization predictable and auditable. Our key idea is to parameterize the learned solution operator as a spectral filter

h_θ(λ)

acting on the eigenvalues of the underlying elliptic operator, implemented via Chebyshev polynomial expansions and trained with a weak-form objective. This parameterization admits an exact decomposition of the energy-norm error into two observable components: a model-dependent spectral approximation term and a distribution-dependent spectral weighting term induced by the input. From this decomposition we derive three diagnostics: a conservative in-band supremum

\vareps_{\mathrm{sup}}

, a global RMS proxy

\vareps_{\mathrm{rms}}

, and a sample-dependent effective metric

\vareps_{\mathrm{eff}}(f)

. These diagnostics can be computed without access to ground-truth solutions. Through four controlled experiments, we show that

\vareps_{\mathrm{eff}}(f)\|f\|

consistently predicts energy error under in-distribution, in-band spectral shift, out-of-band tail, and compound shifts, whereas global metrics can be systematically misleading. Our framework shifts OOD assessment of neural operators from black-box benchmarking to operator-structure diagnostics, providing a practical route to auditable scientific machine learning.

Abstract

Explore similar work

Predicting Out-of-Distribution Generalization of Neural Operators via Observable Spectral Error Decomposition

Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization

On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation