Neural Operators Which Certifiably Simultaneously Solve Infinitely Many Convex Programs using Logarithmic Rank and Linear Complexity

Anastasis Kratsios; Ariel Neufeld; Philipp Schmocker

cs.LGAug 20, 2025

Neural Operators Which Certifiably Simultaneously Solve Infinitely Many Convex Programs using Logarithmic Rank and Linear Complexity

Authors: Anastasis Kratsios, Ariel Neufeld, Philipp Schmocker

Organizations: Department of Mathematics, McMaster University and the Vector Institute, Canada · Division of Mathematical Sciences, Nanyang Technological University, Singapore

Abstract

Neural operators (NOs) are a class of deep learning models designed to simultaneously solve infinitely many related problems by casting them into an infinite-dimensional space, whereon these NOs operate. Such operator learning problems arise naturally in mathematical finance, for instance, when learning families of optimal portfolios or hedging strategies. Two significant gaps, however, remain between theory and practice: 1) most universal approximation guarantees do not describe practically implementable NOs or are pure existence statements; and 2) worst-case parameter bounds from universal approximation theorems suggest that constructed NOs may require an unrealistically large number of parameters to solve most operator learning problems, which stands in direct opposition to a slew of experimental evidence. This paper closes these gaps by exhibiting a short and simple training algorithm to construct a specific class of NOs, proximal equilibrium operators (PEOs), using (realistic) finite-dimensional deep equilibrium layers to solve families of convex optimization problems over a separable Hilbert space $X$ . Here, the inputs are smooth, convex loss functions on $X$ , and the outputs are the associated (approximate) solutions to the optimization problems defined by each input loss. Our main result is a certificate showing that, when the input losses lie in a suitable infinite-dimensional set, our PEOs trained with our algorithm do in fact uniformly approximate the corresponding solutions to arbitrary precision, with rank and width growing only logarithmically and the depth growing linearly in the reciprocal of the approximation error. We then validate both our theoretical results and the trainability of PEOs on three applications: non-linear PDEs, stochastic optimal control, and portfolio optimization and hedging problems in mathematical finance.

Figures & tables

Panel of Figure 1 — Figure 1: Learning the map ( 4.4 ) returning the solution of the parametric PDE ( 4.3 ) by a Proximal Equilibrium Operator $\mathcal{G}$ . In (a), the learning performance is displayed in terms of the mean squared error (MSE) $\frac{1}{|K|}\sum_{k\in K}\|\mathcal{S}(g_{\sigma_{k}})-\mathcal{G}(g_{\sigma_{k}})\|^{2}$ on the training set (label “Train”) and test set (label “Test”). In (b), the predicted solution $\mathcal{G}(g_{\sigma_{k}})$ (label “Predict”) is compared to the true solution $y(T,\cdot)=\mathcal{S}(g_{\sigma_{k}})$ (label “True”) for two $\sigma_{k}$ of the test set.

Table 1: Computation of optimal portfolios for power and logarithmic utility functions given by

u(x)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\frac{(x-x_{0})^{1-\eta}}{1-\eta}

\eta\in(0,\infty)\setminus\{1\}

and

u(x)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\ln(x-x_{0})

\eta=1

, where

x_{0}\geq 0

is the reference point.

Panel of Figure 2 — Figure 2: Learning the solution operator $\mathcal{S}$ of the utility maximization problem ( 4.7 ) by a Proximal Equilibrium Operator $\mathcal{G}$ . In (a), the learning performance is displayed in terms of the mean squared error (MSE) $\frac{1}{|K|}\sum_{k\in K}\|\mathcal{S}(g_{u_{k}})-\mathcal{G}(g_{u_{k}})\|^{2}$ on the training set (label “Train”) and test set (label “Test”). In (b), the predicted solution $\mathcal{G}(g_{u_{k}})$ (label “Predict”) is compared to the true solution $\mathcal{S}(g_{u_{k}})$ (label “True”) for two $\eta_{k}$ of the test set and some fixed $\omega\in\Omega$ .

Panel of Figure 3 — Figure 3: Learning the pricing/hedging operator $\mathcal{S}$ in ( 4.9 ) by a Proximal Equilibrium Operator $\mathcal{G}$ . In (a), the learning performance is displayed in terms of the mean squared error (MSE) $\frac{1}{|K|}\sum_{k\in K}\|\mathcal{S}(g_{H_{k}})-\mathcal{G}(g_{H_{k}})\|^{2}$ on the training set (label “Train”) and test set (label “Test”). In (b), the predicted solution $\mathcal{G}(g_{H_{k}})$ (label “Predict”) is compared to the true solution $\mathcal{S}(g_{H_{k}})$ (label “True”) for one $k$ of the test set and some $\omega\in\Omega$ .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Learning the minimization operator $\mathcal{S}$ in ( A.1 ) by a Proximal Equilibrium Operator $\mathcal{G}$ . In (a), the learning performance is displayed in terms of the mean squared error (MSE) $\frac{1}{|K|}\sum_{k\in K}\|\mathcal{S}(g_{k})-\mathcal{G}(g_{k})\|^{2}$ on the training set (label “Train”) and test set (label “Test”). In (b), the predicted solution $\mathcal{G}(g_{k})$ (label “Predict”) is compared to the true solution $\mathcal{S}(g_{k})$ (label “True”) for four $k$ of the test set.

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 300 papers

Aug 19, 2026

Multi-stage neural operator learning with application for convolutions

Convolution integrals widely exist in applications, and to enable fast and accurate computations, this paper introduces two general multi-stage neural operator learning frameworks. The first, Deep Collocation Neural Operator (DCNO), is a supervised approach that iteratively refines the operator approximation by learning residuals from input-output data pairs. The second, Deep Galerkin Neural Operator (DGNO), is an unsupervised framework applicable when the target operator can be represented by a PDE, leveraging the weak form of the PDE residual for training. Both methods progressively construct basis operators through multiple training stages to enrich the approximation space, leading to significantly improved accuracy over standard one-shot operator learning. We provide theoretical analysis for their approximation capabilities and implement them for learning convolutions. Extensive numerical experiments demonstrate that both DCNO and DGNO achieve high accuracy, approaching machine precision under single float for convolution problems, and offer substantial efficiency gains for numerous queries or parametric variations compared to traditional solvers. We also extend these frameworks to handle multi-input operator learning scenarios involving variations in both the density and kernel of a convolution.

Zhiping Mao, Zhenye Wen, Yong Zhang +1

School of Mathematical Sciences, Eastern Institute of Technology, Ningbo, 315200 Ningbo, China · School of Mathematics and Statistics, Wuhan University, 430072 Wuhan, China · Center for Applied Mathematics and KL-AAGDM, Tianjin University, 300072 Tianjin, China +2

Operator Learning Neural Operators PDE Operator Learning Neural Network Approximation Theory

Ask PDF

May 4, 2026

Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation

Neural operators have emerged as a powerful tool for learning mappings between infinite-dimensional function spaces. However, their approximation properties in Sobolev norms remain poorly quantified, even though these norms control both function values and derivatives and are the natural metrics for PDE well-posedness, stability, and generalization. We develop a functional-analytic framework for operator learning in Sobolev spaces and connect it to the numerical behavior of Fourier Neural Operators (FNOs) on a prototypical PDE. First, for a continuous nonlinear operator

\mathcal{G}: H^{s}(D)\to H^{t}(D')

with

s > d/2

and inputs restricted to a compact subset of

H^{s}(D)

, we prove that

\mathcal{G}

can be uniformly approximated in

H^{t}

-norm by a neural operator with

\mathcal{O}(\varepsilon^{-d/s})

trainable parameters. This yields an explicit complexity--error relation of the form

\|\mathcal{G}-\mathcal{G}_θ\|_{H^{t}} \lesssim C N^{-s/d}

. We then study the one-dimensional viscous Burgers solution operator

\mathcal{G}: u_{0}\mapsto u(\cdot,1)

on a bounded

H^{1}

-ball and train FNOs with an

H^{1}

-loss. Across a sweep of model sizes, we obtain test

H^{1}

-errors down to

\mathcal{O}(10^{-7})

and relative errors of order

10^{-3}

, with predictions accurately matching both solutions and spatial derivatives on held-out data. A log-log plot of Sobolev error versus parameter count exhibits an approximate power law

\|\mathcal{G}-\mathcal{G}_θ\|_{H^{1}} \approx C N^{-α}

with empirical exponent

α\approx 1.4

, and long-horizon training reveals optimization instabilities in large FNOs, providing quantitative evidence that Sobolev-space approximation theory meaningfully predicts neural-operator scaling behavior.

Nicole Hao

Department of Mathematics, Cornell University

Fourier Neural Operator PDE Solving PDE Operator Learning Neural Network Approximation Theory

Ask PDF

May 31, 2026

Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization

Inspired by the function-space theory of neural networks, we formulate and analyze a variation space for nonlinear operators between Hilbert spaces, defined through vector-valued Borel measures of bounded variation. We characterize its unit ball as the closed convex hull of a vector-valued single-neuron dictionary in Bochner spaces. For the ReLU activation, the bounded linear operators in this space are precisely the Schatten-

1

operators, with equivalent norms. For operators in this space, we establish encoder--decoder approximation bounds in the Bochner

L^q

-norm, where the error decomposes into input and output encoding errors and a finite-width term of order

N^{-1/2}

. Under sub-Gaussian assumptions on the input and noise, we further derive high-probability generalization bounds for empirical least squares over path-norm-constrained encoder--decoder networks; the finite-sample contribution to the squared prediction error is of order

K^{-1/2}

up to logarithmic factors. The finite-width and finite-sample constants are independent of the encoding dimensions and bases, with the latter also independent of the network width. When the encoding errors decay algebraically, these bounds yield algebraic approximation and learning rates, in contrast to the complexity barriers for Lipschitz and Fréchet differentiable operator classes.

Jia-Qi Yang, Lei Shi

School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

Neural Network Approximation Theory Operator Learning Neural Operators

Ask PDF

Abstract

Neural Operators

Explore similar work

Multi-stage neural operator learning with application for convolutions

Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation

Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization