A Bregman Proximal Viewpoint on Neural Operators

Abdel-Rahim Mezidi; Jordan Patracone; Saverio Salzo; Amaury Habrard; Massimiliano Pontil; Remi Emonet; Marc Sebban

cs.LGOct 14, 2026

A Bregman Proximal Viewpoint on Neural Operators

Authors: Abdel-Rahim Mezidi, Jordan Patracone, Saverio Salzo, Amaury Habrard, Massimiliano Pontil, Remi Emonet, Marc Sebban

Organizations: Université Jean Monnet Saint-Etienne, CNRS, Institut d’Optique Graduate School, Inria, Laboratoire Hubert Curien UMR 5516, F-42023, SAINT-ETIENNE, France · DIAG, Sapienza University of Rome, 00185 Rome, Italy · Computational Statistics and Machine Learning, IIT, Genova, Italy · Departement of Computer Science, UCL, London, United Kingdom

Abstract

We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activation operator and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the $t$ -th layer of Fourier Neural Operators. The upper branch applies a linear transformation $R_{t}$ to the Fourier modes using the Fourier transform $\mathcal{F}$ and its inverse $\mathcal{F}^{-1}$ . The lower branch performs an affine transformation in the latent space.

Panel of Figure 2 — Figure 2: Illustration of the $t$ -th layer of (Bregman) Neural Operators. On the left, the identity term and the linear term $K_{t}v_{t}+b_{t}$ have been merged into $(I+K_{t})v_{t}=W_{t}v_{t}$ . For both, $\mathcal{K}_{t}^{\rm ac}$ represents any absolutely continuous operator.

Panel of Figure 3 — Figure 3: Illustration of the Bregman proximal viewpoint on operator layers. The action of each operator layer is viewed as the minimizer of the regularized optimization problem where each term in the objective can be linked to a part of the architecture, as evidenced by the color code.

$\mathrm{dom}\psi$	$\psi(t)$	$\psi^{\prime}(t)$	$\psi^{*\prime}(t)$
$[-1,1]$	$-\sqrt{1-t^{2}}$	$t/\sqrt{1-t^{2}}$	ISRU
$[0,1]$	$t\log t+(1-t)\log(1-t)$	$\log\frac{t}{1-t}$	Sigmoid
$[-1,1]$	$\log(1-t^{2})+t\,\mathrm{arctanh}(t)$	$\mathrm{arctanh}$	$\tanh$
$[-1,1]$	$\sqrt{1-t^{2}}+t\arcsin(t)$	$\arcsin$	$\sin$
$\mathbb{R}_{>0}$	$\frac{1}{\beta^{2}}\mathrm{Li}_{2}(\mathrm{e}^{-\beta t})+\frac{t^{2}}{2}$	$\frac{\log(\mathrm{e}^{\beta t}-1)}{\beta}$	SoftPlus β

Table 1: Legendre function

\psi

and its related activation

\psi^{*\prime}

Figure 4:

\ell^{2}

relative errors across different number of layers for 2D Navier Stokes (

\nu=10^{-4}

Figure 5: Models’ weight density distribution.

Figure 6: Comparison of

\ell^{2}

test relative errors across different models, with cross-validated number of layers in

\{4,8,16,32\}

and learning rates, on PDE benchmarks.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Illustration of SoftPlus as a proximity operator.

Dataset	Number of modes	Batch size	Width	Tinit	Tfinal	Train samples
1D Advection	[16]	128	64	0	200	8000
1D Burgers	[16]	128	64	0	200	8000
1D NS	[16]	128	64	10	11	8000
2D NS ( $10^{-4}$ )	[12,12]	16	32	10	15	1000
2D NS ( $10^{-3}$ )	[12,12]	16	32	10	45	1000
2D Darcy	[12,12]	16	32	-	-	1000

Appendix

Table 2: Experimental settings.

Panel of Figure 8 — Figure 8: Predictions and residuals for Navier Stokes $10^{-4}$ .

	1D Advection		1D Burgers		1D NS
	WNO	BWNO	WNO	BWNO	WNO	BWNO
4 layers	$3.0\pm 0.0$ %	$\textbf{2.8}\pm 0.2$ %	$21.5\pm 0.5$ %	$\textbf{21.3}\pm 0.4$ %	$59.2\pm 0.6$ %	$\textbf{58.3}\pm 0.6$ %
8 layers	$2.5\pm 0.1$ %	$\textbf{2.1}\pm 0.1$ %	$19.1\pm 0.6$ %	$\textbf{17.9}\pm 0.6$ %	$59.0\pm 0.6$ %	$\textbf{58.0}\pm 0.6$ %
16 layers	$3.9\pm 0.8$ %	$\textbf{2.0}\pm 0.2$ %	$19.7\pm 0.5$ %	$\textbf{16.4}\pm 0.3$ %	$61.1\pm 0.6$ %	$\textbf{57.6}\pm 0.7$ %
32 layers	$100\pm 0$ %	$\textbf{1.9}\pm 0.1$ %	$100\pm 0$ %	$\textbf{16.4}\pm 0.5$ %	$100\pm 0$ %	$\textbf{57.2}\pm 0.6$ %
64 layers	$100\pm 0$ %	$\textbf{1.8}\pm 0.2$ %	$100\pm 0$ %	$\textbf{16.1}\pm 0.4$ %	$100\pm 0$ %	$\textbf{57.5}\pm 0.6$ %

Appendix

Table 3: Relative error of WNO and BWNO models on benchmark PDEs.

		$T=4$		$T=8$		$T=16$
PDE	Metric	BFNO	FNO	BFNO	FNO	BFNO	FNO
1D Advection	rMSE	$\mathbf{1.55\cdot 10^{-2}}$	$2.43\cdot 10^{-2}$	$\mathbf{1.45\cdot 10^{-2}}$	$3.22\cdot 10^{-2}$	$\mathbf{1.43\cdot 10^{-2}}$	$4.38\cdot 10^{-2}$
	bRMSE	$\mathbf{9.04\cdot 10^{-2}}$	$1.21\cdot 10^{-1}$	$\mathbf{8.28\cdot 10^{-2}}$	$1.51\cdot 10^{-1}$	$\mathbf{7.82\cdot 10^{-2}}$	$2.43\cdot 10^{-1}$
	fRMSE low	$\mathbf{4.56\cdot 10^{-6}}$	$9.38\cdot 10^{-6}$	$\mathbf{4.45\cdot 10^{-6}}$	$1.14\cdot 10^{-5}$	$\mathbf{4.21\cdot 10^{-6}}$	$1.93\cdot 10^{-5}$
	fRMSE mid	$\mathbf{3.89\cdot 10^{-6}}$	$6.81\cdot 10^{-6}$	$\mathbf{3.39\cdot 10^{-6}}$	$8.84\cdot 10^{-6}$	$\mathbf{3.61\cdot 10^{-6}}$	$1.27\cdot 10^{-5}$
	fRMSE high	$\mathbf{3.15\cdot 10^{-7}}$	$4.87\cdot 10^{-7}$	$\mathbf{2.89\cdot 10^{-7}}$	$5.96\cdot 10^{-7}$	$\mathbf{2.76\cdot 10^{-7}}$	$8.08\cdot 10^{-7}$

Appendix

Table 4: Additional comparison of the performance in terms of relative

\ell^{2}

error (rMSE), relative mean squared error on the boundary (rMSE) as well as in the low, mid and high frequency bands (fRMSE low, fRMSE mid, fRMSE high).

Architecture	4 layers	8 layers	16 layers
FNO (ReLU)	13.5 ± 0.1	13.0 ± 0.1	12.6 ± 0.1
BFNO (Softplus)	13.7 ± 0.1	12.6 ± 0.1	12.2 ± 0.1
BFNO (ReLU)	13.4 ± 0.2	12.2 ± 0.2	12.2 ± 0.1

Appendix

Table 5: Comparison BFNO with Softplus or ReLU.

Architecture	4 layers	8 layers	16 layers
FNO	13.5 ± 0.1	13.0 ± 0.1	12.6 ± 0.1
FNO + BN	13.5 ± 0.1	12.8 ± 0.2	12.6 ± 0.1
BFNO	13.7 ± 0.1	12.6 ± 0.1	12.2 ± 0.1
BFNO + BN	13.5 ± 0.1	12.4 ± 0.0	12.3 ± 0.1

Appendix

Table 6: Impact of BatchNormalization (BN) with FNO and BFNO

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 313 papers

Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation
May 4, 2026Nicole HaoFourier Neural Operator PDE Solving
Department of Mathematics, Cornell University
Neural Operators as Efficient Function Interpolators
May 8, 2026Vasilis Niarchos, Angelos Sirbu, Sokratis TrifinopoulosNeural Surrogate Modeling Fourier Neural Operator
Department of Physics, CCTP and ITCP, University of Crete, 71303, Greece · Theoretical Physics Department, CERN, Geneva, Switzerland · Physik-Institut, Universität Zürich, 8057 Zürich, Switzerland
Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization
May 31, 2026Jia-Qi Yang, Lei ShiNeural Network Approximation Theory Operator Learning
School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

May 4, 2026J/K move · Enter open · S save

Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation

Nicole Hao

Department of Mathematics, Cornell University

Fourier Neural Operator PDE Solving PDE Operator Learning Neural Network Approximation Theory

Open paper AskPDF arXiv

Neural operators have emerged as a powerful tool for learning mappings between infinite-dimensional function spaces. However, their approximation properties in Sobolev norms remain poorly quantified, even though these norms control both function values and derivatives and are the natural metrics for PDE well-posedness, stability, and generalization. We develop a functional-analytic framework for operator learning in Sobolev spaces and connect it to the numerical behavior of Fourier Neural Operators (FNOs) on a prototypical PDE. First, for a continuous nonlinear operator

\mathcal{G}: H^{s}(D)\to H^{t}(D')

with

s > d/2

and inputs restricted to a compact subset of

H^{s}(D)

, we prove that

\mathcal{G}

can be uniformly approximated in

H^{t}

-norm by a neural operator with

\mathcal{O}(\varepsilon^{-d/s})

trainable parameters. This yields an explicit complexity--error relation of the form

\|\mathcal{G}-\mathcal{G}_θ\|_{H^{t}} \lesssim C N^{-s/d}

. We then study the one-dimensional viscous Burgers solution operator

\mathcal{G}: u_{0}\mapsto u(\cdot,1)

on a bounded

H^{1}

-ball and train FNOs with an

H^{1}

-loss. Across a sweep of model sizes, we obtain test

H^{1}

-errors down to

\mathcal{O}(10^{-7})

and relative errors of order

10^{-3}

, with predictions accurately matching both solutions and spatial derivatives on held-out data. A log-log plot of Sobolev error versus parameter count exhibits an approximate power law

\|\mathcal{G}-\mathcal{G}_θ\|_{H^{1}} \approx C N^{-α}

with empirical exponent

α\approx 1.4

, and long-horizon training reveals optimization instabilities in large FNOs, providing quantitative evidence that Sobolev-space approximation theory meaningfully predicts neural-operator scaling behavior.

Abstract

Neural Operators

Explore similar work

Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation

Neural Operators as Efficient Function Interpolators

Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization