Organizations: School of Mathematical Sciences, Eastern Institute of Technology, Ningbo, 315200 Ningbo, China · School of Mathematics and Statistics, Wuhan University, 430072 Wuhan, China · Center for Applied Mathematics and KL-AAGDM, Tianjin University, 300072 Tianjin, China · State Key Laboratory of Synthetic Biology, Tianjin University, Tianjin, 300072, China · Computational Sciences Hubei Key Laboratory, Wuhan University, 430072 Wuhan, China
Convolution integrals widely exist in applications, and to enable fast and accurate computations, this paper introduces two general multi-stage neural operator learning frameworks. The first, Deep Collocation Neural Operator (DCNO), is a supervised approach that iteratively refines the operator approximation by learning residuals from input-output data pairs. The second, Deep Galerkin Neural Operator (DGNO), is an unsupervised framework applicable when the target operator can be represented by a PDE, leveraging the weak form of the PDE residual for training. Both methods progressively construct basis operators through multiple training stages to enrich the approximation space, leading to significantly improved accuracy over standard one-shot operator learning. We provide theoretical analysis for their approximation capabilities and implement them for learning convolutions. Extensive numerical experiments demonstrate that both DCNO and DGNO achieve high accuracy, approaching machine precision under single float for convolution problems, and offer substantial efficiency gains for numerous queries or parametric variations compared to traditional solvers. We also extend these frameworks to handle multi-input operator learning scenarios involving variations in both the density and kernel of a convolution.
Figures & tables
Figure 1: Results of DCNO in Algorithm 1 for Example 5.1 : (a) Change of Loss =βs−1Losss during the training; (b) Mean relative L2 error with respect to the number of stages on test set.
DCNO
DGNO
W0=10
W0=30
W0=50
W0=70
W0=10
W0=30
W0=50
W0=70
L=1
1.75 e−3
1.81 e−3
1.20 e−3
5.37 e−4
2.99 e−3
2.15 e−3
1.27 e−3
6.70 e−4
L=2
1.94 e−5
1.80 e−5
1.15 e−5
9.46 e−6
5.80 e−5
3.17 e−5
2.38 e−5
1.62 e−5
L=3
4.01 e−6
3.17 e−6
1.48 e−6
8.29 e−7
1.68 e−5
6.03 e−6
3.47 e−6
9.20 e−7
L=4
1.61 e−6
1.24 e−6
7.04 e−7
5.47 e−7
1.43 e−5
3.10 e−6
1.89 e−6
6.17 e−7
Table 1: Relative L2 -errors obtained by DCNO and DGNO under different network architectures.
Figure 2: Numerical solution (1st row) and pointwise error (2nd row) of DCNO for Example 5.1 on three test samples (a) α=1 ; (b) α=3 ; (c) α=5 .
Figure 3: Basis functions generated by the basis operators of DCNO method.
Figure 4: Relative L2 error of DCNO under varying noise levels (left); A Gaussian noise sample with noise level 0.001 (middle); Pointwise error of DCNO for Example 5.1 on a test sample with α=1 (right).
Figure 5: Results of DGNO in Algorithm 2 for Example 5.1 : (a) Change of Loss during the iterations; (b) Mean relative L2 error with respect to the number of stages.
Stages
Epochs per stage
Training time(s)
Mean relative L2error
supervised DCNO
5
3000
251.23s
1.31E-6
unsupervised DGNO
5
3000
491.60s
6.17E-7
Table 2: Number of stages, epochs per stage, training time and error for DCNO and DGNO.
Figure 6: Solutions for different values of the kernel parameter a (left); Mean relative L2 error with respect to the number of stages for a=0.0001 (right).
Figure 7: Results of DCNO for Example 5.2 : (a) Exact and numerical solutions; (b) Error of the numerical solution; (c) Computational time of the traditional solver and DCNO against the number of computational cases.
a
0.0001
0.001
0.001
0.1
Error
1.20 e−6
3.08 e−6
2.71 e−6
3.48 e−6
Table 3: Mean relative L2 error of numerical solutions from DCNO for varying a .
Method
Evaluation time (s)
Training time (s)
Traditional method
670.44
—
DCNO method
0.021
202.14
Table 4: Comparison between DCNO and the traditional method SOE on test set.
Figure 8: Results of Example 5.3 : (a) Relative L2 error of ψ over time; (b) Total evaluation time of the traditional solver and DGNO for solving the Poisson equation ( 5.3b ) during the simulation; (c) Evolution of the mass over time.
Figure 9: ∣ψ(⋅,t=0)∣2 (left); ∣ψ(⋅,t=1)∣2 by DGNO (middle); Pointwise absolute error at t=1 (right) for Example 5.3 .
Figure 10: Mean relative L2 error with respect to the number of stages: (a) Example 5.4 ; (b) Example 5.5 .
Figure 11: Numerical solution (1st row) and pointwise error (2nd row) of DCNO for Example 5.4 on three test samples: (a) α=τ=2 ; (b) α=τ=1.5 ; (c) α=τ=1 .
Figure 12: Numerical solution (1st row) and pointwise error (2nd row) of DGNO for Example 5.5 on three test samples: (a) α=0.5,γ=2 ; (b) α=0.6,γ=3 ; (c) α=0.7,γ=4 .
Learning mappings between infinite-dimensional objects is a central challenge in scientific machine learning. We introduce a general kernel-based encoder-decoder framework for operator learning that separates observation, representation, learning, and reconstruction. We develop this framework for multi-input, multi-output operator learning, where operators map between products of potentially distinct function spaces. Our approximation theory shows that, although the number of inputs and outputs can increase, the convergence rate is governed by the most challenging constituent approximation problem rather than the overall problem dimension. The framework leads to practical kernel methods with closed-form training and inference, combining mathematical tractability with computational efficiency. We further specialize the approach to multiple operator learning by introducing KernelMO, a family of kernel methods with complementary operator-valued and product-space formulations. Across five families of parametric partial differential equations, the proposed methods achieve competitive or state-of-the-art predictive accuracy while reducing training and inference costs relative to neural operator architectures and deep learning based models, offering an efficient and lightweight alternative.
Adrien Weihs, Chunyang Liao, Jingmin Sun +1
Department of Mathematics, University of California Los Angeles, Los Angeles, CA 90095, USA. · Department of Mathematical Sciences, University of Arkansas, Fayetteville, AR, 72701, USA. · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21218, USA.
The convolution operator is the fundamental building block of modern convolutional neural networks (CNNs), owing to its simplicity, translational equivariance, and efficient implementation. However, its structure as a fixed, linear, locally-averaging operator limits its ability to capture structured signal properties such as low-rank decompositions, adaptive basis representations, and non-uniform spatial dependencies. This paper presents a systematic taxonomy of operators that extend or replace the standard convolution in learning-based image processing pipelines. We organise the landscape of alternative operators into five families: (i) decomposition-based operators, which separate structural and noise components through singular value or tensor decompositions; (ii) adaptive weighted operators, which modulate kernel contributions as a function of spatial position or signal content; (iii) basis-adaptive operators, which optimise the analysis bases together with the network weights; (iv) integral and kernel operators, which generalise the convolution to position-dependent and non-linear kernels; and (v) attention-based operators, which relax the locality assumption entirely. For each family, we provide a formal definition, a discussion of its structural properties with respect to the convolution, and a critical analysis of the tasks for which the operator is most appropriate. We further provide a comparative analysis of all families across relevant dimensions -- linearity, locality, equivariance, computational cost, and suitability for image-to-image and image-to-label tasks -- and outline the open challenges and future directions of this research area.
Operator learning is an emerging field at the intersection of machine learning and scientific computing. By learning mappings between function spaces, neural operators provide data-driven surrogate models for families of partial differential equations (PDEs). Once trained, these models can evaluate solution operators efficiently, making them suitable for many-query applications such as real-time prediction and parameter sweeps. However, maintaining high approximation accuracy and stable long-term predictions remains challenging for complex forward and inverse problems. To address these challenges, we propose the Starter-Iterator Neural Operator (SINO), which incorporates the initialization and residual-correction structures of classical iterative solvers into neural operator learning. The frequency-domain Starter captures dominant global spectral features and provides an informed initial approximation, while the latent-space Iterator applies successive residual-based corrections to refine local and multiscale solution structures. Experiments on representative time-dependent PDEs, including the Navier-Stokes and acoustic wave equations, together with applications to image super-resolution and weather forecasting, show that SINO achieves competitive accuracy and stable performance across the benchmarks considered in this work.