SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators

Mahmoud Elhadidy; Roshan M. D'Souza; Amirhossein Arzani

cs.LGMar 20, 2026

SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators

Authors: Mahmoud Elhadidy, Roshan M. D'Souza, Amirhossein Arzani

Organizations: Department of Mechanical Engineering, University of Utah, Salt Lake City, UT, USA · Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, USA · Department of Mechanical Engineering, University of Wisconsin–Milwaukee, Milwaukee, WI, USA · Department of Biomedical Engineering, University of Utah, Salt Lake City, UT, USA

Abstract

Scientific machine learning is increasingly used to build surrogate models, yet most models are trained under a restrictive assumption in which future data follow the same distribution as the training set. In practice, new experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data. This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting. Such challenges are pronounced in fluid dynamics, where changes in geometry, boundary conditions, or flow regimes induce non-trivial changes to the solution. Here, we introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL. SLE-FNO was compared with a range of established CL methods, including Elastic Weight Consolidation (EWC), Learning without Forgetting (LwF), replay-based approaches, Orthogonal Gradient Descent (OGD), Gradient Episodic Memory (GEM), PiggyBack, and Low-Rank Adaptation (LoRA), within a spatial field-to-field regression setting. The models were trained to map transient concentration fields to time-averaged wall shear stress (TAWSS) in pulsatile aneurysmal blood flow. Tasks were derived from 230 computational fluid dynamics simulations grouped into four sequential and out-of-distribution configurations. Results show that replay-based methods and architecture-based approaches (PiggyBack, LoRA, and SLE-FNO) achieve the best retention, with SLE-FNO providing the strongest overall balance between plasticity and stability, achieving accuracy with zero forgetting and minimal additional parameters. Our findings highlight key differences between CL algorithms and introduce SLE-FNO as a promising strategy for adapting baseline models when extrapolation is required.

Figures & tables

Figure 1: (a) Geometry sketch with four spatial parameters $W,D,N,R$ . We fix $D$ and vary three non-dimensional ratios $W/D$ , $N/D$ , and $R/D$ . (b) Polyhedral mesh of the geometry with a zoom on the inflation layers. (c) The transient boundary conditions used. (d) Extracted results: a 2D slice of concentration at five time steps from 3.15 s to 3.95 s (within the last cardiac cycle), and the corresponding TAWSS projected onto the same slice. (e) Mean spatial parameters for each dataset with the associated inlet flow profiles. (f) Table of parameter ranges and flow rates for each dataset/task. (g) Training/finetuning sequence used throughout the study based on the four datasets/tasks.

Task	Train Samples	Test Samples
A	160	40
B, C, D (each)	8	2

Table 1: The number of training and test samples per task.

Figure 2: The overall Fourier neural operator (FNO) architecture. (a) FNO lifts the input to a higher-dimensional channel space through a neural lifting, applies a stack of Fourier operator layers with nonlinear activations, then projects back to the target dimension with a neural projection to obtain the output. (b) Fourier layer overview: Given input in the feature space, the upper branch applies Fast Fourier Transform (FFT) $\mathcal{F}$ , a learned linear map $R$ on the retained low-frequency modes while discarding higher modes, then applying an inverse transform $\mathcal{F}^{-1}$ . The lower branch applies a local linear transform $W$ and acts as a residual connection.

Figure 3: Regularization-based: (a) Elastic Weight Consolidation (EWC) constrains updates using a Fisher-weighted penalty. (b) Learning without Forgetting (LwF) applies distillation from a frozen previous model. Replay-based: (c) Random reservoir sampling. (d) K-means exemplar replay. Optimization-based: (e) Orthogonal Gradient Descent (OGD) projects gradients to avoid interference with past tasks (recreated from [ 32 ] ). (f) Gradient Episodic Memory (GEM) constrains updates to prevent increases in past-task loss. Architecture-based: (g) PiggyBack learns task-specific masks over a frozen backbone. (h) Low-Rank Adaptation (LoRA) trains per-layer low-rank adapters on a frozen backbone.

Figure 4: Single-Layer Extension with Fourier Neural Operator (SLE-FNO). (a) During training, only a single Fourier layer is trainable while the backbone remains frozen. (b) Kernel PCA is used for routing and out-of-distribution (OOD) detection. (c) Complete framework at inference: KPCA modules select the route, and the prediction is based on the frozen backbone and the SLE-FNO layer.

Figure 5: TAWSS results after the first stage (Training on A) for two randomly selected samples from task A. All units are in dynes/cm 2 .

Method	Stage	A	B	C	D	Avg. Acc.
Baseline	A	0.985 $\pm$ 0.000	-	-	-	0.985
	B	0.663 $\pm$ 0.004	0.910 $\pm$ 0.027	-	-	0.787
	C	0.641 $\pm$ 0.011	0.718 $\pm$ 0.028	0.794 $\pm$ 0.088	-	0.718
	D	0.572 $\pm$ 0.018	0.265 $\pm$ 0.079	0.192 $\pm$ 0.103	0.864 $\pm$ 0.032	0.473
SLE-FNO	A	0.985 $\pm$ 0.000	-	-	-	0.985
	B	0.985 $\pm$ 0.000	0.898 $\pm$ 0.024	-	-	0.942

Table 2: Stage-wise accuracies and average accuracy, as defined in Eq. ( 37 ), for each method. Results are reported as mean

\pm

standard deviation. Best values per stage are highlighted in bold. Each row (A–D) corresponds to a training stage, while each column represents the task accuracy after sequentially fine-tuning the model up to that task.

Method	Stage	$F_{A}$	$F_{B}$	$F_{C}$	$F_{D}$	MeanF
Baseline	B	0.322 $\pm$ 0.004	-	-	-	0.322
	C	0.343 $\pm$ 0.012	0.192 $\pm$ 0.029	-	-	0.268
	D	0.412 $\pm$ 0.026	0.635 $\pm$ 0.079	0.579 $\pm$ 0.127	-	0.542
SLE-FNO	B	0	-	-	-	0
	C	0	0	-	-	0
	D	0	0	0	-	0

Table 3: Forgetting measure

F

(lower is better), defined in Eq. ( 38 ), and the average forgetting measure (MeanF), defined in Eq. ( 39 ), are reported for each method and stage. Results are presented as mean

\pm

standard deviation. MeanF is computed using the mean forgetting values only. Each row (B, C, D) corresponds to a training stage after sequential fine-tuning up to that task.

Figure 6: TAWSS results at the second stage (Fine-Tuning at task B). (a) Results for the worst-performing sample from task B. (b) Results for a representative sample from task A. All units are in dynes/cm 2 .

Figure 7: TAWSS results at the third stage (Fine-Tuning on C). (a) Worst-case sample from task C. (b) Worst-case sample from task B. All units are in dynes/cm 2 .

Figure 8: TAWSS results at the third stage (Fine-Tuning on C). A random sample from task A is shown. All units are in dynes/cm 2 .

Figure 9: TAWSS results at the fourth stage (Fine-Tuning on task D). (a) Worst-case sample from task D. (b) Worst-case sample from task C. All units are in dynes/cm 2 .

Figure 10: TAWSS results at the fourth stage (Fine-Tuning on task D). (a) Worst-case sample from task B. (b) Random sample from task A. All units are in dynes/cm 2 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Task A routing accuracy (%) shown as the mean (solid line) with a shaded region indicating $\pm 1$ standard deviation, plotted as a function of the number of retained KPCA modes. The x-axis represents the number of retained modes, while the y-axis reports the routing accuracy on the Task A test set.

Branch	Score $\uparrow$
Task A
Backbone	0.985
SLE-B	0.637 $\pm$ 0.011
SLE-C	0.464 $\pm$ 0.033
SLE-D	0.551 $\pm$ 0.017
Ensemble	0.665 $\pm$ 0.006

Appendix

Table 5: Performance accuracy without distribution detector. Each entry is reported as mean

\pm

std (higher is better).

Stage	Pool	A	B	C	D	Avg. Acc.
B	$\{A,B\}$	$0.985\pm 0.000$ (100%)	$0.898\pm 0.033$ (100%)	—	—	0.942
C	$\{A,B,C\}$	$0.985\pm 0.000$ (100%)	$0.898\pm 0.033$ (100%)	$0.789\pm 0.050$ (100%)	—	0.891
D	$\{A,B,C,D\}$	$0.985\pm 0.000$ (100%)	$0.898\pm 0.033$ (100%)	$0.789\pm 0.050$ (100%)	$0.778\pm 0.054$ (80%)	0.863

Appendix

Table 6: End-to-end inference accuracy (mean

\pm

std over 10 runs) with the KPCA selector active during inference. Routing accuracy per group across all runs is shown in parentheses. A dash indicates the corrector is not yet available at that stage.

Tier	B	C	D
Small	0.875 $\pm$ 0.020	0.652 $\pm$ 0.096	0.759 $\pm$ 0.056
Medium	0.887 $\pm$ 0.015	0.766 $\pm$ 0.043	0.801 $\pm$ 0.021
Large	0.905 $\pm$ 0.021	0.748 $\pm$ 0.066	0.841 $\pm$ 0.034

Appendix

Table 7: Ablation study on SLE layer size. Performance accuracy is reported at the final step using transformed accuracy (higher is better).

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Replay-Based Continual Learning for Physics-Informed Neural Operators
May 6, 2026Yizheng Wang, Mohammad Sadegh Eshaghi, Xiaoying Zhuang +2OOD Generalization Replay-Based Continual Learning
Department of Engineering Mechanics, Tsinghua University, Beijing 100084, China · Institute of Photonics, Department of Mathematics and Physics, Leibniz University Hannover, Germany · Institute of Structural Mechanics, Bauhaus-Universität Weimar, Marienstr. 15, D-99423 Weimar, Germany
Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary data
Jun 9, 2026Masoume Gholizade, Fabrizio Ruffini, Pietro Ducange +1Continual Learning Federated Continual Learning
Department of Information Engineering, University of Pisa, Largo Lucio Lazzarino 1, Pisa, 56122, Italy
NORACL: Neurogenesis for Oracle-free Resource-Adaptive Continual Learning
Apr 29, 2026Karthik Charan Raghunathan, Christian Metzner, Laura Kriener +1Stability-Plasticity Dilemma Dynamic Neural Networks
Institute of Neuroinformatics University of Zurich & ETH Zurich Zurich, Switzerland

May 6, 2026J/K move · Enter open · S save

Replay-Based Continual Learning for Physics-Informed Neural Operators

Yizheng Wang, Mohammad Sadegh Eshaghi, Xiaoying Zhuang, Timon Rabczuk+1

Department of Engineering Mechanics, Tsinghua University, Beijing 100084, China · Institute of Photonics, Department of Mathematics and Physics, Leibniz University Hannover, Germany · Institute of Structural Mechanics, Bauhaus-Universität Weimar, Marienstr. 15, D-99423 Weimar, Germany

OOD Generalization Replay-Based Continual Learning Physics-Informed Neural Operators Neural Operators

Open paper AskPDF arXiv

Neural operators generally demonstrate strong predictive performance on in-distribution (ID) problems. However, a critical limitation of existing methods is their significant performance degradation when encountering out-of-distribution (OOD) data. To address this issue, this work introduces continual learning into physics-informed neural operators, with particular emphasis on neural operators built upon the Transolver architecture, and proposes a simple yet effective replay-based continual learning strategy. The proposed method is fully physics-informed and does not require labeled data, relying solely on input fields together with physical constraints for training. When new OOD data become available, a small number of past data are incorporated through a distillation-based constraint to preserve previously acquired knowledge and alleviate catastrophic forgetting. Meanwhile, a transfer learning LoRA is employed to enable rapid adaptation to the new data. The proposed framework is systematically validated on three representative physical problems, including the Darcy flow problem in fluid mechanics, a two-dimensional hyperelastic brain tumor problem in biomechanics, and a three-dimensional linear elastic Triply Periodic Minimal Surfaces problem in solid mechanics. The results demonstrate that the proposed method effectively mitigates catastrophic forgetting on previously learned data while maintaining fast adaptability to new data. Compared with conventional joint training strategies, the proposed method significantly improves training efficiency while reducing additional memory usage and computational cost.

Abstract

Continual Learning

Explore similar work

Replay-Based Continual Learning for Physics-Informed Neural Operators

Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary data

NORACL: Neurogenesis for Oracle-free Resource-Adaptive Continual Learning