SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining

Dengdi Sun; Xiaoya Zhou; Xiao Wang; Wanli Lyu; Jin Tang; Bin Luo

cs.LGOct 2, 2026

SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining

Authors: Dengdi Sun, Xiaoya Zhou, Xiao Wang, Wanli Lyu, Jin Tang, Bin Luo

Organizations: School of Artificial Intelligence, Anhui University, Hefei 230601, China · School of Computer Science and Technology, Anhui University, Hefei 230601, China

Abstract

Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still struggle with heterogeneous dynamics, where shared representations may cause knowledge interference, while mixture-of-experts (MoE) architectures suffer from increasing expert redundancy. We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR decouples latent features into low- and high-frequency components, enabling shared modeling of transferable dynamics and specialized learning of PDE-specific patterns. To address expert redundancy, we design a knowledge-guided expert aggregation strategy that measures expert similarity from dataset-specific learned knowledge and routing preferences, enabling the identification and consolidation of similar experts. Experiments on twelve PDE datasets and multiple downstream benchmarks demonstrate superior performance in pre-training, fine-tuning, and transfer learning. Furthermore, our aggregation strategy reduces the number of experts by 50% while maintaining or improving prediction accuracy, achieving a balance between model efficiency and generalization for PDE foundation models.

Figures & tables

Figure 1: Motivation of SPEAR. The upper shows the advantage of frequency decoupling for separating shared and PDE-specific dynamics. The lower highlights knowledge-guided expert merging over routing-based merging.

Figure 2: Overview architecture. The left panel shows the overall pipeline, the upper-right panel presents the FreqMix block, and the lower-right panel illustrates the high-frequency MoE module with low-rank experts.

Figure 3: Knowledge-guided expert merging based on bilinear projection and hierarchical clustering.

	L2RE	Activated	FNO- $\nu$			PDEBench CNS-( $\eta,\zeta$ ), DR, SWE								PDEArena		CFDBench
	Model	Params	1e-5	1e-4	1e-3	1,0.1	1,0.01	Avg.(1)	0.1,0.1	0.1,0.01	Avg.(0.1)	DR	SWE	NS	NS-cond	-
Small model	FNO	0.5M	0.116	0.0922	0.0156	0.151	0.108	0.130	0.230	0.076	0.153	0.0321	0.0091	0.210	0.384	0.0274
	UNet	25M	0.198	0.119	0.0245	0.334	0.291	0.313	0.569	0.357	0.463	0.0971	0.0521	0.102	0.337	0.209
	FFNO	1.3M	0.121	0.0503	0.0099	0.0212	0.0520	0.0366	0.162	0.0452	0.104	0.0571	0.0116	0.0839	0.602	0.0071
	GK-T	1.6M	0.134	0.0792	0.0098	0.0341	0.0377	0.0359	0.0274	0.0366	0.0320	0.0359	0.0069	0.0952	0.423	0.0105
	GNOT	1.8M	0.157	0.0443	0.0125	0.0325	0.0420	0.0373	0.0228	0.0341	0.0285	0.0311	0.0068	0.172	0.325	0.0088

Table 1: The experiments are divided into four parts: standard small models, pre-training performance of foundation models, fine-tuning results, and model merging results. The evaluation metric is L2RE. The best result within each part is highlighted in bold , while the overall best result is emphasized in blue bold.

Model	Activated	FNO- $\nu$		PDEBench			CFDBench
L2RE	Params	1e-5	1e-3	0.1,0.01	SWE	DR	-
FNO	10M	0.1910	0.0245	0.0859	11.0000	0.5300	0.0060
DPOT	7M	0.1070	0.0155	0.0168	0.0063	0.0577	0.0067
MoE-POT	17M	0.0682	0.0077	0.0105	0.0064	0.0411	0.0053
NESTOR	13M	0.0674	0.0076	0.0159	0.0045	0.0184	0.0091
Ours	10M	0.0582	0.0051	0.0165	0.0024	0.0145	0.0056

Table 2: Performance comparison of different methods under the new dataset split. The evaluation metric is L2RE. The best result within each column is highlighted in bold .

Model	Activated Params	Wave-Layer	Wave-Gauss	Turbulence
FNO	0.5M	-	-	0.193
MPP	7M	-	-	0.152
DPOT	7M	0.0095	0.0075	0.135
Poseidon	21M	0.29	0.29	-
MoE-POT	17M	0.05	0.06	0.0621
Ours	10M	0.0085	0.0066	0.0662

Table 3: Comparison of generalization and transfer ability.

Figure 4: Visualization of the predicted physical field at the final time step for Wave-Gauss.

Figure 5: Visualization of the predicted physical field at the final time step for Wave-Layer.

Figure 6: Visualization of the predicted physical variables at the final time step for 2D high-resolution turbulence.

Figure 7: Visualization of the key matrices used in Knowledge-Guided expert merging: (1) dataset similarity matrix $S_{D}$ , (2) dataset–expert preference matrix $R$ , and (3) expert similarity matrix $S_{E}$ .

Num. of experts	1e-5	1e-3	0.1, 0.01	SWE	DR	CFDBench	Avg. $\mathcal{L}_{2}$
4	0.0666	0.0054	0.0162	0.0029	0.0193	0.0079	0.0197
8	0.0679	0.0049	0.0170	0.0029	0.0187	0.0083	0.0200
12	0.0690	0.0050	0.0146	0.0026	0.0190	0.0076	0.0196
16	0.0689	0.0048	0.0165	0.0040	0.0192	0.0077	0.0202
20	0.0700	0.0049	0.0170	0.0028	0.0197	0.0092	0.0206
24	0.0693	0.0049	0.0162	0.0031	0.0187	0.0082	0.0201

Table 4: The impact of the number of non-shared experts.

Method	1e-5	1e-3	0.1, 0.01	SWE	DR	CFDBench	Avg. $\mathcal{L}_{2}$	Promotion
Ours	0.0690	0.0050	0.0148	0.0027	0.0196	0.0077	0.0198	-
w/o MoE	0.0709	0.0054	0.0165	0.0031	0.0196	0.0079	0.0206	0.0008
w/o High-Freq	0.1070	0.0155	0.0168	0.0063	0.0577	0.0067	0.0350	0.0152
w/o Aux Loss	0.0694	0.0055	0.0154	0.0027	0.0185	0.0098	0.0202	0.0004
w/o Orth Loss	0.0692	0.0051	0.0168	0.0026	0.0189	0.0083	0.0202	0.0004

Table 5: Ablation experiments of our proposed model. “w/o” denotes the removal of the corresponding component.

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 310 papers

NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training
Feb 25, 2026Dengdi Sun, Xiaoya Zhou, Xiao Wang +4Neural Operators PDE Surrogate Modeling
School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China
AOT-POT: Adaptive Operator Transformation for Large-Scale PDE Pre-training
May 15, 2026Qitan Lv, Hong Wang, Zhongkai Hao +5Transformer PDE Operator Learning
University of Science and Technology of China · 2Shanghai AI Laboratory · Department of Electronic Engineering, Tsinghua University, China
Don't Fix the Basis -- Learn It: Spectral Representation with Adaptive Basis Learning for PDEs
May 11, 2026Xuxiang Zhao, Angelica I. Aviles-RiveroFourier Neural Operator Representation Learning
Qiuzhen College, Tsinghua University · YMSC, Tsinghua University

Feb 25, 2026J/K move · Enter open · S save

NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training

Dengdi Sun, Xiaoya Zhou, Xiao Wang, Hao Si+3

School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China

Neural Operators PDE Surrogate Modeling Mixture-of-Experts Models

Open paper AskPDF arXiv

Neural operators have emerged as an efficient paradigm for solving PDEs, overcoming the limitations of traditional numerical methods and significantly improving computational efficiency. However, due to the diversity and complexity of PDE systems, existing neural operators typically rely on a single network architecture, which limits their capacity to fully capture heterogeneous features and complex system dependencies. This constraint poses a bottleneck for large-scale PDE pre-training based on neural operators. To address these challenges, we propose a large-scale PDE pre-trained neural operator based on a nested Mixture-of-Experts (MoE) framework. In particular, the image-level MoE is designed to capture global dependencies, while the token-level Sub-MoE focuses on local dependencies. Our model can selectively activate the most suitable expert networks for a given input, thereby enhancing generalization and transferability. We conduct large-scale pre-training on twelve PDE datasets from diverse sources and successfully transfer the model to downstream tasks. Extensive experiments demonstrate the effectiveness of our approach.

Abstract

Neural Operators

Explore similar work

NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training

AOT-POT: Adaptive Operator Transformation for Large-Scale PDE Pre-training

Don't Fix the Basis -- Learn It: Spectral Representation with Adaptive Basis Learning for PDEs