Partial differential equations (PDEs) are fundamental for modeling complex physical systems, yet classical numerical solvers face prohibitive computational costs in high-dimensional and multi-scale regimes. While Transformer-based neural operators have emerged as powerful data-driven alternatives, they conventionally treat all discretized spatial points as uniform, independent tokens. This monolithic approach ignores the intrinsic scale separation of physical fields, applying computationally prohibitive global attention that redundantly mixes smooth large-scale dynamics with high-frequency fluctuations. Rethinking Transformers through the lens of complex dynamics, we propose DynFormer, a novel dynamics-informed neural operator. Rather than applying a uniform attention mechanism across all scales, DynFormer explicitly assigns specialized network modules to distinct physical scales. It leverages a Spectral Embedding to isolate low-frequency modes, enabling a Kronecker-structured attention mechanism to efficiently capture large-scale global interactions with reduced complexity. Concurrently, we introduce a Local-Global-Mixing transformation. This module utilizes nonlinear multiplicative frequency mixing to implicitly reconstruct the small-scale, fast-varying turbulent cascades that are slaved to the macroscopic state, without incurring the cost of global attention. Integrating these modules into a hybrid evolutionary architecture ensures robust long-term temporal stability. Extensive memory-aligned evaluations across four PDE benchmarks demonstrate that DynFormer achieves up to a 95% reduction in relative error compared to state-of-the-art baselines, while significantly reducing GPU memory consumption. Our results establish that embedding first-principles physical dynamics into Transformer architectures yields a highly scalable, theoretically grounded blueprint for PDE surrogate modeling.
Figures & tables
Figure 1: Overview of DynFormer’s performance across four PDE benchmarks and baselines’ peak GPU memory consumption.
Figure 2: Illustration of the DynFormer architecture.
Method
Complexity
Standard Self-Attention
O(N4)
Kronecker Attention
O(N3)
Spectral Embedding
O(M2logM)
Table 1: Computational complexity comparison for 2D spatial domains of size N×N .
Figure 3: Performance comparison of neural operator architectures across diverse PDE benchmarks. (a) Aggregated Log-Min-Max normalized scores (0–100) averaged over all benchmarks, model variants (Tiny/Medium/Large), and random seeds, highlighting DynFormer’s substantial lead over state-of-the-art baselines. (b) Per-benchmark breakdown showing DynFormer’s dominance across 1D Kuramoto-Sivashinsky, 2D Darcy, 2D Navier-Stokes, and 3D Shallow Water equations.
Figure 4: Qualitative visualization and error analysis on the 2D Navier-Stokes benchmark. Top row: Predicted vorticity fields compared against Ground Truth. DynFormer captures fine-scale turbulent structures with high fidelity, whereas baselines exhibit smoothing or artifacts. Bottom left: Error evolution over simulation timesteps, demonstrating DynFormer’s stability. Bottom right: Mean Squared Error (MSE) comparison, where DynFormer achieves an order-of-magnitude reduction compared to competing methods.
Figure 5: Computational efficiency and memory–performance trade-offs. (a) Peak GPU memory consumption (MB) for Tiny, Medium, and Large variants. All baselines are built with comparable consumptions. (b) Score per Memory ratio, illustrating DynFormer’s superior efficiency by reaching 9.6. (a) Scalability analysis across Tiny, Medium, and Large model variants. DynFormer consistently achieves higher performance scores for equivalent or lower memory usage, with its Tiny variant outperforming the Large variants of several baselines.
1D
2D
3D
Model
Sz
Kuramoto-Sivashinsky
Darcy
Navier-Stokes
Shallow Water
Mem
Params
Loss
Time
Mem
Params
Loss
Time
Mem
Params
Loss
Time
Mem
Params
Loss
Time
FactFormer [ 31 ]
T
0.8
6.6K
5.12e-01
2.0
0.7
38.9K
8.12\times10−04
0.4
7.6
25.3K
1.85e-01
12.3
4.1
25.6K
1.24e-01
13.3
M
2.9
51.4K
3.74e-01
3.6
2.9
476.1K
1.40\times10−04
1.0
14.8
99.2K
1.52e-01
13.5
7.7
99.9K
5.58\times10−02
13.4
L
12.3
638.0K
3.38\times10−01
6.8
13.0
7.3M
7.67\times10−05
4.8
23.7
202.6K
1.41\times10−01
16.1
12.3
203.3K
2.66\times10−02
15.1
GNOT [ 18 ]
T
0.9
14.1K
1.00e+00
2.5
0.8
8.0K
7.60e-02
0.5
7.5
3.8K
9.96e-01
11.1
3.9
3.9K
5.94e-01
12.1
Table 2: Comprehensive performance comparison across PDE benchmarks. Metrics reported per model size variant (Tiny/Medium/Large): peak GPU memory usage (GB), relative MSE loss on test dataset, and averaged training time (seconds) per epoch. The Improvement row indicates the percentage reduction in error achieved by DynFormer compared to the previous state-of-the-art (SOTA). Bold , blue , and underline indicate the 1 st , 2 nd , and 3 rd best results among competitor models. DynFormer results are highlighted with gray background.
2D Darcy
2D NS
Variant
Mem
Params
Loss
Time
Mem
Params
Loss
Time
KS Attention
13.1
3.0M
1.68e-04
3.2
21.9
98.2K
5.60e-02
16.9
Classical Attention
17.0
1.3K
1.25e-01
2.4
57.5
49.0B
4.07e-01
9.8
Linear Attention
8.7
857.2K
3.25e-04
3.7
22.2
46.7K
1.93e-01
29.0
Table 3: Ablation Experiment 1 (E1): Attention Mechanisms. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.
2D Darcy
2D NS
Variant
Mem
Params
Loss
Time
Mem
Params
Loss
Time
Scale Decomp. w/ Spectral
13.1
3.0M
1.68e-04
3.2
21.9
98.2K
5.60e-02
16.9
Scale Decomp. w/ Physical
10.7
3.1M
7.60e-05
2.8
21.7
102.2K
1.54e-01
15.4
Table 4: Ablation Experiment 2 (E2): Decomposition Strategy. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.
2D Darcy
2D NS
Variant
Mem
Params
Loss
Time
Mem
Params
Loss
Time
Mixing
13.1
3.0M
1.68e-04
3.2
21.9
98.2K
5.60e-02
16.9
Adding
14.1
3.8M
1.81e-04
3.8
23.8
152.1K
6.02e-02
18.0
Global Only
14.1
3.6M
2.10e-04
3.7
23.8
145.6K
6.50e-02
17.9
Table 5: Ablation Experiment 3 (E3): Connection Method. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss. ’Global Only’ represents the alone applied KS Attention without the mixing by a local transformation.
2D Darcy
2D NS
Variant
Mem
Params
Loss
Time
Mem
Params
Loss
Time
Scale Decomp. w/ Mixing
13.1
3.0M
1.68e-04
3.2
21.9
98.2K
5.60e-02
16.9
No Scale Decomp. w/ Global Only
13.8
3.6M
2.17e-04
3.7
23.6
145.6K
3.11e-01
18.6
No Scale Decomp. w/ Mixing
13.1
3.0M
1.31e-04
3.2
22.2
98.2K
5.76e-02
17.4
Table 6: Ablation Experiment 4 (E4): Mixing Strategies. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.
2D Darcy
2D NS
Variant
Mem
Params
Loss
Time
Mem
Params
Loss
Time
Hybrid
13.1
3.0M
1.68e-04
3.2
21.9
98.2K
5.60e-02
16.9
Parallel
12.3
3.0M
1.67e-04
3.1
20.1
98.2K
7.89e-02
15.7
Sequential
14.0
3.8M
1.51e-04
3.6
20.8
98.1K
6.02e-02
14.4
Table 7: Ablation Experiment 5 (E5): Execution Flow. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Shape
Split
Operator
Mapping
(C,T,S)
( Ntr/Nte )
Type
Task
1D KS
(1,10,256)
1000/200
Evolutionary
Sol → Sol
2D Navier-Stokes
(1,10,64×64)
3D Shallow Water
(2,10,64×32)
2D Darcy
(1,1,49×49)
1000/200
Solution
Param → Sol
Appendix
Table 1: Summary of benchmark datasets and experimental configurations. Tensor shapes follow (C,T,S) where C : channels, T : temporal steps, and S : spatial resolution. Ntrain/Ntest denotes the dataset split.
Dataset
Hidden Dim.
Depth
Heads
Psi Dim.
T
M
L
T
M
L
T
M
L
T
M
L
1D Experiments
1dks
32
64
128
2
4
8
2
4
8
16
32
64
2D Experiments
2ddarcy
36
64
128
2
4
8
2
4
8
16
32
64
2dns
20
24
32
2
3
4
2
3
4
8
12
12
Appendix
Table 2: Hyperparameter settings for ONO. Model sizes are denoted as T (Tiny), M (Medium), and L (Large). Batch size was fixed at 64 across all configurations; gradient clipping was disabled.
Benchmark
Tiny
Medium
Large
P (K)
M (MiB)
F (M)
P (K)
M (MiB)
F (M)
P (M)
M (MiB)
F (G)
1D Ks
21.1
832
53.8
152.1
3024
389.4
1.2
13210
3.0
2D Darcy
25.6
936
61.5
151.0
3050
362.5
1.2
11328
2.8
2D Ns
8.5
8120
348.2
17.3
13950
708.8
0.04
23810
1.6
3D Sw
8.9
4422
182.7
17.8
7530
364.7
0.04
12272
0.81
Appendix
Table 3: Comprehensive model statistics for ONO. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).
Dataset
Hidden Dim.
Depth
Heads
Slice Num.
Ref
T
M
L
T
M
L
T
M
L
T
M
L
1D Experiments
1dks
40
64
128
2
4
8
2
4
8
16
32
64
8
2D Experiments
2ddarcy
36
56
128
2
4
8
2
4
8
16
32
64
8
2dns
20
24
32
2
3
4
2
3
4
16
20
24
8
Appendix
Table 4: Hyperparameter settings for Transolver. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).
Benchmark
Tiny
Medium
Large
P (K)
M (MiB)
F (M)
P (K)
M (MiB)
F (M)
P (M)
M (MiB)
F (G)
1D Ks
80.4
842
202.1
369.7
2668
961.4
2.8
13408
7.4
2D Darcy
65.1
810
153.2
283.6
2534
691.2
2.8
12238
6.9
2D Ns
22.0
7950
893.5
42.9
13866
1800
0.10
23492
4.0
3D Sw
22.4
4662
455.5
43.4
7158
904.9
0.10
12122
2.0
Appendix
Table 5: Comprehensive model statistics for Transolver. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).
Dataset
Hidden Dim.
Depth
Heads
Kernel Multiplier
Latent Multiplier
T
M
L
T
M
L
T
M
L
1D Experiments
1dks
16
32
80
3
4
8
2
4
8
2
2
2D Experiments
2ddarcy
40
80
272
2
6
8
2
4
8
2
2
2dns
32
64
64
2
2
4
2
2
4
2
2
Appendix
Table 6: Hyperparameter settings for FactFormer. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).
Benchmark
Tiny
Medium
Large
P (K)
M (MiB)
F (M)
P (K)
M (MiB)
F (M)
P (M)
M (MiB)
F (G)
1D Ks
6.6
798
12.2
51.4
2858
94.2
0.64
12256
1.2
2D Darcy
38.9
746
40.1
476.1
2874
505.7
7.3
12892
7.8
2D Ns
25.3
7578
450.4
99.2
14774
1700
0.20
23706
3.6
3D Sw
25.6
4020
234.4
99.9
7674
891.5
0.20
12252
1.8
Appendix
Table 7: Comprehensive model statistics for FactFormer. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).
Dataset
Hidden Dim.
Depth
Heads
Scale
T
M
L
T
M
L
T
M
L
1D Experiments
1dks
8
20
64
2
4
8
2
5
8
8
2D Experiments
2ddarcy
8
20
72
2
4
8
2
5
9
8
2dns
4
8
12
2
3
4
1
2
3
8
Appendix
Table 8: Hyperparameter settings for OFormer. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).
Benchmark
Tiny
Medium
Large
P (K)
M (MiB)
F (M)
P (K)
M (MiB)
F (M)
P (M)
M (MiB)
F (G)
1D Ks
11.6
904
31.2
92.0
2698
239.9
1.4
11224
3.5
2D Darcy
11.6
876
29.1
91.9
2588
224.6
1.7
12290
4.1
2D Ns
3.0
7098
134.7
11.6
13618
498.7
0.03
24940
1.4
3D Sw
3.1
3720
68.2
11.7
6988
251.0
0.03
12582
0.71
Appendix
Table 9: Comprehensive model statistics for OFormer. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).
Dataset
Hidden Dim.
Depth
MLP Layers
Heads
N Experts
N Inner
T
M
L
T
M
L
T
M
L
T
M
L
1D Experiments
1dks
16
24
72
1
2
4
1
2
3
2
4
8
4
2
2D Experiments
2ddarcy
12
32
64
1
2
4
1
2
3
2
4
8
4
2
2dns
8
8
10
1
2
3
1
2
3
1
2
2
4
2
Appendix
Table 10: Hyperparameter settings for GNOT. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).
Benchmark
Tiny
Medium
Large
P (K)
M (MiB)
F (M)
P (K)
M (MiB)
F (M)
P (M)
M (MiB)
F (G)
1D Ks
14.1
884
34.6
59.2
3008
147.8
0.99
14630
2.5
2D Darcy
8.0
804
18.3
103.5
3142
243.7
0.79
11544
1.9
2D Ns
3.8
7504
145.8
7.3
13988
279.5
0.02
23816
0.63
3D Sw
3.9
3922
74.7
7.4
6972
141.6
0.02
12280
0.32
Appendix
Table 11: Comprehensive model statistics for GNOT. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).
Dataset
Hidden Dim.
Depth
Heads
Spectral Modes
T
M
L
T
M
L
T
M
L
1D Experiments
1dks
4
8
32
2
3
6
2
4
8
[64, 1]
2D Experiments
2ddarcy
24
32
128
2
6
8
2
4
8
[12, 12]
2dns
16
32
32
2
2
4
2
2
4
[12, 12]
Appendix
Table 12: Hyperparameter settings for DynFormer. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).
Benchmark
Tiny
Medium
Large
P (K)
M (MiB)
F (M)
P (K)
M (MiB)
F (M)
P (M)
M (MiB)
F (G)
1D Ks
1.2
962
2.1
5.9
2776
10.3
0.16
12848
0.29
2D Darcy
32.6
726
32.7
162.9
2362
151.8
3.4
12676
3.1
2D Ns
15.1
6894
263.6
57.9
13206
999.1
0.11
24578
1.8
3D Sw
7.0
3184
51.9
39.7
7068
274.7
0.08
11850
0.55
Appendix
Table 13: Comprehensive model statistics for DynFormer. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).
Figure 1: Visual comparison on 1D Kuramoto-Sivashinsky for the first sample. Each panel displays the evolution of the field u(x,t) over space (horizontal axis) and time (vertical axis).
Figure 2: Visual comparison on 1D Kuramoto-Sivashinsky for the last sample. Each panel displays the evolution of the field u(x,t) over space (horizontal axis) and time (vertical axis).
Figure 3: Visual comparison on 2D Darcy for the first sample. Each panel displays a 2D scalar field (e.g., pressure or concentration) over a square domain
Figure 4: Visual comparison on 2D Darcy for the last sample. Each panel displays a 2D scalar field (e.g., pressure or concentration) over a square domain
Figure 5: Visual comparison on 2D Navier-Stokes for the first sample at the final step. The vorticity variable ω is presented here.
Figure 6: Visual comparison on 2D Navier-Stokes for the last sample at the final step. The vorticity variable ω is presented here.
Figure 7: Visual comparison on 3D Shallow Water for the first sample at the final step. The vorticity variable ω is presented here.
Figure 8: Visual comparison on 3D Shallow Water for the last sample at the final step. The vorticity variable ω is presented here.
Transformer architectures have attracted increasing attention for solving partial differential equations (PDEs), owing to their flexibility in handling irregular discretizations and their ability to capture long-range physical dependencies. However, unlike discrete language tokens or fixed-resolution image patches, observed physical fields are finite samples of underlying infinite-dimensional functions. Consequently, effectively applying Transformers to PDEs requires a tokenizer that respects the functional nature of physical fields and constructs physically expressive tokens from arbitrary discretizations.To this end, we propose \methodname{Physics Transformer}, a function-projection-based Transformer architecture for physical field prediction. Physics Transformer treats a physical field as a continuous function and partitions its discretization into locality-preserving spatial patches. Within each patch, it dynamically learns a set of adaptive local basis functions and projects the sampled field onto these bases to obtain compact physics tokens. The resulting tokens capture diverse latent physical states while preserving fine-scale spatial structures, enabling efficient global interaction through factorized attention across space and physical states. The projected representation further supports efficient decoding at arbitrary query locations. Extensive experiments on diverse benchmarks, ranging from two-dimensional PDE dynamics to industrial-scale three-dimensional CFD simulations, demonstrate that Physics Transformer accurately captures fine-grained physical structures and achieves state-of-the-art predictive performance. These results establish function projection as a practical and effective foundation for designing Transformer architectures for PDE solving.
Guoze Sun, Rui Zhang, Jiankai Tang +4
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Mechanics and Engineering Science, Peking University, Beijing, China
We study the problem of architecture selection for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators. We introduce the Multi-Scale Attention Transformer (\msat{}), a deep learning architecture that encodes spatiotemporal solution histories as token sequences and trains end-to-end via a composite supervised objective with optional physics-informed regularization terms. We conduct a comprehensive empirical evaluation against nine baselines -- including physics-informed neural networks (PINNs), neural operators (FNO, DeepONet, GNOT), and state-space models (Mamba-NO) -- across five benchmark problems from the PINNacle suite, using identical train/test splits and reference data for all methods. \msat{} achieves state-of-the-art generalization on complex geometry problems (Lrel2=0.0101 on Heat2D-CG, a 3.7× improvement over FNO) at 34s total inference vs.\ 120,812s for Mamba-NO. Ablation studies over the physics regularization component reveal a precise inductive bias tradeoff: physics priors reduce test error on diffusion-dominated problems but degrade generalization on chaotic and recirculating-flow regimes, directly characterizing the prior misspecification boundary. Approximation error bounds as a function of domain boundary complexity κ provide a theoretical basis for these empirical findings and a principled rule for architecture selection.
Conventional patchified Transformers operate on uniform spatial partitions, distributing computational effort evenly across the domain irrespective of local features. This inflexible tokenization scheme is inherently limited in its ability to efficiently represent and process solutions to complex PDEs. To address this, we propose MeshTok, an adaptive mesh refinement (AMR)-inspired tokenization and sequence modeling framework. This method selectively refines spatial regions exhibiting sharp gradients, transient features, or multiscale structures, generating a heterogeneous set of multiscale tokens defined on a fixed simulation grid. These tokens are processed within a unified Transformer sequence, enabling the model to simultaneously capture coarse-grained global context and fine-grained local details without requiring specialized architectural components. Although adaptive refinement moderately increases token count, it promotes a more targeted allocation of computational resources to physically informative regions, which we view as a practical inductive bias rather than a formal optimality guarantee. Experimental evaluations across multiple PDE families and benchmark datasets demonstrate that MeshTok consistently improves the efficiency-accuracy trade-off compared to uniform-grid baselines. This suggests adaptive multiscale tokenization as a scalable and generalizable design principle for neural PDE modeling. Code is available at https://github.com/SCAILab-USTC/MeshTok.
Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang +2
School of Mathematical Sciences, University of Science and Technology of China, Hefei 230026, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou 215123, China.