From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs

Pengyu Lai; Yixiao Chen; Dewu Yang; Rui Wang; Feng Wang; Hui Xu

cs.LGMar 3, 2026

From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs

Authors: Pengyu Lai, Yixiao Chen, Dewu Yang, Rui Wang, Feng Wang, Hui Xu

Organizations: School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China

Abstract

Partial differential equations (PDEs) are fundamental for modeling complex physical systems, yet classical numerical solvers face prohibitive computational costs in high-dimensional and multi-scale regimes. While Transformer-based neural operators have emerged as powerful data-driven alternatives, they conventionally treat all discretized spatial points as uniform, independent tokens. This monolithic approach ignores the intrinsic scale separation of physical fields, applying computationally prohibitive global attention that redundantly mixes smooth large-scale dynamics with high-frequency fluctuations. Rethinking Transformers through the lens of complex dynamics, we propose DynFormer, a novel dynamics-informed neural operator. Rather than applying a uniform attention mechanism across all scales, DynFormer explicitly assigns specialized network modules to distinct physical scales. It leverages a Spectral Embedding to isolate low-frequency modes, enabling a Kronecker-structured attention mechanism to efficiently capture large-scale global interactions with reduced complexity. Concurrently, we introduce a Local-Global-Mixing transformation. This module utilizes nonlinear multiplicative frequency mixing to implicitly reconstruct the small-scale, fast-varying turbulent cascades that are slaved to the macroscopic state, without incurring the cost of global attention. Integrating these modules into a hybrid evolutionary architecture ensures robust long-term temporal stability. Extensive memory-aligned evaluations across four PDE benchmarks demonstrate that DynFormer achieves up to a 95% reduction in relative error compared to state-of-the-art baselines, while significantly reducing GPU memory consumption. Our results establish that embedding first-principles physical dynamics into Transformer architectures yields a highly scalable, theoretically grounded blueprint for PDE surrogate modeling.

Figures & tables

Figure 1: Overview of DynFormer’s performance across four PDE benchmarks and baselines’ peak GPU memory consumption.

Figure 2: Illustration of the DynFormer architecture.

Method	Complexity
Standard Self-Attention	$\mathcal{O}(N^{4})$
Kronecker Attention	$\mathcal{O}(N^{3})$
Spectral Embedding	$\mathcal{O}(M^{2}\log M)$

Table 1: Computational complexity comparison for 2D spatial domains of size

N\times N

Figure 3: Performance comparison of neural operator architectures across diverse PDE benchmarks. (a) Aggregated Log-Min-Max normalized scores (0–100) averaged over all benchmarks, model variants (Tiny/Medium/Large), and random seeds, highlighting DynFormer’s substantial lead over state-of-the-art baselines. (b) Per-benchmark breakdown showing DynFormer’s dominance across 1D Kuramoto-Sivashinsky, 2D Darcy, 2D Navier-Stokes, and 3D Shallow Water equations.

Figure 4: Qualitative visualization and error analysis on the 2D Navier-Stokes benchmark. Top row: Predicted vorticity fields compared against Ground Truth. DynFormer captures fine-scale turbulent structures with high fidelity, whereas baselines exhibit smoothing or artifacts. Bottom left: Error evolution over simulation timesteps, demonstrating DynFormer’s stability. Bottom right: Mean Squared Error (MSE) comparison, where DynFormer achieves an order-of-magnitude reduction compared to competing methods.

Figure 5: Computational efficiency and memory–performance trade-offs. (a) Peak GPU memory consumption (MB) for Tiny, Medium, and Large variants. All baselines are built with comparable consumptions. (b) Score per Memory ratio, illustrating DynFormer’s superior efficiency by reaching 9.6. (a) Scalability analysis across Tiny, Medium, and Large model variants. DynFormer consistently achieves higher performance scores for equivalent or lower memory usage, with its Tiny variant outperforming the Large variants of several baselines.

Model

Kuramoto-Sivashinsky

Darcy

Navier-Stokes

Shallow Water

Mem

Params

Loss

Time

Mem

Params

Loss

Time

Mem

Params

Loss

Time

Mem

Params

Loss

Time

FactFormer [ 31 ]

0.8

6.6K

5.12e-01

2.0

0.7

38.9K

8.12\text{\times}{10}^{-04}

0.4

7.6

25.3K

1.85e-01

12.3

4.1

25.6K

1.24e-01

13.3

2.9

51.4K

3.74e-01

3.6

2.9

476.1K

1.40\text{\times}{10}^{-04}

1.0

14.8

99.2K

1.52e-01

13.5

7.7

99.9K

5.58\text{\times}{10}^{-02}

13.4

12.3

638.0K

3.38\text{\times}{10}^{-01}

6.8

13.0

7.3M

7.67\text{\times}{10}^{-05}

4.8

23.7

202.6K

1.41\text{\times}{10}^{-01}

16.1

12.3

203.3K

2.66\text{\times}{10}^{-02}

15.1

GNOT [ 18 ]

0.9

14.1K

1.00e+00

2.5

0.8

8.0K

7.60e-02

0.5

7.5

3.8K

9.96e-01

11.1

3.9

3.9K

5.94e-01

12.1

Table 2: Comprehensive performance comparison across PDE benchmarks. Metrics reported per model size variant (Tiny/Medium/Large): peak GPU memory usage (GB), relative MSE loss on test dataset, and averaged training time (seconds) per epoch. The Improvement row indicates the percentage reduction in error achieved by DynFormer compared to the previous state-of-the-art (SOTA). Bold , blue , and underline indicate the 1 st , 2 nd , and 3 rd best results among competitor models. DynFormer results are highlighted with gray background.

	2D Darcy				2D NS
Variant	Mem	Params	Loss	Time	Mem	Params	Loss	Time
KS Attention	13.1	3.0M	1.68e-04	3.2	21.9	98.2K	5.60e-02	16.9
Classical Attention	17.0	1.3K	1.25e-01	2.4	57.5	49.0B	4.07e-01	9.8
Linear Attention	8.7	857.2K	3.25e-04	3.7	22.2	46.7K	1.93e-01	29.0

Table 3: Ablation Experiment 1 (E1): Attention Mechanisms. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.

	2D Darcy				2D NS
Variant	Mem	Params	Loss	Time	Mem	Params	Loss	Time
Scale Decomp. w/ Spectral	13.1	3.0M	1.68e-04	3.2	21.9	98.2K	5.60e-02	16.9
Scale Decomp. w/ Physical	10.7	3.1M	7.60e-05	2.8	21.7	102.2K	1.54e-01	15.4

Table 4: Ablation Experiment 2 (E2): Decomposition Strategy. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.

	2D Darcy				2D NS
Variant	Mem	Params	Loss	Time	Mem	Params	Loss	Time
Mixing	13.1	3.0M	1.68e-04	3.2	21.9	98.2K	5.60e-02	16.9
Adding	14.1	3.8M	1.81e-04	3.8	23.8	152.1K	6.02e-02	18.0
Global Only	14.1	3.6M	2.10e-04	3.7	23.8	145.6K	6.50e-02	17.9

Table 5: Ablation Experiment 3 (E3): Connection Method. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss. ’Global Only’ represents the alone applied KS Attention without the mixing by a local transformation.

	2D Darcy				2D NS
Variant	Mem	Params	Loss	Time	Mem	Params	Loss	Time
Scale Decomp. w/ Mixing	13.1	3.0M	1.68e-04	3.2	21.9	98.2K	5.60e-02	16.9
No Scale Decomp. w/ Global Only	13.8	3.6M	2.17e-04	3.7	23.6	145.6K	3.11e-01	18.6
No Scale Decomp. w/ Mixing	13.1	3.0M	1.31e-04	3.2	22.2	98.2K	5.76e-02	17.4

Table 6: Ablation Experiment 4 (E4): Mixing Strategies. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.

	2D Darcy				2D NS
Variant	Mem	Params	Loss	Time	Mem	Params	Loss	Time
Hybrid	13.1	3.0M	1.68e-04	3.2	21.9	98.2K	5.60e-02	16.9
Parallel	12.3	3.0M	1.67e-04	3.1	20.1	98.2K	7.89e-02	15.7
Sequential	14.0	3.8M	1.51e-04	3.6	20.8	98.1K	6.02e-02	14.4

Table 7: Ablation Experiment 5 (E5): Execution Flow. Detailed comparison of variants on 2D Darcy and 2D NS. Memory (Mem) is reported in GB, and training time per epoch (Time) is in seconds. Bold indicates the best test loss.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Shape	Split	Operator	Mapping
	$(C,T,S)$	( $N_{\text{tr}}/N_{\text{te}}$ )	Type	Task
1D KS	$(1,10,256)$	$1000/200$	Evolutionary	Sol $\to$ Sol
2D Navier-Stokes	$(1,10,64\times 64)$
3D Shallow Water	$(2,10,64\times 32)$
2D Darcy	$(1,1,49\times 49)$	$1000/200$	Solution	Param $\to$ Sol

Appendix

Table 1: Summary of benchmark datasets and experimental configurations. Tensor shapes follow

(C,T,S)

where

C

: channels,

T

: temporal steps, and

S

: spatial resolution.

N_{\text{train}}/N_{\text{test}}

denotes the dataset split.

Dataset	Hidden Dim.			Depth			Heads			Psi Dim.
Dataset	T	M	L	T	M	L	T	M	L	T	M	L
1D Experiments
1dks	32	64	128	2	4	8	2	4	8	16	32	64
2D Experiments
2ddarcy	36	64	128	2	4	8	2	4	8	16	32	64
2dns	20	24	32	2	3	4	2	3	4	8	12	12

Appendix

Table 2: Hyperparameter settings for ONO. Model sizes are denoted as T (Tiny), M (Medium), and L (Large). Batch size was fixed at 64 across all configurations; gradient clipping was disabled.

Benchmark	Tiny			Medium			Large
Benchmark	P (K)	M (MiB)	F (M)	P (K)	M (MiB)	F (M)	P (M)	M (MiB)	F (G)
1D Ks	21.1	832	53.8	152.1	3024	389.4	1.2	13210	3.0
2D Darcy	25.6	936	61.5	151.0	3050	362.5	1.2	11328	2.8
2D Ns	8.5	8120	348.2	17.3	13950	708.8	0.04	23810	1.6
3D Sw	8.9	4422	182.7	17.8	7530	364.7	0.04	12272	0.81

Appendix

Table 3: Comprehensive model statistics for ONO. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).

Dataset	Hidden Dim.			Depth			Heads			Slice Num.			Ref
Dataset	T	M	L	T	M	L	T	M	L	T	M	L	Ref
1D Experiments
1dks	40	64	128	2	4	8	2	4	8	16	32	64	8
2D Experiments
2ddarcy	36	56	128	2	4	8	2	4	8	16	32	64	8
2dns	20	24	32	2	3	4	2	3	4	16	20	24	8

Appendix

Table 4: Hyperparameter settings for Transolver. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).

Benchmark	Tiny			Medium			Large
Benchmark	P (K)	M (MiB)	F (M)	P (K)	M (MiB)	F (M)	P (M)	M (MiB)	F (G)
1D Ks	80.4	842	202.1	369.7	2668	961.4	2.8	13408	7.4
2D Darcy	65.1	810	153.2	283.6	2534	691.2	2.8	12238	6.9
2D Ns	22.0	7950	893.5	42.9	13866	1800	0.10	23492	4.0
3D Sw	22.4	4662	455.5	43.4	7158	904.9	0.10	12122	2.0

Appendix

Table 5: Comprehensive model statistics for Transolver. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).

Dataset	Hidden Dim.			Depth			Heads			Kernel Multiplier	Latent Multiplier
Dataset	T	M	L	T	M	L	T	M	L	Kernel Multiplier	Latent Multiplier
1D Experiments
1dks	16	32	80	3	4	8	2	4	8	2	2
2D Experiments
2ddarcy	40	80	272	2	6	8	2	4	8	2	2
2dns	32	64	64	2	2	4	2	2	4	2	2

Appendix

Table 6: Hyperparameter settings for FactFormer. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).

Benchmark	Tiny			Medium			Large
Benchmark	P (K)	M (MiB)	F (M)	P (K)	M (MiB)	F (M)	P (M)	M (MiB)	F (G)
1D Ks	6.6	798	12.2	51.4	2858	94.2	0.64	12256	1.2
2D Darcy	38.9	746	40.1	476.1	2874	505.7	7.3	12892	7.8
2D Ns	25.3	7578	450.4	99.2	14774	1700	0.20	23706	3.6
3D Sw	25.6	4020	234.4	99.9	7674	891.5	0.20	12252	1.8

Appendix

Table 7: Comprehensive model statistics for FactFormer. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).

Dataset	Hidden Dim.			Depth			Heads			Scale
Dataset	T	M	L	T	M	L	T	M	L	Scale
1D Experiments
1dks	8	20	64	2	4	8	2	5	8	8
2D Experiments
2ddarcy	8	20	72	2	4	8	2	5	9	8
2dns	4	8	12	2	3	4	1	2	3	8

Appendix

Table 8: Hyperparameter settings for OFormer. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).

Benchmark	Tiny			Medium			Large
Benchmark	P (K)	M (MiB)	F (M)	P (K)	M (MiB)	F (M)	P (M)	M (MiB)	F (G)
1D Ks	11.6	904	31.2	92.0	2698	239.9	1.4	11224	3.5
2D Darcy	11.6	876	29.1	91.9	2588	224.6	1.7	12290	4.1
2D Ns	3.0	7098	134.7	11.6	13618	498.7	0.03	24940	1.4
3D Sw	3.1	3720	68.2	11.7	6988	251.0	0.03	12582	0.71

Appendix

Table 9: Comprehensive model statistics for OFormer. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).

Dataset	Hidden Dim.			Depth			MLP Layers			Heads			N Experts	N Inner
Dataset	T	M	L	T	M	L	T	M	L	T	M	L	N Experts	N Inner
1D Experiments
1dks	16	24	72	1	2	4	1	2	3	2	4	8	4	2
2D Experiments
2ddarcy	12	32	64	1	2	4	1	2	3	2	4	8	4	2
2dns	8	8	10	1	2	3	1	2	3	1	2	2	4	2

Appendix

Table 10: Hyperparameter settings for GNOT. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).

Benchmark	Tiny			Medium			Large
Benchmark	P (K)	M (MiB)	F (M)	P (K)	M (MiB)	F (M)	P (M)	M (MiB)	F (G)
1D Ks	14.1	884	34.6	59.2	3008	147.8	0.99	14630	2.5
2D Darcy	8.0	804	18.3	103.5	3142	243.7	0.79	11544	1.9
2D Ns	3.8	7504	145.8	7.3	13988	279.5	0.02	23816	0.63
3D Sw	3.9	3922	74.7	7.4	6972	141.6	0.02	12280	0.32

Appendix

Table 11: Comprehensive model statistics for GNOT. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).

Dataset	Hidden Dim.			Depth			Heads			Spectral Modes
Dataset	T	M	L	T	M	L	T	M	L	Spectral Modes
1D Experiments
1dks	4	8	32	2	3	6	2	4	8	[64, 1]
2D Experiments
2ddarcy	24	32	128	2	6	8	2	4	8	[12, 12]
2dns	16	32	32	2	2	4	2	2	4	[12, 12]

Appendix

Table 12: Hyperparameter settings for DynFormer. Model sizes are denoted as T (Tiny), M (Medium), and L (Large).

Benchmark	Tiny			Medium			Large
Benchmark	P (K)	M (MiB)	F (M)	P (K)	M (MiB)	F (M)	P (M)	M (MiB)	F (G)
1D Ks	1.2	962	2.1	5.9	2776	10.3	0.16	12848	0.29
2D Darcy	32.6	726	32.7	162.9	2362	151.8	3.4	12676	3.1
2D Ns	15.1	6894	263.6	57.9	13206	999.1	0.11	24578	1.8
3D Sw	7.0	3184	51.9	39.7	7068	274.7	0.08	11850	0.55

Appendix

Table 13: Comprehensive model statistics for DynFormer. The table outlines the resource requirements across three model sizes (Tiny, Medium, and Large) evaluated on various physical simulation benchmarks (1D Ks, 2D Darcy, 2D Ns, and 3D Sw). Metrics reported include the total number of parameters ( P ), memory consumption ( M ), and computational complexity in FLOPs ( F ). Numerical values are abbreviated for clarity: K (thousands), M (millions), G (billions), and MiB (mebibytes).

Figure 1: Visual comparison on 1D Kuramoto-Sivashinsky for the first sample. Each panel displays the evolution of the field $u(x,t)$ over space (horizontal axis) and time (vertical axis).

Figure 2: Visual comparison on 1D Kuramoto-Sivashinsky for the last sample. Each panel displays the evolution of the field $u(x,t)$ over space (horizontal axis) and time (vertical axis).

Figure 3: Visual comparison on 2D Darcy for the first sample. Each panel displays a 2D scalar field (e.g., pressure or concentration) over a square domain

Figure 4: Visual comparison on 2D Darcy for the last sample. Each panel displays a 2D scalar field (e.g., pressure or concentration) over a square domain

Figure 5: Visual comparison on 2D Navier-Stokes for the first sample at the final step. The vorticity variable $\omega$ is presented here.

Figure 6: Visual comparison on 2D Navier-Stokes for the last sample at the final step. The vorticity variable $\omega$ is presented here.

Figure 7: Visual comparison on 3D Shallow Water for the first sample at the final step. The vorticity variable $\omega$ is presented here.

Figure 8: Visual comparison on 3D Shallow Water for the last sample at the final step. The vorticity variable $\omega$ is presented here.

Explore similar work

Physics Transformer: Tailoring Transformer for General PDE Prediction
Jul 27, 2026Guoze Sun, Rui Zhang, Jiankai Tang +4Transformer PDE Surrogate Modeling
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Mechanics and Engineering Science, Peking University, Beijing, China
When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains
May 8, 2026Brandon Yee, Pairie Koh, Jack Rodriguez +1Neural PDE Solvers PDE Solving
Physics Lab, Yee Collins Research Group
MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers
Jun 3, 2026Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang +2Neural PDE Solvers PDE Surrogate Modeling
School of Mathematical Sciences, University of Science and Technology of China, Hefei 230026, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou 215123, China.

Jul 27, 2026J/K move · Enter open · S save

Physics Transformer: Tailoring Transformer for General PDE Prediction

Guoze Sun, Rui Zhang, Jiankai Tang, Mengtao Yan+3

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Mechanics and Engineering Science, Peking University, Beijing, China

Transformer PDE Surrogate Modeling PDE Operator Learning

Open paper AskPDF arXiv

Transformer architectures have attracted increasing attention for solving partial differential equations (PDEs), owing to their flexibility in handling irregular discretizations and their ability to capture long-range physical dependencies. However, unlike discrete language tokens or fixed-resolution image patches, observed physical fields are finite samples of underlying infinite-dimensional functions. Consequently, effectively applying Transformers to PDEs requires a tokenizer that respects the functional nature of physical fields and constructs physically expressive tokens from arbitrary discretizations.To this end, we propose \methodname{Physics Transformer}, a function-projection-based Transformer architecture for physical field prediction. Physics Transformer treats a physical field as a continuous function and partitions its discretization into locality-preserving spatial patches. Within each patch, it dynamically learns a set of adaptive local basis functions and projects the sampled field onto these bases to obtain compact physics tokens. The resulting tokens capture diverse latent physical states while preserving fine-scale spatial structures, enabling efficient global interaction through factorized attention across space and physical states. The projected representation further supports efficient decoding at arbitrary query locations. Extensive experiments on diverse benchmarks, ranging from two-dimensional PDE dynamics to industrial-scale three-dimensional CFD simulations, demonstrate that Physics Transformer accurately captures fine-grained physical structures and achieves state-of-the-art predictive performance. These results establish function projection as a practical and effective foundation for designing Transformer architectures for PDE solving.

Abstract

Explore similar work

Physics Transformer: Tailoring Transformer for General PDE Prediction

When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers