Reconstructing continuous physical fields from sparse, irregular observations is a central challenge in scientific machine learning, particularly for systems governed by partial differential equations (PDEs). Existing physics-informed methods typically enforce governing equations as soft penalty terms during optimization, often leading to gradient imbalance, instability, and degraded physical consistency under limited data. We introduce the Physics-Guided Transformer (PGT), a neural architecture that embeds physical structure directly into the self-attention mechanism. Specifically, PGT incorporates a heat-kernel-derived additive bias into attention logits, encoding diffusion dynamics and temporal causality within the representation. Query coordinates attend to these physics-conditioned context tokens, and the resulting features are decoded using a FiLM-modulated sinusoidal implicit network that adaptively controls spectral response. We evaluate PGT on the one-dimensional heat equation and two-dimensional incompressible Navier-Stokes systems. In sparse 1D reconstruction with 100 observations, PGT achieves a relative L2 error of 5.9e-3, significantly outperforming both PINNs and sinusoidal representations. In the 2D cylinder wake problem, PGT uniquely achieves both low PDE residual (8.3e-4) and competitive relative error (0.034), outperforming methods that optimize only one objective. These results demonstrate that embedding physics within attention improves stability, generalization, and physical fidelity under data-scarce conditions.
Figures & tables
Figure 1: Overview of the PGT architecture. The physics-guided Transformer encoder processes sparse observations into latent context tokens, which are then used to condition the FiLM-modulated SIREN decoder for continuous field reconstruction.
Model
Loss
FLOPs
Train time
Param
Data
PDE
Rel L2
(G)
SIREN
M=100
0.063
0.18
0.54
6.5
126 s
26,419
M=200
0.045
0.14
0.45
6.8
127 s
M=500
0.025
0.18
0.34
7.5
135 s
PINN
M=100
0.026
0.12
0.226
1.69
65 s
66,500
M=200
0.024
0.19
0.332
1.75
65.9 s
Table 1: Comparison of SIREN, PINN, and PGT models across different values of M .
Table 2: Sparse reconstruction performance for the 2D Navier–Stokes cylinder wake at a fixed time snapshot ( Ntrain=1500 ). Best results are shown in bold . Architecture family groups methods: operator-learning (FNO, PI-DeepONet), Transformer-based (PINNsFormer), implicit representation (SIREN, WIRE), classical PINN, and the proposed PGT.
Figure 4: Qualitative comparison between ground truth and PGT reconstruction for the 2D Navier–Stokes problem. Rows correspond to u , v , and p fields. Columns show ground truth, PGT prediction, and absolute error.
Figure 5: Training error convergence for PGT and baseline methods on the 2D Navier–Stokes problem.
Variant
Decoder
Attn. bias Γ
PDE loss LPDE
Reconstruction Error ↓
PDE Residual ↓
PGT (full)
FiLM-SIREN
✓
✓
6.50×10−5
8.30×10−4
No PDE loss
FiLM-SIREN
✓
∘
8.80×10−5
2.90×10−3
No attention bias
FiLM-SIREN
∘
✓
3.00×10−4
1.30×10−2
No physics (data only)
FiLM-SIREN
∘
∘
3.60×10−4
3.50×10−2
SIREN, no FiLM
SIREN
✓
✓
1.83×10−4
1.10×10−3
FiLM-MLP
FiLM-MLP
✓
✓
1.21×10−4
1.05×10−3
Table 3: Ablation study on the 2D Navier–Stokes cylinder wake ( Ntrain=1500 ). Attn. bias Γ : heat-kernel additive bias in self-attention logits (disabled in pure-transformer variants). PDE loss LPDE : momentum and continuity residual terms. Decoder : FiLM-SIREN is the full decoder; SIREN (no FiLM) uses a plain SIREN with context concatenated at the input; MLP replaces sinusoidal activations with GELU; FiLM-MLP retains FiLM conditioning but uses GELU activations. ✓ active; ∘ disabled. Data loss is active in all rows. Best results in bold .
Avg. Rel- ℓ2 ( u,v ) ↓
PDE Residual ↓
η
PGT
PGT-UW
PGT
PGT-UW
0.00
0.0496
0.0410
1.61e-03
4.55e-03
0.01
0.0634
0.0861
1.99e-03
6.80e-03
0.02
0.0406
0.1054
1.86e-03
7.78e-03
0.05
0.0508
0.1102
1.49e-03
7.63e-03
0.10
0.0512
0.1551
1.92e-03
7.74e-03
Table 4: Noise robustness on the 2D Navier–Stokes cylinder wake ( σ=η⋅std(utrain) ). PGT uses a standard MSE data loss; PGT-UW uses a heteroscedastic uncertainty-weighted loss. Best per row in bold .
Figure 6: Reconstruction error (left) and PDE residual (right) as a function of noise level η for PGT and PGT-UW on the 2D Navier–Stokes cylinder-wake problem.
Physics-informed neural networks (PINNs) have garnered significant interest for their potential in solving partial differential equations (PDEs) that govern a wide range of physical phenomena. By incorporating physical laws into the learning process, PINN models have demonstrated the ability to learn physical outcomes reasonably well. However, current PINN approaches struggle to predict or solve new PDEs effectively when there is a lack of training examples, indicating they do not generalize well to unseen problem instances. In this paper, we present a transferable learning approach for PINNs premised on a fast Pseudoinverse PINN framework (Pi-PINN). Pi-PINN learns a transferable physics-informed representation in a shared embedding space and enables rapid solving of both known and unknown PDE instances via closed-form head adaptation using a least-squares-optimal pseudoinverse under PDE constraints. We further investigate the synergies between data-driven multi-task learning loss and physics-informed loss, providing insights into the design of more performant PINNs. We demonstrate the effectiveness of Pi-PINN on various PDE problems, including Poisson's equation, Helmholtz equation, and Burgers' equation, achieving fast and accurate physics-informed solutions without requiring any data for unseen instances. Pi-PINN can produce predictions 100-1000 times faster than a typical PINN, while producing predictions with 10-100 times lower relative error than a typical data-driven model even with only two training samples. Overall, our findings highlight the potential of transferable representations with closed-form head adaptation to enhance the efficiency and generalization of PINNs across PDE families and scientific and engineering applications.
Jian Cheng Wong, Isaac Yin Chung Lai, Pao-Hsiung Chiu +3
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore · National University of Singapore (NUS), Singapore · Indian Institute of Technology Goa (IIT Goa), India +1
We study the problem of architecture selection for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators. We introduce the Multi-Scale Attention Transformer (\msat{}), a deep learning architecture that encodes spatiotemporal solution histories as token sequences and trains end-to-end via a composite supervised objective with optional physics-informed regularization terms. We conduct a comprehensive empirical evaluation against nine baselines -- including physics-informed neural networks (PINNs), neural operators (FNO, DeepONet, GNOT), and state-space models (Mamba-NO) -- across five benchmark problems from the PINNacle suite, using identical train/test splits and reference data for all methods. \msat{} achieves state-of-the-art generalization on complex geometry problems (Lrel2=0.0101 on Heat2D-CG, a 3.7× improvement over FNO) at 34s total inference vs.\ 120,812s for Mamba-NO. Ablation studies over the physics regularization component reveal a precise inductive bias tradeoff: physics priors reduce test error on diffusion-dominated problems but degrade generalization on chaotic and recirculating-flow regimes, directly characterizing the prior misspecification boundary. Approximation error bounds as a function of domain boundary complexity κ provide a theoretical basis for these empirical findings and a principled rule for architecture selection.
Transformer architectures have attracted increasing attention for solving partial differential equations (PDEs), owing to their flexibility in handling irregular discretizations and their ability to capture long-range physical dependencies. However, unlike discrete language tokens or fixed-resolution image patches, observed physical fields are finite samples of underlying infinite-dimensional functions. Consequently, effectively applying Transformers to PDEs requires a tokenizer that respects the functional nature of physical fields and constructs physically expressive tokens from arbitrary discretizations.To this end, we propose \methodname{Physics Transformer}, a function-projection-based Transformer architecture for physical field prediction. Physics Transformer treats a physical field as a continuous function and partitions its discretization into locality-preserving spatial patches. Within each patch, it dynamically learns a set of adaptive local basis functions and projects the sampled field onto these bases to obtain compact physics tokens. The resulting tokens capture diverse latent physical states while preserving fine-scale spatial structures, enabling efficient global interaction through factorized attention across space and physical states. The projected representation further supports efficient decoding at arbitrary query locations. Extensive experiments on diverse benchmarks, ranging from two-dimensional PDE dynamics to industrial-scale three-dimensional CFD simulations, demonstrate that Physics Transformer accurately captures fine-grained physical structures and achieves state-of-the-art predictive performance. These results establish function projection as a practical and effective foundation for designing Transformer architectures for PDE solving.
Guoze Sun, Rui Zhang, Jiankai Tang +4
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Mechanics and Engineering Science, Peking University, Beijing, China