cs.LGMar 30, 2026

Physics-Guided Transformer (PGT): Physics-Aware Attention Mechanism for PINNs

Authors: Ehsan Zeraatkar, Rodion Podorozhny, Jelena Tešić

Organizations: Computer Science, Texas State University, San Marcos, Texas, US

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reconstructing continuous physical fields from sparse, irregular observations is a central challenge in scientific machine learning, particularly for systems governed by partial differential equations (PDEs). Existing physics-informed methods typically enforce governing equations as soft penalty terms during optimization, often leading to gradient imbalance, instability, and degraded physical consistency under limited data. We introduce the Physics-Guided Transformer (PGT), a neural architecture that embeds physical structure directly into the self-attention mechanism. Specifically, PGT incorporates a heat-kernel-derived additive bias into attention logits, encoding diffusion dynamics and temporal causality within the representation. Query coordinates attend to these physics-conditioned context tokens, and the resulting features are decoded using a FiLM-modulated sinusoidal implicit network that adaptively controls spectral response. We evaluate PGT on the one-dimensional heat equation and two-dimensional incompressible Navier-Stokes systems. In sparse 1D reconstruction with 100 observations, PGT achieves a relative L2 error of 5.9e-3, significantly outperforming both PINNs and sinusoidal representations. In the 2D cylinder wake problem, PGT uniquely achieves both low PDE residual (8.3e-4) and competitive relative error (0.034), outperforming methods that optimize only one objective. These results demonstrate that embedding physics within attention improves stability, generalization, and physical fidelity under data-scarce conditions.

Figures & tables

Figure 1: Overview of the PGT architecture. The physics-guided Transformer encoder processes sparse observations into latent context tokens, which are then used to condition the FiLM-modulated SIREN decoder for continuous field reconstruction.
Model LossFLOPsTrain timeParam
DataPDERel L2L^{2}(G) 
SIRENM=100M=1000.0630.180.546.5126 s26,419
M=200M=2000.0450.140.456.8127 s
M=500M=5000.0250.180.347.5135 s
PINNM=100M=1000.0260.120.2261.6965 s66,500
M=200M=2000.0240.190.3321.7565.9 s
Table 1: Comparison of SIREN, PINN, and PGT models across different values of MM .
Figure 3: (a) Error components breakdown. (b) Error components contribution analysis.
ModelParams (M)FLOPs (G)Train TimeRel- L2L^{2}Rel- L2(u)L^{2}(u)Rel- L2(v)L^{2}(v)Rel- L2(p)L^{2}(p)PDE ResidualTrain Error
FNO2.3600.0924.41 s0.7100.22001.49000.7704.70×10−24.70\times 10^{-2}4.20×10−44.20\times 10^{-4}
PI-DeepONet0.215176.1236 s0.0950.06000.23000.2104.20×10−14.20\times 10^{-1}3.90×10−33.90\times 10^{-3}
PINNsFormer0.54513.200296 min0.0800.04000.09000.1108.30×10−28.30\times 10^{-2}8.30×10−48.30\times 10^{-4}
SIREN0.2641.3102.5 min0.6900.31001.00000.7703.90×10−43.90\times 10^{-4}5.50×10−25.50\times 10^{-2}
WIRE0.5285.2006.3 min0.0180.00410.03500.0145.10×10−15.10\times 10^{-1}1.54×10−41.54\times 10^{-4}
PINN0.3361.6002.6 min0.1100.04000.14000.1608.40×10−48.40\times 10^{-4}1.19×10−31.19\times 10^{-3}
Table 2: Sparse reconstruction performance for the 2D Navier–Stokes cylinder wake at a fixed time snapshot ( Ntrain=1500N_{\text{train}}=1500 ). Best results are shown in bold . Architecture family groups methods: operator-learning (FNO, PI-DeepONet), Transformer-based (PINNsFormer), implicit representation (SIREN, WIRE), classical PINN, and the proposed PGT.
Figure 4: Qualitative comparison between ground truth and PGT reconstruction for the 2D Navier–Stokes problem. Rows correspond to uu , vv , and pp fields. Columns show ground truth, PGT prediction, and absolute error.
Figure 5: Training error convergence for PGT and baseline methods on the 2D Navier–Stokes problem.
VariantDecoderAttn. bias Γ\bm{\Gamma}PDE loss LPDE\mathcal{L}_{\text{PDE}}Reconstruction Error ↓\downarrowPDE Residual ↓\downarrow
PGT (full)FiLM-SIREN✓✓6.50×10−5\mathbf{6.50\times 10^{-5}}8.30×10−4\mathbf{8.30\times 10^{-4}}
No PDE lossFiLM-SIREN✓∘\circ8.80×10−58.80\times 10^{-5}2.90×10−32.90\times 10^{-3}
No attention biasFiLM-SIREN∘\circ✓3.00×10−43.00\times 10^{-4}1.30×10−21.30\times 10^{-2}
No physics (data only)FiLM-SIREN∘\circ∘\circ3.60×10−43.60\times 10^{-4}3.50×10−23.50\times 10^{-2}
SIREN, no FiLMSIREN✓✓1.83×10−41.83\times 10^{-4}1.10×10−31.10\times 10^{-3}
FiLM-MLPFiLM-MLP✓✓1.21×10−41.21\times 10^{-4}1.05×10−31.05\times 10^{-3}
Table 3: Ablation study on the 2D Navier–Stokes cylinder wake ( Ntrain=1500N_{\text{train}}=1500 ). Attn. bias Γ\bm{\Gamma} : heat-kernel additive bias in self-attention logits (disabled in pure-transformer variants). PDE loss LPDE\mathcal{L}_{\text{PDE}} : momentum and continuity residual terms. Decoder : FiLM-SIREN is the full decoder; SIREN (no FiLM) uses a plain SIREN with context concatenated at the input; MLP replaces sinusoidal activations with GELU; FiLM-MLP retains FiLM conditioning but uses GELU activations. ✓ active; ∘\circ disabled. Data loss is active in all rows. Best results in bold .
 Avg. Rel- ℓ2\ell_{2} ( u,vu,v ) ↓\downarrowPDE Residual ↓\downarrow
η\etaPGTPGT-UWPGTPGT-UW
0.000.04960.04101.61e-034.55e-03
0.010.06340.08611.99e-036.80e-03
0.020.04060.10541.86e-037.78e-03
0.050.05080.11021.49e-037.63e-03
0.100.05120.15511.92e-037.74e-03
Table 4: Noise robustness on the 2D Navier–Stokes cylinder wake ( σ=η⋅std(utrain)\sigma=\eta\cdot\mathrm{std}(u_{\mathrm{train}}) ). PGT uses a standard MSE data loss; PGT-UW uses a heteroscedastic uncertainty-weighted loss. Best per row in bold .
Figure 6: Reconstruction error (left) and PDE residual (right) as a function of noise level η\eta for PGT and PGT-UW on the 2D Navier–Stokes cylinder-wake problem.
More in this topic

Physics-Informed ML

Physics-informed machine learning combines data-driven models with physical laws to solve, predict, and infer scientific systems.

See all 810 papers

Explore similar work

CardsList

  1. Transferable Physics-Informed Representations via Closed-Form Head Adaptation

    Apr 23, 2026Jian Cheng Wong, Isaac Yin Chung Lai, Pao-Hsiung Chiu +3Representation LearningPDE Solving

    Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore · National University of Singapore (NUS), Singapore · Indian Institute of Technology Goa (IIT Goa), India +1

  2. When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

    May 8, 2026Brandon Yee, Pairie Koh, Jack Rodriguez +1Neural PDE SolversPDE Solving

    Physics Lab, Yee Collins Research Group

  3. Physics Transformer: Tailoring Transformer for General PDE Prediction

    Jul 27, 2026Guoze Sun, Rui Zhang, Jiankai Tang +4TransformerPDE Surrogate Modeling

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Mechanics and Engineering Science, Peking University, Beijing, China