cs.LGSep 7, 2026

PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders

Authors: Nandita N. Patil, Eshwar R. A., Gajanan V. Honnavar

Organizations: QuaNad Research Laboratory, PES University, Bengaluru, India · Independent Researcher, QNu Labs Pvt Ltd, Bengaluru, India. Former Professor at PES University (EC Campus), Bengaluru

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Physics-Informed Neural Networks (PINNs) embed PDE residuals into neural network training, but their internal representations remain opaque: it is unknown what physical features their hidden layers encode or whether those features have a localized causal role. We present PhysSAE, a mechanistic interpretability framework that trains overcomplete sparse autoencoders (SAEs) on PINN penultimate-layer activations and evaluates dictionary atoms through direct causal intervention in the original frozen hidden state: hcf=h−αzkdkh_{\mathrm{cf}} = h - αz_k d_k, bypassing the SAE decoder entirely. Across six PDE families, with 3 PINN seeds and 3 SAE seeds each---we show that (i) Our discovered SAE atoms align with independently-defined physical observables (max Pearson ∣r∣=0.951|r|=0.951, always ≫\gg permutation null), (ii) the causal footprint of top-aligned atom ablation is 1.2--4.2×\times more spatially concentrated canonical than PCA or ICA interventions, and (iii) top-aligned atoms outperform matched random controls on causal localization for structured physical concepts (ESF80_{80} advantage 0.04-0.44). Two-atom bilateral representations improve concept regression R2^2 by ΔR2 ⁣= ⁣0.05-0.15ΔR^2\!=\!0.05\text{-}0.15 over single atoms, while random pairs decrease it by up to 0.60. These results demonstrate that PINNs develop sparse, physically structured latent representations that can be identified and causally interrogated post-hoc, opening a path toward interpretability-aware scientific machine learning.

Figures & tables

PDEConcept nameFormula
HeatAmplitude∣u∗∣|u^{*}|
Gradient∣∂xu∗∣|\partial_{x}u^{*}|
Curvature∣∂xxu∗∣|\partial_{xx}u^{*}|
Temporal rate∣∂tu∗∣|\partial_{t}u^{*}|
Signed valueu∗u^{*}
BurgersAmplitude∣u∗∣|u^{*}|
Table 1: Physical concept fields used for alignment, computed from independent reference solutions. Central finite differences are used for spatial and temporal derivatives. For Schrödinger, h=u+ivh=u+iv is the complex field, ∣h∣|h| its amplitude, and ϕ=arg⁡(h)\phi=\arg(h) its phase.
PDEEquationDomainICRef. typeL2 err
Heatut=uxxu_{t}=u_{xx}[−1,1]×[0,1][-1,1]{\times}[0,1]e−5x2e^{-5x^{2}}Analytic0.025±0.0270.025{\pm}0.027
Burgersut+uux=νuxxu_{t}+uu_{x}=\nu u_{xx} , ν=0.01π\nu{=}\tfrac{0.01}{\pi}[−1,1]×[0,1][-1,1]{\times}[0,1]− ⁣sin⁡(πx)-\!\sin(\pi x)Spectral0.207±0.0680.207{\pm}0.068
Allen–Cahnut=10−4uxx+5u−5u3u_{t}=10^{-4}u_{xx}+5u-5u^{3}[−1,1]×[0,1][-1,1]{\times}[0,1]x2cos⁡(πx)x^{2}\cos(\pi x)Spectral0.930±0.0050.930{\pm}0.005
Conv. easyut+ux=0u_{t}+u_{x}=0[−1,1]×[0,1][-1,1]{\times}[0,1]sin⁡(πx)\sin(\pi x)Analytic0.266±0.0130.266{\pm}0.013
Conv. hardut+30ux=0u_{t}+30u_{x}=0[−1,1]×[0,1][-1,1]{\times}[0,1]sin⁡(πx)\sin(\pi x)Analytic0.951±0.0030.951{\pm}0.003
Schrödingeriht+12hxx+∣h∣2h=0ih_{t}+\tfrac{1}{2}h_{xx}+|h|^{2}h=0[−5,5]×[0,1][-5,5]{\times}[0,1]SolitonAnalytic0.119±0.0620.119{\pm}0.062
Table 2: PDE benchmark. L2 error (mean ±\pm std, 3 PINN seeds) vs. independent reference.
PDEBest conceptMean max ∣r∣|r|Z-score
Heatgradient ∣∂xu∣|\partial_{x}u|0.845±0.0390.845\pm 0.039>100>100
Burgersamplitude ∣u∣|u|0.785±0.0210.785\pm 0.021>80>80
Allen–Cahnsigned uu0.675±0.0250.675\pm 0.025>40>40
Conv. easycharacteristic x−tx{-}t0.841±0.0270.841\pm 0.027>90>90
Conv. hardcharacteristic x−30tx{-}30t0.943±0.0080.943\pm 0.008>120>120
Schrödingeramplitude ∣h∣|h|0.881±0.0520.881\pm 0.052>100>100
Table 3: Maximum top-1 concept alignment. Values are means over 3 PINN seeds (each reported as the max over concepts and seeds). Z-score against permutation null ( n=500n=500 ).
Figure 1: Schrödinger: SAE atom activations (top row) vs. independently-computed reference concept fields (bottom row). Atom 73 tracks the moving soliton envelope ( r=+0.83r{=}+0.83 ). Concepts from analytic references, not the PINN output.
Figure 2: Heat: SAE atom activations (top row) vs. reference concept fields (bottom row). Atoms specialise to gradient ( r=0.89r{=}0.89 ), curvature, and temporal derivative.
Figure 3: Concept alignment matrices (Pearson rr ) for Burgers ( left ) and conv. hard ( right ). Rows: top-20 atoms; columns: physical concepts. Atoms tend to align with at most one or two concepts (monosemanticity).
PDESAE ESF 80PCAICASDSpeedupMax ∣r∣|r|Neg. ctrlBilat. ΔR2\Delta R^{2}
Heat0.165±0.23\mathbf{0.165}{\pm}0.230.7000.5870.2784.2 ×\times0.890−- 0.025++ 0.053
Burgers0.374±0.18\mathbf{0.374}{\pm}0.180.5850.5740.3411.6 ×\times0.804++ 0.005++ 0.097
Allen–Cahn0.476±0.29\mathbf{0.476}{\pm}0.290.5760.6220.4121.2 ×\times0.692++ 0.043++ 0.088
Conv. easy0.366±0.24\mathbf{0.366}{\pm}0.240.5460.5710.4231.5 ×\times0.872++ 0.049++ 0.146
Conv. hard0.373±0.33\mathbf{0.373}{\pm}0.330.7450.6530.2922.0 ×\times0.951++ 0.125++ 0.030
Schrödinger0.179±0.11\mathbf{0.179}{\pm}0.110.4060.5380.3122.3 ×\times0.920−- 0.013++ 0.102
Table 4: Aggregate main results across 6 PDEs ×\times 3 PINN seeds. ESF 80 : fraction of domain capturing 80% of causal effect mass (lower = more localised = better for SAE). Neg. ctrl. adv.: ESF 80 (atom) −- ESF 80 (matched random), negative means atom is more localised. Bilateral ΔR2\Delta R^{2} : two-atom minus single-atom regression improvement. Consistency: Hungarian-matched mean cosine across 3 SAE seeds.
Figure 4: Negative controls: ESF 80 advantage (top-aligned atom minus matched random) split by localised ( left ) vs. global ( right ) concepts. Blue = atom more localised than random (positive result); red = random incidentally more localised (expected for global concepts).
PDEMetricD=128D{=}128D=256D{=}256D=512D{=}512
HeatMean top-1 ∣rˉ∣|\bar{r}|0.8150.8310.836
Recon. MSE0.1280.0750.042
BurgersMean top-1 ∣rˉ∣|\bar{r}|0.8550.9280.964
Recon. MSE0.1920.1060.060
SchrödingerMean top-1 ∣rˉ∣|\bar{r}|0.8170.8360.869
Recon. MSE0.2400.1290.068
Table 5: Overcompleteness ablation: alignment and reconstruction across dictionary sizes DD for three representative PDEs. Mean top-1 alignment ∣rˉ∣|\bar{r}| and reconstruction MSE on held-out activations. SAE trained at λ=0.02\lambda{=}0.02 ; all other settings fixed.
λ\lambdaMean L 0Recon. MSEMean top-1 ∣rˉ∣|\bar{r}|Status
10−410^{-4}259.70.0410.916too dense
10−310^{-3}254.80.0400.918too dense
10−210^{-2}207.60.0490.941good
0.02\mathbf{0.02}149.10.0700.964operative
0.10.175.40.1200.935sparse
0.30.335.00.2340.916recon. degrades
Table 6: Sparsity ablation: alignment and reconstruction vs. mean L 0 across λ\lambda values for Burgers (representative PDE). Operative configuration ( λ=0.02\lambda{=}0.02 ) highlighted; alignment is flat over a wide sparsity range.
PDESingle atom R2R^{2}SAE pair R2R^{2}ΔR2\Delta R^{2}Random pair ΔR2\Delta R^{2}
Heat0.5910.643++ 0.053−- 0.226
Burgers0.3870.483++ 0.097−- 0.203
Allen–Cahn0.2790.367++ 0.088−- 0.189
Conv. easy0.4510.598++ 0.146−- 0.196
Conv. hard0.3320.362++ 0.030−- 0.127
Schrödinger0.5040.607++ 0.102−- 0.215
Table 7: Bilateral alignment: mean R 2 for single atom, SAE atom pair, and random pair per PDE (averaged over concepts and seeds). ΔR2\Delta R^{2} = pair −- single. SAE pairs consistently improve over single atoms; random pairs decrease R 2 .
Table 8: SAE feature consistency across 3 independent SAE seeds per PINN. Mean and median Hungarian-matched cosine similarity between dictionaries; fraction of atoms above thresholds. Low cosine reflects non-uniqueness of sparse coding solutions, not instability of causal localization (ESF 80 std << 0.05 across PINN seeds).
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Burgers atom 115 ( r=−0.52r=-0.52 ). Negative correlation = atom fires where u<0u<0 . Ablation concentrates in the lower-left pre-shock region: the negative- uu bilateral half.
Figure 7: Conv. hard ( β=30\beta=30 , L2 error 95%). Despite PINN failure, the intervention map localises in the characteristic-coordinate direction.
Figure 8: Allen–Cahn (failed PINN, L2 error 93%). The SAE still localises the causal footprint to the region where the PINN did learn structure.
Figure 9: Burgers atom fields. From left: amplitude, shock gradient, curvature, temporal derivative, signed value, shock indicator, spatial coordinate.
Figure 10: Conv. hard: atom for characteristic coordinate x−30tx-30t achieves r=0.95r=0.95 .
Figure 11: Allen–Cahn: alignment weaker (max r=0.69r=0.69 ) reflecting the PINN’s failure to resolve correct interface dynamics.
Figure 12: Conv. easy: characteristic-coordinate atom ( r=0.87r=0.87 ).
Figure 20
Figure 14: Alignment matrices for remaining PDEs.
SettingValue
PINN
Hidden layers / width5 / 128
Activationtanh⁡\tanh
Adam iters / LR8000 / 10−310^{-3}
L-BFGS iters300
Loss weights wBC,wIC,wFw_{\mathrm{BC}},w_{\mathrm{IC}},w_{F}100, 100, 1
Appendix
Table 9: Full experimental hyperparameters.
More in this topic

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Explore similar work

CardsList
Jun 18, 2026

Physics-Informed Neural Network with Squeeze-Excitation-like Attention

We introduce SEA-PINN, a novel architecture that incorporates a Squeeze-Excitation-like attention mechanism into physics-informed neural networks to dynamically recalibrate the importance of neurons across layers. A key feature of SEA-PINN is its highly stable initialization. On 17 out of 20 benchmark problems, SEA-PINN exhibit nearly negligible variance and significantly reduced initial loss, establishing a quasi-deterministic and favorable starting point for optimization. Notably, without employing Fourier feature embeddings or periodic activation functions, SEA-PINN attained competitive accuracy (83% vs. 90% improvement relative to FNN-PINN on the high-frequency case 7) as compared with TSA-PINN-a model specifically engineered for high-frequency problems via learnable frequencies in sinusoidal activations. Furthermore, integrating SEA-PINN into TSA-PINN boosted performance by 42.49%. These results underscore SEA-PINN as a lightweight plug-in module that enhances nonlinear representation power, promotes more robust and efficient convergence, and strengthens the overall reliability of physics-informed learning.

Yun-Fei Song, Long-Gang Pang, Fu-Peng Li +1

Key Laboratory of Quark and Lepton Physics (MOE) & Institute of Particle Physics, Central China Normal University, Wuhan 430079, China · Artificial Intelligence and Computational Physics Research Center, Central China Normal University, Wuhan 430079, China · Key Laboratory of Nuclear Physics and Ion-beam Application (MOE) & Institute of Modern Physics, Fudan University, Shanghai 200433, China +2

Neural PDE SolversPhysics-Informed MLNeural Network Training Dynamics
AskPDF
Mar 3, 2026

Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results

Physics-Informed Neural Networks (PINNs) incorporate physics into neural networks by embedding partial differential equations (PDEs) into their loss function. Despite their success in learning the underlying physics, PINN models remain difficult to train and interpret. In this work, a novel modeling approach is proposed, which relies on the use of Domain-aware Fourier Features (DaFFs) for the positional encoding of the input space. These features encapsulate all the domain-specific characteristics, such as the geometry and boundary conditions, and unlike Random Fourier Features (RFFs), eliminate the need for explicit boundary condition loss terms and loss balancing schemes, while simplifying the optimization process and reducing the computational cost associated with training. We further develop an LRP-based explainability framework tailored to PINNs, enabling the extraction of relevance attribution scores for the input space. It is demonstrated that PINN-DaFFs achieve orders-of-magnitude lower errors and allow faster convergence compared to vanilla PINNs and RFFs-based PINNs. Furthermore, LRP analysis reveals that the proposed leads to more physically consistent feature attributions, while PINN-RFFs and vanilla PINNs display more scattered and less physics-relevant patterns. These results demonstrate that DaFFs not only enhance PINNs' accuracy and efficiency but also improve interpretability, laying the ground for more robust and informative physics-informed learning.

Alberto Miño Calero, Luis Salamanca, Konstantinos E. Tatsis

Department of Engineering Cybernetics NTNU, Trondheim, Norway · Swiss Data Science Center (SDSC) ETH Zürich, Zürich, Switzerland

Fourier Feature EmbeddingsNeural Network InterpretabilityPhysics-Informed MLLayer-Wise Relevance Propagation
AskPDF
May 29, 2026

PINNs Failure Modes are Overfitting

Physics-Informed Neural Networks (PINNs) are a common class of machine learning-based partial differential equation (PDE) solvers which train a network to represent a solution by minimizing a residual loss that encodes the PDE. Despite their successes, they are known to fail on certain simple equations, converging to an incorrect solution despite low loss. These failure modes have garnered significant attention in the literature over the past several years, motivating both architectural and optimization based solutions. By directly visualizing the residual, we show that failure modes are the result of overfitting: the loss is minimized on the collocation points, but not elsewhere. Applying regularization causes the failure modes to vanish. Finally, we extend double backpropagation over the full set of residuals, and use it to achieve state-of-the-art performance on four standard failure mode equations with up to 23×23\times fewer collocation points and a vanilla architecture.

Nigel T. Andersen, Takashi Matsubara

Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan

Neural Network GeneralizationNeural PDE SolversBackpropagationPhysics-Informed ML
AskPDF