PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders

Nandita N. Patil; Eshwar R. A.; Gajanan V. Honnavar

cs.LGSep 7, 2026

PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders

Authors: Nandita N. Patil, Eshwar R. A., Gajanan V. Honnavar

Organizations: QuaNad Research Laboratory, PES University, Bengaluru, India · Independent Researcher, QNu Labs Pvt Ltd, Bengaluru, India. Former Professor at PES University (EC Campus), Bengaluru

Abstract

Physics-Informed Neural Networks (PINNs) embed PDE residuals into neural network training, but their internal representations remain opaque: it is unknown what physical features their hidden layers encode or whether those features have a localized causal role. We present PhysSAE, a mechanistic interpretability framework that trains overcomplete sparse autoencoders (SAEs) on PINN penultimate-layer activations and evaluates dictionary atoms through direct causal intervention in the original frozen hidden state: $h_{\mathrm{cf}} = h - αz_k d_k$ , bypassing the SAE decoder entirely. Across six PDE families, with 3 PINN seeds and 3 SAE seeds each---we show that (i) Our discovered SAE atoms align with independently-defined physical observables (max Pearson $|r|=0.951$ , always $\gg$ permutation null), (ii) the causal footprint of top-aligned atom ablation is 1.2--4.2 $\times$ more spatially concentrated canonical than PCA or ICA interventions, and (iii) top-aligned atoms outperform matched random controls on causal localization for structured physical concepts (ESF $_{80}$ advantage 0.04-0.44). Two-atom bilateral representations improve concept regression R $^2$ by $ΔR^2\!=\!0.05\text{-}0.15$ over single atoms, while random pairs decrease it by up to 0.60. These results demonstrate that PINNs develop sparse, physically structured latent representations that can be identified and causally interrogated post-hoc, opening a path toward interpretability-aware scientific machine learning.

Figures & tables

PDE	Concept name	Formula
Heat	Amplitude	$\|u^{*}\|$
	Gradient	$\|\partial_{x}u^{*}\|$
	Curvature	$\|\partial_{xx}u^{*}\|$
	Temporal rate	$\|\partial_{t}u^{*}\|$
	Signed value	$u^{*}$
Burgers	Amplitude	$\|u^{*}\|$

Table 1: Physical concept fields used for alignment, computed from independent reference solutions. Central finite differences are used for spatial and temporal derivatives. For Schrödinger,

h=u+iv

is the complex field,

|h|

its amplitude, and

\phi=\arg(h)

its phase.

PDE	Equation	Domain	IC	Ref. type	L2 err
Heat	$u_{t}=u_{xx}$	$[-1,1]{\times}[0,1]$	$e^{-5x^{2}}$	Analytic	$0.025{\pm}0.027$
Burgers	$u_{t}+uu_{x}=\nu u_{xx}$ , $\nu{=}\tfrac{0.01}{\pi}$	$[-1,1]{\times}[0,1]$	$-\!\sin(\pi x)$	Spectral	$0.207{\pm}0.068$
Allen–Cahn	$u_{t}=10^{-4}u_{xx}+5u-5u^{3}$	$[-1,1]{\times}[0,1]$	$x^{2}\cos(\pi x)$	Spectral	$0.930{\pm}0.005$
Conv. easy	$u_{t}+u_{x}=0$	$[-1,1]{\times}[0,1]$	$\sin(\pi x)$	Analytic	$0.266{\pm}0.013$
Conv. hard	$u_{t}+30u_{x}=0$	$[-1,1]{\times}[0,1]$	$\sin(\pi x)$	Analytic	$0.951{\pm}0.003$
Schrödinger	$ih_{t}+\tfrac{1}{2}h_{xx}+\|h\|^{2}h=0$	$[-5,5]{\times}[0,1]$	Soliton	Analytic	$0.119{\pm}0.062$

Table 2: PDE benchmark. L2 error (mean

\pm

std, 3 PINN seeds) vs. independent reference.

PDE	Best concept	Mean max $\|r\|$	Z-score
Heat	gradient $\|\partial_{x}u\|$	$0.845\pm 0.039$	$>100$
Burgers	amplitude $\|u\|$	$0.785\pm 0.021$	$>80$
Allen–Cahn	signed $u$	$0.675\pm 0.025$	$>40$
Conv. easy	characteristic $x{-}t$	$0.841\pm 0.027$	$>90$
Conv. hard	characteristic $x{-}30t$	$0.943\pm 0.008$	$>120$
Schrödinger	amplitude $\|h\|$	$0.881\pm 0.052$	$>100$

Table 3: Maximum top-1 concept alignment. Values are means over 3 PINN seeds (each reported as the max over concepts and seeds). Z-score against permutation null (

n=500

Figure 1: Schrödinger: SAE atom activations (top row) vs. independently-computed reference concept fields (bottom row). Atom 73 tracks the moving soliton envelope ( $r{=}+0.83$ ). Concepts from analytic references, not the PINN output.

Figure 2: Heat: SAE atom activations (top row) vs. reference concept fields (bottom row). Atoms specialise to gradient ( $r{=}0.89$ ), curvature, and temporal derivative.

Panel of Figure 3 — Figure 3: Concept alignment matrices (Pearson $r$ ) for Burgers ( left ) and conv. hard ( right ). Rows: top-20 atoms; columns: physical concepts. Atoms tend to align with at most one or two concepts (monosemanticity).

PDE	SAE ESF 80	PCA	ICA	SD	Speedup	Max $\|r\|$	Neg. ctrl	Bilat. $\Delta R^{2}$
Heat	$\mathbf{0.165}{\pm}0.23$	0.700	0.587	0.278	4.2 $\times$	0.890	$-$ 0.025	$+$ 0.053
Burgers	$\mathbf{0.374}{\pm}0.18$	0.585	0.574	0.341	1.6 $\times$	0.804	$+$ 0.005	$+$ 0.097
Allen–Cahn	$\mathbf{0.476}{\pm}0.29$	0.576	0.622	0.412	1.2 $\times$	0.692	$+$ 0.043	$+$ 0.088
Conv. easy	$\mathbf{0.366}{\pm}0.24$	0.546	0.571	0.423	1.5 $\times$	0.872	$+$ 0.049	$+$ 0.146
Conv. hard	$\mathbf{0.373}{\pm}0.33$	0.745	0.653	0.292	2.0 $\times$	0.951	$+$ 0.125	$+$ 0.030
Schrödinger	$\mathbf{0.179}{\pm}0.11$	0.406	0.538	0.312	2.3 $\times$	0.920	$-$ 0.013	$+$ 0.102

Table 4: Aggregate main results across 6 PDEs

\times

3 PINN seeds. ESF 80 : fraction of domain capturing 80% of causal effect mass (lower = more localised = better for SAE). Neg. ctrl. adv.: ESF 80 (atom)

-

ESF 80 (matched random), negative means atom is more localised. Bilateral

\Delta R^{2}

: two-atom minus single-atom regression improvement. Consistency: Hungarian-matched mean cosine across 3 SAE seeds.

Figure 4: Negative controls: ESF 80 advantage (top-aligned atom minus matched random) split by localised ( left ) vs. global ( right ) concepts. Blue = atom more localised than random (positive result); red = random incidentally more localised (expected for global concepts).

PDE	Metric	$D{=}128$	$D{=}256$	$D{=}512$
Heat	Mean top-1 $\|\bar{r}\|$	0.815	0.831	0.836
Heat	Recon. MSE	0.128	0.075	0.042
Burgers	Mean top-1 $\|\bar{r}\|$	0.855	0.928	0.964
Burgers	Recon. MSE	0.192	0.106	0.060
Schrödinger	Mean top-1 $\|\bar{r}\|$	0.817	0.836	0.869
Schrödinger	Recon. MSE	0.240	0.129	0.068

Table 5: Overcompleteness ablation: alignment and reconstruction across dictionary sizes

D

for three representative PDEs. Mean top-1 alignment

|\bar{r}|

and reconstruction MSE on held-out activations. SAE trained at

\lambda{=}0.02

; all other settings fixed.

$\lambda$	Mean L 0	Recon. MSE	Mean top-1 $\|\bar{r}\|$	Status
$10^{-4}$	259.7	0.041	0.916	too dense
$10^{-3}$	254.8	0.040	0.918	too dense
$10^{-2}$	207.6	0.049	0.941	good
$\mathbf{0.02}$	149.1	0.070	0.964	operative
$0.1$	75.4	0.120	0.935	sparse
$0.3$	35.0	0.234	0.916	recon. degrades

Table 6: Sparsity ablation: alignment and reconstruction vs. mean L 0 across

\lambda

values for Burgers (representative PDE). Operative configuration (

\lambda{=}0.02

) highlighted; alignment is flat over a wide sparsity range.

PDE	Single atom $R^{2}$	SAE pair $R^{2}$	$\Delta R^{2}$	Random pair $\Delta R^{2}$
Heat	0.591	0.643	$+$ 0.053	$-$ 0.226
Burgers	0.387	0.483	$+$ 0.097	$-$ 0.203
Allen–Cahn	0.279	0.367	$+$ 0.088	$-$ 0.189
Conv. easy	0.451	0.598	$+$ 0.146	$-$ 0.196
Conv. hard	0.332	0.362	$+$ 0.030	$-$ 0.127
Schrödinger	0.504	0.607	$+$ 0.102	$-$ 0.215

Table 7: Bilateral alignment: mean R 2 for single atom, SAE atom pair, and random pair per PDE (averaged over concepts and seeds).

\Delta R^{2}

= pair

-

single. SAE pairs consistently improve over single atoms; random pairs decrease R 2 .

Panel of Table 8 — Table 8: SAE feature consistency across 3 independent SAE seeds per PINN. Mean and median Hungarian-matched cosine similarity between dictionaries; fraction of atoms above thresholds. Low cosine reflects non-uniqueness of sparse coding solutions, not instability of causal localization (ESF 80 std $<$ 0.05 across PINN seeds).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Burgers atom 115 ( $r=-0.52$ ). Negative correlation = atom fires where $u<0$ . Ablation concentrates in the lower-left pre-shock region: the negative- $u$ bilateral half.

Figure 7: Conv. hard ( $\beta=30$ , L2 error 95%). Despite PINN failure, the intervention map localises in the characteristic-coordinate direction.

Figure 8: Allen–Cahn (failed PINN, L2 error 93%). The SAE still localises the causal footprint to the region where the PINN did learn structure.

Figure 9: Burgers atom fields. From left: amplitude, shock gradient, curvature, temporal derivative, signed value, shock indicator, spatial coordinate.

Figure 10: Conv. hard: atom for characteristic coordinate $x-30t$ achieves $r=0.95$ .

Figure 11: Allen–Cahn: alignment weaker (max $r=0.69$ ) reflecting the PINN’s failure to resolve correct interface dynamics.

Figure 12: Conv. easy: characteristic-coordinate atom ( $r=0.87$ ).

Panel of Figure 14 — Figure 14: Alignment matrices for remaining PDEs.

PINN
Setting	Value
Hidden layers / width	5 / 128
Activation	$\tanh$
Adam iters / LR	8000 / $10^{-3}$
L-BFGS iters	300
Loss weights $w_{\mathrm{BC}},w_{\mathrm{IC}},w_{F}$	100, 100, 1

Appendix

Table 9: Full experimental hyperparameters.

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Physics-Informed Neural Network with Squeeze-Excitation-like Attention
Jun 18, 2026Yun-Fei Song, Long-Gang Pang, Fu-Peng Li +1Neural PDE Solvers Physics-Informed ML
Key Laboratory of Quark and Lepton Physics (MOE) & Institute of Particle Physics, Central China Normal University, Wuhan 430079, China · Artificial Intelligence and Computational Physics Research Center, Central China Normal University, Wuhan 430079, China · Key Laboratory of Nuclear Physics and Ion-beam Application (MOE) & Institute of Modern Physics, Fudan University, Shanghai 200433, China +2
Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results
Mar 3, 2026Alberto Miño Calero, Luis Salamanca, Konstantinos E. TatsisFourier Feature Embeddings Neural Network Interpretability
Department of Engineering Cybernetics NTNU, Trondheim, Norway · Swiss Data Science Center (SDSC) ETH Zürich, Zürich, Switzerland
PINNs Failure Modes are Overfitting
May 29, 2026Nigel T. Andersen, Takashi MatsubaraNeural Network Generalization Neural PDE Solvers
Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan

Jun 18, 2026J/K move · Enter open · S save

Physics-Informed Neural Network with Squeeze-Excitation-like Attention

Yun-Fei Song, Long-Gang Pang, Fu-Peng Li, Jun-Jie Zhang

Key Laboratory of Quark and Lepton Physics (MOE) & Institute of Particle Physics, Central China Normal University, Wuhan 430079, China · Artificial Intelligence and Computational Physics Research Center, Central China Normal University, Wuhan 430079, China · Key Laboratory of Nuclear Physics and Ion-beam Application (MOE) & Institute of Modern Physics, Fudan University, Shanghai 200433, China+2

Neural PDE Solvers Physics-Informed ML Neural Network Training Dynamics

Open paper AskPDF arXiv

We introduce SEA-PINN, a novel architecture that incorporates a Squeeze-Excitation-like attention mechanism into physics-informed neural networks to dynamically recalibrate the importance of neurons across layers. A key feature of SEA-PINN is its highly stable initialization. On 17 out of 20 benchmark problems, SEA-PINN exhibit nearly negligible variance and significantly reduced initial loss, establishing a quasi-deterministic and favorable starting point for optimization. Notably, without employing Fourier feature embeddings or periodic activation functions, SEA-PINN attained competitive accuracy (83% vs. 90% improvement relative to FNN-PINN on the high-frequency case 7) as compared with TSA-PINN-a model specifically engineered for high-frequency problems via learnable frequencies in sinusoidal activations. Furthermore, integrating SEA-PINN into TSA-PINN boosted performance by 42.49%. These results underscore SEA-PINN as a lightweight plug-in module that enhances nonlinear representation power, promotes more robust and efficient convergence, and strengthens the overall reliability of physics-informed learning.

Abstract

Mechanistic Interpretability

Explore similar work

Physics-Informed Neural Network with Squeeze-Excitation-like Attention

Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results

PINNs Failure Modes are Overfitting