On architectural choices for interpretability and thermodynamic consistency in Physically Recurrent Neural Networks in the low-data regime

M. A. Maia; K. A. Meyer; A. M. C. M. van Gils; I. B. C. M. Rocha; F. P. van der Meer

cs.LGOct 2, 2026

On architectural choices for interpretability and thermodynamic consistency in Physically Recurrent Neural Networks in the low-data regime

Authors: M. A. Maia, K. A. Meyer, A. M. C. M. van Gils, I. B. C. M. Rocha, F. P. van der Meer

Organizations: Delft University of Technology, Department of Civil Engineering and Geosciences, P.O.Box 5048, 2600GA, Delft, The Netherlands · Chalmers University of Technology, Department of Mechanical Engineering, Gothenburg, Sweden

Abstract

In this paper, we unravel the effect of different decoder architectures on the interpretability of the latent space of the Physically Recurrent Neural Network. Particular emphasis is given to a new weight normalization constraint, which acts as a regularization technique and enables robust training in the low-data regime. A brief visual exploration illustrates how these changes impact the latent space and how the fictitious stress can align with the true state of the RVE without explicit training. Reaping the benefits of a meaningful latent space, a case study illustrates how information from the microscopic level can be retrieved and incorporated into a multi-task approach that does not require extra parameters or larger training sets. Another key contribution shows that a specific architectural choice can naturally lead to a thermodynamically consistent formulation. By enforcing an adjoint encoder-decoder structure with positive scalar contributions, this modification ensures energy consistency across scales and non-negative dissipation, leading to even lower training requirements. This alternative completes the study on interpretability, inductive bias, and thermodynamic consistency, and demonstrates that data efficiency can be improved with careful architectural choices rooted in the underlying physics.

Figures & tables

Figure 1: FE 2 scheme exemplified for composite RVE with elastoplastic matrix and elastic fibers.

Figure 2: Classical PRNN architecture with dense connections and elastoplastic models embedded in the material layer.

Figure 3: Representation of weights of PRNN with 4 material points: full connectivity between output and material layer allows high expressivity.

Figure 4: Decoder connectivities investigated in this work and their degree of sparsification, interpretability and flexibility, with

m

being the number of fictitious material points.

Figure 5: Thermodynamically consistent and equivalent PRNN architectures: light-blue box implements an energy-based network, while light-brown box corresponds to the conventional stress-based formulation.

Figure 6: Relative error over $\mathcal{T}_{\textrm{GP}}$ for different material layer sizes and training set sizes considering unnormalized decoders..

Panel of Figure 7 — Figure 7: Training and validation losses from different initializations of PRNNs with Dense and TC decoders trained on one curve.

Panel of Figure 8 — Figure 8: Training and test paths for PRNN with TC decoder and 30 material points trained on one loading path.

Panel of Figure 9 — Figure 9: Macroscopic increment of dissipation from representative proportional loading path with two unloading phases (light gray zones) computed with micromodel and PRNN with TC trained on one GP-based path and 4 material points.

Figure 10: Relative error over $\mathcal{T}_{\textrm{GP}}$ for different combinations of material layer and training set size considering normalized decoders.

Figure 11: Relative error over training and validation sets for PRNNs using the Normalized point decoder.

Panel of Figure 12 — Figure 12: Relative error over $\mathcal{T}_{\textrm{GP}}$ of fictitious stresses with different PRNN architectures trained on 5 paths compared to true microscopic stress distribution over matrix phase.

Figure 13: Histogram distributions from the stresses computed in the matrix phase of the micromodel vs the stresses obtained from the fictitious material points of PRNNs with unnormalized decoders (color-coded) and 30 points at several time steps of an unseen GP-based path.

Figure 14: Histogram distributions from the stresses computed in the matrix phase of the micromodel vs the stresses obtained from the fictitious material points of PRNNs with normalized decoders (color-coded) and 30 points at several time steps of an unseen GP-based path.

Figure 15: Relative error between mean fictitious plastic strains and mean microscopic plastic strains (matrix phase only) over paths from

\mathcal{T}_{\textrm{GP}}

using PRNNs with different decoders.

Figure 16: Relative test set errors of Normalized component PRNNs over the unseen task of predicting the maximum hydrostatic matrix stress.

Figure 17: Joint vs sequential learning for double-task model.

Panel of Figure 18 — Figure 18: Comparison of Normalized component PRNNs trained on 5 curves adopting joint vs sequential learning on Maximum and Average tasks over test set $\mathcal{T}_{\textrm{GP}}$ .

Panel of Figure 19 — Figure 19: Test set errors of PRNNs trained for single and double-task and joint learning with limited data sets over the homogenized stress and maximum microscopic hydrostatic stress.

Panel of Figure 20 — Figure 20: Hydrostatic stress from PRNN with 8 material points and trained on 5 GP-based curves on unseen loading path from $\mathcal{T}_{\textrm{GP}}^{\textrm{max.}}$ .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 21 — Figure 21: Relative error of fictitious stresses from normalized and sparse decoders trained on 5 curves compared to true microscopic stresses over matrix phase and full micromodel.

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 302 papers

PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders
Sep 7, 2026Nandita N. Patil, Eshwar R. A., Gajanan V. HonnavarMechanistic Interpretability Causal Interpretability
QuaNad Research Laboratory, PES University, Bengaluru, India · Independent Researcher, QNu Labs Pvt Ltd, Bengaluru, India. Former Professor at PES University (EC Campus), Bengaluru
A Comparative Investigation of Thermodynamic Structure-Informed Neural Networks
Mar 26, 2026Guojie Li, Liu HongPhysics-Informed ML PDE Inverse Problems
School of Mathematics, Sun Yat-sen University, Guangzhou, 510275, China.
pi-Net: Empirical Latent Representations for Differentiable Physics Heads in Data-Scarce Systems
Oct 12, 2026Akshay Dhalpe, Arnab Chattopadhyay, Shahriar Bakrani Balani +2Physics-Informed ML Surrogate Modeling
Tampere University · University of Helsinki

Sep 7, 2026J/K move · Enter open · S save

PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders

Nandita N. Patil, Eshwar R. A., Gajanan V. Honnavar

QuaNad Research Laboratory, PES University, Bengaluru, India · Independent Researcher, QNu Labs Pvt Ltd, Bengaluru, India. Former Professor at PES University (EC Campus), Bengaluru

Mechanistic Interpretability Causal Interpretability Physics-Informed ML Sparse Autoencoders

Open paper AskPDF arXiv

Physics-Informed Neural Networks (PINNs) embed PDE residuals into neural network training, but their internal representations remain opaque: it is unknown what physical features their hidden layers encode or whether those features have a localized causal role. We present PhysSAE, a mechanistic interpretability framework that trains overcomplete sparse autoencoders (SAEs) on PINN penultimate-layer activations and evaluates dictionary atoms through direct causal intervention in the original frozen hidden state:

h_{\mathrm{cf}} = h - αz_k d_k

, bypassing the SAE decoder entirely. Across six PDE families, with 3 PINN seeds and 3 SAE seeds each---we show that (i) Our discovered SAE atoms align with independently-defined physical observables (max Pearson

|r|=0.951

, always

\gg

permutation null), (ii) the causal footprint of top-aligned atom ablation is 1.2--4.2

\times

more spatially concentrated canonical than PCA or ICA interventions, and (iii) top-aligned atoms outperform matched random controls on causal localization for structured physical concepts (ESF

_{80}

advantage 0.04-0.44). Two-atom bilateral representations improve concept regression R

^2

ΔR^2\!=\!0.05\text{-}0.15

over single atoms, while random pairs decrease it by up to 0.60. These results demonstrate that PINNs develop sparse, physically structured latent representations that can be identified and causally interrogated post-hoc, opening a path toward interpretability-aware scientific machine learning.

Abstract

Recurrent Neural Networks

Explore similar work

PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders

A Comparative Investigation of Thermodynamic Structure-Informed Neural Networks

pi-Net: Empirical Latent Representations for Differentiable Physics Heads in Data-Scarce Systems