A comparative study of transformer models and recurrent neural networks for path-dependent composite materials

Petter Uvdal; Mohsen Mirkhalaf

physics.comp-phFeb 17, 2026

A comparative study of transformer models and recurrent neural networks for path-dependent composite materials

Authors: Petter Uvdal, Mohsen Mirkhalaf

Organizations: Department of Physics, University of Gothenburg, Origovägen 6B, 41296 Gothenburg, Sweden

Abstract

Accurate modeling of Short Fiber Reinforced Composites (SFRCs) remains computationally expensive for full-field simulations. Data-driven surrogate models using Artificial Neural Networks (ANNs) have been proposed as an efficient alternative to numerical modeling, where Recurrent Neural Networks (RNNs) are increasingly being used for path-dependent multiscale modeling by predicting the homogenized response of a Representative Volume Element (RVE). However, recently, transformer models have been developed and they offer scalability and efficient parallelization, yet have not been systematically compared with RNNs in this field. In this study, we perform a systematic comparison between RNNs and transformer models trained on sequences of homogenized response of SFRC RVEs. We study the effect on two types of hyperparameters, namely architectural hyperparameters (such as the number of GRU layers, hidden size, number of attention heads, and encoder blocks) and training hyperparameters (such as learning rate and batch size). Both sets of hyperparameters are tuned using Bayesian optimization. We then analyze scaling laws with respect to dataset size and inference accuracy in interpolation and extrapolation regimes. The results show that while transformer models remain competitive in terms of accuracy on large datasets, the RNNs demonstrate better accuracy on small datasets and show better extrapolation performance. Furthermore, under extrapolation, there is a clear difference, where the RNN remains accurate, while the transformer model performs poorly. On the other hand, the transformer model is 7 times faster at inference, requiring 0.5 ms per prediction compared to the 3.5 ms per prediction for the RNN model.

Figures & tables

Figure 1: Fiber direction vector

\boldsymbol{p}

expressed in spherical coordinates

(\theta,\phi)

Figure 2: Example RVEs with their corresponding orientation tensor $\boldsymbol{a}$ .

Figure 3: Schematic of a GRU-based recurrent neural network.

Figure 4: Schematic of the internal computation within a GRU cell, showing update gate, reset gate, and candidate state.

Figure 5: Schematic of the Transformer encoder architecture.

Figure 6: Bayesian optimization progress showing validation RMSE for RNN and Transformer models.

Figure 7: Validation RMSE as a function of attention heads and encoder layers for the transformer.

Figure 8: Validation RMSE as a function of GRU layers and the hidden size for the RNN.

Architectural hyperparameters
Hyperparameter	RNN (GRU)	Transformer
Number of layers ( $L$ )	$[1,9]$	–
Number of encoder blocks ( $N$ )	–	$[1,6]$
Hidden size ( $H$ )	$[50,1000]$	$[50,500]$
Feed-forward size	–	$[50,500]$
Masked-Attention heads ( $A$ )	–	$[1,5]$

Table 1: Hyperparameters included in the Bayesian optimization search space for the RNN and Transformer models.

Hyperparameter	RNN	Transformer
Hidden size	817	56
Feed-forward size	–	343
Layers ( $L$ ) / Encoder Blocks ( $N$ )	2	2
Masked-Attention Heads ( $A$ )	–	1

Table 2: Optimized hyperparameters from Bayesian optimization for the RNN and Transformer models.

Figure 9: Training and validation RMSE for optimized (a) RNN and (b) Transformer.

Figure 10: Scaling behavior with dataset size: test RMSE and MAE versus dataset size. RNN and transformer plotted with consistent color/marker coding across experiments.

Dataset (R20)	Test	Validation	Training
RNN RMSE [MPa]	3.50	3.32	2.84
RNN MaE [MPa]	7.56	7.04	5.99
Transformer RMSE [MPa]	3.64	3.45	1.98
Transformer MaE [MPa]	8.83	7.65	4.38
RNN MeRE	3.79%	3.53%	3.08%
RNN MaRE	7.94%	7.46%	6.43%

Table 3: MeRE and MaRE (relative errors, %) and RMSE and MAE (MPa) for R20

Figure 11: Network predictions against target values of von-Mises stress for two random multiaxial loading paths selected from the test dataset.

Figure 12: A uniaxial loading: micromechanical simulaiton against network predictions showing that RNN outperforms Transformer model in this case which is outside the training data.

Sample	$v_{f}$	$\lambda_{1}$	$\lambda_{2}$	$\lambda_{3}$
1	0.115	0.79	0.20	0.02
2	0.132	0.60	0.35	0.05
3	0.141	0.89	0.08	0.03
4	0.114	0.79	0.17	0.04
5	0.131	0.65	0.28	0.07
6	0.125	0.54	0.27	0.19

Table 4: Fiber volume fraction and eigenvalues (

\lambda_{i}

) of the orientation tensor

\mathbf{a}

of RVEs simulated under the specific loading conditions.

Panel of Figure 13 — Figure 13: Orientation tensor ellipsoid for (a) sample 4 and (b) sample 7 used for the cyclic loading tests.

Dataset	Cyclic loading
RNN RMSE [MPa]	5.74
RNN MaE [MPa]	15.09
Transformer RMSE [MPa]	13.42
Transformer MaE [MPa]	34.78
RNN MeRE	5.37%
RNN MaRE	14.15%

Table 5: The von Mises MeRE and MaRE (relative errors, %) and RMSE and MAE (MPa), averaged for all cyclic loadings

Figure 14: 11 RVEs under the five cyclic loading paths: uniaxial normal stress (

\sigma_{11}

), uniaxial shear stress (

\sigma_{12}

), biaxial stress in two normal directions (

\sigma_{11}{+}\sigma_{22}

), biaxial stress in normal and shear (

\sigma_{11}{+}\sigma_{23}

), and a plane strain (

\epsilon_{11}{+}\epsilon_{22}

). Bars show the MeRE for each loading and error bars indicate the corresponding MaRE.

Neural Surrogate Modeling

Neural surrogate modeling uses learned models to approximate costly simulations, enabling faster prediction, design, control, and inference.

See all 401 papers

Recursive transformers for semiconductor thermo-mechanical reliability
Jul 28, 2026Kart-leong LimNeural Surrogate Modeling PDE Surrogate Modeling
Institute of Microelectronics (IME), Agency for Science, Technology and Research (A*STAR), Singapore
On Surrogate Modeling of Static Response of AM Short-Fiber Thermoplastics Using Graph Neural Networks
Jun 27, 2026Pharindra Pathak, Vipin Kumar, Trenton M. Ricks +2Neural Surrogate Modeling Structural Mechanics
Auburn University, Oakridge National Lab, NASA Glenn Research Center, Auburn University, Auburn University · Graduate Student, Department of Aerospace Engineering, Auburn University, Auburn, AL · Sr. R&D Staff Member, Manufacturing Science Division, ORNL, Knoxville, TN +3
Hybrid Unet-Transformer Model for Generating Stress and Strain Fields from Composite Geometrics
Jun 30, 2026Shrey PatelNeural Surrogate Modeling Hybrid CNN-Transformer Architectures
University of Maryland-College Park United States of America

Jul 28, 2026J/K move · Enter open · S save

Recursive transformers for semiconductor thermo-mechanical reliability

Kart-leong Lim

Institute of Microelectronics (IME), Agency for Science, Technology and Research (A*STAR), Singapore

Neural Surrogate Modeling PDE Surrogate Modeling Recurrent Transformers

Open paper AskPDF arXiv

Transformer-based surrogate models are increasingly used to replace expensive first-principles simulation in engineering design. But conventional transformer architectures are often over parameterized for the small, low-dimensional datasets typical of engineering design spaces, where large simulation data is expensive to generate. Under these conditions, excess parameter capacity leads to overfitting rather than improved accuracy, while also incurring unnecessary memory and compute overhead. This motivates a shift towards architectures that focus on additional compute rather than additional learnable parameters. This paper presents a hardware-aware evaluation of three recursive transformer paradigms for surrogate thermo-mechanical analysis of advanced packages: a)Tiny Recursive Model, b) our proposed Depth Recursive transformer, c) and a simple recursive transformer. We systematically compare their predictive performance (Recall, Mean Reciprocal Rank), parameter count, computational complexity (FLOPs), providing practical design guidelines for selecting recursive transformer architectures under resource-constrained scenarios. We validate this principle on two low-dimensional engineering prediction tasks: 1) thermo-mechanical reliability analysis of advanced semiconductor packages, where stress and warpage from thermal cycling must be evaluated repeatedly across a design-of-experiments sweep under costly finite element analysis (FEA). 2) Laplace PDE iterative numerical solver for capacitance field. Overall, recursive weight-sharing transformers provide an effective and generalizable trade-off between prediction accuracy, parameter efficiency, and computational cost for small data engineering surrogate modeling.

Abstract

Neural Surrogate Modeling

Explore similar work

Recursive transformers for semiconductor thermo-mechanical reliability

On Surrogate Modeling of Static Response of AM Short-Fiber Thermoplastics Using Graph Neural Networks

Hybrid Unet-Transformer Model for Generating Stress and Strain Fields from Composite Geometrics