EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

Yi-Lun Liao; Alexander J. Hoffman; Sabrina C. Shen; Alexandre Duval; Sam Walton Norwood; Tess Smidt

cs.LGApr 10, 2026

EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

Authors: Yi-Lun Liao, Alexander J. Hoffman, Sabrina C. Shen, Alexandre Duval, Sam Walton Norwood, Tess Smidt

Organizations: Massachusetts Institute of Technology (MIT) · Mirror Physics · Entalpic

Abstract

As $SE(3)$ -equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the $SE(3)$ -equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving $1.75\times$ speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU- $S^2$ activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling $S^2$ grids. Together, SwiGLU- $S^2$ activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.

Figures & tables

Figure 1: EquiformerV3 architecture. The proposed improvements are highlighted in red . We encode input 3D atomistic graphs with atom and edge-degree embeddings and process with Transformer blocks, which consist of equivariant merged layer normalization (LN), equivariant graph attention and feedforward networks.

Figure 2: Illustration of how different normalizations calculate statistics. In the proposed equivariant merged layer normalization, we share the merged RMS across all degrees as highlighted in red .

Figure 3: Illustration of different activation functions. The colorful circles are grid features. One circle represents one channel, which contains $R_{\phi}\times R_{\theta}$ grid points on $S^{2}$ . In the proposed SwiGLU- $S^{2}$ activation, we apply both nonlinearity and multiplication to grid features as highlighted in red .

				Number of	Training time
Index	Method	Energy	Force	parameters	(GPU-hours)
1	EquiformerV2 baseline	296	21.23	54M	270
2	+ Predicting total energy	242	19.73	54M	270
3	+ Better implementation	242	19.73	54M	154
4	+ Equivariant merged LN	236	19.28	54M	150
5	+ Improved FFN hyper-parameters	209	18.96	66M	163

Table 1: Ablation studies on EquiformerV3 architecture. All models are trained on the 2M split of the OC20 S2EF dataset, and errors are averaged over the four validation sub-splits. We report mean absolute errors (MAE) for forces in meV/Å and energy in meV, and lower is better. Training time is measured on H100 GPUs. “LN” denotes layer normalization.

						Number of
Index	Model	Prediction	Energy	Force	Stress	parameters
	EquiformerV2-S	Direct	11	49.2	2.4	31M
	EquiformerV2-M	Direct	10	44.8	2.3	87M
	EquiformerV2-L	Direct	9.6	43.1	2.3	154M
	eSEN	Gradient	10.7	47.3	2.6	30M
	UMA-S	Gradient	11.3	57.1	2.9	6M

Table 2: Results on the OMat24 validation set. We report mean absolute errors (MAE) for energy in meV per atom, forces in meV/Å and stress in meV/

\textup{\AA}^{3}

, and lower is better. “Prediction” denotes whether a model uses “Direct” or “Gradient” methods to predict forces and stress. “grad. ft.” denotes gradient fine-tuning.

Model	Prediction	F1 $\uparrow$	RMSD $\downarrow$	$\kappa_{\text{SRME}}\downarrow$	CPS $\uparrow$
MACE-MP-0	Gradient	0.669	0.091	0.682	0.637
ORB v2 MPtrj	Direct	0.765	0.101	1.726	0.470
EquiformerV2 † ‡	Direct	0.815	0.076	1.676	0.522
GRACE-2L-MPtrj	Gradient	0.691	0.090	0.525	0.681
SevenNet-l3i5	Gradient	0.760	0.085	0.550	0.714
eSEN-30M-MP ‡	Gradient	0.831	0.075	0.340	0.797

Table 3: Matbench Discovery results of compliant models trained on only MPtrj. “Prediction” denotes whether a model uses “Direct” or “Gradient” methods to predict forces and stress.

\dagger

corresponds to “eqV2 S DeNS” on the leaderboard.

\ddagger

denotes training with the auxiliary task of DeNS ( Liao et al., 2024a ) .

						Training time
Model	Prediction	F1 $\uparrow$	RMSD $\downarrow$	$\kappa_{\text{SRME}}\downarrow$	CPS $\uparrow$	(GPU-hours)
ORB v2	Direct	0.880	0.097	1.734	0.528	-
EquiformerV2 †	Direct	0.917	0.069	1.771	0.558	-
MACE-MPA-0	Gradient	0.852	0.073	0.412	0.795	-
MatterSim v1 5M	Gradient	0.862	0.073	0.575	0.767	-
eSEN-30M-OAM	Gradient	0.925	0.061	0.170	0.888	-

Table 4: Matbench Discovery results of non-compliant models trained on larger datasets like OMat24 and sAlex. “Prediction” denotes whether a model uses “Direct” or “Gradient” methods to predict forces and stress.

\dagger

corresponds to “eqV2 M” on the leaderboard.

\ddagger

denotes training with the auxiliary task of DeNS. The training time of UMA-M-1.1 is measured on H200 GPUs, and others use H100.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Counterexamples from Pozdnyakov et al. (2020)
	2-body	3-body	4-body
Model		(Fig. 1(b))	(Fig. 2(f))
Gate activation
Number of FFNs = 1	50.0	50.0	50.0
Number of FFNs = 2	50.0	50.0	50.0
Number of FFNs = 3	50.0	50.0	50.0

Appendix

Table 5: Results on the body-order experiments in Joshi et al. (2023) . The

k

-body counterexample contains two geometric graphs that are indistinguishable using

k

-body scalarization. “FFNs” denotes feedforward networks. We mark results where models successfully distinguish two geometric graphs in a counterexample and thus achieve

100\%

accuracy in green . Otherwise, they achieve

50\%

accuracy and are marked in red .

Table 6: Equivariance errors when using different activations in feedforward networks. We consider maximum degree

L_{max}=2,4,6

. For each

L_{max}

, we vary the numbers of grid points

(R_{\phi},R_{\theta})

sampled along

\phi

and

\theta

and report the resulting equivariance errors. The equivariance errors of the gate activation are reported as the baseline of strict equivariance. We mark results achieving similar equivariance errors to the gate activation function in green . Otherwise, they are marked in red .

Table 7: Equivariance errors when using different activations in attention. We consider maximum degree

L_{max}=4,6

and maximum order

M_{max}=2

. For each

L_{max}

, we start with the minimal number of grid points

(R_{\phi},R_{\theta})

that enables the SwiGLU-

S^{2}

activation to have comparable equivariance errors to the gate activation as in Table 6 . Then, we further reduce

R_{\phi}

. We mark results achieving similar equivariance errors to the gate activation function in green . Otherwise, they are marked in red .

Hyper-parameters	Value or description
Optimizer	AdamW
Learning rate scheduling	Cosine learning rate with linear warmup
Warmup epochs	$0.1$
Maximum learning rate	$2\times 10^{-4}$
Batch size	$64$
Number of epochs	$12$

Appendix

Table 8: Hyper-parameters for the base model setting on OC20 S2EF-2M dataset.

Hyper-parameters	Direct pre-training	Gradient fine-tuning
Optimizer	AdamW	AdamW
Learning rate scheduling	Cosine learning rate with linear warmup	Cosine learning rate with linear warmup
Warmup epochs	$0.1$	$0.1$
Maximum learning rate	$2\times 10^{-4}$	$1\times 10^{-4}$
Batch size	$512$	$512$
Number of epochs	$4$	$2$

Appendix

Table 9: Hyper-parameters for the OMat24 dataset.

Hyper-parameters	Direct pre-training	Gradient fine-tuning
Optimizer	AdamW	AdamW
Learning rate scheduling	Cosine learning rate with linear warmup	Cosine learning rate with linear warmup
Warmup epochs	$0.1$	$0.1$
Maximum learning rate	$2\times 10^{-4}$	$5\times 10^{-5}$
Batch size	$512$	$512$
Number of epochs	$70$	$10$

Appendix

Table 10: Hyper-parameters for the MPtrj dataset.

Hyper-parameters	Gradient fine-tuning
Optimizer	AdamW
Learning rate scheduling	Cosine learning rate with linear warmup
Warmup epochs	$0.1$
Maximum learning rate	$5\times 10^{-5}$
Batch size	$256$
Number of epochs	$2$

Appendix

Table 11: Hyper-parameters for fine-tuning OMat24-pretrained models on MPtrj and subsampled Alexandria datasets.

Equivariant GNNs

Equivariant graph neural networks encode how node and edge features transform under symmetries to make geometry-aware predictions.

See all 42 papers

GEqTrain: A Configuration-Driven Framework for Retargeting Equivariant Graph Neural Networks Across 3D Scientific Tasks
Jul 21, 2026Daniele Angioletti, Marco Nobile, Vittorio LimongelliFlow Matching Generative Modeling
Faculty of Biomedical Sciences, Euler Institute, Universit´a della Svizzera italiana, 6900 Lugano, Switzerland
Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning
May 9, 2026Chen Wang, Siyu Hu, Guangming Tan +1Equivariant GNNs Structured Pruning
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
EqGINO: Equivariant Geometry-Informed Fourier Neural Operators for 3D PDEs
Jun 2, 2026Sungwon Kim, Juho Song, Seungmin Shin +3Fourier Neural Operator PDE Surrogate Modeling
Graduate School of Data Science, KAIST, Daejeon, Republic of Korea · Mathematical Sciences, KAIST, Daejeon, Republic of Korea · LG Electronics, Pyeong-taek, Republic of Korea +1

Jul 21, 2026J/K move · Enter open · S save

GEqTrain: A Configuration-Driven Framework for Retargeting Equivariant Graph Neural Networks Across 3D Scientific Tasks

Daniele Angioletti, Marco Nobile, Vittorio Limongelli

Faculty of Biomedical Sciences, Euler Institute, Universit´a della Svizzera italiana, 6900 Lugano, Switzerland

Flow Matching Generative Modeling Equivariant GNNs Equivariant Representation Learning

Open paper AskPDF arXiv

Equivariant graph neural networks provide a powerful modeling language for three-dimensional scientific data, but their reuse is often limited by implementations tied to specific tasks, outputs, and training regimes. We present GEqTrain, a configuration-driven framework that separates dataset semantics, model composition, and training objectives. Raw data are mapped to typed node-, edge-, and graph-level fields, while model stacks, losses, and training workflows are assembled declaratively through Hydra configurations. A shared equivariant backbone and training infrastructure can therefore be retargeted to a new task primarily through configuration. We demonstrate this flexibility on three different problems handled within one software stack: coarse-grained-to-atomistic backmapping of biomolecular systems, prediction of NMR chemical shifts in molecular solids, and equivariant generative modeling. Our aim is not to surpass individually optimized task-specific systems, but to show that a shared representation and training infrastructure can achieve competitive accuracy across qualitatively different tasks at the cost of a configuration change. We further introduce GEqDiff, a generative extension based on equivariant flow matching. GEqDiff treats user-defined equivariant fields as first-class generation targets, jointly transporting Cartesian positions and non-scalar node fields spanning representations up to l=3 within a single equivariant flow. We validate this capability on a controlled synthetic benchmark inspired by protein secondary-structure motifs, showing that fields with heterogeneous transformation properties can be reconstructed jointly and with high fidelity. By reducing the software overhead of moving between predictive and generative, scalar and tensorial settings, GEqTrain aims to make equivariant modeling more reproducible, extensible, and reusable.

Abstract

Equivariant GNNs

Explore similar work

GEqTrain: A Configuration-Driven Framework for Retargeting Equivariant Graph Neural Networks Across 3D Scientific Tasks

Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning

EqGINO: Equivariant Geometry-Informed Fourier Neural Operators for 3D PDEs