Equivariant Spherical Transformer for Efficient Molecular Modeling
Organizations: SAIS · Shanghai Innovation Institute · Xiamen University · Sun Yat-sen University ·
Abstract
Equivariant Graph Neural Networks (GNNs) have significantly advanced the modeling of 3D molecular structure by leveraging group representations. However, message passing built only from exactly equivariant operations can have restricted practical expressiveness at a fixed representation degree. We introduce the Equivariant Spherical Transformer (EST), a plug-and-play module that applies Transformer attention to spherical spatial representations obtained from group representations. EST explicitly mixes information across sampled orientations, unlike prior orientation-wise spherical operations, while retaining a strong approximate equivariant inductive bias through near-uniform spherical sampling. As demonstrated by our experiments on challenging benchmarks like OC20, MPtrj and QM9, EST-based models achieve state-of-the-art performance. For the complex molecular systems within OC20, small models empowered by EST can outperform some larger models and those using additional data. We characterize the ideal closed-domain condition under which the continuous operation is exactly equivariant and empirically quantify the finite-sampling accuracy--equivariance--memory trade-off used in practice.
Figures & tables
| Number of | Throughput | Energy MAE | Force MAE | ||
| Dataset | Model | parameters | Samples/s | ||
| All+MD | EScAIP-Small | 83M | - | 229 | |
| EquiformerV2 ( , 8 layers) | 31M | 7.1 | |||
| All | SchNet | 9.1M | - | ||
| DimeNet++-L-F+E | 10.7M | 4.6 | |||
| SpinConv | 8.5M | 6.0 |
| Energy MAE (meV) | EwT (%) | |||||||
| Model | ID | OOD Ads | OOD Cat | OOD Both | ID | OOD Ads | OOD Cat | OOD Both |
| SchNet | 639 | 734 | 662 | 704 | 2.96 | 2.33 | 2.94 | 2.21 |
| PaiNN | 575 | 783 | 604 | 743 | 3.46 | 1.97 | 3.46 | 2.28 |
| TFN | 584 | 766 | 636 | 700 | 4.32 | 2.51 | 4.55 | 2.66 |
| DimeNet++ | 562 | 725 | 576 | 661 | 4.25 | 2.07 | 4.10 | 2.41 |
| GemNet-dT | 527 | 758 | 549 | 702 | 4.59 | 2.09 | 4.47 | 2.28 |
| Model | Energy (meV/atom) | Forces (meV/Å) | Stress (meV/Å 3 ) |
|---|---|---|---|
| CHGNet | 33 | 79 | 2.19 |
| MACE | 20 | 45 | - |
| EquiformerV2-S | 12.4 | 32.22 | 1.55 |
| + DeNS | 11.43 | 31.67 | 1.44 |
| + EST | 10.80 | 28.85 | 1.36 |
| Task | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Units | meV | meV | meV | D | cal (mol K) | meV | meV | meV | meV | meV | ||
| SchNet | .235 | 63 | 41 | 34 | .033 | .033 | 14 | 14 | .073 | 19 | 14 | 1.70 |
| TFN † | .223 | 58 | 40 | 38 | .064 | .101 | - | - | - | - | - | - |
| DimeNet++ | .044 | 33 | 25 | 20 | .030 | .023 | 8 | 7 | .331 | 6 | 6 | 1.21 |
| PaiNN | .045 | 46 | 28 | 20 | .012 | .024 | 7.35 | 5.98 | .066 | 5.83 | 5.85 | 1.28 |
| TorchMD-NET | .059 | 36 | 20 | 18 | .011 | .026 | 7.62 | 6.16 | .033 | 6.38 | 6.15 | 1.84 |
| GNN Layer | 3 fold | 10 fold | 100 fold | GNN Layer | 2 fold | 3 fold | 5 fold | 10 fold |
|---|---|---|---|---|---|---|---|---|
| EST L=1 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | TFN/MACE L=1 | 50.0 ± 0.0 | 50.0 ± 0.0 | 50.0 ± 0.0 | 50.0 ± 0.0 |
| EST L=2 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | TFN/MACE L=2 | 100.0 ± 0.0 | 50.0 ± 0.0 | 50.0 ± 0.0 | 50.0 ± 0.0 |
| EST L=3 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | TFN/MACE L=3 | 100.0 ± 0.0 | 100.0 ± 0.0 | 50.0 ± 0.0 | 50.0 ± 0.0 |
| EST L=5 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | TFN/MACE L=5 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 50.0 ± 0.0 |
| EST L=10 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | TFN/MACE L=10 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| Sampling Strategy/Number | 1-layer | 2-layer | 3-layer | 4-layer | 5-layer | 6-layer |
|---|---|---|---|---|---|---|
| FL / , w/o optim. | 0.0010 | 0.0011 | 0.0010 | 0.0018 | 0.0022 | 0.0024 |
| FL / , w optim. | 0.0002 | 0.0001 | 0.0003 | 0.0003 | 0.0002 | 0.0002 |
| FL / , w/o optim. | 0.0002 | 0.0002 | 0.0001 | 0.0002 | 0.0001 | 0.0003 |
| e3nn / , w/o optim. | 0.0084 | 0.1006 | 0.0366 | 0.0593 | 0.0046 | 0.0199 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Target | FCTP | EST | S2Activation |
|---|---|---|---|
| FCTP-local | |||
| S2Activation-local | |||
| EST-attention |
| Model | Energy MAE | Force MAE | Force cosine | Force magnitude error |
|---|---|---|---|---|
| SCN-original | 0.54533 | 0.05172 | 0.26900 | 0.08147 |
| SCN–EST | 0.56208 | 0.04931 | 0.29511 | 0.07597 |
| Sampling method | NN-CV | Area-CV | Cap Error | First place |
|---|---|---|---|---|
| e3nn grid | 0.2934 | 0.4348 | 0.1511 | 0/6 |
| Fibonacci | 0.1091 | 0.0666 | 0.0691 | 0/6 |
| Fibonacci + repulsion | 0.0339 | 0.0337 | 0.0646 | 6/6 |
| Model | MAE | Eq. error | Time/batch | Peak memory |
|---|---|---|---|---|
| EqV baseline | 0.0460 | 223 ms | 1,624 MiB | |
| EST, | 0.0414 | 246 ms | 1,944 MiB | |
| EST, | 0.0408 | 255 ms | 2,538 MiB | |
| EST, | 0.0407 | 268 ms | 4,790 MiB | |
| EST, | 0.0405 | 290 ms | 12,424 MiB |
| Setting | Metric | Baseline | Baseline + EST |
|---|---|---|---|
| MD17, gradient force | Mean NVE drift (meV/ps/atom; 8 systems) | 0.036 | 0.022 |
| MPtrj, direct force | Force equivariance MAE (meV/Å; 100 structures) | 0.359 | 0.578 |
| MPtrj, direct force | Force cosine similarity | 0.99988 | 0.99987 |
| MPtrj, direct force | Jacobian asymmetry | 0.258 | 0.464 |
| MPtrj relaxation | Converged / average (eV/Å) | 16/16 / 0.041 | 16/16 / 0.038 |
| MD17 relaxation | Converged / average (eV/Å) | 8/8 / 0.014 | 8/8 / 0.011 |
| Hyper-parameters | S2EF-ALL | IS2RE |
| Optimizer | AdamW | AdamW |
| Learning rate scheduling | Cosine learning rate with linear warmup | Cosine learning rate with linear warmup |
| Warmup epochs | ||
| Maximum learning rate | ||
| Batch size | ||
| Number of epochs |
| Hyper-parameters | EquiformerV2 | EST |
| Optimizer | AdamW | AdamW |
| Learning rate scheduling | Cosine learning rate with | Cosine learning rate with |
| linear warmup | linear warmup | |
| Warmup epochs | ||
| Warmup factor | ||
| Maximum learning rate |
| Hyper-parameters | Full-EST model (Figure 5 ) | EST+Equiformer |
|---|---|---|
| Optimizer | AdamW | AdamW |
| Learning rate scheduling | Cosine learning rate with | Cosine learning rate with |
| linear warmup | linear warmup | |
| Warmup steps | ||
| Maximum learning rate | , | , |
| Batch size |
| Hyper-parameters | EST (with GATA) | |
|---|---|---|
| properties | , , , , , , | , , , , U, ZPVE |
| Loss function | MAE | MSE |
| Warmup steps | 0 | 10000 |
| Weight decay | 0 | 0.01 |
| Intermediate dimension and the number of steerable FFN | 512, 1 | 512, 4 |
| Intermediate dimension and the number of spherical FFN | 512, 1 | 512, 4 |
| Task | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Units | meV | meV | meV | D | cal (mol K) | meV | meV | meV | meV | meV | ||
| GotenNet-B | .032 | 20.5 | 15.2 | 13.0 | .0072 | .019 | 5.19 | 3.44 | .027 | 3.49 | 3.43 | 1.09 |
| + EST | .030 | 31.6 | 15.0 | 13.1 | .0071 | .019 | 5.13 | 3.60 | .028 | 3.58 | 3.55 | 1.17 |
| GotenNet-L | .028 | 19.8 | 13.4 | 12.2 | .0067 | .019 | 4.98 | 3.30 | .024 | 3.41 | 3.37 | 1.08 |
| + EST | .028 | 30.8 | 13.7 | 12.8 | .0070 | .019 | 5.26 | 3.41 | .027 | 3.50 | 3.53 | 1.18 |
| Metric | FCTP | EST | S2Activation | S2MLP |
|---|---|---|---|---|
| Train loss | ||||
| Valid loss |
| Metric | FCTP | EST | S2Activation | S2MLP |
|---|---|---|---|---|
| Train loss | ||||
| Valid loss |
| Model | energy (ID) MAE | energy (OOD Ads) MAE | energy (OOD Cat) MAE | energy (OOD Both) MAE | average |
|---|---|---|---|---|---|
| Equiformer (1 + 0) | 504 | 688 | 521 | 630 | 586 |
| Equiformer + EST (5 + 5) | 501 | 652 | 502 | 578 | 558 |
| Equiformer + EST (0 + 1) | 507 | 664 | 513 | 593 | 569 |
| Equiformer + EST (1 + 1) | 502 | 657 | 504 | 588 | 563 |
| Model | |||
|---|---|---|---|
| Equiformer (1 + 0) | 0.046 | 15 | 6.59 |
| Equiformer + EST (4 + 6) | 0.041 | 14 | 5.64 |
| Equiformer + EST (0 + 1) | 0.045 | 15 | 6.85 |
| Equiformer + EST (1 + 1) | 0.042 | 14 | 5.73 |
| Equiformer + EST (10 + 0) | 0.043 | 15 | 6.47 |
| Equiformer + EST (8 + 0) | 0.044 | 15 | 6.41 |
| Building blocks in EST | MAE | ||||
| LayerNorm | SA | Steerable FFN | Spherical FFN | FL Sampling | |
| - | ✓ | ✓ | ✓ | ✓ | 0.046 |
| - | - | ✓ | ✓ | ✓ | 0.044 |
| ✓ | ✓ | - | ✓ | ✓ | 0.045 |
| - | - | ✓ | - | ✓ | 0.043 |
| ✓ | ✓ | ✓ | ✓ | - | 0.053 |
| model | energy (meV/atom) | forces (meV/Å) | stress (meV/Å 3 ) | forces cos |
|---|---|---|---|---|
| EquiformerV2-S (Repetition) | 11.1 | 29.48 | 1.39 | 0.74 |
| EquiformerV2-S + EST | 10.80 | 28.85 | 1.36 | 0.74 |
| Channels | EST Latency (ms) | DTP Latency (ms) | FCTP Latency (ms) | Speedup vs DTP | Speedup vs FCTP | |
|---|---|---|---|---|---|---|
| 1 | 32 | 0.598 | 0.517 | 1.051 | 0.865 | 1.757 |
| 1 | 64 | 0.540 | 0.512 | 0.975 | 0.948 | 1.805 |
| 1 | 128 | 0.555 | 0.565 | 0.953 | 1.019 | 1.718 |
| 1 | 256 | 0.528 | 0.581 | 0.972 | 1.100 | 1.839 |
| 1 | 512 | 0.522 | 0.538 | 2.654 | 1.031 | 5.081 |
| 2 | 32 | 0.534 | 1.327 | 2.171 | 2.488 | 4.069 |
| Task | ZPVE | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Equiformer + EST update | .041 | 29 | 14 | 13 | .011 | .021 | 7.18 | 6.17 | .227 | 6.35 | 6.32 | 1.27 |
| Full EST | .042 | 28 | 13.4 | 12.4 | .0113 | .022 | 7.03 | 5.94 | .298 | 5.92 | 5.64 | 1.31 |
Equivariant GNNs
Equivariant graph neural networks encode how node and edge features transform under symmetries to make geometry-aware predictions.
See all 51 papers