NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generation
Organizations: Department of Pharmaceutical Sciences, University of Vienna · Christian Doppler Laboratory for Molecular Informatics in the Biosciences · Vienna Doctoral School of Pharmaceutical, Nutritional and Sport Sciences Josef-Holaubek-Platz 2, 1090 Vienna, Austria
Abstract
Transformer-based autoregressive models offer an efficient alternative to diffusion- and flow-based approaches for generating 3D molecules. One challenge remains: standard transformer architectures require a sequential ordering of tokens, which is not inherently defined for the atoms in a molecule. Prior works have addressed this by using canonical atom orderings. However, these approaches are not permutation invariant w.r.t. atoms and bias next-token prediction towards ordering conventions. We overcome this limitation by introducing a novel neighborhood-guided training strategy. Our model, NEAT (Neighborhood-Guided, Efficient, Autoregressive Set Transformer) treats molecular graphs as sets of atoms and learns an order-agnostic distribution over admissible tokens at the graph boundary, thereby ensuring atom-level permutation invariance. NEAT achieves state-of-the-art generation quality on the QM9 and GEOM-Drugs datasets, enables seamless completion of molecular prefixes, and offers a significant speed advantage over existing baselines.
Figures & tables
| Model | atom stable | mol stable | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique | xyz2mol novel |
| QM9 | 99.3 | 94.8 | 97.5 | 97.5 | 100.0 | 100.0 | 99.8 |
| G-SchNet* | 93.7 | 63.6 | 80.1 | 74.7 | 73.0 | 70.6 | 55.1 |
| G-SphereNet* | 67.8 | 14.0 | 16.4 | 3.8 | 37.7 | 7.2 | 6.7 |
| Symphony* | 90.8 | 43.9 | 68.1 | 66.5 | 77.0 | 75.2 | 41.3 |
| Geo2Seq | 98.9 | 93.2 | 97.1 | 81.7 | n.a. | n.a. | n.a. |
| QUETZAL** | 98.3 0.1 | 87.5 0.1 | 94.6 0.1 | 90.1 0.2 | 93.7 0.3 | 89.1 0.2 | 24.7 0.3 |
| Model | atom stable | mol stable | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique | xyz2mol novel |
| FlowMol3 (inpainted) | 76.3 6.5 | 0.4 1.3 | 44.5 16.3 | 36.3 20.1 | 30.2 7.8 | 29.4 7.3 | 29.4 7.3 |
| QUETZAL | 69.0 1.8 | 0.1 0.1 | 27.8 6.8 | 13.6 4.1 | 28.1 4.7 | 17.7 2.8 | 17.7 2.8 |
| NEAT (ours) | 77.0 1.5 | 0.5 0.3 | 97.8 1.8 | 82.2 3.0 | 92.7 2.3 | 77.9 3.0 | 77.9 3.0 |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
| batch size | 2048 |
| 1.5 | |
| bias | true |
| dropout | 0.1 |
| epochs | 10,000 |
| flow matching head hidden dimension | 1,536 |
| Parameter | Value |
| batch size | 512 |
| 1.5 | |
| bias | true |
| dropout | 0.1 |
| epochs | 1,000 |
| flow matching head hidden dimension | 1,536 |
| transformer heads | transformer layers | transformer dimension | model parameters | atom stable | mol stable | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique | xyz2mol novel |
| 6 | 6 | 384 | 89.46 | 96.8 | 74.8 | 89.6 | 84.3 | 91.5 | 86.9 | 52.5 |
| 8 | 8 | 512 | 104.32 | 97.4 | 79.9 | 92.4 | 86.7 | 93.7 | 88.9 | 53.2 |
| 10 | 10 | 640 | 128.62 | 97.6 | 81.6 | 93.2 | 88.0 | 94.2 | 90.0 | 53.3 |
| 12 | 12 | 768 | 164.72 | 97.8 | 83.1 | 93.3 | 88.8 | 95.2 | 91.6 | 53.5 |
| 14 | 14 | 896 | 214.97 | 97.6 | 81.1 | 93.0 | 89.1 | 94.3 | 91.2 | 54.5 |
| transformer heads | transformer layers | transformer dimension | FM head dimension | model parameters | atom stable | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique |
| 6 | 6 | 384 | 1536 | 89.46 | 79.4 | 99.7 | 98.3 | 86.2 | 85.2 |
| 8 | 8 | 512 | 1536 | 104.32 | 80.4 | 99.8 | 97.6 | 91.7 | 89.9 |
| 10 | 10 | 640 | 1536 | 128.62 | 81.5 | 99.8 | 96.7 | 94.0 | 91.4 |
| 12 | 12 | 768 | 1536 | 164.72 | 81.5 | 99.7 | 96.9 | 93.9 | 91.6 |
| 14 | 14 | 896 | 1536 | 214.97 | 80.9 | 99.7 | 96.7 | 94.4 | 91.8 |
| 16 | 16 | 1024 | 1536 | 281.72 | 80.9 | 99.8 | 96.1 | 94.7 | 91.5 |
| centering | noising | sampling | atom stable | mol stable | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique | xyz2mol novel |
| no | no | greedy | 81.1 | 0.5 | 97.9 | 97.4 | 74.2 | 73.7 | 73.7 |
| no | no | multinomial | 78.1 | 1.8 | 98.6 | 91.8 | 50.8 | 47.1 | 47.1 |
| no | no | hybrid | 79.0 | 2.5 | 99.5 | 83.6 | 53.7 | 45.1 | 45.1 |
| yes | no | greedy | 82.3 | 1.6 | 99.2 | 91.7 | 87.8 | 80.9 | 80.9 |
| yes | no | multinomial | 81.6 | 1.6 | 99.2 | 97.7 | 85.4 | 84.2 | 84.2 |
| yes | no | hybrid | 82.7 | 2.5 | 99.7 | 95.5 | 88.7 | 85.2 | 85.2 |
| pooling strategy | atom stable | mol stable | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique | xyz2mol novel |
| additive | 97.8 | 83.1 | 93.3 | 88.8 | 95.2 | 91.6 | 53.5 |
| attention | 95.0 | 63.2 | 83.6 | 81.4 | 89.5 | 87.4 | 56.0 |
| QM9 | GEOM | |||||||
| atom stab. | mol. stab. | lookup valid | lookup unique | xyz2mol valid | xyz2mol unique | atom stab. | lookup valid | |
| Test data | 99.3 | 94.8 | 97.5 | 97.5 | 100 | 100 | 86.2 | 96.6 |
| EDM | 98.7 † | 82.0 † | 91.9 † | 90.7 † | 86.7 †,‡ | 86.0 †,‡ | 81.3 † | 92.6 † |
| GeoLDM | 98.9 | 89.4 | 93.8 | 92.7 | 91.3 †,‡ | 90.3 †,‡ | 84.4 | 99.3 |
| GCDM | 98.7 | 86.0 | 94.9 | 93.4 | n.a. | n.a. | 88.1 | 95.5 |
| GeoBFN | 99.3 | 93.3 | 96.9 | 92.4 | n.a. | n.a. | 86.2 | 91.7 |
| Model | Steps | Runtime | xyz2mol valid | xyz2mol unique |
| SemlaFlow | 100 | 1064 | ||
| SemlaFlow | 60 | 662 | ||
| SemlaFlow | 15 | 188 | ||
| FlowMol3 | 250 | 3420 | ||
| FlowMol3 | 60 | 855 | ||
| FlowMol3 | 15 | 236 |
3D Molecular Generation
3D molecular generation creates molecules by modeling their chemical structures and three-dimensional geometries.
See all 60 papers