NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generation
Authors: Daniel Rose, Roxane Axel Jacob, Johannes Kirchmair, Thierry Langer
Organizations: Department of Pharmaceutical Sciences, University of Vienna · Christian Doppler Laboratory for Molecular Informatics in the Biosciences · Vienna Doctoral School of Pharmaceutical, Nutritional and Sport Sciences Josef-Holaubek-Platz 2, 1090 Vienna, Austria
Transformer-based autoregressive models offer an efficient alternative to diffusion- and flow-based approaches for generating 3D molecules. One challenge remains: standard transformer architectures require a sequential ordering of tokens, which is not inherently defined for the atoms in a molecule. Prior works have addressed this by using canonical atom orderings. However, these approaches are not permutation invariant w.r.t. atoms and bias next-token prediction towards ordering conventions. We overcome this limitation by introducing a novel neighborhood-guided training strategy. Our model, NEAT (Neighborhood-Guided, Efficient, Autoregressive Set Transformer) treats molecular graphs as sets of atoms and learns an order-agnostic distribution over admissible tokens at the graph boundary, thereby ensuring atom-level permutation invariance. NEAT achieves state-of-the-art generation quality on the QM9 and GEOM-Drugs datasets, enables seamless completion of molecular prefixes, and offers a significant speed advantage over existing baselines.
Figures & tables
Figure 1: Autoregressive generation of 3D molecules with NEAT. (1) A set transformer produces an abstract representation zn of the molecule at state n , given the atom types a1:n and the coordinates X1:n . (2) The next atom type an+1 is inferred from zn . (3) The next atom position xn+1 is modeled via a flow head, which concludes the molecule update at state n+1 .
Figure 2: Overview of NEAT’s training workflow through neighborhood guidance. The model takes molecular graphs as input. (1) For each training sample, we randomly construct a connected subgraph via Algorithm 1 . The selected nodes form the source set, and their neighboring boundary nodes define the target set. (2) The source set is encoded using a set transformer, (3) and its representation is used to predict the distribution over the next atom type. (4) For each target-set node, we sample a position from a normal distribution and construct linear interpolation paths between the sampled and true positions. These interpolation paths provide supervision for learning the vector field via flow matching. (5) The cross-entropy and flow matching losses are summed to yield the total loss.
Model
atom stable
mol stable
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
xyz2mol novel
QM9
99.3
94.8
97.5
97.5
100.0
100.0
99.8
G-SchNet*
93.7
63.6
80.1
74.7
73.0
70.6
55.1
G-SphereNet*
67.8
14.0
16.4
3.8
37.7
7.2
6.7
Symphony*
90.8
43.9
68.1
66.5
77.0
75.2
41.3
Geo2Seq
98.9
93.2
97.1
81.7
n.a.
n.a.
n.a.
QUETZAL**
98.3 ± 0.1
87.5 ± 0.1
94.6 ± 0.1
90.1 ± 0.2
93.7 ± 0.3
89.1 ± 0.2
24.7 ± 0.3
Table 1: Benchmarking of NEAT against 3D molecular generators trained on QM9 and GEOM. Metrics are reported as percentages. The first row reports results on the QM9 test set (12,402 molecules). The eighth row reports results on the GEOM test set (146,788 conformers, 30,432 unique molecules). Other rows report results computed from 10,000 generated molecules. Entries for which generated molecules could be retrieved and re-evaluated with our pipeline are marked with a single star (*). Entries for which the source code and model weights were available to generate molecules are marked with two stars (**). For these models and for NEAT, we report the mean ± one standard deviation over three runs. Otherwise, values are taken directly from original publications.
Model
atom stable
mol stable
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
xyz2mol novel
FlowMol3 (inpainted)
76.3 ± 6.5
0.4 ± 1.3
44.5 ± 16.3
36.3 ± 20.1
30.2 ± 7.8
29.4 ± 7.3
29.4 ± 7.3
QUETZAL
69.0 ± 1.8
0.1 ± 0.1
27.8 ± 6.8
13.6 ± 4.1
28.1 ± 4.7
17.7 ± 2.8
17.7 ± 2.8
NEAT (ours)
77.0 ± 1.5
0.5 ± 0.3
97.8 ± 1.8
82.2 ± 3.0
92.7 ± 2.3
77.9 ± 3.0
77.9 ± 3.0
Table 2: Prefix completion performance of NEAT, QUETZAL, and FlowMol3 trained on GEOM. Metrics are reported as percentages. For each of the 100 prefixes, we generate 1,000 molecules and compute average metric values. We then report the mean ± one standard deviation across the 100 prefix-level averages.
Figure 3: Randomly selected examples of molecules generated from three different prefixes: para-substituted benzene, 2,5-substituted furan, and 1,3-substituted indole.
Figure 4: Left : Runtime per step when generating 1000 molecules, averaged over 10 runs. Right : Comparison of xyz2mol validity x uniqueness vs. runtime between NEAT and models trained on GEOM. The number attached to each data point corresponds to the runtime in seconds followed by the number of integration steps (chosen following the recommendations of the authors) in parenthesis. Values are averaged over three runs. Error bars correspond to one standard deviation. Batch size: largest to fit on GPU (FlowMol3: 500; SemlaFlow: 10,000; Megalodon-FM: 50; QUETZAL: 2,000; NEAT: 10,000).
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Relative and absolute size distribution of the source and target set w.r.t. the original graphs in the QM9 dataset. Sampling was performed with β=1.5 and γ=0.45 .
Figure 6: Relative and absolute size distribution of the source and target set w.r.t. the original graphs in the GEOM dataset. Sampling was performed with β=1.5 and γ=0.45 .
Parameter
Value
batch size
2048
β
1.5
bias
true
dropout
0.1
epochs
10,000
flow matching head hidden dimension
1,536
Appendix
Table 3: Hyperparameters of the best performing NEAT model trained on the QM9 dataset.
Parameter
Value
batch size
512
β
1.5
bias
true
dropout
0.1
epochs
1,000
flow matching head hidden dimension
1,536
Appendix
Table 4: Hyperparameters of the best performing NEAT model trained on the GEOM dataset.
transformer heads
transformer layers
transformer dimension
model parameters ×106
atom stable
mol stable
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
xyz2mol novel
6
6
384
89.46
96.8
74.8
89.6
84.3
91.5
86.9
52.5
8
8
512
104.32
97.4
79.9
92.4
86.7
93.7
88.9
53.2
10
10
640
128.62
97.6
81.6
93.2
88.0
94.2
90.0
53.3
12
12
768
164.72
97.8
83.1
93.3
88.8
95.2
91.6
53.5
14
14
896
214.97
97.6
81.1
93.0
89.1
94.3
91.2
54.5
Appendix
Table 5: Model scaling results on the QM9 dataset. Best in bold , second-best underlined . The configuration selected for the final model is highlighted in italics .
transformer heads
transformer layers
transformer dimension
FM head dimension
model parameters ×106
atom stable
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
6
6
384
1536
89.46
79.4
99.7
98.3
86.2
85.2
8
8
512
1536
104.32
80.4
99.8
97.6
91.7
89.9
10
10
640
1536
128.62
81.5
99.8
96.7
94.0
91.4
12
12
768
1536
164.72
81.5
99.7
96.9
93.9
91.6
14
14
896
1536
214.97
80.9
99.7
96.7
94.4
91.8
16
16
1024
1536
281.72
80.9
99.8
96.1
94.7
91.5
Appendix
Table 6: Hyperparameter screening results on the GEOM dataset. Best in bold , second-best underlined . The configuration selected for the final model is highlighted in italics , chosen to balance optimal performance metrics with model size.
centering
noising
sampling
atom stable
mol stable
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
xyz2mol novel
no
no
greedy
81.1
0.5
97.9
97.4
74.2
73.7
73.7
no
no
multinomial
78.1
1.8
98.6
91.8
50.8
47.1
47.1
no
no
hybrid
79.0
2.5
99.5
83.6
53.7
45.1
45.1
yes
no
greedy
82.3
1.6
99.2
91.7
87.8
80.9
80.9
yes
no
multinomial
81.6
1.6
99.2
97.7
85.4
84.2
84.2
yes
no
hybrid
82.7
2.5
99.7
95.5
88.7
85.2
85.2
Appendix
Table 7: Impact of centering, noising, and sampling strategies on the GEOM model. Best in bold , second-best underlined . The configuration selected for the final model is highlighted in italics .
pooling strategy
atom stable
mol stable
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
xyz2mol novel
additive
97.8
83.1
93.3
88.8
95.2
91.6
53.5
attention
95.0
63.2
83.6
81.4
89.5
87.4
56.0
Appendix
Table 8: Impact of additive vs. attention pooling on the QM9 model.
Figure 7: Comparison of generated molecules to the training set across eight molecular properties: molecular weight, Crippen logP, topological polar surface area, ring count, fraction of rotatable bonds, fraction of heteroatoms, fraction of halogen atoms, and fraction of stereocenters. Each panel shows overlaid histograms (training: red; generated: blue), reported as frequency in percent. Fractional properties are normalized by heavy atom count. Integer-valued properties use integer-centered bins.
QM9
GEOM
atom stab.
mol. stab.
lookup valid
lookup unique
xyz2mol valid
xyz2mol unique
atom stab.
lookup valid
Test data
99.3
94.8
97.5
97.5
100
100
86.2
96.6
EDM
98.7 †
82.0 †
91.9 †
90.7 †
86.7 †,‡
86.0 †,‡
81.3 †
92.6 †
GeoLDM
98.9
89.4
93.8
92.7
91.3 †,‡
90.3 †,‡
84.4
99.3
GCDM
98.7
86.0
94.9
93.4
n.a.
n.a.
88.1
95.5
GeoBFN
99.3
93.3
96.9
92.4
n.a.
n.a.
86.2
91.7
Appendix
Table 9: Performance comparison of NEAT with diffusion or Bayesian flow (middle block) and autoregressive (lower block) models trained on the QM9 and GEOM dataset, respectively. All metrics are reported as percentages. The first row reports results on the QM9 and GEOM test set. All other rows report results computed from 10,000 generated molecules. The best and second-best entries are shown in bold and underlined, respectively. Entries for which generated molecules could be retrieved to be re-evaluated with our pipeline are marked with a single star (*). Entries for which the source code and model weights were available to generate molecules are marked with two stars (**). For these models and for NEAT, we report the mean and one standard deviation over three runs. Entries marked by a dagger ( † ) were taken from Cheng et al. [6] . Metrics for all other entries were taken from the original publications. xyz2mol entries marked with a double dagger ( ‡ ) are sourced from publications where the RDKit version is either v2023.3.3 or unspecified; these scores may decrease when re-evaluated with RDKit v2025.9.1.
Figure 8: Qualitative illustration of possible failure modes.
Figure 9: Mean pairwise Jensen-Shannon divergence of next atom-type distributions (left) and mean pairwise Sinkhorn divergence of next-coordinate distributions (right) across N=32 rotated versions of partial molecules, plotted against the number of atoms in the molecule. The shaded regions represent the 90% confidence intervals (CI) across 10 repeated runs.
Figure 10: Log-scaled pairwise Jensen-Shannon divergence of next atom-type distributions (left) and mean pairwise Sinkhorn divergence of next-coordinate distributions (right) across N=32 rotated versions of partial molecules, plotted against the number of atoms in the molecule. Confidence intervals (CI) are omitted for clarity.
Figure 11: Mean probability of the next atom type, averaged over 1000 molecules during the generation process. The plot shows the probabilities of the atom type prediction head before temperature scaling, evaluated across different temperatures.
Figure 12: Correlation between evaluation metrics and various statistics of the generated data distribution w.r.t. temperature-scaled atom type sampling. For each temperature setting, we generate 10,000 molecules and report the mean and standard deviation (error bars) across three random seeds.
Figure 13: xyz2mol validity vs. total number of atoms. Left: distribution of number of atoms. The red bars represent the total number of molecules in each bin, the blue bars represent the absolute number of xyz2mol valid molecules. Right: ratio of xyz2mol valid molecules to the total number of molecules for each bin.
Figure 14: Lookup validity vs. total number of atoms. Left: distribution of number of atoms. The red bars represent the total number of molecules in each bin, the blue bars represent the absolute number of lookup valid molecules. Right: ratio of lookup valid molecules to the total number of molecules for each bin.
Model
Steps
Runtime
xyz2mol valid
xyz2mol unique
SemlaFlow
100
1064
86.6±0.5
86.5±0.5
SemlaFlow
60
662
85.5±0.1
85.5±0.1
SemlaFlow
15
188
80.4±0.4
80.4±0.4
FlowMol3
250
3420
95.8±0.7
95.7±0.7
FlowMol3
60
855
92.2±0.3
91.9±0.3
FlowMol3
15
236
55.5±0.2
55.5±0.2
Appendix
Table 10: Average runtime and performance metrics for both SemlaFlow and FlowMol3 with three choices for the number of integration steps: a) the number recommended by the authors of SemlaFlow and FlowMol3, b) 60 (same as NEAT), and c) 15 (resulting in runtimes matching NEAT’s). Mean ± one standard deviation are reported over 3 independent runs.
Figure 15: Randomly selected molecules generated by NEAT trained on the QM9 dataset. Implicit hydrogen atoms have been removed for the sake of clarity. 3D plots of the same molecules are shown in the next figure.
Figure 16: Randomly selected molecules generated by NEAT trained on the QM9 dataset (white: hydrogen, gray: carbon, blue: nitrogen, red: oxygen, brown: fluorine). 2D plots of the same molecules are shown in the previous figure.
Figure 17: Randomly selected molecules generated by NEAT trained on the GEOM-Drugs dataset. Implicit hydrogen atoms have been removed for the sake of clarity. 3D plots of the same molecules are shown in the next figure.
Figure 18: Randomly selected molecules generated by NEAT trained on the GEOM-Drugs dataset (white: hydrogen, gray: carbon, blue: nitrogen, red: oxygen, yellow: sulfur, brown: fluorine, green: chlorine). 2D plots of the same molecules are shown in the previous figure.
Figure 19: Prefixes used for the prefix completion task (1/3).
Figure 20: Prefixes used for the prefix completion task (2/3).
Figure 21: Prefixes used for the prefix completion task (3/3).
Figure 22: Iterative build of a molecule with NEAT, trained on QM9 (1/3).
Figure 23: Iterative build of a molecule with NEAT, trained on QM9 (2/3).
Figure 24: Iterative build of a molecule with NEAT, trained on QM9 (3/3).
Molecular size is intrinsically linked to composition, structure, and properties, but most 3D molecular generators treat size as a predefined input rather than a quantity to be generated. We introduce Equivariant-Free Transformer-Autoencoded Latent Flow Matching, a two-stage framework that samples a fixed-dimensional latent vector using flow matching and employs an autoregressive Transformer to determine molecular size, atom types, coordinates, and chemical attributes. Canonical atom ordering and rigid-pose alignment enable Transformers without equivariant layers, while decoded attributes guide bond reconstruction. On PCQM4Mv2, unconditional generation yields 87.9% unique and novel molecules passing sanitization and PoseBusters checks, exceeding baselines with lower end-to-end training and sampling time and higher end-to-end throughput. Across ten target HOMO-LUMO gaps, internal ranking retains 30% of the screened candidates and increases the density functional theory-verified hit rate within 0.1 eV of the target from 25.0% to 52.4%, while largely preserving novelty and diversity. These results demonstrate fixed-dimensional latent generation with autoregressive decoding as a practical approach to molecular design without prespecifying size.
Weichi Yao, Cameron Gruich, Bryan R. Goldsmith +1
Michigan Institute for Data & AI in Society · Department of Chemical Engineering, University of Michigan, Ann Arbor, USA · Department of Statistics, University of Michigan, Ann Arbor, USA
Three-dimensional (3D) molecule generation has been dominated by diffusion models, which achieve strong generation quality but typically require the molecular size to be specified a priori. Recent autoregressive approaches have substantially narrowed the performance gap while naturally supporting variable-length generation and conditioning on partial molecular context. However, balancing unconditional and context-conditioned generation remains challenging. We introduce KRONOS, a latent autoregressive diffusion framework that generates molecules in the latent space of a pre-trained autoencoder, jointly modeling molecular graph topology and geometry, while retaining the flexibility of autoregressive generation. We further introduce a mixed training strategy inspired by Fill-in-the Middle (FIM) paradigm, enabling both unconditional and fragment-conditioned molecular generation within a single left-to-right autoregressive model. Experiments on QM9 and GEOM-Drugs demonstrate that KRONOS achieves leading unconditional generation performance among autoregressive methods, while remaining competitive with diffusion models. Moreover, fragment-conditioned generation is achieved with negligible impact on unconditional generation performance, demonstrating that both generation paradigms can be supported within a single architecture.
Federico Ottomano, Gaopeng Ren, Yingzhen Li +2
Department of Chemistry Imperial College London · Department of Computing Imperial College London
AI-driven de novo molecular design offers a promising route to accelerate early-stage drug discovery by generating novel ligands directly within target protein binding pockets. We present NEAT-POCKET, a pocket-conditioned extension of the autoregressive NEAT model for 3D molecular generation. NEAT-POCKET generates molecules atom by atom in protein pocket environments while preserving atom permutation invariance and explicitly modeling hydrogen atoms. Benchmarks on the CrossDocked and SPINDR datasets show that NEAT-POCKET achieves competitive structure-based generation performance while sampling substantially faster than existing baselines. Beyond full-molecule generation, NEAT-POCKET naturally enables pocket-conditioned fragment completion, a task directly relevant to lead optimization and scaffold elaboration. These results position NEAT-POCKET as a fast, flexible, and practical framework for structure-based drug design.
Roxane Axel Jacob, Daniel Rose, Thierry Langer +1
Department of Pharmaceutical Sciences, Faculty of Life Sciences, University of Vienna, Josef-Holaubek-Platz 2, 1090 Vienna, Austria. · Christian Doppler Laboratory for Molecular Informatics in the Biosciences, Department of Pharmaceutical Sciences, Faculty of Life Sciences, Josef-Holaubek-Platz 2, 1090 Vienna, Austria. · Vienna Doctoral School of Pharmaceutical, Nutritional and Sport Sciences (PhaNuSpo), Josef-Holaubek-Platz 2, 1090 Vienna, Austria.