Synthetic Crystal Data Generation for Diffusion Pretraining in Material Property Prediction
Organizations: Department of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332
Abstract
Diffusion-based pretraining has emerged as a powerful strategy for learning transferable representations in graph neural networks for materials property prediction without requiring quantum-mechanical labels such as energies, forces, and stresses for every pretraining structure. However, the method retains a critical dependency: it requires structures residing near local minima of a potential energy surface, which have traditionally been obtained through costly DFT relaxations of known or physically synthesized compounds. This reliance constrains both the scale and the structural diversity of available pretraining data, leaving large regions of crystal structure space unexplored. In this work, we connect diffusion-based pretraining with synthetic structure generation to reduce this bottleneck. We introduce a synthetic data generation pipeline combining three complementary strategies: perturb-and-relax, generative sampling via MatterGen, and protostructure enumeration. Where structural relaxation is required, we use fast machine-learned interatomic potentials instead of performing a new DFT relaxation for each generated structure. We further compare a coordinate-denoising control with continuous-time variance-preserving (VP) and sub-VP diffusion objectives using a randomly initialized Orb-v2 backbone. The source comparison covers seven MatBench property-prediction tasks, and an eighth task, mp\_gap, is included in the protostructure scaling study. Every synthetic-pretraining condition numerically outperforms random initialization on all seven source-comparison tasks. Scaling the protostructure corpus from 20k to 165k under a fixed 400-epoch schedule numerically matches or exceeds MP-20k pretraining on most tasks; because this protocol increases both data diversity and optimization updates, the result reflects joint data-and-compute scaling.
Figures & tables
| MAE ( ) | |||
| Initialization / Pretraining | jdft2d | log_gvrh | dielectric |
| meV/atom | - | ||
| Coordinate denoising ( Zaidi et al., 2022 ) | |||
| VP | 0.190 | ||
| sub-VP | 20.60 | 0.063 | |
| Orb-v2 pretrained | |||
| Mean MAE SE ( ) | |||||||
|---|---|---|---|---|---|---|---|
| Pretraining data | log_gvrh | log_kvrh | jdft2d | dielectric | perovskites | phonons | 2d |
| MatterGen | |||||||
| Perturb-and-relax | |||||||
| Protostructure | |||||||
| MP-20k | |||||||
| Random initialization | |||||||
| Mean MAE SE ( ) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Size | log_gvrh | log_kvrh | jdft2d | dielectric | perovskites | phonons | 2d | mp_gap |
| 20k | ||||||||
| 50k | ||||||||
| 75k | ||||||||
| 110k | ||||||||
| 165k | ||||||||
| Component | Repository | Commit | Role |
|---|---|---|---|
| Orb noise-head fork | https://github.com/aamodvarma/orb-models-noisehead | 8d5b40059e48 | Diffusion backbone and noise head |
| MatterTune fork | https://github.com/aamodvarma/MatterTune_diffusion | f2a464b3e97b | Pretraining and downstream fine-tuning |
| Protostructure code | https://github.com/aamodvarma/protostructure | 9464e1457135 | Enumeration, population, and screening |
| MatterGen | https://github.com/microsoft/mattergen | ac9ddd406171 | Unconditional structure generation |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Pretraining data | Wall-clock (h) | Min / epoch |
|---|---|---|
| MP-20k | 22.5 | 3.4 |
| Perturb-and-relax (20k) | 27.2 | 4.1 |
| MatterGen (20k) | 29.8 | 4.5 |
| Protostructure 20k | 48.6 | 7.3 |
| Protostructure 50k | 134.1 | 20.1 |
| Protostructure 75k | 134.3 | 20.2 |
| Task | Mean (h) | Median (h) | Mean epochs |
|---|---|---|---|
| jdft2d | 0.27 | 0.18 | 92 |
| phonons | 0.59 | 0.52 | 158 |
| dielectric | 1.89 | 1.40 | 100 |
| 2d | 2.05 | 2.03 | 147 |
| log_kvrh | 2.39 | 2.32 | 80 |
| log_gvrh | 4.32 | 3.38 | 121 |
Synthetic Data Pretraining
Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.
See all 130 papers