Quantitative Assessment of Crystal Structure Prediction
Organizations: Universidad de Los Andes, Colombia
Abstract
Crystal Structure Prediction (CSP) aims to predict a material's 3D structure from its chemical composition, with applications from energy storage to biomedical devices. Despite its importance, existing evaluation frameworks suffer from key limitations: they rely on threshold-sensitive or incomparable metrics, focus predominantly on geometric performance while overlooking physical plausibility, and fail to account for polymorphism, where a chemical composition can crystallize into multiple stable structures. In parallel, recent Symmetry-Informed CSP (SICSP) methods often report results alongside standard CSP baselines while receiving crystallographic templates at inference, which constrain the search space and make SICSP a fundamentally different task. To address these limitations, we introduce an Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) that evaluates CSP and SICSP as separate tasks. We evaluate 7 CSP and 2 SICSP methods under OPAL-Bench across 4 datasets using threshold-free geometric and physical distance metrics. Our results show that no method dominates across all metrics, and that geometric performance is not strongly correlated with physical plausibility. We also release OPAL-CSP, an SE(3)-equivariant flow-matching baseline for the CSP track that performs competitively on geometric metrics and ranks among the best on physical ones. By unifying geometric and physical metrics under a polymorph-aware protocol, OPAL-Bench enables a more comprehensive assessment of generative quality in CSP.
Figures & tables
| Geometric Distances (Å) | Physical Distances | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Chamf. | Haus. | Sink. | RMSE | (g/cm 3 ) | SOAP | XRD | FE (eV/at.) | R. FE (eV/at.) | |
| Perov-5 | FlowLLM [ 14 ] | 1.218 | 0.623 | 0.970 | 0.610 | 0.434 | 8.05 | 2.164 | 0.175 | |
| CDVAE [ 9 ] | 2.295 | 1.272 | 1.655 | 1.172 | 0.413 | 10.81 | 3.180 | 3.008 | ||
| CrystalFlow [ 12 ] | 1.328 | 0.698 | 1.058 | 0.670 | 1.141 | 12.11 | 3.116 | 0.422 | ||
| MatterGen [ 11 ] | 1.231 | 0.631 | 0.955 | 0.619 | 0.251 | 6.48 | 1.678 | 0.101 | ||
| DiffCSP [ 10 ] | 1.229 | 0.625 | 0.921 | 0.616 | 0.236 | 6.53 | 1.672 | 0.116 | ||
| Geometric Distances (Å) | Physical Distances | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Chamf. | Haus. | Sink. | RMSE | (g/cm 3 ) | SOAP | XRD | FE (eV/at.) | R. FE (eV/at.) | |
| Perov-5 | SGEquiDiff [ 20 ] (CSPML) | 1.171 | 0.594 | 0.895 | 0.588 | 0.315 | 7.21 | 1.872 | 0.152 | |
| DiffCSP++ [ 19 ] (CSPML) | 1.200 | 0.609 | 0.901 | 0.602 | 0.337 | 7.39 | 1.936 | 0.164 | ||
| SGEquiDiff (GT) | 0.076 | 0.040 | 0.058 | 0.038 | 0.175 | 1.96 | 0.809 | 0.013 | ||
| DiffCSP++ (GT) | 0.323 | 0.165 | 0.245 | 0.162 | 0.173 | 2.09 | 0.856 | 0.016 | ||
| Carbon-24 | SGEquiDiff (CSPML) | 1.752 | 1.563 | 13.211 | 1.048 | 0.489 | 83.14 | 5.410 | 1.488 | |
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Method A | Method B | A wins | B wins | Reversal rate |
|---|---|---|---|---|
| CrystalFlow | DiffCSP | 445 | 126 | 22.1% |
| CDVAE | MatterGen | 115 | 455 | 20.1% |
| CrystalFlow | MatterGen | 495 | 76 | 13.3% |
| CDVAE | DiffCSP | 39 | 532 | 6.8% |
| CDVAE | CrystalFlow | 0 | 571 | 0.0% |
| DiffCSP | MatterGen | 571 | 0 | 0.0% |
| Model | Perov-5 | Carbon-24 | MP-20 | MPTS-52 | ||||
|---|---|---|---|---|---|---|---|---|
| MR | RMSE | MR | RMSE | MR | RMSE | MR | RMSE | |
| CDVAE | 0.447 | 0.330 | 0.194 | 0.355 | 0.333 | 0.123 | 0.042 | 0.280 |
| DiffCSP | 0.516 | 0.081 | 0.155 | 0.299 | 0.490 | 0.065 | 0.127 | 0.149 |
| MatterGen | 0.507 | 0.073 | 0.167 | 0.337 | 0.293 | 0.063 | 0.100 | 0.124 |
| FlowLLM | 0.520 | 0.097 | 0.194 | 0.419 | 0.223 | 0.080 | 0.006 | 0.395 |
| CrystalFlow | 0.523 | 0.098 | 0.176 | 0.316 | 0.618 | 0.051 | 0.223 | 0.138 |
| Model | Perov-5 | Carbon-24 | MP-20 | MPTS-52 | ||||
|---|---|---|---|---|---|---|---|---|
| MR | RMSE | MR | RMSE | MR | RMSE | MR | RMSE | |
| SGEquiDiff (CSPML) | 0.530 | 0.098 | 0.103 | 0.231 | 0.635 | 0.041 | 0.277 | 0.078 |
| DiffCSP++ (CSPML) | 0.511 | 0.093 | 0.290 | 0.171 | 0.705 | 0.030 | 0.361 | 0.093 |
| SGEquiDiff (GT) | 0.981 | 0.025 | 0.288 | 0.147 | 0.756 | 0.039 | 0.417 | 0.093 |
| DiffCSP++ (GT) | 0.984 | 0.051 | 0.288 | 0.112 | 0.787 | 0.034 | 0.432 | 0.096 |
| Dataset | Split | Structures | Poly. Groups | Polymorphism Rate | Max Polymorphs |
|---|---|---|---|---|---|
| Perov-5 | train | 11,356 | 3,330 | 58.6% | 2 |
| val | 3,787 | 362 | 19.1% | 2 | |
| test | 3,785 | 376 | 19.9% | 2 | |
| MPTS-52 | train | 27,380 | 1,261 | 10.4% | 8 |
| val | 5,000 | 116 | 4.8% | 5 | |
| test | 8,096 | 190 | 5.0% | 6 |
| Dataset | Unique SGs | Top 1 SG | Top 5 SGs | Top 20 SGs |
|---|---|---|---|---|
| Perov-5 | 5 | 56% | 100% | 100% |
| Carbon-24 | 56 | 33% | 73% | 96% |
| MP-20 | 179 | 15% | 31% | 65% |
| MPTS-52 | 208 | 9% | 26% | 56% |
| Perov-5 | Carbon-24 | MP-20 | MPTS-52 | |
|---|---|---|---|---|
| Cutoff (Å) | 10 | 5 | 12 | 12 |
| Layers | 3 | 6 | 6 | 6 |
| Hyperparameter | Value |
|---|---|
| Max degree | 6 |
| Sphere channels | 128 |
| Max neighbours | 20 |
| Attention heads | 8 |
| Attention hidden channels | 64 |
| FFN hidden channels | 128 |
| Model | Chamf. | Haus. | Sink. | RMSE | SOAP | XRD | FE | R. FE | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Perov-5 | DiffCSP++ (CSPML) | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| SGEquiDiff (CSPML) | 100.0% | 100.0% | 100.0% | 100.0% | 99.9% | 99.9% | 99.9% | 100.0% | 99.9% | |
| DiffCSP++ (GT) | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | |
| SGEquiDiff (GT) | 99.9% | 99.9% | 99.9% | 99.9% | 99.9% | 99.9% | 99.9% | 99.9% | 99.9% | |
| Intersection | 99.8% | 99.8% | 99.8% | 99.8% | 99.8% | 99.8% | 99.8% | 99.8% | 99.8% | |
| Carbon-24 | DiffCSP++ (CSPML) | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Metric | Parameter | Range swept | Default | Worst-case |
|---|---|---|---|---|
| SOAP | (Å) | 3.5–8.0 | 4.0 | 0.96 |
| SOAP | 4–13 | 12 | 0.95 | |
| SOAP | 3–12 | 9 | 1.00 | |
| SOAP | (Å) | 0.2–1.0 | 0.5 | 0.89 |
| XRD | ( ∘ ) | 3–10 | 5 | 0.94 |
| XRD | ( ∘ ) | 60–120 | 90 | 0.96 |
| Geometric Distances (Å) | Physical Distances | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Chamf. | Haus. | Sink. | RMSE | (g/cm 3 ) | SOAP | XRD | FE (eV/at.) | R. FE (eV/at.) | |
| Perov-5 | FlowLLM | 1.234 | 0.633 | 1.010 | 0.621 | 0.448 | 8.07 | 2.187 | 0.175 | |
| CDVAE | 2.327 | 1.291 | 1.676 | 1.188 | 0.439 | 10.95 | 3.221 | 3.008 | ||
| CrystalFlow | 1.380 | 0.733 | 1.107 | 0.696 | 1.161 | 12.28 | 3.147 | 0.422 | ||
| MatterGen | 1.255 | 0.643 | 1.004 | 0.629 | 0.263 | 6.56 | 1.688 | 0.101 | ||
| DiffCSP | 1.244 | 0.638 | 0.963 | 0.625 | 0.242 | 6.63 | 1.697 | 0.116 | ||
| Geometric Distances (Å) | Physical Distances | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Chamf. | Haus. | Sink. | RMSE | (g/cm 3 ) | SOAP | XRD | FE (eV/at.) | R. FE (eV/at.) | |
| Perov-5 | FlowLLM | 1.218 | 0.623 | 0.971 | 0.610 | 0.459 | 8.30 | 2.225 | 0.197 | |
| CDVAE | 2.295 | 1.272 | 1.655 | 1.172 | 0.438 | 10.95 | 3.242 | 3.038 | ||
| CrystalFlow | 1.328 | 0.698 | 1.058 | 0.670 | 1.178 | 12.42 | 3.174 | 0.450 | ||
| MatterGen | 1.231 | 0.631 | 0.955 | 0.619 | 0.276 | 7.01 | 1.773 | 0.128 | ||
| DiffCSP | 1.229 | 0.625 | 0.922 | 0.616 | 0.257 | 6.98 | 1.802 | 0.141 | ||
| Dataset | Geometric | Physical | Geometric vs. physical | MR vs. RMSE |
|---|---|---|---|---|
| Perov-5 | 0.846 | 0.832 | 0.608 | |
| Carbon-24 | 0.786 | 0.543 | 0.059 | |
| MP-20 | 0.976 | 0.604 | 0.521 | 0.649 |
| MPTS-52 | 0.964 | 0.914 | 0.243 | 0.714 |
Crystal Structure Prediction
Crystal structure prediction seeks stable atomic arrangements from composition, molecular identity, or experimental measurements.
See all 39 papers