To address pressing scientific challenges such as climate change, increasingly sophisticated generative models are being developed to efficiently sample the large chemical space of potential functional materials. The proliferation of these models has necessitated the establishment of rigorous evaluation metrics. While uniqueness (U), novelty (N), and stability (S) of samples serve as standard metrics, their current formulations show several limitations. U and N rely on binary comparisons of crystals, rendering them dependent on heuristic thresholds, incapable of quantifying the degree of similarity, sensitive to atomic coordinate perturbations, and not invariant to sample permutation. Similarly, the binary assessment of S risks a premature exclusion of marginally unstable yet potentially novel candidates. These limitations are addressed by making the aforementioned metrics continuous. Furthermore, we integrate them into a unified metric ``continuous SUN" (cSUN), which offers a smoother score distribution and greater tunability than the conventional binary SUN metric. Experimental results demonstrate that our continuous metrics provide granular insights into sample distributions and secondarily serve as a soft screening score for selecting samples for further analysis. Finally, the use of cSUN as a reward signal in reinforcement learning is explored, showing that its adjustable weighting scheme effectively enhances sample diversity and convergence to a better optima in a Chemeleon2 case study.
Figures & tables
Figure 1: Illustration of the three primary evaluation metrics: Uniqueness, Novelty, and Stability. Uniqueness quantifies the internal diversity of generated samples, while novelty assesses their dissimilarity from the training samples. Both metrics fundamentally rely on the distance function d to compare two crystals. Stability evaluates thermodynamic plausibility by calculating the energy above the convex hull relative to a reference database of known materials.
Figure 2: Unit cell (and 2×2×2 supercell*) of example crystal structures used in Table 1 .
compositional
structural
both
discrete
continuous
discrete
continuous
discrete
continuous
dcomp
delm
dwyckoff
dam
dsmat
delm+am
wz-ZnO vs wz-ZnO (supercell)
0
0.000
0
0.000
0
0.000
wz-ZnO vs rs-ZnO
0
0.000
1
0.510
1
0.114
wz-ZnO vs wz-GaN
1
0.875
0
0.108
1
0.699
wz-ZnO vs BiX2TeX3
1
0.915
1
0.783
1
0.881
Table 1: Distances between example pairs of crystals based on different metrics. Continuous distances ( delm , dam , and delm+am ) offer more granular insights into crystal similarity than their discrete counterparts ( dcomp , dwyckoff , and dsmat ).
compositional
structural
both
discrete
continuous
discrete
continuous
discrete
continuous
dcomp
delm
dwyckoff
dam
dsmat
delm+am
Isometry invariance
✓
✓
✓
✓
✓
Lipschitz continuity
✓
✓
✓
✓
Permutation invariance of U
✓
✓
✓
✓
✓
Table 2: Summary of the theoretical properties of crystal distance functions. Continuous distances satisfy all three robustness requirements, whereas the conventional dsmat fails to do so.
seed
0
1
2
3
4
Usmat
0.9942
0.9947
0.9940
0.9944
0.9943
Table 3: Variation of Usmat under sample permutation. Scores were computed on a fixed set of 10k CDVAE samples shuffled with different seeds, confirming that the metric is not invariant to sample order.
Figure 3: Uniqueness scores U for each generative model based on different distances. The evaluation is based on 10k unconditioned samples. Each subplot shows U computed with a specific distance. The horizontal gray dashed line represents U of the MP20 test set. Since U is guaranteed to be ≤ 1.0, the area with U>1 is shaded in each subplot.
Figure 4: Novelty scores N for each generative model based on different distances. Each subplot shows N computed with a specific distance. The horizontal gray dashed line represents N of the MP20 test set. Since N is guaranteed to be ≤ 1.0, the area with N>1 is shaded in each subplot.
Figure 5: Top-5 most novel crystal samples from an early-days model, CDVAE, identified by cNelm+am . Space groups are determined with the same tolerance threshold as the Materials Project. The energy above the convex hull [eV/atom] ( Ehull ) values are computed against the Materials Project using the MACE-MPA-0 force field [ 40 ] . While CDVAE achieves the highest Nelm+am in Figure 4 , the high positive Ehull values suggest that this high novelty is due to the generation of physically implausible crystals.
Figure 6: (a): Cumulative distribution of Ehull . Approximately 20% of the experimentally synthesized crystals in the Materials Project (theoretical=False) have Ehull values greater than the typical threshold of 0.1 [eV/atom]. (b): Comparison of stability score functions. While S drops down at Ehull=0.1 [eV/atom], cS decreases gradually from Ehull=0 [eV/atom] to Ehull=0.4289 [eV/atom]. The value of 0.4289 is the 99.9th percentile in the MP20 test data shown in (a).
Figure 7: (a): Ratio of samples whose binary stability score S(x) is one. The majority of the generated samples are classified as unstable (S=0), which means they are simply discarded during the evaluation process. (b): Distribution of continuous stability score cS(x) . cS offers a smoother score distribution, and fewer samples are merely removed.
Figure 8: (a): Comparison of the SUNsmat score for 10k crystal samples from MatterGen. (b): Distribution of the cSUNelm+am score for 10k samples from MatterGen. SUNsmat and cSUNelm+am positively correlate with each other, but cSUNelm+am offers a more detailed ranking of candidates.
Figure 9: SUN scores for each generative model based on different distances. The evaluation is based on 10k unconditioned samples. Each subplot shows SUN computed with a specific distance. The horizontal gray dashed line represents SUN for the MP20 test set. Since SUN is guaranteed to be ≤ 1.0, the area with SUN>1 is shaded in each subplot.
Figure 10: Ranking of models based on SUN scores using different continuous distances with different weights for the underlying cSUN. Specifically, we set (wS,wU,wN)=(wS,1,2−wS) . A larger wS indicates that stability is prioritized over novelty during evaluation.
Figure 11: Top-5 samples from MatterGen identified by cSUNelm+am . Space groups are determined with the same tolerance threshold as the Materials Project. Ehull [eV/atom] values are computed against the Materials Project. Unlike Figure 5 , these candidates exhibit significantly greater structural plausibility, since stability is considered.
Figure 12: Training dynamics of Chemeleon2 under different reward function regimes for reinforcement learning. To make visualization easier, each reward value is normalized between 0 and 1 and smoothed out by taking the average of the last 50 steps at each time step. The model achieves effective convergence regardless of whether the reward signal is binary or continuous.
Reward function
default
SUNsmat
SUNelm+am
SUNelm+am ( wU=10 )
Evaluation metric
SUNsmat
0.2651
0.4273
0.0478
0.3946
SUNelm+am
0.1701
0.3087
0.5396
0.5643
Table 4: Comparative assessment of SUNsmat and SUNelm+am scores for Chemeleon2 under different reward signals for reinforcement learning. The highest score is in bold, and the second highest score is underlined. Notably, adjusting wU to 10 in SUNelm+am significantly improves performance in both metrics compared to the default weights, indicating that high flexibility in the continuous reward can help optimization in this Chemeleon2/GRPO setting.
Reward function
default
SUNsmat
SUNelm+am
SUNelm+am ( wU=10 )
Number of unique compositions
6807
4067
287
1980
Most frequent composition
K (37)
GdX5IrX2 (611)
CsHgX5 (900)
AcScPdX2 (147)
Second frequent composition
LiX3CoX4OX8 (22)
AcX4Pt (141)
AcScHgX2 (750)
AcPbX2Bi (145)
Third frequent composition
TbZnGa (21)
AcX12NiX6Pb (86)
CsX6RbHg (743)
AcScPbBi (124)
Table 5: Number of unique compositions and the top 3 most frequent compositions in the 10k samples from Chemeleon2. Each column corresponds to a different reward function for reinforcement learning. Reinforcement learning with SUNsmat or SUNelm+am results in extremely focusing on specific compositions. Enhancing diversity by setting wU to 10 in SUNelm+am mitigates the issue.
CDVAE
DiffCSP
DiffCSP++
MatterGen
Chemeleon-DNG
ADiT
Chemeleon2
MP20 Test
dcomp
0.9717
0.9461
0.9522
0.9523
0.9373
0.7743
0.6807
0.9434
delm
0.9379
0.9446
0.9440
0.9441
0.9440
0.9402
0.9341
0.9439
dwyckoff
0.0045
0.0326
0.1910
0.0470
0.0640
0.0038
0.0039
0.2125
dam
0.5032
0.5435
0.5389
0.5445
0.5528
0.5131
0.5384
0.5481
dsmat
0.9945
0.9771
0.9806
0.9838
0.9791
0.8841
0.7936
0.9940
delm+am
0.8406
0.8549
0.8534
0.8547
0.8565
0.8447
0.8456
0.8553
Table S1 : The uniqueness scores U for each generative model based on different distances. The highest score in each row is in bold, and the second highest score is underlined. Note that the MP20 Test set is included as a baseline and is excluded from the ranking.
CDVAE
DiffCSP
DiffCSP++
MatterGen
Chemeleon-DNG
ADiT
Chemeleon2
MP20 Test
dcomp
0.9221
0.8207
0.8121
0.8156
0.7497
0.2352
0.8090
0.7904
delm
0.5230
0.3954
0.3951
0.3883
0.3306
0.0655
0.3348
0.1973
dwyckoff
0.0352
0.0814
0.0098
0.0967
0.1137
0.0425
0.0371
0.0500
dam
0.1627
0.1095
0.1082
0.1199
0.1073
0.1176
0.1367
0.0705
dsmat
0.9892
0.9050
0.8925
0.9158
0.8630
0.4582
0.9731
0.9608
delm+am
0.4920
0.3717
0.3725
0.3681
0.3150
0.0923
0.3302
0.1913
Table S2 : The novelty scores N for each generative model based on different distances. The highest score in each row is in bold, and the second highest score is underlined. Note that the MP20 Test set is included as a baseline and is excluded from the ranking.
CDVAE
DiffCSP
DiffCSP++
MatterGen
Chemeleon-DNG
ADiT
Chemeleon2
MP20 Test
dcomp
0.0290
0.1741
0.1551
0.2234
0.2078
0.0565
0.1932
0.7300
delm
0.1188
0.1371
0.0938
0.1891
0.1410
0.0178
0.1896
0.1714
dwyckoff
0.0000
0.0071
0.0010
0.0149
0.0138
0.0012
0.0011
0.0420
dam
0.0209
0.0227
0.0152
0.0362
0.0255
0.0272
0.0476
0.0358
dsmat
0.0318
0.2106
0.1820
0.2800
0.2562
0.0680
0.2651
0.9021
delm+am
0.1003
0.1191
0.0828
0.1640
0.1223
0.0259
0.1701
0.1503
Table S3 : The SUN scores for each generative model based on different distances. The highest score in each row is in bold, and the second highest score is underlined. Note that the MP20 Test set is included as a baseline and is excluded from the ranking.
Figure S1 : Uam , Nam , and SUNam scores for each generative model under different AMD vector lengths kmax . Evaluation is performed using 10k unconditioned samples. The horizontal gray dashed line indicates the corresponding score for the MP20 test set. Although the absolute scores gradually increase with kmax , the distributional patterns remain nearly identical within each row.
Figure S2 : Spearman rank correlations between model scores obtained using different AMD vector lengths kmax . All pairs show strong positive correlations, indicating that the model rankings for each score are robust to the choice of kmax .
Figure S3 : Uam , Uelm , Nam , and Nelm scores for each generative model under different normalization methods for the distance functions. Evaluation is performed using 10k unconditioned samples. The horizontal gray dashed line indicates the corresponding score for the MP20 test set. Because U and N are bounded above by 1.0, regions where U>1 or N>1 are shaded in each subplot. Although the absolute scores vary with the normalization method, the model ranking trends are largely preserved.
Figure S4 : Spearman rank correlations between model scores obtained under different distance-normalization methods. Most pairs show positive correlations, indicating that the model rankings for each score are robust to the choice of normalization method, except for d=1−exp(−d′) in Uelm .
Figure S5 : Model rankings for Uelm+am , Nelm+am , and SUNelm+am under different weight settings for delm+am . Increasing welm shifts the evaluation focus toward compositional uniqueness and novelty. The black dashed vertical line indicates the default weight used in this study, welm≈0.78 , which was statistically derived from the MP20 test set.
Figure S6 : Visualizations of the three functional forms for continuous stability (cS), used in our analysis. The threshold hyperparameter τ is set to 0.3 [eV/atom].
Figure S7 : Model rankings for SUNelm+am across different functional forms and thresholds for the continuous stability term. The black dashed vertical line marks the default stability function. Greater emphasis on stability, through a smaller τ or a larger exponent, increases the rank of the MP20 test set and decreases that of CDVAE. Apart from these two extreme cases, the remaining model rankings are largely stable across settings.
Figure S8 : Models achieving the highest and second-highest SUNelm+am scores under different component weights (wS,wU,wN) . All weights are varied over {0.0,0.5,1.0,1.5,2.0,2.5,3.0} while satisfying the constraint wS+wU+wN=3 . The default setting is located at the center of each triangle. The rankings are most strongly influenced by wS .
Figure S9 : Distributions of continuous uniqueness cUelm+am and novelty cNelm+am scores for 10k samples generated by different models.
Figure S10 : Pairwise energy distance and earth mover’s distance between 1,000-sample sets from each generative model and the MP20 test set. delm+am is used as the underlying distance function for both metrics.