MOTIF: LLM-Defined Material Taxonomies for OOD Property Prediction via Invariant Feature Selection
Authors: Gwangseon Jang, Sieun Yun, Dong Ho Lee, Kyong-Ha Lee, Chanuk Lim, Jin Woo Choi, Han Seul Kim
Organizations: Large-scale AI Research Center, Korea Institute of Science and Technology Information (KISTI) · Department of Advanced Materials Engineering, Chungbuk National University · Department of Data Information and Physics, Kongju National University
Out-of-distribution (OOD) generalization remains a central challenge in materials property prediction, where models must extrapolate to novel chemistries beyond the training distribution. While large language models (LLMs) have demonstrated strong scientific reasoning, their direct use for quantitative prediction remains limited. To address this, we systematically evaluate multiple roles for LLM assistance, finding that only taxonomy-based grouping consistently improves OOD generalization over a strong kNN+GBT baseline. Building on this, we here propose MOTIF, a framework that leverages LLMs not as predictors, but as generators of candidate taxonomies that partition materials into chemically meaningful environments. These taxonomies are evaluated via training-only pseudo-OOD validation, and only those that improve generalization are retained. Invariant feature selection is then applied within each accepted taxonomy, followed by gated aggregation across taxonomies. We evaluate on 2D halide perovskites, where bandgap prediction must generalize across distinct organic spacer families. On this benchmark, MOTIF reduces mean OOD MAE from 0.122 to 0.111 eV. Under a frozen dev/test protocol, the deployable configuration further improves errors from 0.123 to 0.117 eV (electronic bandgap) and from 0.181 to 0.173 eV (optical bandgap). Beyond this primary task, we validate transferability across additional materials datasets, including 3D perovskite solar cell efficiencies and Matbench benchmarks with structure-based splits. Across domains, MOTIF consistently improves robustness under distribution shift, outperforming expert-defined groupings, rule-based heuristics, and clustering baselines, while remaining competitive with strong classical selection methods. These results demonstrate that LLMs are most effective not as predictors, but as tools for discovering structure in data.
Figures & tables
Figure 1: Motif pipeline. Stage 1 : an LLM classifies each spacer along multiple chemical axes once, offline. Stage 2 : within each taxonomy, a Chow test filters features whose relationship to the target is stable across groups, and kNN/GBT are trained only in that invariant subspace. Stage 3 : a pseudo-OOD gate keeps only taxonomies that beat the baseline, and gated aggregation combines the surviving models. No LLM call is made at inference time.
Method
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean
Δ
kNN (30-dim)
0.113
0.075
0.142
0.092
0.183
0.165
0.128
—
GBT (30-dim)
0.125
0.104
0.207
0.058
0.164
0.148
0.134
—
kNN+GBT Blend (baseline)
0.123
0.136
0.119
0.110
0.137
0.105
0.122
—
Self-Refine [ 12 ] (best LLM)
0.425
0.282
0.644
0.419
0.658
0.335
0.461
+278%
Motif (gated median)
0.128
0.129
0.095
0.099
0.098
0.119
0.111
−8.5%
Motif (best single axis; post hoc)
0.118
0.130
0.109
0.094
0.118
0.094
0.110
−9.3%
Table 1: 2D halide perovskite bandgap: six-fold leave-one-spacer-family-out OOD benchmark (MAE in eV). Motif is our proposed method. kNN, GBT, and their blend are non-LLM baselines; Self-Refine [ 12 ] is the best-performing direct LLM predictor. “Best single axis” and “oracle” are post-hoc upper bounds, not deployable configurations.
Method
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Motif
0.128
0.129
0.095
0.099
0.098
0.119
kNN+GBT Blend
0.123
0.136
0.119
0.110
0.137
0.105
kNN
0.113
0.075
0.142
0.092
0.183
0.165
GBT
0.125
0.104
0.207
0.058
0.164
0.148
Self-Refine
0.425
0.282
0.644
0.419
0.658
0.335
Table 2: 2D halide perovskite bandgap: fold-level OOD MAE matrix. Motif beats the kNN+GBT blend on Alkyl-L, Phenyl, Hetero, and Cyclo, but not on Alkyl-S or Special.
Figure 2: Role-level summary on the six-fold 2D halide perovskite bandgap benchmark. Across 19 published LLM methods (plus 4 ML baselines and 65+ in-house variants; Appendix P ), methods that inject numerical values do not beat the classical baseline. The only role that consistently improves over the baseline is offline classification.
Figure 3: Mechanistic diagnostics for Motif on the 2D halide perovskite bandgap benchmark. Left: the strongest drops come from removing invariant kNN or the balance filter, and random groupings nearly collapse to the baseline. Right: a pilot taxonomy bank produced degenerate partitions, whereas the final spacer-centric design yields mostly usable axes.
Table 3: Competitive controls for the 2D halide perovskite task. Panels A–C use the frozen training-only protocol; Panel D adds the strongest direct-LLM and RAG baselines on the same six-fold leave-one-spacer-family-out task for scale. Motif rows are our proposed method. “Baseline” is the kNN+GBT blend without taxonomy-conditioned invariant filtering. See Appendix U for method details.
Figure 4: 2D halide perovskite bandgap: failure-mode analysis. Left: the best taxonomy changes across folds, so the remaining gap is a routing problem. Right: Motif improves by trimming large OOD errors, not by uniformly improving already-easy examples.
Dataset
Target
N
Folds
BL MAE
Gated
Safe
Best Δ
Matbench Dielec.
dielectric
4,764
7
0.608
0.584
0.571
−6.2%
Matbench 2D Exfol.
exfol. en.
636
6
72.2
70.1
68.7
−4.9%
OQMD ABX3
form. en.
4,000
7
0.435
0.427
0.420
−3.6%
MP ABX3
bandgap
4,557
7
0.845
0.847
0.817
−3.3%
Table 4: Additional transfer benchmarks beyond the primary 2D perovskite task. We report both the automatic gated-median ensemble and the deployable safe-blend ensemble against the same kNN+GBT baseline. All four benchmarks use LLM taxonomies ( gpt-4o-mini ) with leave-one-group-out OOD splits.
Method
OOD MAE (eV)
Cost (753 queries)
Hardware
Motif (after caching)
0.111
$0.00
CPU only
Motif (incl. taxonomy gen.)
0.111
$0.03–0.37
CPU only
Self-Refine (best LLM)
0.461
$45.18
API
LATS
0.567
$112.95
API
Table 5: Cost–accuracy comparison on the 2D perovskite bandgap benchmark (753 OOD queries).
Appendix figures & tables41 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Crystal structures of 2D halide perovskites with different spacer configurations: Ruddlesden–Popper (left) and Dion–Jacobson (right) phases. The Ruddlesden–Popper structure consists of perovskite layers separated by bilayer organic spacers, whereas the Dion–Jacobson structure features monolayer spacers that bridge adjacent inorganic slabs, leading to distinct interlayer coupling and electronic properties.
Figure 6: Schematic representation of the 2D halide perovskite dataset. Each material is described by its compositional and structural components, including (a) B-site metal, (b) halide composition, (c) organic spacer cation, and (d) the number of inorganic layers (n). These factors collectively determine the (e) experimentally measured bandgap, which serves as the prediction target. The dataset spans diverse chemical environments arising from variations in spacer chemistry and layer thickness.
Fold (Held-Out Family)
Held-Out
Train
Mean BG (eV)
Std BG (eV)
Alkyl-short
232
521
2.41
0.40
Alkyl-long
79
674
2.31
0.26
Phenyl
166
587
2.45
0.44
Heterocyclic
128
625
2.38
0.36
Cycloalkyl
37
716
2.70
0.44
Special
111
642
2.33
0.34
Appendix
Table 6: 2D perovskite bandgap: OOD fold statistics. “Held-Out” counts the bandgap-valid OOD queries, “Train” is the training size after excluding that family.
Figure 7: Chemical superfamilies of organic spacer cations used to define the leave-one-spacer-family-out OOD evaluation protocol: (a) alkyl chains (short: C1–C4; long: C5+), (b) phenyl-containing, (c) heterocyclic, (d) cycloalkyl, and (e) special structures including ammonium salts, amino acids, and other unique chemistries. Each superfamily defines a distinct chemical environment, and is held out in turn to evaluate generalization to unseen spacer chemistries.
Figure 8: Bandgap distributions per spacer superfamily. Each panel shows the histogram and kernel density estimate for the held-out family (colored) against the remaining training distribution (gray). The vertical dashed line indicates the family-specific mean bandgap.
Figure 9: Overview of the feature groups used in this study. The 30 features are organized into five categories: structural descriptors, B-site identity, halide composition, spacer identity, and intralayer cation descriptors. These groups capture key chemical and structural factors—such as composition, lattice geometry, and spacer chemistry—that define distinct chemical environments and govern structure–property relationships in 2D halide perovskites.
Method
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean
Cat.
Motif (oracle)
0.117
0.118
0.092
0.086
0.097
0.089
0.100
Tax.
Motif (best single)
0.118
0.130
0.109
0.094
0.118
0.094
0.110
Tax.
Motif (gated median)
0.128
0.129
0.095
0.099
0.098
0.119
0.111
Tax.
kNN+GBT Blend
0.123
0.136
0.119
0.110
0.137
0.105
0.122
ML
kNN (30-dim)
0.113
0.075
0.142
0.092
0.183
0.165
0.128
ML
Random Forest
0.131
0.136
0.128
0.104
0.131
0.109
0.123
ML
Appendix
Table 7: 2D perovskite bandgap: methods with full fold-wise OOD logs. MAE in eV. Bold : best per column.
Method
Mean MAE
Cat.
Evidence Wt. Mean
0.304
RAG
Step-Back
0.463
Prompt
Self-Consistency
0.492
Prompt
Self-Discover
0.506
Prompt
LLM Process
0.511
Reason
CoVe
0.513
Reason
Appendix
Table 8: 2D perovskite bandgap: stored mean-only results for methods whose archives did not retain per-fold decompositions.
Method
MAE (eV)
vs. Baseline
Description
Zero-Shot
0.552
+352%
Direct question, no examples
Chain-of-Thought [ 24 ]
0.563
+361%
Step-by-step reasoning
Self-Consistency
0.492
+303%
Majority vote over 5 CoT samples
Self-Refine
0.461
+278%
Iterative refinement (3 rounds)
Step-Back Abstraction
0.463
+279%
Abstract principle then predict
Analogical Reasoning
0.542
+344%
Find similar compounds then predict
Appendix
Table 9: 2D perovskite bandgap: direct LLM prediction methods. All use gpt-4o . MAE is 6-fold mean OOD (eV).
Figure 10: Example illustrating the mismatch between text-based similarity and property similarity in RAG-based prediction. Although “(PEA) 2 PbI 4 ” retrieves “(PEA) 2 SnI 4 ” due to similar chemical naming, substitution of Pb with Sn leads to a significant change in electronic structure and bandgap. This highlights the limitation of retrieval methods that rely on textual similarity rather than underlying chemical factors.
Method
MAE (eV)
vs. Baseline
Description
BM25-RAG
2.427
+1889%
BM25 text retrieval + LLM
Embedding-RAG
2.430
+1892%
Embedding similarity retrieval + LLM
Evidence Weighted Mean
0.304
+149%
Weighted mean of retrieved bandgaps
CRAG
0.680
+457%
Corrective RAG with self-grading
Appendix
Table 10: 2D perovskite bandgap: RAG methods. MAE is 6-fold mean OOD (eV).
Variant
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean
Full Motif
0.128
0.129
0.095
0.099
0.098
0.119
0.111
w/o invariant kNN
0.130
0.134
0.098
0.103
0.106
0.120
0.115
w/o CV gate
0.131
0.130
0.096
0.100
0.100
0.120
0.113
w/o balance filter
0.134
0.135
0.101
0.106
0.108
0.125
0.118
w/o strict α
0.132
0.131
0.098
0.103
0.104
0.121
0.115
Single taxonomy
0.130
0.131
0.096
0.101
0.102
0.119
0.113
Appendix
Table 11: 2D perovskite bandgap: per-fold ablation results (OOD MAE in eV). Each row is an ablation variant; each column is a held-out fold. Bold : best per column.
Figure 11: Balance analysis comparing the pilot taxonomy design (5 axes, imbalanced) and the final taxonomy bank (9 axes, balanced). Left: Maximum group fraction per axis. Pilot axes have max-group fractions of 72–100%, while the final bank is below 85%. Right: Effect on Chow test power—imbalanced groups produce statistically unreliable invariance decisions.
Pilot axis
# Groups
Max Group %
Status
Problem
Ionic character
3
85%
Filtered
Most spacers classified as “ionic”
Structural motif
3
89%
Filtered
“chain” dominates over “ring” and “cage”
Electronic coupling
3
72%
Passed
Marginal; weak group produces unreliable Chow test
Confinement
2
100%
Filtered
All 2D perovskites are “quantum confined”
Stability class
3
99%
Filtered
Nearly all classified as “air-stable”
Appendix
Table 12: Pilot taxonomy design and its balance statistics. Axes with Max % >85% are highlighted in red.
V2 Axis
# Groups
Max Group %
Balance Assessment
Spacer size
4
50%
Excellent (4 roughly equal groups)
Conjugation
3
56%
Good
Polarity
3
57%
Good
Steric profile
4
42%
Excellent (most balanced axis)
H-bonding
3
57%
Good
Flexibility
3
37%
Excellent
Appendix
Table 13: Final taxonomy bank balance statistics over the 313 unique spacer cations. All 9 axes pass the 85% balance threshold.
Axis
# Grp
Group 1
Group 2
Group 3
Group 4
Max %
Spacer size
4
287 (tiny)
231 (small)
156 (medium)
79 (large)
38.1%
Conjugation
3
198 (saturated)
247 (partial)
308 (full)
—
40.9%
Polarity
3
312 (nonpolar)
289 (polar)
152 (amphiphilic)
—
41.4%
Steric profile
4
186 (linear)
134 (branched)
209 (cyclic)
224 (planar)
29.7%
H-bonding
3
267 (strong)
312 (weak)
174 (bifunctional)
—
41.4%
Flexibility
3
278 (rigid)
256 (semi-flex)
219 (flexible)
—
36.9%
Appendix
Table 14: Distribution of bandgap-valid compounds per group for each taxonomy axis.
Figure 12: Invariant feature heatmap across all 9 taxonomy axes and 30 features. Green cells indicate invariant features (Chow p>0.01 ); red cells indicate variant features. Structural, B-site, halide, and A-site features are almost universally invariant, while spacer-related features (17–24) show taxonomy-dependent variance patterns.
Structural
B-site
Halide
Spacer
A-site
Axis
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Size
✓
✓
✓
✓
✓
✓
✓
✓
×
✓
✓
✓
✓
✓
✓
✓
×
×
×
×
×
×
×
×
✓
✓
✓
✓
✓
✓
Conj.
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
×
✓
✓
✓
×
✓
×
✓
×
×
✓
✓
×
×
✓
✓
✓
✓
✓
✓
Polar.
✓
✓
✓
✓
✓
✓
✓
×
✓
✓
✓
✓
✓
✓
✓
✓
×
×
✓
×
✓
×
×
×
✓
✓
✓
✓
✓
✓
Steric
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
×
✓
✓
✓
✓
×
×
×
×
×
✓
×
×
✓
✓
✓
✓
✓
✓
H-bond
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
×
×
×
✓
✓
×
×
✓
✓
✓
✓
✓
✓
✓
Appendix
Table 15: Invariant feature selection heatmap. “ ✓ ” = invariant (Chow p>0.01 ), “ × ” = variant. Shown for the full training set (no fold held out). Feature indices refer to the feature-group summary earlier in this appendix.
Figure 13: Sensitivity to α threshold. Left: Mean OOD MAE (6-fold) vs. α for the gated median ensemble and the best-single-taxonomy strategy. The optimum occurs at α=0.01 for both. Right: Per-fold MAE vs. α showing that all six folds share a similar optimum region.
α
# Inv
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean (Best)
Mean (Gated)
Std
0.001
18.2
0.121
0.132
0.098
0.096
0.101
0.098
0.108
0.114
0.0147
0.005
20.6
0.119
0.130
0.096
0.095
0.099
0.096
0.106
0.112
0.0138
0.01
22.1
0.118
0.130
0.095
0.094
0.098
0.094
0.105
0.111
0.0144
0.02
23.8
0.120
0.131
0.098
0.098
0.102
0.098
0.108
0.115
0.0122
0.05
25.4
0.122
0.133
0.104
0.103
0.108
0.103
0.112
0.115
0.0108
Appendix
Table 17: Complete α sensitivity results under best-single-taxonomy strategy. “# Inv” is the average number of invariant features across all 9 taxonomy axes.
α
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean
Std
0.001
0.131
0.133
0.099
0.103
0.104
0.112
0.1130
0.0147
0.005
0.130
0.131
0.097
0.101
0.101
0.112
0.1120
0.0138
0.01
0.128
0.129
0.095
0.099
0.098
0.119
0.1113
0.0144
0.02
0.129
0.131
0.101
0.102
0.103
0.123
0.1148
0.0122
0.05
0.130
0.133
0.106
0.105
0.108
0.109
0.1152
0.0108
Appendix
Table 18: α sensitivity under the gated median ensemble strategy. Per-fold OOD MAE (eV).
Figure 14: Per-fold OOD MAE for all 8 ensemble strategies plus baseline and oracle. The gated median (red) achieves the best or near-best performance on the phenyl, heterocyclic, and cycloalkyl folds, while remaining competitive on alkyl folds. The oracle (dashed) shows the upper bound achievable with perfect taxonomy selection.
Strategy
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean
Simple Average
0.128
0.131
0.098
0.101
0.102
0.114
0.1123
Median
0.129
0.132
0.097
0.100
0.103
0.115
0.1127
Trimmed Mean
0.129
0.131
0.097
0.101
0.103
0.115
0.1127
CV-weighted
0.128
0.131
0.097
0.100
0.102
0.115
0.1122
Top-1 CV Select
0.127
0.132
0.098
0.100
0.101
0.115
0.1122
Diversity-weighted
0.128
0.131
0.098
0.101
0.102
0.114
0.1123
Appendix
Table 19: 2D perovskite bandgap: complete per-fold OOD MAE (eV) for all 8 ensemble strategies. Bold : best automatic strategy per column.
Figure 15: Full-fold bandgap comparison of reasoning-model choices for taxonomy generation. o3 is the strongest bandgap annotator overall, but the fold winners still change across families and no single backbone dominates every held-out family.
LLM
Bandgap Top-1
Bandgap Safe
Optical Top-1
Optical Safe
One-time Cost
gpt-4o
0.1168
0.1174
0.2056
0.1732
$0.37
o3-mini
0.1173
0.1170
0.2042
0.1823
—
o3
0.1148
0.1159
0.2298
0.1859
$5.22
o1
0.1225
0.1188
0.1942
0.1762
$134.02
Appendix
Table 20: Reasoning-model comparison on the full control benchmark (mean MAE in eV). One-time annotation cost is for the full 313×9 taxonomy bank.
Figure 16: Oracle gap analysis. Left: Per-fold comparison of gated median, best-single-taxonomy, and oracle MAE, with the gap highlighted. Right: Oracle taxonomy configuration. The gap is largest on the special fold (0.030 eV) and smallest on the cycloalkyl fold (0.001 eV).
Fold
Gated
Oracle
Gap
Best Axis
α
# Inv Features
Alkyl-short
0.128
0.117
0.011
electron_effect
0.01
24
Alkyl-long
0.129
0.118
0.011
steric
0.01
22
Phenyl
0.095
0.092
0.003
flexibility
0.01
24
Heterocyclic
0.099
0.086
0.013
spacer_size
0.005
20
Cycloalkyl
0.098
0.097
0.001
spacer_size
0.01
22
Special
0.119
0.089
0.030
spacer_size
0.01
22
Appendix
Table 21: Oracle per-fold configurations. “Best Axis” is the single taxonomy axis that produces the lowest MAE. “Gap” is the difference between gated median and oracle.
Figure 17: 6-fold mean OOD MAE for all 6 refined pipeline variants (R1–R6) compared to the baseline and the main Motif gated median. R4_strict is the strongest refined variant on average, while R3_alpha overfits despite its additional tuning freedom.
Variant
Alkyl-S
Alkyl-L
Phenyl
Hetero
Cyclo
Special
Mean
R1_gated (gated ensemble)
0.1239
0.1323
0.1114
0.1042
0.1207
0.1054
0.1163
R2_hybrid (env+standard)
0.1176
0.1257
0.1120
0.1002
0.1185
0.1051
0.1132
R3_alpha (per-fold α )
0.1243
0.1351
0.1098
0.1053
0.1284
0.1261
0.1215
R4_strict (strict inv.)
0.1176
0.1295
0.1095
0.1014
0.1170
0.0940
0.1115
R5_knn_env (env-aware kNN)
0.1187
0.1324
0.1083
0.1012
0.1140
0.1041
0.1131
R6_best (best single tax.)
0.1192
0.1341
0.1109
0.1038
0.1192
0.1072
0.1157
Appendix
Table 22: 2D perovskite bandgap: complete per-fold results for refined pipeline variants (OOD MAE in eV). Bold : best per column among R1–R6.
Figure 18: Absolute-error distribution analysis. Left: Density of absolute errors for the baseline kNN+GBT, Motif gated median, and Self-Refine. Right: Empirical CDF of absolute errors. Motif shifts probability mass toward smaller errors and trims the large-error tail relative to the classical baseline.
Compound
True BG
BL Error
Motif Error
Change
Fold
Largest improvements (Motif reduces error)
(CHA) 2 PbBr 4
3.41
0.41
0.12
−71%
Cyclo
(PyrA) 2 SnI 4
1.83
0.35
0.09
−74%
Hetero
(4-FPEA) 2 PbBr 4
3.52
0.38
0.14
−63%
Phenyl
(ThMA) 2 PbI 4
2.27
0.29
0.08
−72%
Hetero
(CyPA) 2 PbI 4
2.35
0.32
0.11
−66%
Cyclo
Appendix
Table 23: 2D perovskite bandgap: representative compounds where Motif produces the largest improvement and degradation relative to the baseline.
Component
Time
API Cost
Hardware
Notes
Motif Pipeline
Taxonomy generation ( gpt-4o-mini )
3 min
$0.03
CPU
One-time, cached
Taxonomy generation ( gpt-4o )
5 min
$0.37
CPU
One-time, cached
Taxonomy generation ( o3-mini )
—
—
CPU
First cached run predates usage logging
Taxonomy generation ( o3 )
—
$5.22
CPU
One-time, cached
Taxonomy generation ( o1 )
—
$134.02
CPU
One-time, cached
Appendix
Table 24: Computational cost breakdown for Motif and baselines.
Subcategory
# Configs
Best MAE
vs. BL
Feature augmentation
12
0.127
−1%
Feature imputation
10
0.128
+0%
LLM embeddings
8+
0.131
+2%
Appendix
Table 25: In-house LLM feature-generation experiments. All results are 6-fold mean OOD MAE (eV). Baseline: 0.122 eV.
Subcategory
# Configs
Best MAE
vs. BL
Importance weighting
8
0.126
−2%
Sample reranking
7
0.127
−1%
Selective correction
5+
0.130
+2%
Appendix
Table 26: In-house LLM sample-selection experiments. All results are 6-fold mean OOD MAE (eV). Baseline: 0.122 eV.
Subcategory
# Configs
Best MAE
vs. BL
GP prior
5
0.132
+3%
Physics constraints
6
0.134
+5%
Bayesian blending
4+
0.133
+4%
Appendix
Table 27: In-house LLM prior/constraint experiments. All results are 6-fold mean OOD MAE (eV). Baseline: 0.122 eV.
Category
# Configs
Best MAE (eV)
Best Δ
(iii) LLM-generated features
30+
0.127
−1%
(iv) LLM-guided sample selection
20+
0.126
−2%
(v) LLM as prior/constraint
15+
0.132
+3%
Total
65+
0.126
−2%
Appendix
Table 28: Summary of in-house configuration variants by category. “Best Δ ” is the best improvement over the 0.122 eV baseline within that category.
Comparison
Backbone
Δ MAE
95% CI
p
BL vs Safe Blend
gpt-4o
+ 0.0041
[0.0014, 0.0068]
0.002
BL vs Top-1 CV
gpt-4o
+ 0.0021
[ − 0.0030, 0.0072]
0.206
BL vs Safe Blend
o3
+ 0.0058
[0.0031, 0.0086]
< 0.001
BL vs Top-1 CV
o3
+ 0.0042
[ − 0.0008, 0.0094]
0.051
BL vs Safe Blend
gpt-5.2
+ 0.0052
[0.0026, 0.0079]
< 0.001
BL vs Top-1 CV
gpt-5.2
+ 0.0045
[ − 0.0003, 0.0099]
0.036
Appendix
Table 29: Query-level paired bootstrap CIs for the 2D perovskite bandgap control benchmark. All MAE values in eV.
Fold
n
BL MAE
Motif MAE
Δ [95% CI]
p
Alkyl-short
232
0.130
0.128
+ 0.002 [ − 0.003, 0.007]
0.201
Alkyl-long
79
0.124
0.120
+ 0.004 [ − 0.001, 0.009]
0.060
Phenyl
166
0.133
0.123
+ 0.010 [0.003, 0.017]
0.004
Heterocyclic
128
0.111
0.108
+ 0.002 [ − 0.004, 0.009]
0.220
Cycloalkyl
37
0.137
0.117
+ 0.020 [0.004, 0.037]
0.006
Special
111
0.106
0.110
− 0.003 [ − 0.008, 0.001]
0.926
Appendix
Table 30: Per-fold bootstrap CIs on 2D perovskite bandgap (Baseline vs Safe Blend, gpt-4o ). Positive Δ = Motif better.
Fold
Role
BL MAE
Motif MAE
Δ%
Alkyl-short
Dev
0.130
0.128
−1.6%
Phenyl
Dev
0.133
0.123
−7.3%
Alkyl-long
Test
0.124
0.120
−3.2%
Heterocyclic
Test
0.111
0.108
−2.2%
Cycloalkyl
Test
0.137
0.117
−14.7%
Special
Test
0.106
0.110
+3.0%
Appendix
Table 31: Frozen dev/test results on 2D perovskite bandgap. Dev folds were used for α selection only; test folds were never used for any design decision.
Fold
Test Spacers
Train Spacers
Overlap
Test-Only
Alkyl-short
92
197
0
92
Alkyl-long
20
269
0
20
Phenyl
73
216
0
73
Heterocyclic
55
234
0
55
Cycloalkyl
14
275
0
14
Special
48
241
0
48
Appendix
Table 32: Spacer overlap verification on 2D perovskite benchmark. In every fold, the overlap between test and training spacers is zero, confirming no leakage.
Selector
Mean MAE (eV)
vs. Baseline
Chow test ( α=0.01 ) + forward sel.
0.1144
−7.2%
Bonferroni Chow ( α/24 )
0.1233
0.0%
Permutation test (200 perms)
0.1233
0.0%
MI-stability ( σr<0.05 )
0.1244
+0.9%
Baseline (no invariance filter)
0.1233
—
Appendix
Table 33: Alternative invariance selectors on 2D perovskite bandgap. All use the same gated-median ensemble and taxonomy bank. Mean MAE is across 6 OOD folds.
Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel molecules. Discovering novel molecules requires accurate out-of-distribution (OOD) predictions, but ML models struggle to generalize OOD. Currently, no systematic benchmarks exist for molecular OOD prediction tasks. We present BOOM, benchmarks for out-of-distribution molecular property predictions: a chemically-informed benchmark for OOD performance on common molecular property prediction tasks. We evaluate over 150 model-task combinations to benchmark deep learning models on OOD performance. Overall, we find that no existing model achieves strong generalization across all tasks: even the top-performing model exhibited an average OOD error 3x higher than in-distribution. Current chemical foundation models do not show strong OOD extrapolation, while models with high inductive bias can perform well on OOD tasks with simple, specific properties. We perform extensive ablation experiments, highlighting how data generation, pre-training, hyperparameter optimization, model architecture, and molecular representation impact OOD performance. Developing models with strong OOD generalization is a new frontier challenge in chemical ML. This open-source benchmark is available at https://github.com/FLASK-LLNL/BOOM
Evan R. Antoniuk, Shehtab Zaman, Tal Ben-Nun +10
Lawrence Livermore National Laboratory, Livermore, CA · Binghamton University, Binghamton, NY
General-purpose Large Language Models (LLMs) increasingly rival domain-specific pipelines on scientific prediction, but their performance under distribution shift is untested. Across six alloy and crystal-structure datasets, we benchmark LLMs against traditional, BERT-based, TabPFN, and text-regression baselines under random cross-validation and six out-of-distribution (OOD) protocols shifting input and target distributions, with severity quantified via a standardized Wasserstein distance. We isolate the LLM's advantage over k-nearest-neighbor (k-NN) retrieval: it beats k-NN on all eight tasks at a fixed 5% reference budget, on seven of eight when independently optimized, and in 72.5% of 120 OOD task--protocol configurations against an optimized baseline. No model family is uniformly best; the LLM is often strongest under severe shift, and error increases with target-space but not representation-space severity. Self-reported LLM confidence tracks both shift and error, more consistently than a density-only control from the same retrieved samples, and a matched-cohort check finds no substantial evidence of memorization.
YuJie Liu, Dezhen Xue, Jun Sun +3
State Key Laboratory for Mechanical Behavior of Materials, State Key Laboratory of Porous Metal Materials, School of Materials Science and Engineering,Xi’an Jiatong University, Xi’an 710049, China · AiMaterial Research, Santa Fe, New Mexico, USA 87501
Out-of-distribution (OOD) detection is essential for deploying machine learning models in high stakes scientific applications, yet methods that flag OOD inputs without retraining the model have been validated almost exclusively on image classification. We introduce MOOD (Materials Out-of-Distribution Detection), a benchmark for evaluating whether these post-hoc OOD detectors transfer to graph neural networks (GNNs) on materials property prediction. MOOD evaluates 15 post-hoc detectors and 2 supervised diagnostic probes across 4 GNN architectures, 5 MatBench tasks, and 15 physically motivated compositional and structural shifts, yielding 264 evaluation settings and over 13,000 AUROC measurements across three encoder training seeds. Our evaluation reveals three key findings. First, current post-hoc detectors perform near chance; all 15 detectors achieve an AUROC around 0.5. Second, detector failures decompose into two regimes: method bottlenecks and encoder bottlenecks. Method bottlenecks arise when supervised probes can separate in-distribution (ID) from OOD samples but unsupervised detectors fail, meaning that the embeddings contain OOD signal that current detectors do not extract. Encoder bottlenecks arise when even supervised probes fail to separate ID from OOD samples, indicating that the relevant shift is not accessible in the frozen GNN representation. Third, when OOD prediction error exceeds ID error, the AUROC of distance-based and density-based detectors correlates with the size of that gap. Collectively, these findings indicate that reliable OOD detection in scientific domains requires representation learning strategies that explicitly preserve OOD relevant signal, particularly for tasks where current GNN architectures encode no extractable signal at all.
Shirley X. Yu, Amey P. Pasarkar, Adji Bousso Dieng
Department of Computer Science, Princeton University · Vertaix · Lewis-Sigler Institute for Integrative Genomics, Princeton University