Out-of-distribution (OOD) detection is essential for deploying machine learning models in high stakes scientific applications, yet methods that flag OOD inputs without retraining the model have been validated almost exclusively on image classification. We introduce MOOD (Materials Out-of-Distribution Detection), a benchmark for evaluating whether these post-hoc OOD detectors transfer to graph neural networks (GNNs) on materials property prediction. MOOD evaluates 15 post-hoc detectors and 2 supervised diagnostic probes across 4 GNN architectures, 5 MatBench tasks, and 15 physically motivated compositional and structural shifts, yielding 264 evaluation settings and over 13,000 AUROC measurements across three encoder training seeds. Our evaluation reveals three key findings. First, current post-hoc detectors perform near chance; all 15 detectors achieve an AUROC around 0.5. Second, detector failures decompose into two regimes: method bottlenecks and encoder bottlenecks. Method bottlenecks arise when supervised probes can separate in-distribution (ID) from OOD samples but unsupervised detectors fail, meaning that the embeddings contain OOD signal that current detectors do not extract. Encoder bottlenecks arise when even supervised probes fail to separate ID from OOD samples, indicating that the relevant shift is not accessible in the frozen GNN representation. Third, when OOD prediction error exceeds ID error, the AUROC of distance-based and density-based detectors correlates with the size of that gap. Collectively, these findings indicate that reliable OOD detection in scientific domains requires representation learning strategies that explicitly preserve OOD relevant signal, particularly for tasks where current GNN architectures encode no extractable signal at all.
Figures & tables
Figure 1: MatBench tasks with sample sizes and OOD splits applied to each. Every panel shows a representative material. On the left is a general example from the task’s dataset: rutile TiO 2 for dielectric, MgB 2 for log_gvrh, NaCl for log_kvrh, SrTiO 3 for perovskites, diamond Si for phonons. On the right is an example belonging to the split’s OOD class: SiC for carbide, NaCl for cubic, SrTiO 3 for perovskite, etc.
Family
Detector
Mean AUROC ↑
Median AUROC ↑
Win count ↑
% settings >0.55↑
Distance
KNN-k20-norm
0.512 ± 0.001
0.504 ± 0.001
13.2 ± 3.0
10.4 ± 0.8%
KNN-k50-norm
0.508 ± 0.001
0.503 ± 0.001
7.7 ± 3.5
9.3 ± 1.8%
Norm
0.500 ± 0.001
0.500 ± 0.001
23.3 ± 4.0
6.1 ± 1.1%
Cosine
0.531 ± 0.001
0.507 ± 0.001
54.8 ± 8.6
13.5 ± 0.6%
LOF
0.526 ± 0.001
0.505 ± 0.001
30.2 ± 3.3
13.1 ± 0.2%
ASH+KNN
0.516 ± 0.002
0.505 ± 0.002
30.8 ± 5.3
11.3 ± 0.6%
Table 1: Per-detector performance summary across Nall=264 settings (mean ± std over three encoder training seeds). Win count splits ties equally; % settings >0.55 uses the above chance threshold of § 3 . Supervised probes (bottom two rows) are diagnostic ceilings, not deployable detectors. Bold is the best unsupervised detector per family. ↑ indicates higher is better.
Figure 2: Headline AUROC across Nall=264 settings. Every unsupervised detector median is within 0.01 of chance. The supervised probes extend to AUROC ≈1.0 on multiple settings (circled in red), indicating extractable OOD signal exists in some embeddings.
Figure 3: Family level analysis. (A) Per-setting family-max AUROC. Error bars are ±1 Standard Error of the Mean (SEM) across Nall=264 settings. (B) Win rate per family, including the supervised probes as a fifth group, with ties split equally.
dielectric
log_gvrh
log_kvrh
perovskites
phonons
N
56
60
60
36
52
Linear ↑
0.599 ± 0.001
0.600 ± 0.002
0.494 ± 0.004
0.500 ± 0.005
0.499 ± 0.009
MLP ↑
0.601 ± 0.005
0.606 ± 0.003
0.490 ± 0.003
0.499 ± 0.002
0.493 ± 0.008
Mean det. ↑
0.522 ± 0.002
0.524 ± 0.001
0.504 ± 0.001
0.500 ± 0.001
0.502 ± 0.003
Best det. ↑
0.569 ± 0.003
0.560 ± 0.001
0.505 ± 0.001
0.502 ± 0.001
0.506 ± 0.003
Gap (mean) ↑
+0.078 ± 0.002
+0.076 ± 0.002
− 0.010 ± 0.004
+0.001 ± 0.005
− 0.003 ± 0.010
Table 2: Bottleneck dichotomy by task (mean ± std over three encoder training seeds). Linear/MLP are the supervised probe AUROCs. Mean/Best detector are the mean across all 15 unsupervised detectors and Cosine (highest-mean detector). Gap (mean) = Linear − Mean detector (sets the Bottleneck label); Gap (best) = Linear − Best detector. On both method-bottlenecked tasks the probe exceeds even the best detector, by less than 0.05 (§3.4, Appendix H). Method columns shaded yellow, Encoder columns blue. Bold is the best value per row. ↑ indicates higher is better.
Figure 4: Method vs. encoder bottlenecks. (A) Linear vs. MLP probe AUROC; close agreement ( r=0.971 ) rules out nonlinearity as the explanation for the encoder bottleneck. (B) Linear probe vs. Cosine (best deployable detector); two clusters emerge separating method-bottlenecked tasks (probe high, detector low) from encoder-bottlenecked ones (both near chance). (C) Linear probe AUROC by task; yellow ( Method ) and blue ( Encoder ) shading mark the two bottleneck regimes.
Figure 5: Detector AUROC tracks regression failure magnitude. Per-setting AUROC against log10(MAEOOD/MAEID) for one representative detector per family. Blue lines fit Nall=264 settings; red lines fit the degraded subset Ndeg=153 . Per-detector grid in Appendix D , Figure 7 .
Detector
β (all) ↑
R2 (all) ↑
β (degraded) ↑
R2 (degraded) ↑
Cosine (Distance)
+0.322
0.27
+0.598
0.45
MDS-BIC (Density)
+0.228
0.23
+0.309
0.21
IForest (Isolation)
+0.160
0.21
+0.185
0.15
ViM (Boundary / Hybrid)
+0.184
0.19
+0.230
0.15
Linear probe (Supervised)
+0.321
0.13
+0.745
0.38
MLP probe (Supervised)
+0.361
0.15
+0.795
0.40
Table 3: Regression of detector AUROC against log10(MAEOOD/MAEID) . OLS slope β and R2 for one representative detector per family plus the supervised probes, fit on the seed-averaged per-setting points (AUROC and MAE ratio averaged over the three encoder seeds) for all Nall=264 settings and the degraded subset Ndeg=153 . Per-seed fits ( β± std) are in Appendix Table 14 . Bold is the best value per column. ↑ indicates higher is better.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Target
MatBench n
Splits
Archs
Cells
Mean ∣IDtest∣
Mean ∣OODtest∣
dielectric
refractive index
4,764
14
4
56
361
361
log_gvrh
log shear modulus
10,987
15
4
60
691
691
log_kvrh
log bulk modulus
10,987
15
4
60
712
712
perovskites
formation energy
18,928
9
4
36
1,274
1,274
phonons
phonon frequency
1,265
13
4
52
98
98
Appendix
Table 4: Per-dataset benchmark statistics. For each MatBench task we report the prediction target, total task size, number of OOD splits retained after the ≥50 minority class filter, number of architectures evaluated, total number of settings (arch × split), and mean balanced ID/OOD test sizes across settings.
Split rule
Type
Definition
comp_oxide
composition
structure contains at least one O atom
comp_halide
composition
structure contains at least one F/Cl/Br/I atom
comp_chalcogenide
composition
structure contains at least one S/Se/Te atom
comp_nitride
composition
structure contains at least one N atom
comp_carbide
composition
structure contains at least one C atom
comp_intermetallic
composition
structure contains only metallic elements
Appendix
Table 5: OOD split rule definitions. Each rule maps a structure to a binary OOD label.
Dataset
Split
Arch
Reason
Details
Splits dropped at construction (minority class < 50):
dielectric
comp_carbide
all
minority class < 50
N<50
perovskites
comp_carbide
all
minority class < 50
N<50
perovskites
comp_intermetallic
all
minority class < 50
N<50
perovskites
struct_hexagonal
all
minority class < 50
N<50
perovskites
struct_low_spacegroup
all
minority class < 50
N<50
Appendix
Table 6: Excluded splits and seed 42 collapses. One filter reduces the candidate 5×15×4=300 settings to the 264 reported in the main paper. Top: splits dropped at construction because the minority OOD class has fewer than 50 samples in the source dataset ( N=9 splits, all 4 architectures). Bottom: CGCNN/phonons encoders that collapse to constant embeddings under seed 42 only ( N=7 settings); these train normally under seeds 123 and 456, are retained, and are averaged over the two valid seeds.
Figure 6: MOOD benchmark axes. 5 MatBench datasets, 4 GNN architectures, 15 OOD splits (6 compositional, 9 structural), and 17 detectors across 5 families. The full grid yields 300 candidate (dataset, architecture, split) settings; 264 are kept after dropping splits with fewer than 50 minority class samples. Seven CGCNN/phonons settings collapse under seed 42 only and are averaged over the two valid seeds. Running 17 detection methods on each setting yields 4,488 AUROC measurements per seed.
Family
Detector
Mean ↑
Median ↑
Mean (deg.) ↑
Probe gap ↓
Wins ↑
% > 0.55 ↑
Distance
KNN-k20-norm
0.512
0.504
0.523
+0.031
13.2
10.4%
KNN-k50-norm
0.508
0.503
0.518
+0.034
7.7
9.3%
Norm
0.500
0.500
0.504
+0.043
23.3
6.1%
Cosine
0.531
0.507
0.549
+0.012
54.8
13.5%
LOF
0.526
0.505
0.542
+0.017
30.2
13.1%
ASH+KNN
0.516
0.505
0.530
+0.026
30.8
11.3%
Appendix
Table 7: Per-detector performance summary (full version with degraded subset and probe gap columns; the main paper Table 1 omits these for width; central values are means over three encoder seeds). Mean / Median AUROC: across all Nall=264 settings. Mean AUROC (degraded): restricted to the Ndeg=153 settings where seed mean OOD MAE > seed mean ID MAE. Probe gap: mean per-setting (linear probe AUROC − this AUROC); smaller is better. Win count: ties split equally. Bold is the best value per column among unsupervised detectors. ↑ indicates higher is better; ↓ indicates lower is better.
Alignn
CGCNN
SchNet
DimeNet
Dataset
ID ↓
OOD ↓
ID ↓
OOD ↓
ID ↓
OOD ↓
ID ↓
OOD ↓
dielectric
0.447 ± 0.020
0.480 ± 0.018
0.567 ± 0.007
0.605 ± 0.021
0.417 ± 0.028
0.463 ± 0.010
0.430 ± 0.025
0.476 ± 0.034
log_gvrh
0.074 ± 0.002
0.098 ± 0.001
0.097 ± 0.002
0.128 ± 0.007
0.094 ± 0.003
0.118 ± 0.004
0.098 ± 0.006
0.124 ± 0.008
log_kvrh
0.060 ± 0.003
0.061 ± 0.003
0.079 ± 0.005
0.079 ± 0.006
0.070 ± 0.002
0.071 ± 0.002
0.069 ± 0.001
0.069 ± 0.000
perovskites
0.034 ± 0.001
0.033 ± 0.001
0.069 ± 0.005
0.068 ± 0.005
0.046 ± 0.005
0.046 ± 0.005
0.045 ± 0.007
0.045 ± 0.007
phonons
34.9 ± 8.2
38.3 ± 7.6
78.4 ± 3.6
85.8 ± 1.6
58.3 ± 4.3
65.2 ± 4.3
58.1 ± 5.7
66.2 ± 8.4
Appendix
Table 8: Mean ID and OOD MAE per dataset and architecture (mean ± std over three encoder seeds). Averaged across the OOD splits retained for each dataset. ↓ indicates lower is better.
Dataset
Split
Arch
ID MAE
OOD MAE
dielectric
comp_chalcogenide
alignn
0.4153 ± 0.0313
0.3598 ± 0.0395
cgcnn
0.5958 ± 0.0585
0.5928 ± 0.1980
dimenet
0.6899 ± 0.3254
0.3388 ± 0.0031
schnet
0.3327 ± 0.0181
0.5772 ± 0.0804
comp_halide
alignn
0.4861 ± 0.0517
1.5563 ± 0.0865
cgcnn
0.5829 ± 0.0062
1.2402 ± 0.3578
Appendix
Table 9: Per-setting held-out ID and OOD MAE (mean ± std over three encoder seeds). Each row is one retained (dataset, split, arch) setting ( N=264 ). log10 gap =log10(seed mean OOD MAE/seed mean ID MAE) .
Architecture
Split type
Detector
Alignn ↑
Cgcnn ↑
Schnet ↑
Dimenet ↑
Comp. ↑
Struct. ↑
KNN-k20
0.502 ± 0.002
0.518 ± 0.007
0.509 ± 0.003
0.519 ± 0.002
0.529 ± 0.002
0.501 ± 0.001
KNN-k50
0.497 ± 0.002
0.514 ± 0.008
0.506 ± 0.003
0.517 ± 0.001
0.523 ± 0.003
0.499 ± 0.002
Norm
0.512 ± 0.001
0.503 ± 0.002
0.491 ± 0.001
0.494 ± 0.001
0.502 ± 0.003
0.498 ± 0.001
Cosine
0.530 ± 0.001
0.535 ± 0.003
0.529 ± 0.001
0.530 ± 0.002
0.556 ± 0.001
0.515 ± 0.001
LOF
0.536 ± 0.002
0.530 ± 0.003
0.515 ± 0.002
0.522 ± 0.001
0.545 ± 0.002
0.513 ± 0.001
Appendix
Table 10: Per-detector AUROC, stratified by architecture and split type (mean ± std over three encoder seeds). Compositional N=104 , structural N=160 settings. No detector exceeds 0.55 in any column except Cosine on the compositional subset ( 0.556 ). Bold is the best value per column among unsupervised detectors. ↑ indicates higher is better.
Family
All ↑
dielectric ↑
log_gvrh ↑
log_kvrh ↑
perovskites ↑
phonons ↑
Distance
0.516 ± 0.000
0.531 ± 0.001
0.530 ± 0.001
0.506 ± 0.000
0.501 ± 0.001
0.502 ± 0.003
Density
0.515 ± 0.001
0.530 ± 0.002
0.534 ± 0.001
0.505 ± 0.001
0.500 ± 0.001
0.499 ± 0.003
Isolation
0.500 ± 0.002
0.494 ± 0.004
0.498 ± 0.003
0.502 ± 0.001
0.497 ± 0.001
0.507 ± 0.003
Boundary / Hybrid
0.502 ± 0.001
0.501 ± 0.003
0.507 ± 0.002
0.500 ± 0.001
0.497 ± 0.001
0.505 ± 0.004
Supervised
0.543 ± 0.001
0.600 ± 0.003
0.603 ± 0.002
0.492 ± 0.003
0.499 ± 0.004
0.496 ± 0.008
Appendix
Table 11: Mean AUROC by detector family and dataset (mean ± std over three encoder seeds). Each row aggregates the per-setting AUROCs of every detector in that family. The supervised row is a diagnostic ceiling, excluded from rankings. No unsupervised family reaches ≥0.55 on any dataset. Bold is the best value per column among unsupervised detectors. ↑ indicates higher is better.
Detector
All ↓
dielectric ↓
log_gvrh ↓
log_kvrh ↓
perovskites ↓
phonons ↓
MDS-BIC
0.935 ± 0.002
0.926 ± 0.004
0.917 ± 0.003
0.946 ± 0.001
0.953 ± 0.001
0.939 ± 0.007
KNN-k10
0.942 ± 0.002
0.935 ± 0.005
0.937 ± 0.004
0.945 ± 0.002
0.949 ± 0.002
0.946 ± 0.005
kPCA
0.938 ± 0.002
0.926 ± 0.004
0.929 ± 0.005
0.945 ± 0.002
0.950 ± 0.001
0.944 ± 0.006
Norm
0.947 ± 0.002
0.953 ± 0.003
0.939 ± 0.002
0.948 ± 0.001
0.951 ± 0.001
0.946 ± 0.006
LOF
0.928 ± 0.001
0.910 ± 0.005
0.894 ± 0.004
0.945 ± 0.002
0.951 ± 0.003
0.953 ± 0.006
OCSVM
0.946 ± 0.002
0.943 ± 0.004
0.945 ± 0.005
0.948 ± 0.000
0.950 ± 0.002
0.944 ± 0.004
Appendix
Table 12: FPR95 per detector and per dataset (mean ± std over three encoder seeds). Lower is better. Covers 14 of 15 unsupervised detectors (Flow FPR95 is deferred to v2). KNN-k10 is the default k=10 KNN from the per-setting evaluation outputs, distinct from the k=20 / k=50 variants in Table 1 . Bold is the best value per column among unsupervised detectors. ↓ indicates lower is better.
Dataset
N
Mean linear probe
% p<0.01
% p<0.05
Bottleneck
dielectric
56
0.599 ± 0.001
28.6 ± 0.0 (16/56, 16/56, 16/56)
31.5 ± 1.0 (18/56, 17/56, 18/56)
Method
log_gvrh
60
0.600 ± 0.002
26.7 ± 0.0 (16/60, 16/60, 16/60)
26.7 ± 0.0 (16/60, 16/60, 16/60)
Method
log_kvrh
60
0.494 ± 0.004
0.0 ± 0.0 (0/60, 0/60, 0/60)
4.4 ± 4.8 (1/60, 1/60, 6/60)
Encoder
perovskites
36
0.500 ± 0.005
0.0 ± 0.0 (0/36, 0/36, 0/36)
2.8 ± 2.8 (0/36, 1/36, 2/36)
Encoder
phonons
52
0.499 ± 0.009
0.6 ± 1.1 (0/45, 0/52, 1/52)
1.9 ± 1.9 (0/45, 1/52, 2/52)
Encoder
Appendix
Table 13: Per-setting permutation test of the linear probe ( B=100 , same 5-fold protocol), per encoder seed. p=(1+#{null≥real})/(B+1) . Rejection rates are mean ± std across seeds; per-seed counts given as c42/c123/c456 out of N . The linear probe column is from the canonical probe pipeline. Under the null, ∼ 1% of settings reach p<0.01 by chance.
Detector
β (all) ↑
R2 (all) ↑
β (degraded) ↑
R2 (degraded) ↑
KNN-k20-norm
+0.215 ± 0.020
0.22 ± 0.01
+0.295 ± 0.009
0.20 ± 0.01
KNN-k50-norm
+0.191 ± 0.019
0.18 ± 0.01
+0.243 ± 0.013
0.15 ± 0.01
Norm
+0.069 ± 0.018
0.04 ± 0.02
+0.042 ± 0.016
0.01 ± 0.00
Cosine
+0.311 ± 0.027
0.26 ± 0.02
+0.579 ± 0.024
0.43 ± 0.02
LOF
+0.259 ± 0.042
0.22 ± 0.06
+0.466 ± 0.027
0.36 ± 0.03
ASH+KNN
+0.215 ± 0.032
0.21 ± 0.04
+0.347 ± 0.053
0.29 ± 0.08
Appendix
Table 14: Full per-detector OLS of AUROC against log10(MAEOOD/MAEID) (mean ± std over three encoder seeds), on all Nall=264 settings and the degraded subset. Bold is the best value per column. ↑ indicates higher is better.
Figure 7: Detector AUROC vs log10(OOD MAE/ID MAE) , one panel per detector. Full 4×5 grid ( 17 panels) corresponding to the compacted main-paper Figure 5 (one panel per family). Blue regression lines fit all Nall=264 settings; red regression lines fit the degraded subset ( Ndeg=153 , seed mean OOD MAE > seed mean ID MAE).
Dataset
ALIGNN ↑
CGCNN ↑
SchNet ↑
DimeNet++ ↑
Spread
Label
dielectric
0.612 ± 0.003
0.599 ± 0.002
0.596 ± 0.002
0.591 ± 0.004
0.022
Method
log_gvrh
0.612 ± 0.002
0.608 ± 0.001
0.595 ± 0.003
0.585 ± 0.003
0.027
Method
log_kvrh
0.488 ± 0.004
0.502 ± 0.010
0.496 ± 0.006
0.491 ± 0.007
0.014
Encoder
perovskites
0.495 ± 0.014
0.504 ± 0.006
0.504 ± 0.002
0.498 ± 0.005
0.009
Encoder
phonons
0.521 ± 0.008
0.498 ± 0.019
0.491 ± 0.011
0.485 ± 0.020
0.036
Encoder
Appendix
Table 15: Linear-probe AUROC by dataset and architecture (mean ± std over three seeds). All 20 cells fall on the same side of the 0.55 threshold as the dataset-level label. Bold is the best value per row. ↑ indicates higher is better.
Detector
N
Mean AURC ↓
Median AURC ↓
Mean SPE ↑
Median SPE ↑
% SPE > 0 ↑
MDS-BIC
257
6.936
0.088
+0.340
+0.349
94.2%
KNN k=10
257
7.571
0.084
+0.363
+0.348
93.4%
LOF
257
6.970
0.082
+0.347
+0.306
98.8%
kPCA
257
6.700
0.085
+0.372
+0.374
99.2%
OCSVM
257
7.212
0.082
+0.317
+0.289
99.6%
IForest
257
6.591
0.086
+0.354
+0.333
99.2%
Appendix
Table 16: Risk-coverage summary per detector. Per-setting AURC (area under the risk-coverage curve, lower is better) and SPE (selective-prediction effectiveness; SPE = 1 matches an oracle abstaining on worst error samples first, SPE = 0 matches random abstention, SPE < 0 is worse than random) aggregated across N=257 seed 42 settings. Unlike the other appendix tables, these values are computed on the seed 42 encoder run only: the risk-coverage curves require per-sample detector scores, which are not stored per seed, so a 3-seed mean ± std is deferred to the next benchmark version. The eight detectors with directly comparable score direction conventions (Mahalanobis variants, KNN k=10 , LOF, kPCA, OCSVM, IForest, Norm, normalising flow density) are reported. Here KNN k=10 is the default k=10 KNN from the per-setting evaluation outputs, distinct from the k=20 / k=50 variants in Table 1 . Bold is the best value per column. ↓ indicates lower is better; ↑ indicates higher is better.
Dataset
Split
ID MAE ↓
OOD MAE ↓
log gap
Maha AUROC ↑
KNN AUROC ↑
dielectric
LOCO
0.429
0.408
-0.021
0.503
0.501
SparseXcluster
0.469
0.537
+0.068
0.507
0.519
SparseYcluster
0.463
0.685
+0.222
0.502
0.509
log_gvrh
LOCO
0.089
0.089
-0.001
0.500
0.498
SparseXcluster
0.090
0.091
+0.001
0.544
0.547
SparseYcluster
0.090
0.081
-0.009
0.528
0.523
Appendix
Table 17: Omee distribution-shift cross-check. The Omee suite ( Omee et al. [2024] ) defines three alternative split protocols: LOCO (leave-one-cluster-out), SparseXcluster (sparse covariate clusters), and SparseYcluster (sparse target clusters). We rerun the two detectors with directly comparable implementations (Mahalanobis and KNN) on these splits using the same ALIGNN and CGCNN embeddings as the main paper, 50 folds per split. Mean AUROC remains within ∼ 0.05 of chance, consistent with the near chance story under physics motivated splits. ↓ indicates lower is better. ↑ indicates higher is better.
bond directionality, anisotropy [ De Jong et al., 2015 ]
partial
✓
log_kvrh
logKVRH (bulk)
mean bond stiffness, density [ De Jong et al., 2015 ]
weak
weak
perovskites
formation energy
cohesive energy within ABX 3 [ Castelli et al., 2012 ]
weak
weak
phonons
highest optical phonon freq.
reduced mass, local force constants [ Petretto et al., 2018 ]
weak
weak
Appendix
Table 18: Physical drivers of MatBench targets and split-axis alignment. Compositional axes shift element families (oxide, halide, etc.); structural axes shift lattice symmetry (cubic, low-spacegroup, etc.). A shift is property relevant if the property’s physical drivers are sensitive to that axis.
Figure 8: Probe AUROC against composition-only shift magnitude. The two regimes are distinguished by AUROC range, not slope: encoder-bottlenecked datasets are pinned at chance regardless of compositional shift (OLS slope 0.000 ), while method-bottlenecked datasets reach probe AUROCs up to 1.0 but the highest values cluster at low Magpie shift (OLS slope −0.030 ), showing that probe recoverability is governed by property relevant encoding rather than crude compositional distance.
Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel molecules. Discovering novel molecules requires accurate out-of-distribution (OOD) predictions, but ML models struggle to generalize OOD. Currently, no systematic benchmarks exist for molecular OOD prediction tasks. We present BOOM, benchmarks for out-of-distribution molecular property predictions: a chemically-informed benchmark for OOD performance on common molecular property prediction tasks. We evaluate over 150 model-task combinations to benchmark deep learning models on OOD performance. Overall, we find that no existing model achieves strong generalization across all tasks: even the top-performing model exhibited an average OOD error 3x higher than in-distribution. Current chemical foundation models do not show strong OOD extrapolation, while models with high inductive bias can perform well on OOD tasks with simple, specific properties. We perform extensive ablation experiments, highlighting how data generation, pre-training, hyperparameter optimization, model architecture, and molecular representation impact OOD performance. Developing models with strong OOD generalization is a new frontier challenge in chemical ML. This open-source benchmark is available at https://github.com/FLASK-LLNL/BOOM
Evan R. Antoniuk, Shehtab Zaman, Tal Ben-Nun +10
Lawrence Livermore National Laboratory, Livermore, CA · Binghamton University, Binghamton, NY
When testing data and training data come from different distributions, deep neural networks (DNNs) will face significant safety risks in practical applications. Therefore, out-of-distribution (OOD) detection techniques, which can identify OOD samples at test time and alert the system, are urgently needed. Existing graph OOD detection methods usually characterize fine-grained in-distribution (ID) patterns from multiple perspectives, and train end-to-end graph neural networks (GNNs) for prediction. However, due to the unavailability of OOD data during training, the absence of explicit supervision signals could lead to sub-optimal performance of end-to-end encoders. To address this issue, we follow the pre-training+prompting paradigm to utilize pre-trained GNN encoders, and propose Disentangled Graph Prompting (DGP), to capture fine-grained ID patterns with the help of ID graph labels. Specifically, we design two prompt generators that respectively generate class-specific and class-agnostic prompt graphs by modifying the edge weights of an input graph. We also design several effective losses to train the prompt generators and prevent trivial solutions. We conduct extensive experiments on ten datasets to demonstrate the superiority of our proposed DGP, which achieves a relative AUC improvement of 3.63% over the best graph OOD detection baseline. Ablation studies and hyper-parameter experiments further show the effectiveness of DGP. Code is available at https://github.com/BUPT-GAMMA/DGP.
Cheng Yang, Yu Hao, Qi Zhang +1
the Beijing Key Lab of Intelligent Telecommunications Software and Multimedia, Beijing University of Posts and Telecommunications, China · China Mobile Group Shaanxi Co., Ltd.
Out-of-distribution (OOD) detection is essential for reliable deployment of deep learning systems, yet the majority of existing methods are evaluated on small, visually homogeneous benchmarks. In this work, we study six OOD detection methods spanning post-hoc scoring, auxiliary objectives, energy-based models, and constrained optimization on the Plant Pathology 2021 dataset, a fine-grained task with natural distribution shifts. Energy-based fine-tuning performs best across OOD settings, improving detection over the softmax baseline while preserving in-distribution accuracy. Analysis shows these gains stem from both a restructuring of the embedding space alongside calibration of the scoring function. We further document practical training instabilities that arise when scaling constrained optimization methods to moderate-sized datasets, findings that are largely absent from existing literature. Our results demonstrate that principled OOD detection is achievable on real-world domain-specific data and that benchmark evaluations alone may not capture the challenges that emerge in practice.