MOOD: Benchmarking Post-Hoc OOD Detection for Materials Property Prediction
Organizations: Department of Computer Science, Princeton University · Vertaix · Lewis-Sigler Institute for Integrative Genomics, Princeton University
Abstract
Out-of-distribution (OOD) detection is essential for deploying machine learning models in high stakes scientific applications, yet methods that flag OOD inputs without retraining the model have been validated almost exclusively on image classification. We introduce MOOD (Materials Out-of-Distribution Detection), a benchmark for evaluating whether these post-hoc OOD detectors transfer to graph neural networks (GNNs) on materials property prediction. MOOD evaluates 15 post-hoc detectors and 2 supervised diagnostic probes across 4 GNN architectures, 5 MatBench tasks, and 15 physically motivated compositional and structural shifts, yielding 264 evaluation settings and over 13,000 AUROC measurements across three encoder training seeds. Our evaluation reveals three key findings. First, current post-hoc detectors perform near chance; all 15 detectors achieve an AUROC around 0.5. Second, detector failures decompose into two regimes: method bottlenecks and encoder bottlenecks. Method bottlenecks arise when supervised probes can separate in-distribution (ID) from OOD samples but unsupervised detectors fail, meaning that the embeddings contain OOD signal that current detectors do not extract. Encoder bottlenecks arise when even supervised probes fail to separate ID from OOD samples, indicating that the relevant shift is not accessible in the frozen GNN representation. Third, when OOD prediction error exceeds ID error, the AUROC of distance-based and density-based detectors correlates with the size of that gap. Collectively, these findings indicate that reliable OOD detection in scientific domains requires representation learning strategies that explicitly preserve OOD relevant signal, particularly for tasks where current GNN architectures encode no extractable signal at all.
Figures & tables
| Family | Detector | Mean AUROC | Median AUROC | Win count | % settings |
|---|---|---|---|---|---|
| Distance | KNN-k20-norm | 0.512 0.001 | 0.504 0.001 | 13.2 3.0 | 10.4 0.8% |
| KNN-k50-norm | 0.508 0.001 | 0.503 0.001 | 7.7 3.5 | 9.3 1.8% | |
| Norm | 0.500 0.001 | 0.500 0.001 | 23.3 4.0 | 6.1 1.1% | |
| Cosine | 0.531 0.001 | 0.507 0.001 | 54.8 8.6 | 13.5 0.6% | |
| LOF | 0.526 0.001 | 0.505 0.001 | 30.2 3.3 | 13.1 0.2% | |
| ASH+KNN | 0.516 0.002 | 0.505 0.002 | 30.8 5.3 | 11.3 0.6% |
| dielectric | log_gvrh | log_kvrh | perovskites | phonons | |
|---|---|---|---|---|---|
| 56 | 60 | 60 | 36 | 52 | |
| Linear | 0.599 0.001 | 0.600 0.002 | 0.494 0.004 | 0.500 0.005 | 0.499 0.009 |
| MLP | 0.601 0.005 | 0.606 0.003 | 0.490 0.003 | 0.499 0.002 | 0.493 0.008 |
| Mean det. | 0.522 0.002 | 0.524 0.001 | 0.504 0.001 | 0.500 0.001 | 0.502 0.003 |
| Best det. | 0.569 0.003 | 0.560 0.001 | 0.505 0.001 | 0.502 0.001 | 0.506 0.003 |
| Gap (mean) | +0.078 0.002 | +0.076 0.002 | 0.010 0.004 | +0.001 0.005 | 0.003 0.010 |
| Detector | (all) | (all) | (degraded) | (degraded) |
|---|---|---|---|---|
| Cosine (Distance) | 0.27 | 0.45 | ||
| MDS-BIC (Density) | 0.23 | 0.21 | ||
| IForest (Isolation) | 0.21 | 0.15 | ||
| ViM (Boundary / Hybrid) | 0.19 | 0.15 | ||
| Linear probe (Supervised) | 0.13 | 0.38 | ||
| MLP probe (Supervised) | 0.15 | 0.40 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Target | MatBench | Splits | Archs | Cells | Mean | Mean |
|---|---|---|---|---|---|---|---|
| dielectric | refractive index | 4,764 | 14 | 4 | 56 | 361 | 361 |
| log_gvrh | log shear modulus | 10,987 | 15 | 4 | 60 | 691 | 691 |
| log_kvrh | log bulk modulus | 10,987 | 15 | 4 | 60 | 712 | 712 |
| perovskites | formation energy | 18,928 | 9 | 4 | 36 | 1,274 | 1,274 |
| phonons | phonon frequency | 1,265 | 13 | 4 | 52 | 98 | 98 |
| Split rule | Type | Definition |
| comp_oxide | composition | structure contains at least one O atom |
| comp_halide | composition | structure contains at least one F/Cl/Br/I atom |
| comp_chalcogenide | composition | structure contains at least one S/Se/Te atom |
| comp_nitride | composition | structure contains at least one N atom |
| comp_carbide | composition | structure contains at least one C atom |
| comp_intermetallic | composition | structure contains only metallic elements |
| Dataset | Split | Arch | Reason | Details |
|---|---|---|---|---|
| Splits dropped at construction (minority class 50): | ||||
| dielectric | comp_carbide | all | minority class 50 | |
| perovskites | comp_carbide | all | minority class 50 | |
| perovskites | comp_intermetallic | all | minority class 50 | |
| perovskites | struct_hexagonal | all | minority class 50 | |
| perovskites | struct_low_spacegroup | all | minority class 50 | |
| Family | Detector | Mean | Median | Mean (deg.) | Probe gap | Wins | % 0.55 |
|---|---|---|---|---|---|---|---|
| Distance | KNN-k20-norm | 0.512 | 0.504 | 0.523 | +0.031 | 13.2 | 10.4% |
| KNN-k50-norm | 0.508 | 0.503 | 0.518 | +0.034 | 7.7 | 9.3% | |
| Norm | 0.500 | 0.500 | 0.504 | +0.043 | 23.3 | 6.1% | |
| Cosine | 0.531 | 0.507 | 0.549 | +0.012 | 54.8 | 13.5% | |
| LOF | 0.526 | 0.505 | 0.542 | +0.017 | 30.2 | 13.1% | |
| ASH+KNN | 0.516 | 0.505 | 0.530 | +0.026 | 30.8 | 11.3% |
| Alignn | CGCNN | SchNet | DimeNet | |||||
|---|---|---|---|---|---|---|---|---|
| Dataset | ID | OOD | ID | OOD | ID | OOD | ID | OOD |
| dielectric | 0.447 0.020 | 0.480 0.018 | 0.567 0.007 | 0.605 0.021 | 0.417 0.028 | 0.463 0.010 | 0.430 0.025 | 0.476 0.034 |
| log_gvrh | 0.074 0.002 | 0.098 0.001 | 0.097 0.002 | 0.128 0.007 | 0.094 0.003 | 0.118 0.004 | 0.098 0.006 | 0.124 0.008 |
| log_kvrh | 0.060 0.003 | 0.061 0.003 | 0.079 0.005 | 0.079 0.006 | 0.070 0.002 | 0.071 0.002 | 0.069 0.001 | 0.069 0.000 |
| perovskites | 0.034 0.001 | 0.033 0.001 | 0.069 0.005 | 0.068 0.005 | 0.046 0.005 | 0.046 0.005 | 0.045 0.007 | 0.045 0.007 |
| phonons | 34.9 8.2 | 38.3 7.6 | 78.4 3.6 | 85.8 1.6 | 58.3 4.3 | 65.2 4.3 | 58.1 5.7 | 66.2 8.4 |
| Dataset | Split | Arch | ID MAE | OOD MAE |
|---|---|---|---|---|
| dielectric | comp_chalcogenide | alignn | 0.4153 0.0313 | 0.3598 0.0395 |
| cgcnn | 0.5958 0.0585 | 0.5928 0.1980 | ||
| dimenet | 0.6899 0.3254 | 0.3388 0.0031 | ||
| schnet | 0.3327 0.0181 | 0.5772 0.0804 | ||
| comp_halide | alignn | 0.4861 0.0517 | 1.5563 0.0865 | |
| cgcnn | 0.5829 0.0062 | 1.2402 0.3578 |
| Architecture | Split type | |||||
|---|---|---|---|---|---|---|
| Detector | Alignn | Cgcnn | Schnet | Dimenet | Comp. | Struct. |
| KNN-k20 | 0.502 0.002 | 0.518 0.007 | 0.509 0.003 | 0.519 0.002 | 0.529 0.002 | 0.501 0.001 |
| KNN-k50 | 0.497 0.002 | 0.514 0.008 | 0.506 0.003 | 0.517 0.001 | 0.523 0.003 | 0.499 0.002 |
| Norm | 0.512 0.001 | 0.503 0.002 | 0.491 0.001 | 0.494 0.001 | 0.502 0.003 | 0.498 0.001 |
| Cosine | 0.530 0.001 | 0.535 0.003 | 0.529 0.001 | 0.530 0.002 | 0.556 0.001 | 0.515 0.001 |
| LOF | 0.536 0.002 | 0.530 0.003 | 0.515 0.002 | 0.522 0.001 | 0.545 0.002 | 0.513 0.001 |
| Family | All | dielectric | log_gvrh | log_kvrh | perovskites | phonons |
|---|---|---|---|---|---|---|
| Distance | 0.516 0.000 | 0.531 0.001 | 0.530 0.001 | 0.506 0.000 | 0.501 0.001 | 0.502 0.003 |
| Density | 0.515 0.001 | 0.530 0.002 | 0.534 0.001 | 0.505 0.001 | 0.500 0.001 | 0.499 0.003 |
| Isolation | 0.500 0.002 | 0.494 0.004 | 0.498 0.003 | 0.502 0.001 | 0.497 0.001 | 0.507 0.003 |
| Boundary / Hybrid | 0.502 0.001 | 0.501 0.003 | 0.507 0.002 | 0.500 0.001 | 0.497 0.001 | 0.505 0.004 |
| Supervised | 0.543 0.001 | 0.600 0.003 | 0.603 0.002 | 0.492 0.003 | 0.499 0.004 | 0.496 0.008 |
| Detector | All | dielectric | log_gvrh | log_kvrh | perovskites | phonons |
|---|---|---|---|---|---|---|
| MDS-BIC | 0.935 0.002 | 0.926 0.004 | 0.917 0.003 | 0.946 0.001 | 0.953 0.001 | 0.939 0.007 |
| KNN-k10 | 0.942 0.002 | 0.935 0.005 | 0.937 0.004 | 0.945 0.002 | 0.949 0.002 | 0.946 0.005 |
| kPCA | 0.938 0.002 | 0.926 0.004 | 0.929 0.005 | 0.945 0.002 | 0.950 0.001 | 0.944 0.006 |
| Norm | 0.947 0.002 | 0.953 0.003 | 0.939 0.002 | 0.948 0.001 | 0.951 0.001 | 0.946 0.006 |
| LOF | 0.928 0.001 | 0.910 0.005 | 0.894 0.004 | 0.945 0.002 | 0.951 0.003 | 0.953 0.006 |
| OCSVM | 0.946 0.002 | 0.943 0.004 | 0.945 0.005 | 0.948 0.000 | 0.950 0.002 | 0.944 0.004 |
| Dataset | Mean linear probe | % | % | Bottleneck | |
|---|---|---|---|---|---|
| dielectric | 56 | 0.599 0.001 | 28.6 0.0 (16/56, 16/56, 16/56) | 31.5 1.0 (18/56, 17/56, 18/56) | Method |
| log_gvrh | 60 | 0.600 0.002 | 26.7 0.0 (16/60, 16/60, 16/60) | 26.7 0.0 (16/60, 16/60, 16/60) | Method |
| log_kvrh | 60 | 0.494 0.004 | 0.0 0.0 (0/60, 0/60, 0/60) | 4.4 4.8 (1/60, 1/60, 6/60) | Encoder |
| perovskites | 36 | 0.500 0.005 | 0.0 0.0 (0/36, 0/36, 0/36) | 2.8 2.8 (0/36, 1/36, 2/36) | Encoder |
| phonons | 52 | 0.499 0.009 | 0.6 1.1 (0/45, 0/52, 1/52) | 1.9 1.9 (0/45, 1/52, 2/52) | Encoder |
| Detector | (all) | (all) | (degraded) | (degraded) |
|---|---|---|---|---|
| KNN-k20-norm | +0.215 0.020 | 0.22 0.01 | +0.295 0.009 | 0.20 0.01 |
| KNN-k50-norm | +0.191 0.019 | 0.18 0.01 | +0.243 0.013 | 0.15 0.01 |
| Norm | +0.069 0.018 | 0.04 0.02 | +0.042 0.016 | 0.01 0.00 |
| Cosine | +0.311 0.027 | 0.26 0.02 | +0.579 0.024 | 0.43 0.02 |
| LOF | +0.259 0.042 | 0.22 0.06 | +0.466 0.027 | 0.36 0.03 |
| ASH+KNN | +0.215 0.032 | 0.21 0.04 | +0.347 0.053 | 0.29 0.08 |
| Dataset | ALIGNN | CGCNN | SchNet | DimeNet++ | Spread | Label |
|---|---|---|---|---|---|---|
| dielectric | 0.612 0.003 | 0.599 0.002 | 0.596 0.002 | 0.591 0.004 | 0.022 | Method |
| log_gvrh | 0.612 0.002 | 0.608 0.001 | 0.595 0.003 | 0.585 0.003 | 0.027 | Method |
| log_kvrh | 0.488 0.004 | 0.502 0.010 | 0.496 0.006 | 0.491 0.007 | 0.014 | Encoder |
| perovskites | 0.495 0.014 | 0.504 0.006 | 0.504 0.002 | 0.498 0.005 | 0.009 | Encoder |
| phonons | 0.521 0.008 | 0.498 0.019 | 0.491 0.011 | 0.485 0.020 | 0.036 | Encoder |
| Detector | Mean AURC | Median AURC | Mean SPE | Median SPE | % SPE 0 | |
|---|---|---|---|---|---|---|
| MDS-BIC | 257 | 6.936 | 0.088 | +0.340 | +0.349 | 94.2% |
| KNN k=10 | 257 | 7.571 | 0.084 | +0.363 | +0.348 | 93.4% |
| LOF | 257 | 6.970 | 0.082 | +0.347 | +0.306 | 98.8% |
| kPCA | 257 | 6.700 | 0.085 | +0.372 | +0.374 | 99.2% |
| OCSVM | 257 | 7.212 | 0.082 | +0.317 | +0.289 | 99.6% |
| IForest | 257 | 6.591 | 0.086 | +0.354 | +0.333 | 99.2% |
| Dataset | Split | ID MAE | OOD MAE | log gap | Maha AUROC | KNN AUROC |
|---|---|---|---|---|---|---|
| dielectric | LOCO | 0.429 | 0.408 | -0.021 | 0.503 | 0.501 |
| SparseXcluster | 0.469 | 0.537 | +0.068 | 0.507 | 0.519 | |
| SparseYcluster | 0.463 | 0.685 | +0.222 | 0.502 | 0.509 | |
| log_gvrh | LOCO | 0.089 | 0.089 | -0.001 | 0.500 | 0.498 |
| SparseXcluster | 0.090 | 0.091 | +0.001 | 0.544 | 0.547 | |
| SparseYcluster | 0.090 | 0.081 | -0.009 | 0.528 | 0.523 |
| Dataset | Target | Dominant drivers | Comp. rel. | Struct. rel. |
| dielectric | refractive index | electronic polarizability, bandgap [ Shannon, 1993 , Petousis et al., 2017 ] | ✓ | partial |
| log_gvrh | (shear) | bond directionality, anisotropy [ De Jong et al., 2015 ] | partial | ✓ |
| log_kvrh | (bulk) | mean bond stiffness, density [ De Jong et al., 2015 ] | weak | weak |
| perovskites | formation energy | cohesive energy within ABX 3 [ Castelli et al., 2012 ] | weak | weak |
| phonons | highest optical phonon freq. | reduced mass, local force constants [ Petretto et al., 2018 ] | weak | weak |
OOD Detection
Out-of-distribution (OOD) detection identifies inputs unlike a model’s training data so systems can flag uncertainty, reject inputs, or adapt safely.
See all 239 papers