cs.LGAug 18, 2026
SaveOOD Detection for EEG-based Machine Learning in High-Risk Environments
Organizations: School of Informatics, University of Edinburgh, United Kingdom.
Abstract
Machine learning models for electroencephalography (EEG) analysis show great promise across a wide range of applications, but their deployment in high-risk domains is hindered by their vulnerability to distribution shifts. Encountering out-of-distribution (OOD) data can lead to catastrophic, overconfident predictive failures. While OOD detection methods can mitigate these risks, they remain heavily under-explored for EEG. Moreover, evaluations in the broader literature typically evaluate OOD detection performance in isolation, ignoring their practical impact on downstream applications. To bridge this gap, we introduce a benchmark for EEG OOD detection, evaluate a broad range of methods, and furthermore evaluate their value in two clinical downstream prediction task. Our results disentangle OOD detection and model uncertainty estimation capabilities, which are frequently conflated in the literature, provide actionable insights about the current state of the art for EEG OOD detection and model uncertainty estimation, and demonstrate how complementary methods for both aspects can be combined to form a robust safety net for the deployment of EEG-based machine learning models in real-world applications.
Figures & tables
Figure 1: OOD Detection Evaluation Concept. Starting from an EEG dataset with annotations for a prediction task (here TUAB with EEG normality labels), the data is first partitioned into training and test splits. The training data is then used to train two models: a generative model trained using only the EEG without labels (here a UNet trained via flow-matching), and a discriminative model trained through supervised learning (here a TCN model). The test data then serves as in-distribution test data, for which corresponding out-of-distribution data is generated through controlled perturbations. Finally, we evaluate a range of OOD detection methods based on either the generative or discriminative model. In addition to OOD detection performance, we also evaluate its impact on downstream performance, quantifying how the prediction accuracy of the discriminative model varies across samples with different OOD scores of the different methods.
Figure 2: Illustration of EEG OOD Perturbations. An unperturbed (ID) EEG segment is shown (top left panel) along with corresponding OOD samples obtained through the different perturbations. For visualisation purposes, only a subset of channels are shown.
| Perturbation | Severity | MSP | Energy | ODIN | ASH | LL | Typ. | DoSE | SITN |
|---|---|---|---|---|---|---|---|---|---|
| Resampling | 125 Hz | 0.517 | 0.500 | 0.514 | 0.533 | 0.369 | 0.559 | 0.556 | 0.947 |
| 150 Hz | 0.457 | 0.479 | 0.453 | 0.534 | 0.275 | 0.646 | 0.647 | 0.981 | |
| 200 Hz | 0.383 | 0.451 | 0.377 | 0.522 | 0.143 | 0.816 | 0.826 | 0.989 | |
| Channel Shuffle | 10% | 0.549 | 0.494 | 0.549 | 0.513 | 0.607 | 0.521 | 0.543 | 0.754 |
| 50% | 0.553 | 0.431 | 0.555 | 0.494 | 0.793 | 0.682 | 0.711 | 0.924 | |
| 100% | 0.526 | 0.391 | 0.529 | 0.460 | 0.823 | 0.726 | 0.748 | 0.942 |
Table 1: OOD detection performance on TUAB. Performance is reported in terms of AUROC and bold values highlight the best result for each perturbation and severity level. Discriminative-model-based methods (MSP, Energy, ODIN, ASH) perform largely at chance level, whereas generative-model-based methods achieve considerably better OOD detection with increasing performance for higher severity levels. SITN consistently achieves the best performance.
Figure 3: Model performance for pathology prediction on the TUAB dataset. A ID samples and corresponding OOD samples obtained with the resampling perturbation (severity 200 Hz). Performance is shown in terms of accuracy across percentile bins, where the x ticks indicate the upper edge of the corresponding bin (i.e. the first bin contains samples from the 0-0.1 percentile, the second one 0.1-1, etc.). The histograms indicate the number of ID and OOD samples in each bin. The arrows in the top left/right of each plot indicate the OOD direction for the corresponding metric. See Section 13 for full downstream performance results across all perturbations and OOD detection methods. B The same visualisations restricted to only in-distribution data. MSP is a good indicator of model performance across in-distribution samples except for a failure mode at the highest MSP scores, whereas SITN provides little information about model performance between ID samples.
Figure 4: Decision Diagram for Combined OOD and Uncertainty Detection. Samples are jointly probed by an OOD detector and a model uncertainty estimator, each of which can trigger the system to abstain from prediction or defer to a human expert to prevent misclassifications.
| ID Only | ID and OOD | |||
|---|---|---|---|---|
| Rejection Method | Coverage | Accuracy | Coverage | Accuracy |
| Baseline (No Rejection) | 100.0% | 80.5% | 100.0% | 67.5% |
| SITN (OOD Detection) | 98.5% | 80.8% | 49.5% | 80.7% |
| MSP (Model Uncertainty) | 88.1% | 83.8% | 89.2% | 69.3% |
| Combined (SITN + MSP) | 86.9% | 84.1% | 43.7% | 84.1% |
Table 2: Selective Classification Performance for OOD Detection and Model Uncertainty Estimation. OOD Detection (using SITN), Model Uncertainty (using MSP), and their combination (SITN+MSP) are compared on ID-only data and ID+OOD data. OOD data was generated with the resampling perturbation at 200 Hz. MSP provides an accuracy improvement on in-distribution data, but only SITN provides effective protection against OOD data. Combining both methods enables the highest accuracy across both in- and out-of-distribution data.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Perturbation | Severity | MSP | Energy | ODIN | ASH | LL | Typ. | DoSE | SITN |
|---|---|---|---|---|---|---|---|---|---|
| Resampling | 125 Hz | 0.531 | 0.517 | 0.540 | 0.540 | 0.286 | 0.587 | 0.760 | 0.956 |
| 150 Hz | 0.497 | 0.520 | 0.514 | 0.590 | 0.166 | 0.732 | 0.886 | 0.984 | |
| 200 Hz | 0.470 | 0.531 | 0.495 | 0.658 | 0.056 | 0.906 | 0.965 | 0.992 | |
| Channel Shuffle | 10% | 0.496 | 0.502 | 0.505 | 0.518 | 0.622 | 0.552 | 0.711 | 0.759 |
| 50% | 0.443 | 0.490 | 0.470 | 0.570 | 0.841 | 0.766 | 0.968 | 0.956 | |
| 100% | 0.408 | 0.483 | 0.436 | 0.583 | 0.879 | 0.819 | 0.982 | 0.968 |
Table 3: OOD detection performance in terms of AUROC across different perturbations and severity levels on CAUEEG. Bold values indicate the best performing metric for a given perturbation and severity. Discriminative-model-based methods (MSP, Energy, ODIN, ASH) perform largely at chance level, whereas generative-model-based methods achieve considerably better OOD detection with increasing performance for higher severity levels.
| Perturbation | Severity | TUAB | CAUEEG |
|---|---|---|---|
| Resampling | 125 Hz | 0.548 | 0.544 |
| 150 Hz | 0.594 | 0.586 | |
| 200 Hz | 0.654 | 0.640 | |
| Channel Shuffle | 10% | 0.583 | 0.604 |
| 50% | 0.741 | 0.763 | |
| 100% | 0.775 | 0.799 |
Table 4: Mahalanobis distance OOD detection performance in terms of AUROC across different perturbations and severity levels. Compared to other discriminative approaches, Mahalanobis distances show better performance for the Resampling, Channel Shuffle, and Re-referencing perturbations, but also a strong failure for the High-pass Filter perturbation, where they consistently underperform even random guessing.
Figure 5: Sample Size Dependence of OOD Detection Performance. Starting from a baseline of just 25 participants and 5 segments per participant (= 125 samples in total), the training data was gradually increased by either increasing the number of participants (solid lines) or segments per participant (dashed lines). The sampling rate perturbation at 200 Hz was used to generate OOD samples. OOD detection performance generally increased with the amount of training data (see Section 9 for a discussion of the likelihood). Even in the smallest data regimes, generative methods dominated discriminative approaches. Note that MSP is occluded by ODIN, which showed almost identical performance across sample sizes.
Figure 6: Likelihood Limitations. A Distribution of log-likelihoods across ID samples (100 Hz) and OOD samples obtained with the resampling perturbation at increasing severities of 125, 150, and 200 Hz. As previously observed in the context of computer vision datasets, OOD samples are consistently assigned higher likelihoods than the in-distribution data. B EEG segments that were assigned the lowest and highest likelihoods, respectively. As in computer vision, where “simpler” images like constant-colour images are assigned high likelihoods, EEG segments with completely flat channels also result in high likelihoods. High-amplitude artefacts result in the opposite extreme and are assigned very small likelihoods.
| Condition | MSP | Energy | ODIN | ASH | LL | Typ. | DoSE | SITN |
|---|---|---|---|---|---|---|---|---|
| TUAB CAUEEG | 0.594 | 0.478 | 0.598 | 0.517 | 0.669 | 0.432 | 0.462 | 0.544 |
| CAUEEG TUAB | 0.489 | 0.558 | 0.477 | 0.514 | 0.398 | 0.643 | 0.703 | 0.740 |
Table 7: Cross-dataset Discrimination Across OOD Detection Methods. Models were trained on one dataset and then evaluated using the test split of the training dataset as ID and the test split of the other dataset as OOD data (e.g. TUAB CAUEEG denotes trained on TUAB and using CAUEEG as OOD). The degree to which OOD scores discriminate between samples from the two datasets is reported in terms of AUROC. Note that ID and OOD should be interpreted with a grain of salt in this setup as it is unclear to what degree cross-dataset samples are OOD.
Figure 7: Cross-dataset Subgroup Analysis of OOD Scores Across Diagnosis Labels on CAUEEG. The results show normalised participant-level OOD scores for samples from the CAUEEG dataset assigned by methods trained on TUAB. Scores are largely overlapping without a consistent trend across diagnosis groups.
Figure 8: Cross-dataset Subgroup Analysis of OOD Scores Across Age on CAUEEG. The results show normalised participant-level OOD scores for samples from the CAUEEG dataset assigned by methods trained on TUAB. No clear trend is observable.
Figure 9: Cross-dataset Subgroup Analysis of OOD Scores Across EEG Normality Labels on TUAB. The results show normalised participant-level OOD scores for samples from the TUAB dataset assigned by methods trained on CAUEEG. Scores are largely overlapping, although SITN assigned somewhat higher OOD scores to abnormal EEG recordings.
Figure 10: Cross-dataset Subgroup Analysis of OOD Scores Across Age on TUAB. The results show normalised participant-level OOD scores for samples from the TUAB dataset assigned by methods trained on CAUEEG. No clear trends are discernable.
Figure 11: Cross-participant Experiments on the BNCI 2014-001 Motor Imagery Dataset. Numbers next to the points indicate participant id. Methods were trained on the training data from participant 1 and evaluated on the test data from all participants. OOD scores were aggregated on the participant-level through averaging and aligned such that higher scores always indicate more OOD. Only SITN clearly separates test data from the training participant (ID) from the other participants (OOD). MSP assigns the lowest OOD scores to two participants where model performance was even higher than for the training participant, but also the 3rd lowest score to the participant where model performance was lowest and which was flagged as highly OOD by most other methods.
Figure 12: Normality Prediction on TUAB — Resampling Perturbation (200 Hz)
Figure 13: Normality Prediction on TUAB — Channel Shuffle Perturbation (100%)
Figure 14: Normality Prediction on TUAB — Re-referencing Perturbation (bipolar) Note: The irregular bin sizes at the upper percentiles of the SITN plot are due to metric saturation, where a large number of tied maximum scores are grouped into the final bin.
Figure 15: Normality Prediction on TUAB — High-pass Filter Perturbation (8 Hz)
Figure 16: Normality Prediction on TUAB — Low-pass Filter Perturbation (15 Hz)
Figure 17: Dementia Diagnosis on CAUEEG — Resampling Perturbation (200 Hz)
Figure 18: Dementia Diagnosis on CAUEEG — Channel Shuffle Perturbation (100%)
Figure 19: Dementia Diagnosis on CAUEEG — Re-referencing Perturbation (bipolar) Note: The irregular bin sizes at the upper percentiles of the SITN plot are due to metric saturation, where a large number of tied maximum scores are grouped into the final bin.
Figure 20: Dementia Diagnosis on CAUEEG — High-pass Filter Perturbation (8 Hz)
Figure 21: Dementia Diagnosis on CAUEEG — Low-pass Filter Perturbation (15 Hz)
More in this topic
OOD Detection
Out-of-distribution (OOD) detection identifies inputs unlike a model’s training data so systems can flag uncertainty, reject inputs, or adapt safely.
See all 222 papers