Machine Learning classifiers used in Brain-Computer Interfaces make classifications based on the distribution of data they were trained on. When they need to make inferences on samples that fall outside of this distribution, they can only make blind guesses. Instead of allowing random guesses, these Out-of-Distribution (OOD) samples should be detected and rejected. We study OOD detection in Motor Imagery BCIs by training a model on some classes and observing whether unfamiliar classes can be detected based on increased uncertainty. We test seven different OOD detection techniques and one more method that has been claimed to boost the quality of OOD detection. Our findings show that OOD detection for Brain-Computer Interfaces is more challenging than in other machine learning domains due to the high uncertainty inherent in classifying EEG signals. For many subjects, uncertainty for in-distribution classes can still be higher than for out-of-distribution classes. As a result, many OOD detection methods prove to be ineffective, though MC Dropout performed best. Additionally, we show that high in-distribution classification performance predicts high OOD detection performance, suggesting that improved accuracy can also lead to improved robustness. Our research demonstrates a setup for studying how models deal with unfamiliar EEG data and evaluates methods that are robust to these unfamiliar inputs. OOD detection can improve the overall safety and reliability of BCIs.
Figures & tables
Figure 1: Concept of out-of-distribution detection. A model is trained on several known classes. It has to be able to recognize when unknown data does not belong to any of the known classes, while still classifying the known classes correctly.
Figure 2: A visual explanation of three different OOD detection approaches. The data points belong to user nine of the BNCI2014 dataset. In this illustration feet is considered OOD on a model trained on the right hand and left hand classes.
Figure 3: Performance of OOD detection methods across three datasets. Colours indicate the following UQ categories: Bayesian , Density , Distance , Baseline .
Figure 4: Performance of OOD detection for different classes plotted against the different UQ methods.“Left”, “Right” and“Both” refer to hand movements. Whilst the performance for the Left and Right hand classes stays relatively similar, we find a significant increase in AUROC for the Both hands and Feet classes.
Method
Median
IQR
pReAct
pcorrelation
ρcorrelation
On-Task AUROC
BNCI2014
Deep Ensemble
0.542
0.102
0.026
3.877⋅10−13
0.586
0.852
MC Dropout
0.597
0.124
0.145
8.857⋅10−12
0.557
0.837
DDU
0.519
0.174
0.945
0.558
0.052
0.837
Energy
0.530
0.136
0.537
4.400⋅10−12
0.563
0.837
DUQ
0.509
0.130
0.873
.037
0.185
0.772
Table 3: The OOD detection performance ( Median ) and spread ( IQR ) measured through AUROC per method resulting from the OOD approach comparison. Using a Wilcoxon signed-rank test ( pReAct ), we find that ReAct does not improve OOD optimisation. Using Spearman’s rank correlation test ( pcorrelation , ρcorrelation ) between classification performance and OOD detection performance without ReAct, we find a non-linear correlation for multiple results. Bonferroni corrected α=2.381⋅10−3 . On-Task AUROC refers to the average classification performance on ID data for a model trained on three ID classes. Several OOD detection methods do not affect ID classification and therefore have the same on-task performance.
Figure 5: OOD detection vs on-task classification performance on Bayesian approaches showing a non-linear correlation on the Stieger2021 dataset.
Figure 6: Difference in AUROC when using ReACt aggregated over all datasets. Smoothing was done using KDE. The dotted line indicates no difference
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : OOD detection vs on-task classification performance for models trained on two and three ID classes.
Figure B1 : The difference between OOD detection when a model is trained on three classes or two classes . Colours indicate the following UQ categories: Bayesian , Density , Distance , Baseline .
Method
BNCI2014
Schirrmeister2017
Stieger2021
p
Effect Size
p
Effect Size
p
Effect Size
Deep Ensemble
0.070
1866
0.058
4769
8.440⋅10−4
76575
MC Dropout
0.466
1669
0.335
4419
1.319⋅10−3
77056
DDU
0.941
1550
0.192
4547
0.158
94683
Energy
0.102
1834
7.250⋅10−4
5327
1.100⋅10−3
76858
DUQ
0.756
1593
0.268
4473
0.388
92592
Appendix
Table B1 : Comparison of p-values and effect sizes from a Mann–Whitney U test across the three datasets for different methods. Bonferroni corrected α=2.381 e-3
Contrast
Estimate
SE
z
p
padj
DDU - DUQ
0.025
0.011
2.368
0.0179
0.1073
DDU - Deep Ensemble
-0.089
0.011
-8.408
4.164⋅10−17
6.662⋅10−16
DDU - Energy
-0.066
0.011
-6.255
3.987⋅10−10
3.987⋅10−9
DDU - d-KNN
-0.056
0.011
-5.309
1.103⋅10−7
9.924⋅10−7
DDU - MC Dropout
-0.141
0.011
-13.296
2.443⋅10−40
4.887⋅10−39
DDU - Softmax
-0.066
0.011
-6.275
3.503⋅10−10
3.854⋅10−9
Appendix
Table C2 : Pairwise contrast estimates with standard errors, z-scores, and adjusted p-values.
Class
k
Gamma ( γ )
Centroid size
Penalty
BNCI2014
Left hand
1
0.999
64
1.55×10−3
Right hand
2
0.995
128
1.20×10−4
Feet
12
0.913
64
1.19×10−5
Tongue
106
0.995
128
1.20×10−4
BNCI2014 inversion
Appendix
Table D1 : Hyperparameters for d-KNN ( k ) and DUQ ( γ , centroid size, penalty) across all datasets. γ=1−one_minus_gamma . Penalty refers to the magnitude of the gradient penalty during training.
Hyperparameter
Search space
Scale
1−γ
[10−3,10−1]
Log-uniform
γ
1−(1−γ)
Derived
Centroid size
{32,64,128,256}
Categorical
penalty
[10−5,5×10−2]
Log-uniform
Appendix
Table D2 : Hyperparameter search space used during optimization of the DUQ model.
Machine learning models for electroencephalography (EEG) analysis show great promise across a wide range of applications, but their deployment in high-risk domains is hindered by their vulnerability to distribution shifts. Encountering out-of-distribution (OOD) data can lead to catastrophic, overconfident predictive failures. While OOD detection methods can mitigate these risks, they remain heavily under-explored for EEG. Moreover, evaluations in the broader literature typically evaluate OOD detection performance in isolation, ignoring their practical impact on downstream applications. To bridge this gap, we introduce a benchmark for EEG OOD detection, evaluate a broad range of methods, and furthermore evaluate their value in two clinical downstream prediction task. Our results disentangle OOD detection and model uncertainty estimation capabilities, which are frequently conflated in the literature, provide actionable insights about the current state of the art for EEG OOD detection and model uncertainty estimation, and demonstrate how complementary methods for both aspects can be combined to form a robust safety net for the deployment of EEG-based machine learning models in real-world applications.
Philipp Bomatter, Henry Gouk
School of Informatics, University of Edinburgh, United Kingdom.
Out-of-distribution (OOD) detection is essential for deploying machine learning models in open-world and safety-critical scenarios, where test inputs may deviate from the training distribution and overconfident predictions on unknown samples can lead to unreliable decisions. Outlier Exposure (OE) has emerged as a promising OOD detection paradigm by introducing auxiliary outliers during training to enlarge the margin between in-distribution (ID) and OOD samples. Existing OE-based methods typically enlarge this margin by employing uniform labels to maximize the entropy of OOD samples over ID categories. However, we theoretically show that uniform labels inevitably disregard the relations between OOD samples and ID categories, termed the over-softening effect, leading to a suboptimal margin bound. Our theoretical analysis further reveals that explicitly exploiting such relations can instead yield improved OOD detection performance. Motivated by this insight, we propose Adaptive Confidence OE (AOE), a simple yet effective method that leverages temperature scaling to recalibrate outlier labels. Specifically, AOE generates adaptive soft targets from temperature-scaled model predictions for OOD samples, where the learnable temperature smooths the prediction distribution without fully erasing class-wise relational information. By supervising OOD samples with these adaptive soft targets, AOE preserves the semantic proximity between OOD samples and ID categories while encouraging the softened targets to approach a high-entropy distribution, thereby suppressing overconfident OOD predictions and enlarging the separation margin. Extensive experiments across diverse benchmarks demonstrate the effectiveness of AOE.
Fengqiang Wan, Qing-Yuan Jiang, Yang Yang
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China.
Out-of-distribution (OOD) detection is essential for deploying deep learning models reliably, yet no single method performs consistently across architectures and datasets -- a scorer that leads on one benchmark often falters on another. We attribute this inconsistency to a shared structural limitation: logit-based methods see only the classifier's confidence signal, while feature-based methods attempt to measure membership in the training distribution but do so in the full feature space where confidence and membership are entangled, inheriting architecture-sensitive failure modes. We observe that penultimate features naturally decompose into two orthogonal subspaces: a classifier-aligned component encoding confidence, and a residual the classifier discards. We discover that this residual carries a class-specific directional signature for in-distribution data -- a membership signal invisible to logit-based methods and entangled with noise in feature-based methods. We propose CORE (COnfidence + REsidual), which disentangles the two signals by scoring each subspace independently and combines them via normalized summation. Because the two signals are orthogonal by construction, their failure modes are approximately independent, producing robust detection where either view alone is unreliable. CORE achieves competitive or state-of-the-art performance across five architectures and five benchmark configurations, ranking first in three of five settings and achieving the highest grand average AUROC with negligible computational overhead.