Machine Learning classifiers used in Brain-Computer Interfaces make classifications based on the distribution of data they were trained on. When they need to make inferences on samples that fall outside of this distribution, they can only make blind guesses. Instead of allowing random guesses, these Out-of-Distribution (OOD) samples should be detected and rejected. We study OOD detection in Motor Imagery BCIs by training a model on some classes and observing whether unfamiliar classes can be detected based on increased uncertainty. We test seven different OOD detection techniques and one more method that has been claimed to boost the quality of OOD detection. Our findings show that OOD detection for Brain-Computer Interfaces is more challenging than in other machine learning domains due to the high uncertainty inherent in classifying EEG signals. For many subjects, uncertainty for in-distribution classes can still be higher than for out-of-distribution classes. As a result, many OOD detection methods prove to be ineffective, though MC Dropout performed best. Additionally, we show that high in-distribution classification performance predicts high OOD detection performance, suggesting that improved accuracy can also lead to improved robustness. Our research demonstrates a setup for studying how models deal with unfamiliar EEG data and evaluates methods that are robust to these unfamiliar inputs. OOD detection can improve the overall safety and reliability of BCIs.
Figures & tables
Figure 1: Concept of out-of-distribution detection. A model is trained on several known classes. It has to be able to recognize when unknown data does not belong to any of the known classes, while still classifying the known classes correctly.
Figure 2: A visual explanation of three different OOD detection approaches. The data points belong to user nine of the BNCI2014 dataset. In this illustration feet is considered OOD on a model trained on the right hand and left hand classes.
Figure 3: Performance of OOD detection methods across three datasets. Colours indicate the following UQ categories: Bayesian , Density , Distance , Baseline .
Figure 4: Performance of OOD detection for different classes plotted against the different UQ methods.“Left”, “Right” and“Both” refer to hand movements. Whilst the performance for the Left and Right hand classes stays relatively similar, we find a significant increase in AUROC for the Both hands and Feet classes.
Method
Median
IQR
pReAct
pcorrelation
ρcorrelation
On-Task AUROC
BNCI2014
Deep Ensemble
0.542
0.102
0.026
3.877⋅10−13
0.586
0.852
MC Dropout
0.597
0.124
0.145
8.857⋅10−12
0.557
0.837
DDU
0.519
0.174
0.945
0.558
0.052
0.837
Energy
0.530
0.136
0.537
4.400⋅10−12
0.563
0.837
DUQ
0.509
0.130
0.873
.037
0.185
0.772
Table 3: The OOD detection performance ( Median ) and spread ( IQR ) measured through AUROC per method resulting from the OOD approach comparison. Using a Wilcoxon signed-rank test ( pReAct ), we find that ReAct does not improve OOD optimisation. Using Spearman’s rank correlation test ( pcorrelation , ρcorrelation ) between classification performance and OOD detection performance without ReAct, we find a non-linear correlation for multiple results. Bonferroni corrected α=2.381⋅10−3 . On-Task AUROC refers to the average classification performance on ID data for a model trained on three ID classes. Several OOD detection methods do not affect ID classification and therefore have the same on-task performance.
Figure 5: OOD detection vs on-task classification performance on Bayesian approaches showing a non-linear correlation on the Stieger2021 dataset.
Figure 6: Difference in AUROC when using ReACt aggregated over all datasets. Smoothing was done using KDE. The dotted line indicates no difference
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : OOD detection vs on-task classification performance for models trained on two and three ID classes.
Figure B1 : The difference between OOD detection when a model is trained on three classes or two classes . Colours indicate the following UQ categories: Bayesian , Density , Distance , Baseline .
Method
BNCI2014
Schirrmeister2017
Stieger2021
p
Effect Size
p
Effect Size
p
Effect Size
Deep Ensemble
0.070
1866
0.058
4769
8.440⋅10−4
76575
MC Dropout
0.466
1669
0.335
4419
1.319⋅10−3
77056
DDU
0.941
1550
0.192
4547
0.158
94683
Energy
0.102
1834
7.250⋅10−4
5327
1.100⋅10−3
76858
DUQ
0.756
1593
0.268
4473
0.388
92592
Appendix
Table B1 : Comparison of p-values and effect sizes from a Mann–Whitney U test across the three datasets for different methods. Bonferroni corrected α=2.381 e-3
Contrast
Estimate
SE
z
p
padj
DDU - DUQ
0.025
0.011
2.368
0.0179
0.1073
DDU - Deep Ensemble
-0.089
0.011
-8.408
4.164⋅10−17
6.662⋅10−16
DDU - Energy
-0.066
0.011
-6.255
3.987⋅10−10
3.987⋅10−9
DDU - d-KNN
-0.056
0.011
-5.309
1.103⋅10−7
9.924⋅10−7
DDU - MC Dropout
-0.141
0.011
-13.296
2.443⋅10−40
4.887⋅10−39
DDU - Softmax
-0.066
0.011
-6.275
3.503⋅10−10
3.854⋅10−9
Appendix
Table C2 : Pairwise contrast estimates with standard errors, z-scores, and adjusted p-values.
Class
k
Gamma ( γ )
Centroid size
Penalty
BNCI2014
Left hand
1
0.999
64
1.55×10−3
Right hand
2
0.995
128
1.20×10−4
Feet
12
0.913
64
1.19×10−5
Tongue
106
0.995
128
1.20×10−4
BNCI2014 inversion
Appendix
Table D1 : Hyperparameters for d-KNN ( k ) and DUQ ( γ , centroid size, penalty) across all datasets. γ=1−one_minus_gamma . Penalty refers to the magnitude of the gradient penalty during training.
Hyperparameter
Search space
Scale
1−γ
[10−3,10−1]
Log-uniform
γ
1−(1−γ)
Derived
Centroid size
{32,64,128,256}
Categorical
penalty
[10−5,5×10−2]
Log-uniform
Appendix
Table D2 : Hyperparameter search space used during optimization of the DUQ model.