GALA: Gallery-Level Alignment for Real-Life Vision Decoding from EEG
Organizations: IMT Atlantique, France · Center for Mind/Brain Sciences (CIMeC), University of Trento, Italy · Australian Institute for Machine Learning, University of Adelaide, Australia
Abstract
Neural decoding of visual experience from electroencephalography (EEG) has achieved strong performance in intra-subject settings, but current methods fail to generalize across individuals: on the standard 200-way benchmark THINGS-EEG-2, top-1 accuracy drops from above 80% within subjects to about 32% for an unseen subject. Adapting a pre-trained encoder to new subjects and sessions typically requires labeled data, a bottleneck whose collection time severely limits practical device deployment. In a practical Brain-Computer Interface (BCI), however, the possible outcomes are known in advance and form a gallery of candidates, and the unlabeled EEG of the user accumulates as soon as the device is used. We propose a new setting, incremental transductive calibration, in which the decoder is calibrated regularly on all the unlabeled EEG received so far and on the known gallery, without any labels. This setting is closer to the conditions of a deployed BCI than the zero-shot and batch evaluations in current use. We evaluate it with a protocol in which queries arrive one at a time, with unknown and possibly unequal frequencies, and each query is decoded before it is used for calibration. In this setting, we introduce GALA : GAllery-Level Alignment, a label-free calibration method which registers the unlabeled EEG embeddings onto the gallery through shrinkage whitening, hubness correction and a Sinkhorn--Procrustes alignment with query frequencies estimated from the buffer, updated after every query. On THINGS-EEG-2, with a gallery of 200 images, it raises the top-1 accuracy on a new subject from 28% to 67% over 53 minutes of use. Across 60 evaluation settings that vary the gallery size, the query frequencies and the number of repetitions averaged per query, GALA is the most accurate label-free method in every setting, 6.6 points above the best competing method on average at the end of the stream.
Figures & tables
| Uniform | Skewed | Partial | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | 1 min | 5 min | 20 min | 53 min | 1 min | 5 min | 20 min | 1 min | 5 min | 20 min |
| Zero-shot | 30.1 | 27.8 | 28.4 | 27.9 | 29.5 | 27.4 | 28.7 | 28.8 | 28.2 | 28.0 |
| CSLS | 35.8 | 34.9 | 35.4 | 34.2 | 34.6 | 34.2 | 35.7 | 35.1 | 35.5 | 34.5 |
| AdaBN [ 50 ] | 30.1 | 28.9 | 30.3 | 29.1 | 29.5 | 29.0 | 30.6 | 28.8 | 31.9 | 29.6 |
| Tent [ 77 ] | 30.1 | 28.6 | 30.0 | 29.0 | 29.5 | 28.7 | 30.4 | 28.8 | 31.5 | 29.4 |
| SATTC [ 34 ] | 28.2 | 32.0 | 35.5 | 35.9 | 27.5 | 32.2 | 35.7 | 28.3 | 33.2 | 34.3 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | S10 | Top-1 | Top-5 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Inductive | NICE [ 72 ] | 7.6 | 5.9 | 6.0 | 6.3 | 4.4 | 5.6 | 5.6 | 6.3 | 5.7 | 8.4 | 6.2 | 21.4 |
| ATM-S [ 48 ] | 10.5 | 7.1 | 11.9 | 14.7 | 7.0 | 11.1 | 16.1 | 15.0 | 4.9 | 20.5 | 11.9 | 33.8 | |
| Neural-MCRL [ 51 ] | 13.0 | 12.0 | 14.5 | 12.5 | 11.5 | 13.5 | 14.0 | 18.5 | 13.5 | 17.0 | 14.0 | 34.3 | |
| UBP [ 79 ] | 11.5 | 15.5 | 9.8 | 13.0 | 8.8 | 11.7 | 10.2 | 12.2 | 15.5 | 16.0 | 12.4 | 33.4 | |
| NeuroBridge [ 83 ] | 23.2 | 21.2 | 13.2 | 17.0 | 14.5 | 25.0 | 15.3 | 20.1 | 13.7 | 27.2 | 19.0 | 45.9 | |
| Shallow Align. [ 21 ] | 23.5 | 30.6 | 10.0 | 19.5 | 18.1 | 22.7 | 18.6 | 17.3 | 23.0 | 34.4 | 21.8 | 49.4 |
| Alljoined-1.6M, | Alljoined-1.6M, | THINGS-MEG, | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Uniform | Skewed | Partial | Uniform | Skewed | Partial | Uniform | Skewed | Partial |
| Zero-shot | 6.9 | 7.1 | 7.1 | 8.7 | 8.7 | 8.4 | 4.6 | 4.7 | 4.6 |
| CSLS | 7.6 | 7.6 | 7.4 | 10.6 | 10.5 | 10.5 | 5.4 | 5.7 | 5.8 |
| SATTC [ 34 ] | 7.3 | 7.5 | 6.7 | 9.8 | 9.5 | 8.8 | 5.2 | 5.1 | 5.7 |
| Hungarian assignment | 8.6 | 8.6 | 8.1 | 12.2 | 11.7 | 11.0 | 6.2 | 7.1 | 6.0 |
| SCORE with ATM | 8.5 | 9.0 | 8.9 | 12.5 | 12.2 | 12.3 | 6.0 | 6.2 | 6.7 |
| Uniform | Skewed | Partial | |||||
|---|---|---|---|---|---|---|---|
| Variant | 1 | 20 | 53 | 1 | 20 | 1 | 20 |
| GALA | 37.2 | 59.1 | 67.3 | 37.4 | 59.0 | 37.4 | 56.1 |
| w/o whitening | 37.4 | 55.7 | 60.8 | 37.7 | 55.8 | 37.5 | 53.0 |
| w/o CSLS | 36.2 | 55.8 | 58.6 | 35.4 | 55.3 | 36.4 | 54.5 |
| uniform masses | 37.1 | 58.4 | 69.4 | 37.3 | 58.5 | 37.1 | 55.0 |
| w/o column masses | 37.9 | 58.5 | 65.8 | 37.9 | 58.3 | 37.1 | 55.6 |
| Uniform | Skewed | Partial | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Encoder | Method | 1 | 5 | 10 | 20 | 53 | 1 | 5 | 10 | 20 | 1 | 5 | 10 | 20 |
| ATM | Zero-shot | 30.1 | 27.8 | 28.0 | 28.4 | 27.9 | 29.5 | 27.4 | 28.8 | 28.7 | 28.8 | 28.2 | 29.1 | 28.0 |
| SATTC | 28.2 | 32.0 | 34.2 | 35.5 | 35.9 | 27.5 | 32.2 | 35.1 | 35.7 | 28.3 | 33.2 | 34.3 | 34.3 | |
| SCORE | 29.3 | 42.3 | 48.2 | 52.0 | 53.4 | 29.3 | 42.2 | 48.1 | 52.1 | 29.9 | 43.9 | 48.8 | 50.1 | |
| GALA | 37.2 | 46.2 | 51.6 | 59.1 | 67.3 | 37.4 | 45.3 | 52.0 | 59.0 | 37.4 | 47.4 | 51.7 | 56.1 | |
| TSConv | Zero-shot | 21.7 | 24.6 | 23.5 | 25.1 | 22.1 | 30.0 | 22.8 | 25.2 | 24.7 | 23.3 | 25.6 | 24.8 | 23.4 |
EEG-Based Visual Decoding
EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.
See all 57 papers