GALA: Gallery-Level Alignment for Real-Life Vision Decoding from EEG
Authors: Philippe Forest, Giulia Lioi, Mai Pham Quyen, Alix Lamouroux, Javen Shi, Adrien Merlini
Organizations: IMT Atlantique, France · Center for Mind/Brain Sciences (CIMeC), University of Trento, Italy · Australian Institute for Machine Learning, University of Adelaide, Australia
Neural decoding of visual experience from electroencephalography (EEG) has achieved strong performance in intra-subject settings, but current methods fail to generalize across individuals: on the standard 200-way benchmark THINGS-EEG-2, top-1 accuracy drops from above 80% within subjects to about 32% for an unseen subject. Adapting a pre-trained encoder to new subjects and sessions typically requires labeled data, a bottleneck whose collection time severely limits practical device deployment. In a practical Brain-Computer Interface (BCI), however, the possible outcomes are known in advance and form a gallery of candidates, and the unlabeled EEG of the user accumulates as soon as the device is used. We propose a new setting, incremental transductive calibration, in which the decoder is calibrated regularly on all the unlabeled EEG received so far and on the known gallery, without any labels. This setting is closer to the conditions of a deployed BCI than the zero-shot and batch evaluations in current use. We evaluate it with a protocol in which queries arrive one at a time, with unknown and possibly unequal frequencies, and each query is decoded before it is used for calibration. In this setting, we introduce GALA : GAllery-Level Alignment, a label-free calibration method which registers the unlabeled EEG embeddings onto the gallery through shrinkage whitening, hubness correction and a Sinkhorn--Procrustes alignment with query frequencies estimated from the buffer, updated after every query. On THINGS-EEG-2, with a gallery of 200 images, it raises the top-1 accuracy on a new subject from 28% to 67% over 53 minutes of use. Across 60 evaluation settings that vary the gallery size, the query frequencies and the number of repetitions averaged per query, GALA is the most accurate label-free method in every setting, 6.6 points above the best competing method on average at the end of the stream.
Figures & tables
Figure 1: Evaluation settings. Top: in the standard zero-shot evaluation, each query is decoded alone against the gallery. In the standard transductive batch evaluation, the 200 test queries are calibrated together and each gallery image is queried exactly once. Bottom: in incremental transductive calibration, the gallery is known, but queries arrive one at a time with unequal frequencies, some gallery images are never queried, and the calibration at each step uses all the queries received so far.
Figure 2: GALA. The unlabeled EEG embeddings of an unseen subject are whitened, compared with the gallery through cross-domain similarity local scaling, and registered onto it by a Sinkhorn normalization with estimated column masses bj and a soft Procrustes alignment. Each new query (blue) triggers one pass of this loop.
Figure 3: Incremental transductive calibration on THINGS-EEG-2 (10 subjects, q=20 ). Top-1 accuracy of the incoming queries, each decoded before it is used for calibration, in windows of one fifteenth of the stream, against stimulus time, for the uniform, skewed and partial streams (mean ± one standard deviation over three training seeds). The figure shows four methods, all with the ATM encoder; Table 1 reports every method.
Uniform
Skewed
Partial
Method
1 min
5 min
20 min
53 min
1 min
5 min
20 min
1 min
5 min
20 min
Zero-shot
30.1
27.8
28.4
27.9
29.5
27.4
28.7
28.8
28.2
28.0
CSLS
35.8
34.9
35.4
34.2
34.6
34.2
35.7
35.1
35.5
34.5
AdaBN [ 50 ]
30.1
28.9
30.3
29.1
29.5
29.0
30.6
28.8
31.9
29.6
Tent [ 77 ]
30.1
28.6
30.0
29.0
29.5
28.7
30.4
28.8
31.5
29.4
SATTC [ 34 ]
28.2
32.0
35.5
35.9
27.5
32.2
35.7
28.3
33.2
34.3
Table 1: Incremental transductive calibration: top-1 accuracy (%) of the queries received after 1, 5 and 20 minutes of stimulus time, and at the end of the uniform stream. Every query is decoded before it is used for calibration ( q=20 , mean over 10 subjects, three training seeds and ten stream orders; one seed for SCORE with its own encoder).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Calibration gain across encoders and image targets in the batch evaluation. Fraction of the zero-shot errors fixed by GALA against (a) the zero-shot top-1 accuracy and (b) the RSA between EEG and image embeddings, for the six encoders of Table 5 (circles) and for ATM trained against five other image targets (diamonds).
Figure 6: Uniform stream for each menu size K (rows) and number of averaged repetitions q (columns). Top-1 accuracy of the incoming queries, in windows of one fifteenth of the stream, against stimulus time, at 200 ms per repetition.
Figure 7: Skewed stream for each menu size K (rows) and number of averaged repetitions q (columns), as in Fig. 6 .
Figure 8: Partial stream for each menu size K (rows) and number of averaged repetitions q (columns), as in Fig. 6 .
Method
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
Top-1
Top-5
Inductive
NICE [ 72 ]
7.6
5.9
6.0
6.3
4.4
5.6
5.6
6.3
5.7
8.4
6.2
21.4
ATM-S [ 48 ]
10.5
7.1
11.9
14.7
7.0
11.1
16.1
15.0
4.9
20.5
11.9
33.8
Neural-MCRL [ 51 ]
13.0
12.0
14.5
12.5
11.5
13.5
14.0
18.5
13.5
17.0
14.0
34.3
UBP [ 79 ]
11.5
15.5
9.8
13.0
8.8
11.7
10.2
12.2
15.5
16.0
12.4
33.4
NeuroBridge [ 83 ]
23.2
21.2
13.2
17.0
14.5
25.0
15.3
20.1
13.7
27.2
19.0
45.9
Shallow Align. [ 21 ]
23.5
30.6
10.0
19.5
18.1
22.7
18.6
17.3
23.0
34.4
21.8
49.4
Appendix
Table 2: Standard batch evaluation: top-1 accuracy (%) of 200-way cross-subject retrieval on THINGS-EEG-2, per held-out subject, with one 80-repetition query per gallery image.
Alljoined-1.6M, q=20
Alljoined-1.6M, q=40
THINGS-MEG, q=6
Method
Uniform
Skewed
Partial
Uniform
Skewed
Partial
Uniform
Skewed
Partial
Zero-shot
6.9
7.1
7.1
8.7
8.7
8.4
4.6
4.7
4.6
CSLS
7.6
7.6
7.4
10.6
10.5
10.5
5.4
5.7
5.8
SATTC [ 34 ]
7.3
7.5
6.7
9.8
9.5
8.8
5.2
5.1
5.7
Hungarian assignment
8.6
8.6
8.1
12.2
11.7
11.0
6.2
7.1
6.0
SCORE with ATM
8.5
9.0
8.9
12.5
12.2
12.3
6.0
6.2
6.7
Appendix
Table 3: Incremental transductive calibration on Alljoined-1.6M and THINGS-MEG: top-1 accuracy (%) on the last 30% of the queries of each stream (ATM, one training seed).
Uniform
Skewed
Partial
Variant
1
20
53
1
20
1
20
GALA
37.2
59.1
67.3
37.4
59.0
37.4
56.1
w/o whitening
37.4
55.7
60.8
37.7
55.8
37.5
53.0
w/o CSLS
36.2
55.8
58.6
35.4
55.3
36.4
54.5
uniform masses
37.1
58.4
69.4
37.3
58.5
37.1
55.0
w/o column masses
37.9
58.5
65.8
37.9
58.3
37.1
55.6
Appendix
Table 4: Ablation of GALA components in the incremental setting: top-1 accuracy (%) after 1 and 20 minutes of each stream, and at the end of the uniform stream.
Uniform
Skewed
Partial
Encoder
Method
1
5
10
20
53
1
5
10
20
1
5
10
20
ATM
Zero-shot
30.1
27.8
28.0
28.4
27.9
29.5
27.4
28.8
28.7
28.8
28.2
29.1
28.0
SATTC
28.2
32.0
34.2
35.5
35.9
27.5
32.2
35.1
35.7
28.3
33.2
34.3
34.3
SCORE
29.3
42.3
48.2
52.0
53.4
29.3
42.2
48.1
52.1
29.9
43.9
48.8
50.1
GALA
37.2
46.2
51.6
59.1
67.3
37.4
45.3
52.0
59.0
37.4
47.4
51.7
56.1
TSConv
Zero-shot
21.7
24.6
23.5
25.1
22.1
30.0
22.8
25.2
24.7
23.3
25.6
24.8
23.4
Appendix
Table 5: Top-1 accuracy (%) across encoders after 1, 5, 10 and 20 minutes of the uniform, skewed and partial streams, and at the end of the uniform stream (53 minutes).
Accurate visual decoding can reveal how the brain represents visual information and recover perceived content from neural signals such as electroencephalography (EEG), with potential for neural communication. However, current EEG-to-image retrieval methods perform far below their within-subject counterparts for new users without labeled calibration, limiting real-world deployment. To understand this gap, we analyze EEG features across subjects and find that different subjects preserve similar relationships among concepts but express them along different coordinate directions. We therefore propose Subject Coordinate Recovery (SCORE), a target label-free framework combining recovery-aware source training with coordinate alignment at deployment. During training, SCORE aligns source subject EEG with a common image space and simulates unseen-subject recovery through source-only episodes. At deployment, with both encoders frozen, SCORE selects reliable EEG-image landmarks through hubness-corrected matching and estimates an orthogonal transformation to recover target EEG coordinates without source data or target labels. In 200-way retrieval on two public benchmarks, SCORE outperforms the unadapted baseline for every target subject and achieves the best overall accuracy. It reaches 53.23%/83.55% and 12.01%/32.16% Top-1/Top-5 on THINGS-EEG2 and Alljoined-1.6M, respectively, surpassing the strongest baselines by 17.45/15.70 and 3.08/4.62 percentage points. Without target labels or encoder updates, SCORE brings brain-based visual decoding closer to robust, practical, low-latency deployment across users.
Zhenyao Cui, Siyuan Kan, Siyang Li +2
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China
Electroencephalography (EEG) signals drift over time, which can cause static brain-computer interface (BCI) models to degrade in practice. We present a benchmark for online adaptation and compare two widely used pipeline families, Common Spatial Patterns (CSP) and Riemannian covariance-based methods, under time-ordered prequential (test-then-train) evaluation. We examine (i) which pipelines benefit most from label-revealed updates, (ii) whether controlled forgetting of older data improves robustness, and (iii) how a minimal-calibration cold start compares with starting from a pretrained model. Across four datasets (three motor-imagery datasets and one movement-decoding dataset), label-revealed online updates improve 13 of 14 model/dataset pairs on the two largest streams, with relative accuracy gains of up to about 18% over a frozen model. A Shapley-based data-valuation analysis over temporal blocks assigns the largest mean value to the most recent block in each of the three analyzed datasets, while older blocks retain positive value.
Bogdan Kozyrskiy, Artem Grachev, Abraham I. Camelo Guerrero
Independent Researcher, France · Independent Researcher, Germany · School of Information Technology, York University, Toronto, Canada
Cross-subject EEG-to-image retrieval for visual decoding is challenged by subject shift and hubness in the embedding space, which distort similarity geometry and destabilize top-k rankings, making small-k shortlists unreliable. We introduce SATTC (Structure-Aware Test-Time Calibration), a label-free calibration head that operates directly on the similarity matrix of frozen EEG and image encoders. SATTC combines a geometric expert, subject-adaptive whitening of EEG embeddings with an adaptive variant of Cross-domain Similarity Local Scaling (CSLS), and a structural expert built from mutual nearest neighbors, bidirectional top-k ranks, and class popularity, fused via a simple Product-of-Experts rule. On THINGS-EEG2 under a strict leave-one-subject-out protocol, standardized inference with cosine similarities, L2-normalized embeddings, and candidate whitening already yields a strong cross-subject baseline over the original ATM retrieval setup. Building on this baseline, SATTC further improves Top-1 and Top-5 accuracy, reduces hubness and per-class imbalance, and produces more reliable small-k shortlists. These gains transfer across multiple EEG encoders, supporting SATTC as an encoder-agnostic, label-free test-time calibration layer for cross-subject neural decoding.