Accurate visual decoding can reveal how the brain represents visual information and recover perceived content from neural signals such as electroencephalography (EEG), with potential for neural communication. However, current EEG-to-image retrieval methods perform far below their within-subject counterparts for new users without labeled calibration, limiting real-world deployment. To understand this gap, we analyze EEG features across subjects and find that different subjects preserve similar relationships among concepts but express them along different coordinate directions. We therefore propose Subject Coordinate Recovery (SCORE), a target label-free framework combining recovery-aware source training with coordinate alignment at deployment. During training, SCORE aligns source subject EEG with a common image space and simulates unseen-subject recovery through source-only episodes. At deployment, with both encoders frozen, SCORE selects reliable EEG-image landmarks through hubness-corrected matching and estimates an orthogonal transformation to recover target EEG coordinates without source data or target labels. In 200-way retrieval on two public benchmarks, SCORE outperforms the unadapted baseline for every target subject and achieves the best overall accuracy. It reaches 53.23%/83.55% and 12.01%/32.16% Top-1/Top-5 on THINGS-EEG2 and Alljoined-1.6M, respectively, surpassing the strongest baselines by 17.45/15.70 and 3.08/4.62 percentage points. Without target labels or encoder updates, SCORE brings brain-based visual decoding closer to robust, practical, low-latency deployment across users.
Figures & tables
Figure 1: Cross-subject EEG-to-image retrieval and the idea behind SCORE . (a) Source subjects provide labeled EEG–image pairs, whereas an unseen target subject provides unlabeled EEG and a candidate gallery. (b) Each subject expresses the same concept relationships along its own coordinate directions, which SCORE recovers in the image space.
Figure 2: Source and target EEG RSMs on encoder-unseen concepts (THINGS-EEG2), for one LOSO fold. (a)-(c) Two of the nine source subjects and the held-out target subject, under a shared fixed z-score scale. (d) All held-out concept pairs for the displayed source–target pair, with the target subject on the horizontal axis and source B on the vertical axis ( r=0.713 ).
Alignment
Top-1 ↑
Top-5 ↑
Direct
16.89
41.95
Ridge map
20.01
47.15
Orthogonal map
28.22
59.16
Table 1: Target-to-source EEG retrieval before and after coordinate mapping on THINGS-EEG2.
Figure 3: Orthogonal coordinate recovery for one target–source pair, projected onto a two-dimensional plane chosen from the fitting concepts.
Figure 4: Overview of SCORE . Left: recovery-aware source training aligns source subjects and simulates target recovery. Right: at deployment, coordinate recovery recovers the coordinates of unlabeled target EEG features before image retrieval.
Source training
Deployment method
THINGS-EEG2
Alljoined-1.6M
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Original
None
26.22±1.08
57.98±0.88
6.08±0.33
20.55±0.79
EA
27.45±0.82
59.37±1.07
6.31±0.19
20.47±0.67
SATTC (SAW)
30.98±0.35
60.85±1.00
8.93±0.34
27.54±1.19
SATTC
26.12±0.45
52.73±1.82
6.78±0.15
20.70±0.98
CSLS
35.78±1.62
67.85±1.21
8.04±0.35
25.39±0.54
Table 2: Cross-subject retrieval on two EEG-to-image benchmarks, with the SAMGA encoder throughout. EA is Euclidean Alignment, which normalizes each subject before source training and is then deployed without further adaptation. SATTC (SAW) denotes SATTC’s subject-adaptive whitening operator, applied on its own. Values are means and standard deviations across all subjects and three seeds.
Figure 5: Retrieval accuracy for every held-out subject and the mean (THINGS-EEG2), in (a) Top-1 and (b) Top-5.
Figure 6: Retrieval examples from one held-out subject. Each method shows its top three images, the green border marks the viewed image, and the rightmost column its rank in the 200-image gallery before and after recovery.
EEG encoder
Standard
Recovery-Aware
Standard + Recovery
Complete SCORE
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
NICE ( Song et al. 2024 )
27.60
60.38
31.08
63.50
50.27
81.05
54.85
83.75
ShallowFBCSPNet ( Schirrmeister et al. 2017 )
10.63
32.67
11.37
33.83
28.97
61.45
30.73
62.92
EEGNet ( Lawhern et al. 2018 )
15.95
42.80
17.22
45.68
44.28
76.72
46.95
78.65
Table 3: Generalization across EEG encoders on THINGS-EEG2. Values are mean Top-1 and Top-5 accuracy across three seeds.
Phase
Configuration
Top-1 ↑
Top-5 ↑
Train
SAMGA objective
26.22±1.08
57.98±0.88
+ multi-positive
28.63±1.75
62.02±1.73
+ simulated recovery
29.33±0.57
62.42±1.60
Test
CSLS ranking
39.08±0.81
71.23±0.70
+ mean and scale
43.80±1.57
75.68±0.58
+ recovery
50.98±1.53
80.38±0.78
Table 4: Cumulative ablation of recovery-aware training and coordinate recovery. Each row adds one component to the row above it.
Gallery size
Raw
SATTC (SAW)
CSLS
SCORE
200
29.33
35.28
39.08
53.23
700
15.72
17.98
21.75
29.47
1,200
12.25
12.80
17.03
22.27
1,854
9.47
9.92
13.33
17.03
Table 5: Top-1 retrieval as training-concept distractors are added to the 200 matching images. Gallery size counts the 200 matching images plus the added distractors. Every column uses the same recovery-aware representation.
Update
Ranking
Top-1 ↑
Top-5 ↑
None
Cosine
39.53±0.97
75.59±0.81
None
CSLS
50.41±0.60
83.16±0.77
Frozen map
Cosine
56.35±1.14
87.38±0.78
Frozen map
CSLS
64.16±1.59
91.51±0.91
Table 6: Transfer of a frozen map between disjoint 100-concept galleries.
Zero-shot EEG-to-image retrieval aims to decode perceived visual content from electroencephalography (EEG) by aligning neural responses with pretrained visual representations, providing a promising route toward scalable visual neural decoding and practical brain-computer interfaces. However, robust EEG-to-image retrieval remains challenging, because prior methods usually rely on either a single fixed visual target or a subject-invariant target construction scheme. Such designs overlook two important properties of visually evoked EEG signals: they preserve information across multiple representational scales, and the visual granularity best matched to EEG may vary across subjects. To address these issues, subject-aware multi-granularity alignment (SAMGA) framework is proposed for zero-shot EEG-to-image retrieval. SAMGA first constructs a subject-aware visual supervision target by adaptively aggregating multiple intermediate representations from a pretrained vision encoder, allowing the model to absorb subject-dependent granularity deviations during training while preserving subject-agnostic inference. Building on this adaptive target construction, a coarse-to-fine cross-modal alignment strategy is further designed with a shared encoder wherein the coarse stage stabilizes the shared semantic geometry and reduces subject-induced distribution shift, and the fine stage further improves instance-level retrieval discrimination. Extensive experiments on the THINGS-EEG benchmark demonstrate that the proposed method achieves 91.3% Top-1 and 98.8% Top-5 accuracy in the intra-subject setting, and 34.4% Top-1 and 64.8% Top-5 accuracy in the inter-subject setting, outperforming recent state-of-the-art methods.
Lin Jiang, Qingshan She, Jiale Xu +3
School of Automation, Hangzhou Dianzi University, and also with Zhejiang Provincial Key Laboratory of Brain Computer Collaborative Intelligence Technology and Applications, Hangzhou, Zhejiang 310018, China
Cross-subject EEG-to-image retrieval requires a neural represen- tation trained on source subjects to remain aligned with a visual embedding space for an unseen subject. Whereas existing methods primarily focus on the EEG side, we address this problem from the perspective of the visual target. Our approach preserves the spatial information of the Perception Encoder, converts its patch grid into a compact set of learned visual views, and aggregates them for each image with a block-structured, content-dependent router. The target is learned jointly with the EEG encoder through contrastive learning with MMD regularization across source subjects. For deployment, we propose a training-free representation refinement that aligns frozen embeddings without updating either encoder. Under leave- one-subject-out evaluation on THINGS-EEG2, the structured target achieves 35.3%/65.6% Top-1/Top-5 accuracy, the best among com- pared methods. Refinement raises this to 48.1%/77.1%, an 18.5% Top-1 gain over the strongest compared method, improving all ten held-out subjects.
Salini Yadav, Taveena Lotey, Mickaël Coustaty +2
Indian Institute of Technology Roorkee, India · University of La Rochelle, France · Indian Institute of Technology (ISM) Dhanbad, India
Cross-subject EEG-to-image retrieval for visual decoding is challenged by subject shift and hubness in the embedding space, which distort similarity geometry and destabilize top-k rankings, making small-k shortlists unreliable. We introduce SATTC (Structure-Aware Test-Time Calibration), a label-free calibration head that operates directly on the similarity matrix of frozen EEG and image encoders. SATTC combines a geometric expert, subject-adaptive whitening of EEG embeddings with an adaptive variant of Cross-domain Similarity Local Scaling (CSLS), and a structural expert built from mutual nearest neighbors, bidirectional top-k ranks, and class popularity, fused via a simple Product-of-Experts rule. On THINGS-EEG2 under a strict leave-one-subject-out protocol, standardized inference with cosine similarities, L2-normalized embeddings, and candidate whitening already yields a strong cross-subject baseline over the original ATM retrieval setup. Building on this baseline, SATTC further improves Top-1 and Top-5 accuracy, reduces hubness and per-class imbalance, and produces more reliable small-k shortlists. These gains transfer across multiple EEG encoders, supporting SATTC as an encoder-agnostic, label-free test-time calibration layer for cross-subject neural decoding.