Complementarity-Driven Dual-Teacher Hierarchical Alignment for Zero-Shot EEG-to-Image Retrieval
Authors: Lukun Wu, Jie Li, Ziqi Ren, Ruiyang Xia, Yan Gao, Xinbo Gao
Organizations: Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China · Center for Biomedical-photonics and Molecular Imaging (BMI), School of Life Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China · School of Electronic Information (School of Artificial Intelligence), Northwest University, Xi’an 710127, China
Decoding visual information from EEG is central to brain--computer interfaces and brain-inspired intelligence, and contrastive alignment with pretrained vision foundation models (VFMs) has become the dominant paradigm. However, existing methods align EEG solely with CLIP-family VFMs, leaving hierarchical visual information in neural responses under-exploited. To address this gap, we formulate a mechanism-paradigm correspondence view linking hierarchical visual processing (from appearance to semantics) to the complementary biases of cross-paradigm VFMs, with image-only models (e.g., DINO) and image--text models (e.g., CLIP) capturing distinct facets of this hierarchy. Guided by this view, we propose the Dual-Teacher Hierarchical Alignment (DTHA) framework. Concretely, a shared EEG encoder is supervised by frozen multi-blur CLIP and shallow DINOv2 teachers via dual-branch contrastive alignment, so each branch fully supervises the visual cues it emphasizes. A fused-retrieval objective learns adaptive weights to combine their complementary visual representations for zero-shot EEG-to-image retrieval. For the shared EEG encoder, we introduce the Attentive Modulated Scalp-Pattern Bank Encoder (AMSPBE), combining temporo-spatial attention with a learnable scalp-pattern bank for accurate, interpretable visual feature extraction, while modeling regional temporal dynamics and filter-bank topographies. On THINGS-EEG zero-shot retrieval, DTHA achieves 92.4% Top-1 accuracy, exceeding prior state of the art by 12.4 percentage points. Spatial, spectral, and representational-similarity analyses, together with THINGS-MEG evaluation, validate DTHA's effectiveness, biological plausibility, and cross-modality generalization. This complementarity-driven perspective shows that supervision from cross-paradigm VFMs enables more complete extraction of hierarchical visual information from brain activity.
Figures & tables
Fig. 1: Mechanism–paradigm correspondence between hierarchical visual processing (appearance-related structure to semantic understanding) and emergent VFM representations from distinct training paradigms (image-only self-supervised vs. image–text contrastive learning), motivating complementarity-driven dual-teacher EEG alignment beyond CLIP-family VFMs alone.
Fig. 2: Overview of complementarity-driven dual-teacher hierarchical alignment (DTHA) for zero-shot EEG-to-image retrieval. Train: frozen multi-blur CLIP-RN50 and shallow DINOv2 ViT-L/14 teachers supply zvclip and zvdino through learnable hierarchical weighting modules fHclip and fHdino ; shared AMSPBE ( fB ) maps RSVP-evoked EEG to brain embedding zb , which is aligned to both teachers via branch-wise SCE losses LSCEclip and LSCEdino . Fusion weights wclip and wdino combine similarity matrices Sclip and Sdino into Sfused for fused-retrieval learning LSCEfused (similarity-level fusion, not feature concatenation). Test: frozen fB , fHclip , and fHdino encode query EEG and gallery images; learned wclip and wdino fuse Sclip and Sdino for gallery ranking.
Fig. 3: Architecture of AMSPBE ( fB ). EEG undergoes temporo-spatial attention modulation (TSAM; tri-region temporal attention and per-channel attention), temporal encoding ( k=7 , tpool=128 ), and a K=16 positivity-constrained scalp-pattern filter bank with an MLP projection, yielding brain embedding zb . Right: representative learned scalp-pattern topographies.
Type
Sub
Chan
Train ∗
Test ∗
SOA(ms)
EEG
10
63
1654 ∣ 10 ∣ 4
200 ∣ 1 ∣ 80
200 (100)
MEG
4
271
1854 ∣ 12 ∣ 1
200 ∣ 1 ∣ 12
1500 ± 200 (500)
TABLE I: Configuration details of the THINGS-EEG and THINGS-MEG datasets (Sub: subjects, Chan: channels).
Method
Subject 1
Subject 2
Subject 3
Subject 4
Avg
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
Intra-subject : train and test on one subject
NICE
9.6
27.8
18.5
47.8
14.2
41.6
9.0
26.6
12.8
36.0
UBP
15.0
38.0
46.0
80.5
27.3
59.0
18.5
43.5
26.7
55.2
ATS
18.0
43.0
56.0
89.0
36.0
67.0
19.5
50.0
32.4
62.3
HVF
14.0
31.8
63.8
91.8
41.0
78.3
17.0
41.0
33.9
60.7
TABLE III: Top-1 and Top-5 accuracy (%) for 200-way zero-shot retrieval on THINGS-MEG (four-subject mean; three-seed average).
Fig. 4: CLIP vs. DINO representational differences on THINGS-EEG training images. (A) Category-semantic RSA for CLIP-RN50 and DINOv2 ViT-L/14: CLIP shows block-diagonal clustering; DINO does not. (B) CLIP category-semantic RSA at each blur level ( σ=1 – 63 ): block-diagonal clustering weakens as σ increases. (C) Spearman ρ of CLIP RDMs with category-semantic and chroma reference RDMs across blur levels.
Fig. 5: Layer-wise Spearman ρ of teacher RDMs with category-semantic and chroma reference RDMs (CLIP, PE-Core, DINO; THINGS-EEG test , n=200 ). Gold dashed lines mark semantic–chroma ρ crossovers.
Fig. 6: Layer-wise Spearman ρ of DINOv2 ViT-L/14 RDMs with six low-level reference RDMs—chroma, luminance, edge density, and spectral statistics—(THINGS-EEG test , n=200 ). Shallow layers show generally higher ρ .
Fig. 7: Single-teacher retrieval sweep (branch Top-1, %; 10-subject mean, seed 2025). Each condition uses branch SCE only (Eq. ( 2 )), without fused-retrieval. Top: CLIP/PE-Core blur sweeps; bottom: DINOv2/v3 layer spans on ViT and ConvNeXt. Solid/hatched bars mark shallow/deep mid-depth splits.
I. Single-teacher training vs. DTHA fused retrieval
Setting
THINGS-EEG
THINGS-MEG
Top-1
Top-5
Δfused
Top-1
Top-5
Δfused
CLIP (single-teacher)
80.6
96.9
−11.8
52.1
81.3
−12.3
DINO (single-teacher)
88.7
99.2
−3.7
56.7
84.0
−7.7
DTHA fused (default)
92.4
99.5
–
64.4
86.8
–
II. Per-branch retrieval: DTHA joint training vs. single-teacher training
TABLE V: Overall dual-teacher comparison on THINGS-EEG (10-subject mean) and THINGS-MEG (4-subject mean); three-seed averages. Part I: fused Sfused vs. single-teacher CLIP/DINO (branch SCE only); Part II: Sclip / Sdino branch retrieval under full DTHA. Δfused : row Top-1 − DTHA fused; Δsingle : Part II − matching Part I single-teacher row.
(a) Training objective
Branch losses (α,β) and fused term γ
Top-1
Top-5
Δ
Default (1:1:0.25)
92.6
99.5
–
CLIP branch only (1:0:0.25)
90.1
99.2
−2.5
DINO branch only (0:1:0.25)
91.5
99.2
−1.1
Fused-retrieval only (0:0:1)
88.8
98.9
−3.8
Loss weight γ ( α=β=1 )
Top-1
Top-5
Δ
TABLE VI: Fusion and training ablations on THINGS-EEG (CLIP + DINOv2, 10-subject mean; seed 2025). Δ : Top-1 gap vs. default DTHA.
Brain encoder
THINGS-EEG
THINGS-MEG
Top-1
Top-5
Top-1
Top-5
AMSPBE (proposed)
92.6
99.5
64.6
87.6
SpatialConv [ 25 ] LVBP
91.5 ∗
99.5
63.0
86.5
EEGNet [ 38 ] JNE’18
72.1 ∗∗
94.0
36.2
64.1
TSConv [ 4 ] NICE
66.4 ∗∗
91.0
29.5
60.2
STAE [ 29 ] ATS
62.2 ∗∗
90.0
43.3
72.9
TABLE VII: Brain-encoder comparison under a fixed DTHA recipe (seed 2025). EEG Top-1 stars: one-sided paired Wilcoxon vs. AMSPBE.
Variant
THINGS-EEG
THINGS-MEG
Top-1
Top-5
Top-1
Top-5
Full AMSPBE (default)
92.6
99.5
64.6
87.6
Main component ablations
w/o spatial filter bank ( K=16 )
70.8
94.5
39.9
67.6
w/o TSAM & depthwise temporal conv
92.1
99.4
63.4
88.0
TSAM and depthwise temporal conv ablations
TABLE VIII: AMSPBE component ablation under the fixed DTHA dual-teacher recipe (fused Top-1/Top-5, %; seed 2025). EEG: THINGS-EEG ( n=10 ); MEG: THINGS-MEG ( n=4 ). TSAM abbreviation is defined in Fig. 3 .
Fig. 8: Spatio-spectral interpretability on THINGS-EEG and THINGS-MEG (10-subject / 4-subject mean; seed 2025). (A–D) Learned tri-region temporal attention and scalp-pattern topographies. (E) Fused Top-1 under five scalp partitions (F/C/P/T/O) vs. full montage (All); unselected channels zero-masked. (F) Fused Top-1 under δ (0.5–4), θ (4–8), α (8–13), β (13–30), γ (30–100; MEG clipped to Nyquist −1 ) vs. broadband (All); zero-phase 4th-order Butterworth.
Region (channels)
CLIP
DINO
Fused
DINO − CLIP
wdino
Occipital (8)
73.5
85.3
87.9
+11.8
0.502
Parietal (16)
40.6
40.2
47.3
−0.4
0.366
Temporal (10)
16.3
13.0
18.0
−3.4
0.323
Central (14)
5.9
4.4
6.0
−1.6
0.308
Frontal (15)
2.9
1.7
3.0
−1.2
0.326
Posterior V17 (17)
78.4
87.1
90.9
+8.7
0.492
TABLE IX: Partition-wise DTHA retrieval under regional partitioning (THINGS-EEG, 10-subject mean; seed 2025). Train/test on masked channels only; unselected channels are zero-masked. CLIP and DINO: branch Top-1 (%); Fused: fused Top-1 (%); DINO − CLIP: branch gap (pp). wdino : subject-averaged learned fusion weight on the DINO branch ( wclip+wdino=1 ).
Fig. 9: CLIP-branch, DINO-branch, and fused Top-1 retrieval on four representative THINGS-EEG test queries. Green/red borders mark correct/incorrect Top-1 hits. Rows 1–2 (aircraft carrier, SUV): fusion-gain cases—both branches miss and fusion succeeds; row 3 (banana): CLIP fails but DINO succeeds and fusion follows; row 4 (cupcake): DINO succeeds but fusion fails—an occasional conflict case under similarity-level fusion.
Fig. 10: Cross-modal EEG–image similarity on THINGS-EEG. sˉintra and sˉinter denote mean normalized similarities within and across superordinate blocks; Δ=sˉintra−sˉinter (larger Δ = stronger intra-category aggregation).
Fig. 11: Representative fused Top-5 EEG-to-image retrieval on THINGS-EEG test queries. Each row shows the query and top-5 gallery hits ranked by Sfused (green border: query and correct Top-1 hit). All shown queries recover the correct target at Top-1; ranks 2–5 illustrate how semantic and appearance-related cues combine in the ranked gallery. Buggy, cat, and meatloaf: remaining hits are largely superordinate-consistent neighbors; banana and basketball: remaining hits are largely appearance-related matches across labels; for ferry , remaining hits mix ship-consistent matches (e.g., sailboat, cruise ship) with the appearance-driven hit freezer .
Zero-shot EEG-to-image retrieval aims to decode perceived visual content from electroencephalography (EEG) by aligning neural responses with pretrained visual representations, providing a promising route toward scalable visual neural decoding and practical brain-computer interfaces. However, robust EEG-to-image retrieval remains challenging, because prior methods usually rely on either a single fixed visual target or a subject-invariant target construction scheme. Such designs overlook two important properties of visually evoked EEG signals: they preserve information across multiple representational scales, and the visual granularity best matched to EEG may vary across subjects. To address these issues, subject-aware multi-granularity alignment (SAMGA) framework is proposed for zero-shot EEG-to-image retrieval. SAMGA first constructs a subject-aware visual supervision target by adaptively aggregating multiple intermediate representations from a pretrained vision encoder, allowing the model to absorb subject-dependent granularity deviations during training while preserving subject-agnostic inference. Building on this adaptive target construction, a coarse-to-fine cross-modal alignment strategy is further designed with a shared encoder wherein the coarse stage stabilizes the shared semantic geometry and reduces subject-induced distribution shift, and the fine stage further improves instance-level retrieval discrimination. Extensive experiments on the THINGS-EEG benchmark demonstrate that the proposed method achieves 91.3% Top-1 and 98.8% Top-5 accuracy in the intra-subject setting, and 34.4% Top-1 and 64.8% Top-5 accuracy in the inter-subject setting, outperforming recent state-of-the-art methods.
Lin Jiang, Qingshan She, Jiale Xu +3
School of Automation, Hangzhou Dianzi University, and also with Zhejiang Provincial Key Laboratory of Brain Computer Collaborative Intelligence Technology and Applications, Hangzhou, Zhejiang 310018, China
Zero-shot brain-to-image retrieval requires robust alignment between noisy EEG responses and visual representations. Existing EEG-vision alignment methods often operate in sensor space and apply fixed visual supervision to all responses, ignoring both spatial mixing in scalp EEG and response-wise variability in alignment reliability. We propose an adaptive cortically constrained EEG-vision alignment method for zero-shot brain-to-image retrieval. The method reconstructs EEG responses into predefined ROI-level source-pattern representations and encodes them with a Neuro-ROI Attention Encoder. To handle response-wise variability, we introduce an evidence-based adaptive visual supervision strategy that weights detail-controlled visual targets using model-based alignment evidence. On THINGS-EEG, the proposed method achieves strong 200-way zero-shot retrieval performance, with ROI-level attribution providing post hoc interpretability of the learned source-pattern representations. These results show that cortically constrained representation learning and adaptive supervision can jointly support EEG-vision alignment for zero-shot brain-to-image retrieval.
Ye Wang, Haokun Ren, Wei Wu +4
School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, China · School of Medicine, Shanghai Jiaotong University, Shanghai, China · National Center for Applied Mathematics in Chongqing, Chongqing Normal University, Chongqing, China +1
Zero-shot visual decoding from electroencephalography (EEG) aims to infer visual semantics from non-invasive neural recordings, but remains challenging due to the low signal-to-noise ratio, non-stationarity, and limited spatial resolution of EEG. Existing EEG-vision alignment methods often rely on holistic EEG embeddings, which can obscure the complementary temporal, spectral, and spatial structure underlying visual perception. We introduce a unified multiview EEG representation learning framework for aligning brain responses with visual semantic embeddings. Our method builds an EEG encoder that jointly models three complementary views: input-conditioned state-space temporal dynamics, learnable wavelet-based spectral decomposition for sample-adaptive frequency modeling, and attention-modulated graph learning for structured electrode interactions. The resulting multiview EEG embeddings are fused and aligned with pretrained visual representations in a shared semantic space using contrastive learning with EEG-specific regularization, enabling 200-way zero-shot visual classification. Experiments on THINGS-EEG benchmark show that our method achieves state-of-the-art performance, with 54.8% Top-1 and 85.6% Top-5 accuracy in the within-subject setting and 15.3% Top-1 and 45.4% Top-5 accuracy in the cross-subject setting. We further present the first systematic cross-session EEG-image decoding evaluation, achieving 40.8% Top-1 and 78.0% Top-5 accuracy. These results suggest that explicitly modeling multiview neural structure improves both semantic alignment and generalization in EEG-based visual decoding.
Salini Yadav, Taveena Lotey, Pravendra Singh +1
Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India