Complementarity-Driven Dual-Teacher Hierarchical Alignment for Zero-Shot EEG-to-Image Retrieval
Authors: Lukun Wu, Jie Li, Ziqi Ren, Ruiyang Xia, Yan Gao, Xinbo Gao
Organizations: Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China · Center for Biomedical-photonics and Molecular Imaging (BMI), School of Life Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China · School of Electronic Information (School of Artificial Intelligence), Northwest University, Xi’an 710127, China
Decoding visual information from EEG is central to brain--computer interfaces and brain-inspired intelligence, and contrastive alignment with pretrained vision foundation models (VFMs) has become the dominant paradigm. However, existing methods align EEG solely with CLIP-family VFMs, leaving hierarchical visual information in neural responses under-exploited. To address this gap, we formulate a mechanism-paradigm correspondence view linking hierarchical visual processing (from appearance to semantics) to the complementary biases of cross-paradigm VFMs, with image-only models (e.g., DINO) and image--text models (e.g., CLIP) capturing distinct facets of this hierarchy. Guided by this view, we propose the Dual-Teacher Hierarchical Alignment (DTHA) framework. Concretely, a shared EEG encoder is supervised by frozen multi-blur CLIP and shallow DINOv2 teachers via dual-branch contrastive alignment, so each branch fully supervises the visual cues it emphasizes. A fused-retrieval objective learns adaptive weights to combine their complementary visual representations for zero-shot EEG-to-image retrieval. For the shared EEG encoder, we introduce the Attentive Modulated Scalp-Pattern Bank Encoder (AMSPBE), combining temporo-spatial attention with a learnable scalp-pattern bank for accurate, interpretable visual feature extraction, while modeling regional temporal dynamics and filter-bank topographies. On THINGS-EEG zero-shot retrieval, DTHA achieves 92.4% Top-1 accuracy, exceeding prior state of the art by 12.4 percentage points. Spatial, spectral, and representational-similarity analyses, together with THINGS-MEG evaluation, validate DTHA's effectiveness, biological plausibility, and cross-modality generalization. This complementarity-driven perspective shows that supervision from cross-paradigm VFMs enables more complete extraction of hierarchical visual information from brain activity.
Figures & tables
Fig. 1: Mechanism–paradigm correspondence between hierarchical visual processing (appearance-related structure to semantic understanding) and emergent VFM representations from distinct training paradigms (image-only self-supervised vs. image–text contrastive learning), motivating complementarity-driven dual-teacher EEG alignment beyond CLIP-family VFMs alone.
Fig. 2: Overview of complementarity-driven dual-teacher hierarchical alignment (DTHA) for zero-shot EEG-to-image retrieval. Train: frozen multi-blur CLIP-RN50 and shallow DINOv2 ViT-L/14 teachers supply zvclip and zvdino through learnable hierarchical weighting modules fHclip and fHdino ; shared AMSPBE ( fB ) maps RSVP-evoked EEG to brain embedding zb , which is aligned to both teachers via branch-wise SCE losses LSCEclip and LSCEdino . Fusion weights wclip and wdino combine similarity matrices Sclip and Sdino into Sfused for fused-retrieval learning LSCEfused (similarity-level fusion, not feature concatenation). Test: frozen fB , fHclip , and fHdino encode query EEG and gallery images; learned wclip and wdino fuse Sclip and Sdino for gallery ranking.
Fig. 3: Architecture of AMSPBE ( fB ). EEG undergoes temporo-spatial attention modulation (TSAM; tri-region temporal attention and per-channel attention), temporal encoding ( k=7 , tpool=128 ), and a K=16 positivity-constrained scalp-pattern filter bank with an MLP projection, yielding brain embedding zb . Right: representative learned scalp-pattern topographies.
Type
Sub
Chan
Train ∗
Test ∗
SOA(ms)
EEG
10
63
1654 ∣ 10 ∣ 4
200 ∣ 1 ∣ 80
200 (100)
MEG
4
271
1854 ∣ 12 ∣ 1
200 ∣ 1 ∣ 12
1500 ± 200 (500)
TABLE I: Configuration details of the THINGS-EEG and THINGS-MEG datasets (Sub: subjects, Chan: channels).
Method
Subject 1
Subject 2
Subject 3
Subject 4
Avg
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
Intra-subject : train and test on one subject
NICE
9.6
27.8
18.5
47.8
14.2
41.6
9.0
26.6
12.8
36.0
UBP
15.0
38.0
46.0
80.5
27.3
59.0
18.5
43.5
26.7
55.2
ATS
18.0
43.0
56.0
89.0
36.0
67.0
19.5
50.0
32.4
62.3
HVF
14.0
31.8
63.8
91.8
41.0
78.3
17.0
41.0
33.9
60.7
TABLE III: Top-1 and Top-5 accuracy (%) for 200-way zero-shot retrieval on THINGS-MEG (four-subject mean; three-seed average).
Fig. 4: CLIP vs. DINO representational differences on THINGS-EEG training images. (A) Category-semantic RSA for CLIP-RN50 and DINOv2 ViT-L/14: CLIP shows block-diagonal clustering; DINO does not. (B) CLIP category-semantic RSA at each blur level ( σ=1 – 63 ): block-diagonal clustering weakens as σ increases. (C) Spearman ρ of CLIP RDMs with category-semantic and chroma reference RDMs across blur levels.
Fig. 5: Layer-wise Spearman ρ of teacher RDMs with category-semantic and chroma reference RDMs (CLIP, PE-Core, DINO; THINGS-EEG test , n=200 ). Gold dashed lines mark semantic–chroma ρ crossovers.
Fig. 6: Layer-wise Spearman ρ of DINOv2 ViT-L/14 RDMs with six low-level reference RDMs—chroma, luminance, edge density, and spectral statistics—(THINGS-EEG test , n=200 ). Shallow layers show generally higher ρ .
Fig. 7: Single-teacher retrieval sweep (branch Top-1, %; 10-subject mean, seed 2025). Each condition uses branch SCE only (Eq. ( 2 )), without fused-retrieval. Top: CLIP/PE-Core blur sweeps; bottom: DINOv2/v3 layer spans on ViT and ConvNeXt. Solid/hatched bars mark shallow/deep mid-depth splits.
I. Single-teacher training vs. DTHA fused retrieval
Setting
THINGS-EEG
THINGS-MEG
Top-1
Top-5
Δfused
Top-1
Top-5
Δfused
CLIP (single-teacher)
80.6
96.9
−11.8
52.1
81.3
−12.3
DINO (single-teacher)
88.7
99.2
−3.7
56.7
84.0
−7.7
DTHA fused (default)
92.4
99.5
–
64.4
86.8
–
II. Per-branch retrieval: DTHA joint training vs. single-teacher training
TABLE V: Overall dual-teacher comparison on THINGS-EEG (10-subject mean) and THINGS-MEG (4-subject mean); three-seed averages. Part I: fused Sfused vs. single-teacher CLIP/DINO (branch SCE only); Part II: Sclip / Sdino branch retrieval under full DTHA. Δfused : row Top-1 − DTHA fused; Δsingle : Part II − matching Part I single-teacher row.
(a) Training objective
Branch losses (α,β) and fused term γ
Top-1
Top-5
Δ
Default (1:1:0.25)
92.6
99.5
–
CLIP branch only (1:0:0.25)
90.1
99.2
−2.5
DINO branch only (0:1:0.25)
91.5
99.2
−1.1
Fused-retrieval only (0:0:1)
88.8
98.9
−3.8
Loss weight γ ( α=β=1 )
Top-1
Top-5
Δ
TABLE VI: Fusion and training ablations on THINGS-EEG (CLIP + DINOv2, 10-subject mean; seed 2025). Δ : Top-1 gap vs. default DTHA.
Brain encoder
THINGS-EEG
THINGS-MEG
Top-1
Top-5
Top-1
Top-5
AMSPBE (proposed)
92.6
99.5
64.6
87.6
SpatialConv [ 25 ] LVBP
91.5 ∗
99.5
63.0
86.5
EEGNet [ 38 ] JNE’18
72.1 ∗∗
94.0
36.2
64.1
TSConv [ 4 ] NICE
66.4 ∗∗
91.0
29.5
60.2
STAE [ 29 ] ATS
62.2 ∗∗
90.0
43.3
72.9
TABLE VII: Brain-encoder comparison under a fixed DTHA recipe (seed 2025). EEG Top-1 stars: one-sided paired Wilcoxon vs. AMSPBE.
Variant
THINGS-EEG
THINGS-MEG
Top-1
Top-5
Top-1
Top-5
Full AMSPBE (default)
92.6
99.5
64.6
87.6
Main component ablations
w/o spatial filter bank ( K=16 )
70.8
94.5
39.9
67.6
w/o TSAM & depthwise temporal conv
92.1
99.4
63.4
88.0
TSAM and depthwise temporal conv ablations
TABLE VIII: AMSPBE component ablation under the fixed DTHA dual-teacher recipe (fused Top-1/Top-5, %; seed 2025). EEG: THINGS-EEG ( n=10 ); MEG: THINGS-MEG ( n=4 ). TSAM abbreviation is defined in Fig. 3 .
Fig. 8: Spatio-spectral interpretability on THINGS-EEG and THINGS-MEG (10-subject / 4-subject mean; seed 2025). (A–D) Learned tri-region temporal attention and scalp-pattern topographies. (E) Fused Top-1 under five scalp partitions (F/C/P/T/O) vs. full montage (All); unselected channels zero-masked. (F) Fused Top-1 under δ (0.5–4), θ (4–8), α (8–13), β (13–30), γ (30–100; MEG clipped to Nyquist −1 ) vs. broadband (All); zero-phase 4th-order Butterworth.
Region (channels)
CLIP
DINO
Fused
DINO − CLIP
wdino
Occipital (8)
73.5
85.3
87.9
+11.8
0.502
Parietal (16)
40.6
40.2
47.3
−0.4
0.366
Temporal (10)
16.3
13.0
18.0
−3.4
0.323
Central (14)
5.9
4.4
6.0
−1.6
0.308
Frontal (15)
2.9
1.7
3.0
−1.2
0.326
Posterior V17 (17)
78.4
87.1
90.9
+8.7
0.492
TABLE IX: Partition-wise DTHA retrieval under regional partitioning (THINGS-EEG, 10-subject mean; seed 2025). Train/test on masked channels only; unselected channels are zero-masked. CLIP and DINO: branch Top-1 (%); Fused: fused Top-1 (%); DINO − CLIP: branch gap (pp). wdino : subject-averaged learned fusion weight on the DINO branch ( wclip+wdino=1 ).
Fig. 9: CLIP-branch, DINO-branch, and fused Top-1 retrieval on four representative THINGS-EEG test queries. Green/red borders mark correct/incorrect Top-1 hits. Rows 1–2 (aircraft carrier, SUV): fusion-gain cases—both branches miss and fusion succeeds; row 3 (banana): CLIP fails but DINO succeeds and fusion follows; row 4 (cupcake): DINO succeeds but fusion fails—an occasional conflict case under similarity-level fusion.
Fig. 10: Cross-modal EEG–image similarity on THINGS-EEG. sˉintra and sˉinter denote mean normalized similarities within and across superordinate blocks; Δ=sˉintra−sˉinter (larger Δ = stronger intra-category aggregation).
Fig. 11: Representative fused Top-5 EEG-to-image retrieval on THINGS-EEG test queries. Each row shows the query and top-5 gallery hits ranked by Sfused (green border: query and correct Top-1 hit). All shown queries recover the correct target at Top-1; ranks 2–5 illustrate how semantic and appearance-related cues combine in the ranked gallery. Buggy, cat, and meatloaf: remaining hits are largely superordinate-consistent neighbors; banana and basketball: remaining hits are largely appearance-related matches across labels; for ferry , remaining hits mix ship-consistent matches (e.g., sailboat, cruise ship) with the appearance-driven hit freezer .
School of Automation, Hangzhou Dianzi University, and also with Zhejiang Provincial Key Laboratory of Brain Computer Collaborative Intelligence Technology and Applications, Hangzhou, Zhejiang 310018, China
School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, China · School of Medicine, Shanghai Jiaotong University, Shanghai, China · National Center for Applied Mathematics in Chongqing, Chongqing Normal University, Chongqing, China +1