Organizations: School of Automation, Guangdong University of Technology, Guangzhou 510006, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
Visual neural decoding aims to infer perceived visual stimuli from brain activity, offering important insights into human visual cognition while enabling transformative applications in brain computer interfaces and artificial intelligence. Recent advances in pretrained vision-language models have provided a semantically rich representation space for this task. However, reliable alignment between EEG/MEG signals and visual embeddings remains challenging because neural responses are noisy, highly variable across trials, and subject dependent. To address this issue, we propose a unified neural-to-image retrieval framework that improves retrieval performance by aligning neural and image representations in a shared semantic space. The framework integrates dual-stream adaptation, frequency-domain calibration, and phase-guided temperature modulation to promote more robust cross-modal semantic alignment. Specifically, dual-stream adaptation models a low-dimensional shared subspace with correlation regularization to suppress redundant noise and enhance cross-modal consistency. Frequency-domain calibration conditionally modulates neural representations using amplitude and phase statistics. Phase-guided temperature modulation further adjusts the supervision strength in contrastive learning according to the stability of neural responses. Experiments on THINGS-EEG and THINGS-MEG under both intra-subject and inter-subject settings demonstrate that the proposed framework consistently outperforms strong baselines. These results suggest that robust neural-to-image retrieval under heterogeneous neural conditions benefits from a stable shared semantic space, frequency-informed representation calibration, and adaptive supervision. Code will be released at github.com/wanfkuabc/Geometry-Regularized-Brain-Embeddings.
Figures & tables
Fig. 1: Overview of the proposed neural-to-image retrieval framework. Left: A frozen CLIP visual encoder first defines the target image semantic space, and the brain encoder maps neural signals into this shared space. To improve the stability of cross-modal alignment, the framework combines dual-stream adaptation, batch-wise standardization, and low-dimensional shared-subspace regularization. In parallel, a frequency-domain branch extracts amplitude and phase statistics from neural signals to conditionally calibrate neural embeddings. Phase-circular variance is further used to derive a phase-stability-based control signal, which generates sample-specific temperatures for adaptive contrastive learning. By jointly integrating frequency-based calibration, phase-based supervision control, and cross-modal structural regularization, the framework improves the robustness and retrieval performance of neural-to-image alignment. Right: Internal structure of the low-dimensional adapter. The adapter takes the input embedding, applies normalization and two lightweight linear transformations, and produces both the refined output and its low-dimensional bottleneck representation.
Method
Subj. 1
Subj. 2
Subj. 3
Subj. 4
Subj. 5
Subj. 6
Subj. 7
Subj. 8
Subj. 9
Subj. 10
Avg
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Intra-subject: train and test on one subject
BraVL [ 2 ]
6.1
17.9
4.9
14.9
5.6
17.4
5.0
15.1
4.0
13.4
6.0
18.2
6.5
20.4
8.8
23.7
4.3
14.0
7.0
19.7
5.8
17.5
NICE [ 10 ]
13.2
39.5
13.5
40.3
14.5
42.7
20.6
52.7
10.1
31.5
16.5
44.0
17.0
42.1
22.9
56.1
15.4
41.6
17.4
45.8
16.1
43.6
NICE-SA [ 10 ]
13.3
40.2
12.1
36.1
15.3
39.6
15.9
49.0
9.8
34.4
14.2
42.4
17.9
43.6
18.2
50.2
14.4
38.7
16.0
42.8
14.7
41.7
NICE-GA [ 10 ]
15.2
40.1
13.9
40.1
14.7
42.7
17.6
48.9
9.0
29.7
16.4
44.4
14.9
43.1
20.3
52.1
14.1
39.7
19.6
46.7
15.6
42.8
TABLE I: Top-1 and Top-5 accuracy (%) for 200-way zero-shot retrieval on THINGS-EEG. The table reports results in both the intra-subject setting, where training and testing are performed on the same subject, and the inter-subject setting, where one subject is held out for testing.
Method
Subj. 1
Subj. 2
Subj. 3
Subj. 4
Avg
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Intra-subject: train and test on one subject
NICE
9.6
27.8
18.5
47.8
14.2
41.6
9.0
26.6
12.8
36.0
NICE-SA
9.8
27.8
18.6
46.4
10.5
38.4
11.7
27.2
12.7
35.0
NICE-GA
8.7
30.5
21.8
56.6
16.5
49.7
10.3
32.3
14.3
42.3
UBP
15.0
38.0
46.0
80.5
27.3
59.0
18.5
43.5
26.7
55.2
TABLE II: Top-1 and Top-5 accuracy (%) for 200-way zero-shot retrieval on THINGS-MEG. The table reports results in both the intra-subject setting and the inter-subject setting with leave one subject out evaluation.
Fig. 2: Qualitative comparison of neural-to-image retrieval results. The first column shows the ground-truth target image (GT), and the remaining columns show the Top-1 to Top-5 retrieved results for each method. Red boxes denote correct matches.
Fig. 3: Visualization of EEG–image cross-modal alignment. Left: correlation heatmap between EEG features and image features in the shared embedding space. Right: retrieval examples, where red boxes denote the ground-truth target images.
Comparison
Δ Margin
Δ Rank
Δ Top-1
Hardest20 Δ Margin
Hardest20 Δ Rank
Hardest20 Δ Top-1
Easiest20 Δ Margin
Easiest20 Δ Rank
Easiest20 Δ Top-1
Baseline vs. frequency-only
0.0004±0.0021
−0.064±0.177
−0.003±0.010
0.0043±0.0028
0.055±0.427
0.005±0.011
−0.0057±0.0039
0.000±0.000
0.000±0.000
Baseline vs. frequency-and-adaptive
0.0106±0.0029
0.464±0.452
0.034±0.019
−0.0658±0.0098
4.790±2.298
0.130±0.021
0.0740±0.0104
−0.075±0.098
−0.030±0.033
TABLE III: Summary of the effects of adaptive supervision, frequency modulation, and their joint design relative to the baseline over five random seeds. Each entry is reported as mean ± standard deviation. Positive Δ Rank indicates that the compared method achieves a better, that is, smaller, retrieval rank than the baseline.
Fig. 4: Gain trends of the frequency-only and joint frequency-and-adaptive variants across bins defined by the baseline retrieval margin. Lower baseline margin indicates more challenging samples. The frequency-only variant remains near zero across most bins, whereas the joint variant shows larger gains in the lower-margin region.
Fig. 5: Gain analysis across five bins defined by the phase-derived control score estimated from the adaptive-only model. The bins reflect control-score grouping rather than ground-truth difficulty strata.
Setting
Metric
Pooled Pearson
Pooled Spearman
Subject-wise Pearson
Subject-wise Spearman
Intra-subject
Control signal vs. Margin
-0.0188
-0.0388
−0.0057±0.0864
−0.0318±0.0685
Control signal vs. Negative Rank
0.0119
-0.0323
0.0449±0.0673
−0.0268±0.0631
Control signal vs. Top-1 Correct
-0.0349
-0.0375
−0.0259±0.0703
−0.0325±0.0620
Inter-subject
Control signal vs. Margin
-0.0030
-0.0024
−0.0032±0.0676
−0.0075±0.0671
Control signal vs. Negative Rank
0.0049
0.0002
0.0044±0.0595
−0.0105±0.0646
Control signal vs. Top-1 Correct
-0.0095
-0.0058
−0.0344±0.0657
−0.0387±0.0658
TABLE IV: Correlation analysis between the phase-derived control signal and retrieval diagnostics under intra-subject and inter-subject settings. We report both pooled sample-level correlations and subject-wise mean ± standard deviation across subjects.
Fig. 6: Diagnostic visualization of the relationship between the phase-derived control signal and retrieval diagnostics. The pooled sample-level trend is nearly flat, and the subject-wise correlations remain centered near zero, suggesting that the current control signal is not strongly associated with sample-wise retrieval outcomes.
Projectors
Adapters
BWS
Amp Phase
Adaptive Temp.
Top-1
Top-5
Img
EEG
Img
EEG
×
×
×
×
×
×
×
50.15
80.00
✓
×
×
×
×
×
×
54.70
81.60
×
✓
×
×
×
×
×
53.35
78.55
✓
✓
×
×
×
×
×
55.10
81.65
✓
✓
✓
✓
×
×
×
58.25
84.75
TABLE V: Ablation study of the proposed framework on THINGS-EEG.
Fig. 7: Performance improvement of the proposed method over the UBP baseline across different combinations of EEG backbones and image backbones on THINGS-EEG. Each cell reports the absolute gain (%) achieved by our method. The last row and last column denote the average improvements across different backbones.
Amp
Phase
CircVar
AdaTemp
Top-1 (%)
Top-5 (%)
61.45
86.95
✓
61.20
86.60
✓
61.60
87.10
✓
✓
61.65
87.10
✓
✓
62.90
86.30
✓
✓
✓
61.35
86.20
TABLE VI: Ablation study of frequency-domain statistics and phase-guided adaptive temperature on THINGS-EEG in the intra-subject setting. Amp denotes amplitude statistics, Phase denotes phase statistics, CircVar denotes phase-circular variance, and AdaTemp denotes the phase-guided adaptive temperature mechanism.
Fig. 8: Ablation of the low-dimensional bottleneck size. The model is evaluated with bottleneck sizes ranging from 64 to 2048. The results show that a moderate bottleneck dimension provides a better balance between discriminative capacity and alignment stability, with r=128 achieving the best Top-1 performance.
Method
Top-1 (%)
Top-5 (%)
Fixed temperature
63.00
87.30
Global learnable temperature
62.40
87.15
Simple sample weighting
63.15
86.55
Phase-guided adaptive temperature
63.40
86.85
TABLE VII: Comparison of different temperature control strategies. Sample-wise indicates whether the temperature or weighting is assigned at the individual-sample level. Phase-guided indicates whether the temperature control is derived from phase stability.
Visual neural decoding seeks to reconstruct or infer perceived visual stimuli from brain activity patterns, providing critical insights into human cognition and enabling transformative applications in brain-computer interfaces and artificial intelligence. Current approaches, however, remain constrained by the scarcity of high-quality stimulus-brain response pairs and the inherent semantic mismatch between neural representations and visual content. Inspired by perceptual variability and co-adaptive strategy of the biological systems, we propose a novel self-supervised architecture, named NeuroBridge, which integrates Cognitive Prior Augmentation (CPA) with Shared Semantic Projector (SSP) to promote effective cross-modality alignment. Specifically, CPA simulates perceptual variability by applying asymmetric, modality-specific transformations to both EEG signals and images, enhancing semantic diversity. Unlike previous approaches, SSP establishes a bidirectional alignment process through a co-adaptive strategy, which mutually aligns features from two modalities into a shared semantic space for effective cross-modal learning. NeuroBridge surpasses previous state-of-the-art methods under both intra-subject and inter-subject settings. In the intra-subject scenario, it achieves the improvements of 12.3% in top-1 accuracy and 10.2% in top-5 accuracy, reaching 63.2% and 89.9% respectively on a 200-way zero-shot retrieval task. Extensive experiments demonstrate the effectiveness, robustness, and scalability of the proposed framework for neural visual decoding.
Wenjiang Zhang, Sifeng Wang, Yuwei Su +3
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China · Queen Mary School Hainan, Beijing University of Posts and Telecommunications, Hainan 570100, China
Zero-shot EEG-to-image retrieval aims to decode perceived visual content from electroencephalography (EEG) by aligning neural responses with pretrained visual representations, providing a promising route toward scalable visual neural decoding and practical brain-computer interfaces. However, robust EEG-to-image retrieval remains challenging, because prior methods usually rely on either a single fixed visual target or a subject-invariant target construction scheme. Such designs overlook two important properties of visually evoked EEG signals: they preserve information across multiple representational scales, and the visual granularity best matched to EEG may vary across subjects. To address these issues, subject-aware multi-granularity alignment (SAMGA) framework is proposed for zero-shot EEG-to-image retrieval. SAMGA first constructs a subject-aware visual supervision target by adaptively aggregating multiple intermediate representations from a pretrained vision encoder, allowing the model to absorb subject-dependent granularity deviations during training while preserving subject-agnostic inference. Building on this adaptive target construction, a coarse-to-fine cross-modal alignment strategy is further designed with a shared encoder wherein the coarse stage stabilizes the shared semantic geometry and reduces subject-induced distribution shift, and the fine stage further improves instance-level retrieval discrimination. Extensive experiments on the THINGS-EEG benchmark demonstrate that the proposed method achieves 91.3% Top-1 and 98.8% Top-5 accuracy in the intra-subject setting, and 34.4% Top-1 and 64.8% Top-5 accuracy in the inter-subject setting, outperforming recent state-of-the-art methods.
Lin Jiang, Qingshan She, Jiale Xu +3
School of Automation, Hangzhou Dianzi University, and also with Zhejiang Provincial Key Laboratory of Brain Computer Collaborative Intelligence Technology and Applications, Hangzhou, Zhejiang 310018, China
EEG-based visual neural decoding aims to align neural responses with visual stimuli for tasks such as image retrieval. However, limited paired data and a fundamental mismatch between high-fidelity digital images and biological visual perception - distorted by retinotopic mapping and subject-specific neuroanatomy - severely impede cross-modal alignment. To address this, we propose MB2L, a Multi-Level Bidirectional Biomimetic Learning framework that incorporates structured physiological inductive biases into representation learning. Specifically, we propose Adaptive Blur with Visual Priors to mitigate perceptual-structural mismatch by reweighting visual inputs according to retinotopic priors. We further propose Biomimetic Visual Feature Extraction to learn multi-level visual representations consistent with hierarchical cortical processing, enhancing subject-invariant encoding. These modules are jointly optimized via Multi-level Bidirectional Contrastive Learning, which aligns EEG and visual features in a shared semantic space through bidirectional contrastive objectives. Experiments show MB2L achieves 80.5% Top-1 and 97.6% Top-5 accuracy on zero-shot EEG-to-image retrieval, significantly outperforming prior methods and demonstrating strong generalization across subjects and experimental settings.
Jingtao Liu, Peiliang Gong, Chuhang Zheng +2
Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · Department of Electrical and Information Engineering, Tianjin University, Tianjin, China