Stabilizing Neural-to-Image Retrieval with Frequency-Domain Priors and Adaptive Supervision
Organizations: School of Automation, Guangdong University of Technology, Guangzhou 510006, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
Abstract
Visual neural decoding aims to infer perceived visual stimuli from brain activity, offering important insights into human visual cognition while enabling transformative applications in brain computer interfaces and artificial intelligence. Recent advances in pretrained vision-language models have provided a semantically rich representation space for this task. However, reliable alignment between EEG/MEG signals and visual embeddings remains challenging because neural responses are noisy, highly variable across trials, and subject dependent. To address this issue, we propose a unified neural-to-image retrieval framework that improves retrieval performance by aligning neural and image representations in a shared semantic space. The framework integrates dual-stream adaptation, frequency-domain calibration, and phase-guided temperature modulation to promote more robust cross-modal semantic alignment. Specifically, dual-stream adaptation models a low-dimensional shared subspace with correlation regularization to suppress redundant noise and enhance cross-modal consistency. Frequency-domain calibration conditionally modulates neural representations using amplitude and phase statistics. Phase-guided temperature modulation further adjusts the supervision strength in contrastive learning according to the stability of neural responses. Experiments on THINGS-EEG and THINGS-MEG under both intra-subject and inter-subject settings demonstrate that the proposed framework consistently outperforms strong baselines. These results suggest that robust neural-to-image retrieval under heterogeneous neural conditions benefits from a stable shared semantic space, frequency-informed representation calibration, and adaptive supervision. Code will be released at github.com/wanfkuabc/Geometry-Regularized-Brain-Embeddings.
Figures & tables
| Method | Subj. 1 | Subj. 2 | Subj. 3 | Subj. 4 | Subj. 5 | Subj. 6 | Subj. 7 | Subj. 8 | Subj. 9 | Subj. 10 | Avg | |||||||||||
| Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | |
| Intra-subject: train and test on one subject | ||||||||||||||||||||||
| BraVL [ 2 ] | 6.1 | 17.9 | 4.9 | 14.9 | 5.6 | 17.4 | 5.0 | 15.1 | 4.0 | 13.4 | 6.0 | 18.2 | 6.5 | 20.4 | 8.8 | 23.7 | 4.3 | 14.0 | 7.0 | 19.7 | 5.8 | 17.5 |
| NICE [ 10 ] | 13.2 | 39.5 | 13.5 | 40.3 | 14.5 | 42.7 | 20.6 | 52.7 | 10.1 | 31.5 | 16.5 | 44.0 | 17.0 | 42.1 | 22.9 | 56.1 | 15.4 | 41.6 | 17.4 | 45.8 | 16.1 | 43.6 |
| NICE-SA [ 10 ] | 13.3 | 40.2 | 12.1 | 36.1 | 15.3 | 39.6 | 15.9 | 49.0 | 9.8 | 34.4 | 14.2 | 42.4 | 17.9 | 43.6 | 18.2 | 50.2 | 14.4 | 38.7 | 16.0 | 42.8 | 14.7 | 41.7 |
| NICE-GA [ 10 ] | 15.2 | 40.1 | 13.9 | 40.1 | 14.7 | 42.7 | 17.6 | 48.9 | 9.0 | 29.7 | 16.4 | 44.4 | 14.9 | 43.1 | 20.3 | 52.1 | 14.1 | 39.7 | 19.6 | 46.7 | 15.6 | 42.8 |
| Method | Subj. 1 | Subj. 2 | Subj. 3 | Subj. 4 | Avg | |||||
| Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | |
| Intra-subject: train and test on one subject | ||||||||||
| NICE | 9.6 | 27.8 | 18.5 | 47.8 | 14.2 | 41.6 | 9.0 | 26.6 | 12.8 | 36.0 |
| NICE-SA | 9.8 | 27.8 | 18.6 | 46.4 | 10.5 | 38.4 | 11.7 | 27.2 | 12.7 | 35.0 |
| NICE-GA | 8.7 | 30.5 | 21.8 | 56.6 | 16.5 | 49.7 | 10.3 | 32.3 | 14.3 | 42.3 |
| UBP | 15.0 | 38.0 | 46.0 | 80.5 | 27.3 | 59.0 | 18.5 | 43.5 | 26.7 | 55.2 |
| Comparison | Margin | Rank | Top-1 | Hardest20 Margin | Hardest20 Rank | Hardest20 Top-1 | Easiest20 Margin | Easiest20 Rank | Easiest20 Top-1 |
| Baseline vs. frequency-only | |||||||||
| Baseline vs. frequency-and-adaptive |
| Setting | Metric | Pooled Pearson | Pooled Spearman | Subject-wise Pearson | Subject-wise Spearman |
| Intra-subject | Control signal vs. Margin | -0.0188 | -0.0388 | ||
| Control signal vs. Negative Rank | 0.0119 | -0.0323 | |||
| Control signal vs. Top-1 Correct | -0.0349 | -0.0375 | |||
| Inter-subject | Control signal vs. Margin | -0.0030 | -0.0024 | ||
| Control signal vs. Negative Rank | 0.0049 | 0.0002 | |||
| Control signal vs. Top-1 Correct | -0.0095 | -0.0058 |
| Projectors | Adapters | BWS | Amp Phase | Adaptive Temp. | Top-1 | Top-5 | ||
| Img | EEG | Img | EEG | |||||
| 50.15 | 80.00 | |||||||
| ✓ | 54.70 | 81.60 | ||||||
| ✓ | 53.35 | 78.55 | ||||||
| ✓ | ✓ | 55.10 | 81.65 | |||||
| ✓ | ✓ | ✓ | ✓ | 58.25 | 84.75 | |||
| Amp | Phase | CircVar | AdaTemp | Top-1 (%) | Top-5 (%) |
| 61.45 | 86.95 | ||||
| ✓ | 61.20 | 86.60 | |||
| ✓ | 61.60 | 87.10 | |||
| ✓ | ✓ | 61.65 | 87.10 | ||
| ✓ | ✓ | 62.90 | 86.30 | ||
| ✓ | ✓ | ✓ | 61.35 | 86.20 |
| Method | Top-1 (%) | Top-5 (%) |
| Fixed temperature | 63.00 | 87.30 |
| Global learnable temperature | 62.40 | 87.15 |
| Simple sample weighting | 63.15 | 86.55 |
| Phase-guided adaptive temperature | 63.40 | 86.85 |
EEG-Based Visual Decoding
EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.
See all 57 papers