Few-Shot Cross-Subject Neural--Image Retrieval via Stimulus-Anchored Latent Bias Calibration
Authors: Xizhang Yao, Li Zhang, Gan Huang, Zhen Liang
Organizations: School of Biomedical Engineering, Health Science Center, Shenzhen University, and the Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, China
In cross-subject neural--image retrieval, comparing the mean of a few stimulus-matched target observations with the global source mean conflates subject-specific representation shifts with stimulus-composition effects. To address this problem, we propose Stimulus-Anchored Latent Bias Calibration (SA-LBC), which estimates the first-order shift under fixed preprocessing and a frozen source-trained representation by averaging residuals between target responses and same-stimulus source-population anchors, then shrinks the correction using source-derived uncertainty. Under uniform sampling without replacement from a finite stimulus population, we derive exact finite-population MSE expressions for the unshrunk anchored and global-mean estimators, including residual--anchor cross-covariance, and establish a necessary and sufficient condition for anchoring to reduce MSE relative to global-mean estimation. In a 10-subject leave-one-subject-out evaluation on THINGS-EEG2, SA-LBC increases mean reciprocal rank (MRR) for the Attention--CLIP ViT-B/32 configuration from 0.1867 without calibration to 0.2013 and 0.2336 with M=2 and M=16 stimulus-level calibration samples, respectively. Across six neural--visual configurations and all calibration budgets, stimulus-matched anchors yield higher MRR than anchors with shuffled stimulus identities, but retrieval gains over no calibration remain representation-dependent. At (M=2), shrinkage improves MRR over the unshrunk anchored estimator in all six configurations, but its effects at larger budgets vary across representations; the relative ranking of translation and orientation corrections also depends on regularization strength. Thus, lower shift-estimation MSE does not guarantee higher retrieval performance. The source code is publicly available at: https://github.com/Xizhang-Yao/SA-LBC.
Figures & tables
Fig. 1: Overview of Stimulus-Anchored Latent Bias Calibration (SA-LBC), illustrated for EEG–image retrieval after fixed preprocessing. (a) Source-only representation training is followed by calibration from M pre-test target observations and retrieval on concept-disjoint TEST stimuli. (b) The global-mean and anchored estimates differ by the sampled-stimulus composition term aˉM=pˉM−μP . (c) Source embeddings are averaged across subjects for each stimulus to form population anchors; averaging target–anchor residuals yields the shift estimate bT . (d) Source-only leave-one-out (LOO) residual uncertainty, the sampling budget, and the estimated shift energy determine the plug-in shrinkage factor β . The fixed correction is applied before ℓ2 normalization and cosine ranking. Stimulus IDs select the source anchors; latent calibration uses no image pixels or visual features, target TEST statistics, or retrieval feedback, and does not update the representation. In THINGS-EEG2, each calibration observation averages four responses, and participant-specific TRAIN whitening uses additional target recordings outside the M -observation budget.
Method
M=2
M=4
M=8
M=16
No adaptation
0.1867 ± 0.0495
0.1867 ± 0.0495
0.1867 ± 0.0495
0.1867 ± 0.0495
Global mean
0.1673 ± 0.0265
0.1994 ± 0.0338
0.2213 ± 0.0412
0.2347 ± 0.0459
Adaptive global mean
0.1998 ± 0.0459
0.2107 ± 0.0460
0.2219 ± 0.0475
0.2318 ± 0.0489
Shuffled anchor
0.1953 ± 0.0456
0.2056 ± 0.0451
0.2166 ± 0.0456
0.2275 ± 0.0470
Raw anchored ( β=1 )
0.1822 ± 0.0341
0.2090 ± 0.0401
0.2271 ± 0.0450
0.2381 ± 0.0481
SA-LBC
0.2013 ± 0.0464
0.2120 ± 0.0476
0.2239 ± 0.0495
0.2336 ± 0.0504
TABLE I: Attention–CLIP ViT-B/32 retrieval after participant-specific preprocessing. MRR is the primary endpoint; values are mean ± SD over 10 subjects after averaging 200 paired draws and three seeds within subject. Both adaptive estimators share the noise term but compute their own coefficient; raw anchoring uses β=1 . Bold marks the highest rounded mean within each calibration budget. Complete Top-1 and Top-5 results are in Supplementary Table S12.
Fig. 2: Paired MRR effects for Attention–CLIP ViT-B/32: SA-LBC minus (a) no adaptation, (b) adaptive global mean, and (c) bank-shuffled anchoring. Error bars are 95% subject-bootstrap intervals after within-subject draw/seed averaging. The controls test net utility, the adaptive-reference increment, and reference-composition matching, respectively; their effects are not interchangeable.
EEG encoder
Visual backbone
T
R0.1
(T+R)0.1
R1
(T+R)1
Attention
CLIP ViT-B/32
0.23364
0.16749
0.20697
0.19326
0.24114
EEGProject
CLIP ViT-B/32
0.25908
0.17956
0.21624
0.22060
0.26695
TSConv
CLIP ViT-B/32
0.21047
0.18012
0.18171
0.21503
0.21582
Attention
CLIP RN101
0.21311
0.15846
0.19034
0.18487
0.22101
Attention
DINOv2 ViT-B/14
0.18318
0.13609
0.16186
0.15945
0.18855
TSConv
DINOv2 ViT-B/14
0.18264
0.15619
0.15840
0.18479
0.18641
TABLE II: Regularization-dependent geometric ordering at M=16 . Entries are group-average MRR from the same cache within each configuration. Comparisons of correction types are made within each row. ρ=0.1 is the primary setting; ρ=1 is a fixed member of the sensitivity grid, not a TEST-selected deployment parameter. The complete grid is reported in the Supplementary Material.
EEG encoder
Visual backbone
Baseline MRR
Δ MRR ( M=2 )
pH
Δ MRR ( M=16 )
pH
Attention
CLIP ViT-B/32
0.18675
+0.01458
0.0078
+0.04690
0.0078
EEGProject
CLIP ViT-B/32
0.21393
+0.01224
0.0234
+0.04515
0.0078
TSConv
CLIP ViT-B/32
0.20947
-0.00330
0.0078
+0.00100
0.6172
Attention
CLIP RN101
0.18022
+0.00944
0.0215
+0.03290
0.0078
Attention
DINOv2 ViT-B/14
0.15587
+0.00800
0.0078
+0.02730
0.0078
TSConv
DINOv2 ViT-B/14
0.18122
-0.00236
0.0547
+0.00143
0.2754
TABLE III: Calibration utility across six source-trained neural–visual configurations. Baseline MRR and paired changes use the same re-encoded cache within each row; Attention–CLIP ViT-B/32 is counted once. Δ is SA-LBC minus no adaptation. pH is Holm-adjusted across M∈{2,4,8,16} separately within each configuration and contrast, not across the six rows. These tests do not compare encoders. The full four-budget results are provided in the Supplementary Material.
Decoding visual information from brain signals probes neural representations and enables neuro-rehabilitation and dream decoding. Recent brain-to-image retrieval approaches have achieved promising performance, typically by averaging many (up to 80) neural trials per image, requiring repeated stimulus presentation that increases latency, cost, and user burden. When only one or a few repetitions are available, the retrieval accuracy drops sharply. This drop is commonly attributed to query noise because averaging suppresses noise and increases signal stability. However, we find a non-transitive alignment pattern: the low-repetition query signal and the image representation each align with the high-repetition center, but not directly with each other. This pattern shows that query noise is only part of the problem and that gallery placement also affects retrieval. We therefore propose a neural-anchor-based retrieval (NEAR) framework that treats the high-repetition center as an anchor and approaches it from both sides: a denoiser pulls the noisy query toward the true anchor, and a small network predicts each candidate's pseudo anchor from its image and pulls the image toward it. Across four datasets spanning EEG, MEG and fMRI, NEAR consistently improved retrieval in the few-repetition regime. On THINGS-EEG2, it improved 200-way Top-1 accuracy by 5.7 and 9.3 percentage points respectively, when averaging one and four repetitions. By anchoring neural and visual representations, NEAR reduces reliance on repeated acquisition and brings neural retrieval closer to real-world deployment.
Zhenyao Cui, Siyuan Kan, Dingkun Liu +1
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China
Cross-subject EEG-to-image retrieval for visual decoding is challenged by subject shift and hubness in the embedding space, which distort similarity geometry and destabilize top-k rankings, making small-k shortlists unreliable. We introduce SATTC (Structure-Aware Test-Time Calibration), a label-free calibration head that operates directly on the similarity matrix of frozen EEG and image encoders. SATTC combines a geometric expert, subject-adaptive whitening of EEG embeddings with an adaptive variant of Cross-domain Similarity Local Scaling (CSLS), and a structural expert built from mutual nearest neighbors, bidirectional top-k ranks, and class popularity, fused via a simple Product-of-Experts rule. On THINGS-EEG2 under a strict leave-one-subject-out protocol, standardized inference with cosine similarities, L2-normalized embeddings, and candidate whitening already yields a strong cross-subject baseline over the original ATM retrieval setup. Building on this baseline, SATTC further improves Top-1 and Top-5 accuracy, reduces hubness and per-class imbalance, and produces more reliable small-k shortlists. These gains transfer across multiple EEG encoders, supporting SATTC as an encoder-agnostic, label-free test-time calibration layer for cross-subject neural decoding.
Visual neural decoding aims to infer perceived visual stimuli from brain activity, offering important insights into human visual cognition while enabling transformative applications in brain computer interfaces and artificial intelligence. Recent advances in pretrained vision-language models have provided a semantically rich representation space for this task. However, reliable alignment between EEG/MEG signals and visual embeddings remains challenging because neural responses are noisy, highly variable across trials, and subject dependent. To address this issue, we propose a unified neural-to-image retrieval framework that improves retrieval performance by aligning neural and image representations in a shared semantic space. The framework integrates dual-stream adaptation, frequency-domain calibration, and phase-guided temperature modulation to promote more robust cross-modal semantic alignment. Specifically, dual-stream adaptation models a low-dimensional shared subspace with correlation regularization to suppress redundant noise and enhance cross-modal consistency. Frequency-domain calibration conditionally modulates neural representations using amplitude and phase statistics. Phase-guided temperature modulation further adjusts the supervision strength in contrastive learning according to the stability of neural responses. Experiments on THINGS-EEG and THINGS-MEG under both intra-subject and inter-subject settings demonstrate that the proposed framework consistently outperforms strong baselines. These results suggest that robust neural-to-image retrieval under heterogeneous neural conditions benefits from a stable shared semantic space, frequency-informed representation calibration, and adaptive supervision. Code will be released at github.com/wanfkuabc/Geometry-Regularized-Brain-Embeddings.
YaoYang Liu, ZhuoShi Tang, Haopeng Lai +3
School of Automation, Guangdong University of Technology, Guangzhou 510006, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China