SIRAZ: Sparse Inter-Layer Residual Aggregation for Zero-Shot Visual Neural Decoding
Organizations: Department of Computer Science and Engineering, Indian Institute of Technology Ropar, Rupnagar, 140001, Punjab, India
Abstract
Brain-computer interfaces aim to establish direct communication between neural activity and external systems, making interpretation of brain signals an important problem. In this context, neural visual decoding seeks to infer visual information from neural activity by learning a mapping between brain responses and the corresponding visual stimuli. It investigates how visual information emerges from neural signals and relates to perceptual processing while enabling the reconstruction or retrieval of visual stimuli from neural signals. Our brain primarily follows a hierarchical pathway to process visual information captured by the eye. However, neural representations learned by existing methods differ fundamentally from those produced by the biological visual system. To bridge this gap and mimic the human vision hierarchy, we introduce Sparse Inter-layer Residual Aggregation for Zero-shot (SIRAZ). SIRAZ is a framework that aligns neural visual representations by jointly modeling early visual processes via fovea- and magnocellular-pathway-inspired Difference-of-Gaussians transformations. To align the semantic information with neural information, it extracts multiple intermediate representations of the image at multiple depths and passes them through the Residual Gated Adapter Mechanism. To study the temporal dynamics of neural signals, the Direction-aware Multi-Scale Temporal Attention Encoder is introduced to align the learned neural representation with the final visual representation. The proposed framework achieves 92.5% top-1 and 99.27% top-5 accuracy in the intra-subject setting on the THINGS-EEG dataset, and 50.25% and 75% top-1 and top-5 accuracy in the THINGS-MEG dataset. The results indicate that combining biologically motivated visual representations with the structured modalities of neural dynamics improves cross-modal alignment and zero-shot image retrieval.
Figures & tables
| Method | Metric | Sub1 | Sub2 | Sub3 | Sub4 | Sub5 | Sub6 | Sub7 | Sub8 | Sub9 | Sub10 | Average |
| Intra-subject: Subject-Specific Evaluation | ||||||||||||
| BraVL Du et al. (2023) | Top-1 | 6.1 | 4.9 | 5.6 | 5.0 | 4.0 | 6.0 | 6.5 | 8.8 | 4.3 | 7.0 | 5.8 |
| Top-5 | 17.9 | 14.9 | 17.4 | 15.1 | 13.4 | 18.2 | 20.4 | 23.7 | 14.0 | 19.7 | 17.5 | |
| NICE Song et al. (2024) | Top-1 | 12.3 | 10.4 | 13.1 | 16.4 | 8.0 | 14.1 | 15.2 | 20.0 | 13.3 | 14.9 | 13.8 |
| Top-5 | 36.6 | 33.9 | 39.0 | 47.0 | 26.9 | 40.6 | 42.1 | 49.9 | 37.1 | 41.9 | 39.5 | |
| UBP Wu et al. (2025) | Top-1 | 41.2 | 51.2 | 51.2 | 51.1 | 42.2 | 57.5 | 49.0 | 58.6 | 45.1 | 61.5 | 50.9 |
| Method | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Average | |||||
| Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | |
| Intra-subject: Subject-Specific Evaluation | ||||||||||
| NICE Song et al. (2024) | 9.6 | 27.8 | 18.5 | 47.8 | 14.2 | 41.6 | 9.0 | 26.6 | 12.8 | 36.0 |
| UBP Wu et al. (2025) | 15.0 | 38.0 | 46.0 | 80.5 | 27.3 | 59.0 | 18.5 | 43.5 | 26.7 | 55.2 |
| ATS Wu et al. (2026) | 18.0 | 43.0 | 56.0 | 89.0 | 36.0 | 67.0 | 19.5 | 50.0 | 32.4 | 62.3 |
| NeuroBridge Zhang et al. (2026) | 16.5 | 41.6 | 53.7 | 85.3 | 40.4 | 73.2 | 18.1 | 43.1 | 32.2 | 60.8 |
| Configuration | Top-1 | Gain | Top-5 | Gain |
| DMSTAE | 62.65 | – | 85.90 | – |
| + Intermediate layers | 72.85 | +10.20 | 95.25 | +9.35 |
| + Fovea Blur + Magno (DoG) | 75.35 | +2.50 | 95.05 | -0.20 |
| + ReGRAM | 89.85 | +14.50 | 99.00 | +3.95 |
| + Residual connection | 92.50 | +2.65 | 99.25 | +0.25 |
Neural Decoding
Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.
See all 202 papers