Neuro-Symbolic Decoding of Neural Activity

Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Building on the Brain Interaction Transformer (Brain-IT), our method decodes language tokens from brain activity and integrates them with a language model to answer visual questions. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.

Decoding visual experience from fMRI is important for understanding human perception and developing brain-computer interfaces. Recent generative decoding methods can reconstruct plausible images and videos, but visual plausibility alone does not ensure that the output is faithfully grounded in brain activity. In this work, we propose Neurons++, a visual-stream fMRI reconstruction framework for both static image and dynamic video reconstruction. Compared with our conference version Neurons, Neurons++ extends the paradigm from video-only reconstruction to visual-stream reconstruction for both images and videos, and replaces parallel task conditioning with serial cortex-inspired decoding. The model first uses image-fMRI pretraining to obtain a stable spatial-semantic initialization, and then performs subject-specific fine-tuning for image and video reconstruction separately. The serial design organizes fMRI-derived representations into explicit spatial, semantic, language-level, and temporal cues. Extensive experiments show that Neurons++ achieves state-of-the-art video reconstruction and competitive image reconstruction across standard benchmarks. Beyond reconstruction fidelity, neural-to-visual analyses indicate that Neurons++ exhibits qualitative directional controllability under neural perturbation, that its fMRI encoder read-out is sensitive to stimulus-relevant cortical features, and that it supports ROI-guided semantic modulation. Code and model weights are available at: https://github.com/xmed-lab/NeuronsPlusPlus.

Functional magnetic resonance imaging (fMRI)-based visual decoding aims to recover visual information from measured brain activity, commonly by mapping fMRI responses into latent visual features for downstream decoding tasks. Most existing methods learn mappings from fMRI responses to visual features extracted by artificial neural networks (ANNs), yet it remains unclear whether ANN-derived features provide suitable targets for brain decoding. In this study, we investigate spiking neural network (SNN)-derived visual features as alternative targets for fMRI-based visual decoding. We compare an ANN baseline with four SNN variants from the same architectural family, which differ in their spiking dynamics. To isolate the effect of the target features, all models use the same L2-regularized linear fMRI-to-feature decoder, while only the feature vectors used as regression targets are varied. Compared with the ANN baseline, SNN-derived features exhibit stronger alignment with fMRI responses and improve visual semantic decoding performance. For instance, on the GoD dataset, SNN-derived features reduce feature-prediction error from 0.7707 to 0.0282 and improve top-1 semantic decoding accuracy from 0.1800 to 0.4400. Ablation results further indicate that both spiking neural dynamics and temporal simulation steps contribute to the observed advantage. These findings support SNN-derived features as effective brain-decodable visual representations and highlight target feature design as an important component of fMRI-based visual decoding.