Neuro-Symbolic Decoding of Neural Activity

Yanchen Wang; Joy Hsu; Ehsan Adeli; Jiajun Wu

q-bio.NCFeb 22, 2026

Neuro-Symbolic Decoding of Neural Activity

Authors: Yanchen Wang, Joy Hsu, Ehsan Adeli, Jiajun Wu

Organizations: Columbia University · Stanford University

Abstract

We propose NEURONA, a neuro-symbolic framework for fMRI decoding and concept grounding in neural activity. Leveraging image- and video-based fMRI question-answering datasets, NEURONA learns to decode interacting concepts from visual stimuli based on patterns of fMRI responses, integrating symbolic reasoning and compositional execution with fMRI grounding across brain regions. We demonstrate that incorporating structural priors (e.g., compositional predicate-argument dependencies between concepts) into the decoding process significantly improves both decoding accuracy over precise queries, and notably, generalization to unseen queries at test time. With NEURONA, we highlight neuro-symbolic frameworks as promising tools for understanding neural activity.

Figures & tables

Figure 1: NEURONA is a neuro-symbolic framework for neural decoding that parses each query into a symbolic expression and maps the accompanying fMRI recording into candidate parcel-level embeddings. It grounds concepts in the expression to these candidate parcels with learned linear concept modules, optionally guided by predicate-argument structure, and composes the grounded scores to answer the question. Supervision is provided only by the final answer, which enables learning of intermediate groundings.

Figure 2: We include example queries and dataset distribution overviews for BOLD5000-QA and CNeuroMod-QA; both datasets span diverse queries and tasks.

	BOLD5000-QA				CNeuroMod-QA
Method	Overall	Action	Position	T/F	Overall	Action	Position	T/F
Linear	$0.4692$	$0.2069$	$0.1778$	$0.5260$	$0.4638$	$0.3043$	$0.1285$	$0.5192$
UMBRAE	$0.4754$	$0.2069$	$0.1746$	$0.5340$	$0.4642$	$0.1432$	$0.1762$	$0.5378$
SDRecon	$0.4711$	$0.2414$	$0.1937$	$0.5248$	$0.4430$	$0.1350$	$0.1481$	$0.5238$
BrainCap	$0.4773$	$0.1937$	$0.1724$	$0.5551$	$0.4417$	$0.1257$	$0.1477$	$0.5112$
NEURONA (Ours)	$\mathbf{0.7041}$	$\mathbf{0.6207}$	$\mathbf{0.5079}$	$\mathbf{0.7407}$	$\mathbf{0.7046}$	$\mathbf{0.6514}$	$\mathbf{0.5746}$	$\mathbf{0.7250}$

Table 1: We evaluate NEURONA on BOLD5000-QA and CNeuroMod-QA, comparing its performance to prior fMRI language decoding models and a linear baseline.

	BOLD5000-QA				CNeuroMod-QA
Method	Overall	Action	Position	T/F	Overall	Action	Position	T/F
Linear	$0.4587$	$0.0690$	$0.0794$	$0.5231$	$0.4143$	$0.0398$	$0.0323$	$0.5003$
UMBRAE	$0.4501$	$0.1379$	$0.0984$	$0.5191$	$0.4502$	$0.1205$	$0.1384$	$0.5225$
SDRecon	$0.4702$	$0.2414$	$0.1937$	$0.5237$	$0.4341$	$0.1236$	$0.1473$	$0.5008$
BrainCap	$0.4754$	$0.1724$	$0.1937$	$0.5311$	$0.4347$	$0.1198$	$0.1402$	$0.5042$
NEURONA (Ours)	$\mathbf{0.6840}$	$\mathbf{0.6207}$	$\mathbf{0.4984}$	$\mathbf{0.7184}$	$\mathbf{0.6583}$	$\mathbf{0.4365}$	$\mathbf{0.5261}$	$\mathbf{0.6991}$

Table 2: Generalization results of NEURONA and prior work on unseen queries.

	BOLD5000-QA
Method	Overall	Action	Position	T/F
Single region	$0.6451\pm 0.0161$	$0.2973\pm 0.0361$	$0.2005\pm 0.0047$	$0.7293\pm 0.0191$
Multi-region (MR)	$0.6476\pm 0.0048$	$0.2973\pm 0.0361$	$0.2005\pm 0.0047$	$0.7324\pm 0.0056$
Subject-guided MR	$0.6678\pm 0.0029$	$0.2881\pm 0.0299$	$0.3469\pm 0.0134$	$0.7308\pm 0.0024$
Object-guided MR	$0.6733\pm 0.0051$	$0.3892\pm 0.0752$	$0.3429\pm 0.0164$	$0.7363\pm 0.0045$
Full argument-guided MR	$\mathbf{0.7102\pm 0.0053}$	$\mathbf{0.5965\pm 0.0322}$	$\mathbf{0.5378\pm 0.0135}$	$\mathbf{0.7425\pm 0.0057}$

Table 3: We ablate our hypotheses about the structure of concept grounding on BOLD5000-QA.

	CNeuroMod-QA
Method	Overall	Action	Position	T/F
Single region	$0.6429\pm 0.0013$	$0.2165\pm 0.0193$	$0.2445\pm 0.0009$	$0.7400\pm 0.0016$
Multi-region (MR)	$0.6162\pm 0.0027$	$0.2165\pm 0.0193$	$0.2445\pm 0.0009$	$0.7042\pm 0.0023$
Subject-guided MR	$0.6265\pm 0.0042$	$0.2339\pm 0.0043$	$0.3722\pm 0.0045$	$0.7008\pm 0.0051$
Object-guided MR	$0.6872\pm 0.0040$	$0.6320\pm 0.0019$	$0.4933\pm 0.0172$	$0.7149\pm 0.0045$
Full argument-guided MR	$\mathbf{0.7189\pm 0.0009}$	$\mathbf{0.6422\pm 0.0072}$	$\mathbf{0.5931\pm 0.0084}$	$\mathbf{0.7417\pm 0.0005}$

Table 4: We evaluate our hypotheses about structural priors on CNeuroMod-QA.

	BOLD5000-QA	CNeuroMod-QA
Null	$0.6267\pm 0.0067$	$0.6424\pm 0.0057$
NEURONA	$\mathbf{0.8475\pm 0.0186}$	$\mathbf{0.8592\pm 0.0084}$

Table 5: Consistency of concept grounding between NEURONA and a multinomial null model.

Figure 3: We show examples of learned grounding from NEURONA. On both BOLD5000-QA and CNeuroMod-QA, we see that predicate concepts ground to regions that their constituent objects arguments are grounded to, following hierarchical predicate-argument structure.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

BOLD5000-QA	Overall	Action	Position	T/F
Yeo-7	0.6864	0.5517	0.4730	0.7270
Yeo-17	0.7041	0.6207	0.5079	0.7407
DiFuMo-64	0.6992	0.5517	0.5524	0.7282
DiFuMo-128	0.7026	0.5862	0.5460	0.7327

Appendix

Table 6: NEURONA’s performance with coarse and fine-grained atlases on BOLD5000-QA.

CNeuroMod-QA	Overall	Action	Position	T/F
Yeo-7	0.6969	0.6459	0.5577	0.7180
Yeo-17	0.7046	0.6514	0.5746	0.7250
Schaefer-100	0.7043	0.6549	0.5614	0.7258

Appendix

Table 7: NEURONA’s performance with coarse and fine-grained atlases on CNeuroMod-QA.

BOLD5000-QA	Yeo-7	Yeo-17	Difumo-64	Difumo-128
Yeo-7	-	-0.017	-0.133	-0.012
Yeo-17	0.017	-	-0.116	0.006
Difumo-64	0.133	0.116	-	0.121
Difumo-128	0.012	-0.006	-0.121	-

Appendix

Table 8: Effect sizes between atlases in BOLD5000-QA.

CNeuroMod-QA	Yeo-7	Yeo-17	Schaefer-100
Yeo-7	-	0.094	0.089
Yeo-17	-0.094	-	-0.006
Schaefer-100	-0.089	0.006	-

Appendix

Table 9: Effect sizes between atlases in CNeuroMod-QA.

BOLD5000-QA	Overall	Unary Concept	Relational Concept
Yeo-7 Null	0.5738	–	–
Yeo-7 NEURONA	0.8207	0.8283	0.7343
Yeo-17 Null	0.5357	–	–
Yeo-17 NEURONA	0.8220	0.8351	0.6646
DiFuMo-64 Null	0.5075	–	–
DiFuMo-64 NEURONA	0.8462	0.8644	0.6064

Appendix

Table 10: Concept grounding consistency in BOLD5000-QA.

CNeuroMod-QA	Overall	Unary Concept	Relational Concept
Yeo-7 Null	0.5740	–	–
Yeo-7 NEURONA	0.8437	0.8564	0.7563
Yeo-17 Null	0.5358	–	–
Yeo-17 NEURONA	0.8700	0.8967	0.6812
Schaefer-100 Null	0.5029	–	–
Schaefer-100 NEURONA	0.8346	0.8695	0.5838

Appendix

Table 11: Concept grounding consistency in CNeuroMod-QA.

BOLD5000-QA	Overall	Action	Position	T/F
Linear	0.4702 ± 0.0073	0.0677 ± 0.0815	0.1817 ± 0.0289	0.5280 ± 0.0068
UMBRAE	0.4948 ± 0.0098	0.2488 ± 0.0572	0.2092 ± 0.0401	0.5401 ± 0.0097
SDRecon	0.4751 ± 0.0064	0.2887 ± 0.0499	0.2005 ± 0.0047	0.5266 ± 0.0075
BrainCap	0.4842 ± 0.0052	0.2412 ± 0.0470	0.2005 ± 0.0047	0.5383 ± 0.0061
NEURONA	0.7102 ± 0.0053	0.5965 ± 0.0322	0.5378 ± 0.0135	0.7425 ± 0.0057
CNeuroMod-QA	Overall	Action	Position	T/F

Appendix

Table 12: Results across subjects for BOLD5000-QA and CNeuroMod-QA.

BOLD5000-QA	Overall	Action	Position	T/F
Linear	0.4595 ± 0.0093	0.0308 ± 0.0312	0.1291 ± 0.0545	0.5250 ± 0.0059
UMBRAE	0.4886 ± 0.0068	0.1214 ± 0.0217	0.0914 ± 0.0212	0.5392 ± 0.0080
SDRecon	0.4752 ± 0.0064	0.2973 ± 0.0361	0.2005 ± 0.0047	0.5266 ± 0.0075
BrainCap	0.4838 ± 0.0055	0.2412 ± 0.0470	0.1996 ± 0.0040	0.5380 ± 0.0064
NEURONA	0.6812 ± 0.0055	0.4952 ± 0.0682	0.4696 ± 0.0190	0.7217 ± 0.0057
CNeuroMod-QA	Overall	Action	Position	T/F

Appendix

Table 13: Generalization results across subjects for BOLD5000-QA and CNeuroMod-QA.

	BOLD5000-QA	CNeuroMod-QA
Null (random)	0.5276	0.5267
Null (multinomial)	0.6045	0.6481
NEURONA	0.7000	0.7027

Appendix

Table 14: Cross-subject consistency of concept grounding.

	Overall	Action	Position	T/F
UMBRAE	0.4494	0.0106	0.0485	0.5036
NEURONA (Ours)	0.5535	0.7237	0.5246	0.5481

Appendix

Table 15: Cross-dataset generalization results, where models are trained on BOLD5000 and tested on CNeuroMod.

Selected network	Overall	Action	Position	T/F
VisCent VisPeri	0.6750 ± 0.0067	0.3577 ± 0.0459	0.3765 ± 0.0159	0.7330 ± 0.0072
SomMotA/B	0.6724 ± 0.0069	0.3480 ± 0.0317	0.3620 ± 0.0131	0.7326 ± 0.0079
DorsAttnA/B	0.6753 ± 0.0073	0.3281 ± 0.0786	0.3850 ± 0.0282	0.7324 ± 0.0076
SalVentAttnA/B	0.6709 ± 0.0069	0.2910 ± 0.0655	0.3839 ± 0.0267	0.7281 ± 0.0060
LimbicA/B	0.6669 ± 0.0101	0.3379 ± 0.0918	0.3512 ± 0.0603	0.7282 ± 0.0044
ContA/B/C	0.6785 ± 0.0038	0.3380 ± 0.0808	0.4129 ± 0.0173	0.7310 ± 0.0035

Appendix

Table 16: Network ablations to evaluate the contributions of each network to decoding performance.

Figure 4: Statistical tests evaluating the effects of the five hypotheses across subjects in BOLD5000.

Figure 5: Statistical tests evaluating the effects of the five hypotheses across subjects in CNeuroMod.

BOLD5000-QA	Argument swapping	Predicate transfer	Role systematicity	Other	All (T/F)
Unseen	0.7451	0.75	0.7391	0.7096	0.7184
All	0.7562	0.6667	0.8030	0.7393	0.7407
CNeuroMod-QA	Argument swapping	Predicate transfer	Role systematicity	Other	All (T/F)
Unseen	0.7526	0.7333	0.7390	0.6842	0.6991
All	0.7011	0.5088	0.7020	0.7307	0.7250

Appendix

Table 17: Fine-grained generalization analyses in BOLD5000-QA and CNeuroMod-QA.

	BOLD5000-QA	CNeuroMod-QA
Unguided-predicate / subject	-0.0940	0.0230
Unguided-predicate / object	0.0048	0.0019
Unguided-predicate / guided-predicate	0.0336	0.1488
Guided-predicate / subject	0.2020	0.1095
Guided-predicate / object	0.2975	0.2552

Appendix

Table 18: Results of predicate argument binding.

Table 24

Figure 6: Accuracy confusion matrix across concepts in BOLD5000-QA.

Figure 7: Accuracy confusion matrix across concepts in CNeuroMod-QA

Figure 8: We show examples of learned concept grounding by NEURONA on BOLD5000-QA and CNeuroMod-QA, across subject, object, and predicate concepts.

Figure 9: Examples of queries in BOLD5000-QA and CNeuroMod-QA.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 180 papers

Brain-IT-VQA: From Brain Signals to Answers
May 28, 2026Roman Beliy, Matias Cosarinsky, Oliver Heinimann +2Visual Question Answering Neural Decoding
Weizmann Institute of Science
Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability
Oct 9, 2026Haonan Wang, Yuchen Wang, Jingyu Lu +4fMRI-Based Visual Decoding Neural Decoding
The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China
Spiking Neural Networks for fMRI-Based Visual Semantic Decoding
Jul 21, 2026Jiahong Zhang, Jinning Zhao, Sijun Shen +3Visual Representation Learning Neural Decoding
Institute of Automation, Chinese Academy of Sciences, Beijing 100045, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China

May 28, 2026J/K move · Enter open · S save

Brain-IT-VQA: From Brain Signals to Answers

Roman Beliy, Matias Cosarinsky, Oliver Heinimann, Navve Wasserman+1

Weizmann Institute of Science

Visual Question Answering Neural Decoding Visual Neuroscience Neuroimaging fMRI-Based Visual Decoding Functional Magnetic Resonance Imaging

Open paper AskPDF arXiv

Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Building on the Brain Interaction Transformer (Brain-IT), our method decodes language tokens from brain activity and integrates them with a language model to answer visual questions. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.

Abstract

Neural Decoding

Explore similar work

Brain-IT-VQA: From Brain Signals to Answers

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

Spiking Neural Networks for fMRI-Based Visual Semantic Decoding