AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding

Shiqi Shen; Shurui Li; Xilin Zhang; Yuanning Li

cs.CVNov 2, 2026

AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding

Authors: Shiqi Shen, Shurui Li, Xilin Zhang, Yuanning Li

Organizations: ShanghaiTech University · South China Normal University · XS Vision

Abstract

Visual mental imagery (VMI) decoding uses brain activity to reconstruct internally generated visual scenes, providing a potential pathway for externalizing internal experiences and developing neuroprosthetic communication interfaces. However, progress is limited by the lack of dedicated, training-scale VMI datasets. Current VMI paradigms often rely on visual prompts or short-term memory images, introducing visual and mnemonic confounds that make it difficult to disentangle neural activity associated with endogenous image construction from signals driven by prior or concurrent visual input. Furthermore, existing VMI datasets offer limited descriptive control over individual visual attributes, making it difficult to systematically assess fine grained visual details. To address these gaps, we introduce AVID, the first dedicated training-scale 5.0 Tesla functional magnetic resonance imaging (fMRI) resource for auditory-induced VMI decoding. AVID comprises approximately 67 hours of dense recordings from nine participants. The dataset is structured into two complementary splits: a scene-level complex split using naturalistic Mandarin descriptions and a controlled simple split with explicitly specified visual attributes. We further formalize a standardized evaluation protocol that decouples source-image consistency from semantic alignment, allowing for a more nuanced assessment of imagery reconstruction. Baseline qualitative and quantitative results show that current reconstruction pipelines remain limited on AVID, although the most successful reconstructions preserve recognizable scene level semantics from spoken descriptions. By combining dedicated training-scale neural recordings and standardized evaluation resources, AVID provides both a data foundation for imagery-specific model development and a benchmark for evaluating VMI decoding with substantially reduced sensory contamination.

Figures & tables

Figure 1: Overview of our benchmark. (A) Source images are converted into spoken Mandarin descriptions, which participants hear during eyes-closed imagery in the 5T scanner. The fMRI data are preprocessed to estimate trial-wise responses and define ROI-based neural inputs for decoding. (B) AVID includes approximately 67 hours of recordings from 9 participants across complex scene-level and controlled attribute splits, using a trial structure with an auditory cue, imagery period, vividness rating, and reset interval. (C) Decoders are evaluated on fixed AVID splits, with reconstructions scored by source-image consistency and spoken-description alignment.

Split	Participants	Stimuli	Total trials	Task time (h)	Repetitions
Complex	9	800	14,400	48.0	2
Simple	7	400	5,600	18.7	2
Total	9	1,200	20,000	66.7	2

Table 1: Summary of AVID dataset statistics.

Figure 2: Qualitative comparison of mental imagery generation results. (A) Examples from the complex stimuli split. (B) Examples from the simple stimuli split. For each stimulus, we compare the source image with reconstructions generated by LDM, MindEye1, MindEye2, and MindBridge. The results illustrate how different methods recover semantic content and visual structure across natural images and simple shape stimuli.

	Image-image				Image-text
Method	PixCorr	SSIM	Inception	CLIP-I	CLIP-T	SigLIP	AltCLIP	TIFA
LDM	0.038	0.185	59.09%	59.61%	66.81%	68.54%	67.34%	32.56%
MindEye1	0.056	0.291	65.14%	65.94%	67.83%	70.63%	69.83%	30.27%
MindEye2	0.034	0.262	55.75%	57.81%	57.95%	59.08%	58.82%	24.34%
MindBridge	0.040	0.170	65.22%	65.80%	67.66%	69.20%	69.61%	32.25%

Table 2: Complex split baseline results. Metrics are grouped by source image consistency and spoken description alignment.

	Image-image				Image-text
Method	PixCorr	SSIM	Inception	CLIP-I	CLIP-T	SigLIP	AltCLIP	TIFA
LDM	-0.001	0.636	54.07%	51.75%	55.59%	57.16%	57.25%	48.92%
MindEye1	-0.028	0.962	53.19%	52.75%	53.19%	54.33%	52.83%	44.34%
MindEye2	0.004	0.684	53.54%	51.41%	52.09%	52.81%	51.53%	39.34%
MindBridge	-0.006	0.906	57.49%	55.69%	55.69%	57.13%	56.69%	40.32%

Table 3: Simple split baseline results. Metrics are grouped by source image consistency and spoken description alignment.

Method	Color	Position	Visual identity
LDM	$30.13\pm 5.58\%$	$24.07\pm 5.46\%$	$2.36\pm 1.79\%$
MindEye1	$31.65\pm 7.83\%$	$35.34\pm 8.56\%$	$3.38\pm 1.96\%$
MindEye2	$32.59\pm 10.47\%$	$29.39\pm 4.73\%$	$2.07\pm 1.25\%$
MindBridge	$35.55\pm 4.04\%$	$26.99\pm 6.81\%$	$2.70\pm 1.30\%$
Chance	$25.76\%$	$20.00\%$	$1.05\%$

Table 4: Attribute-level evaluation on the simple split. Reconstruction scores are class-balanced accuracies, reported as mean

\pm

sample standard deviation across seven participants. The chance row gives the analytical expected score under uniform selection from the corresponding answer options. Differences between model configurations are descriptive.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 5 — Table 5: Example complex stimuli.

Panel of Table 6 — Table 6: Example simple stimuli.

Figure 4: Temporal signal-to-noise ratio (tSNR) across subjects and runs in AVID. (A) Distribution of run-wise median tSNR values for each subject, where tSNR was computed voxel-wise after quadratic detrending and summarized within the top 10% highest-tSNR voxels in each run. (B) Run-by-run median tSNR trajectories for all subjects, with colored lines indicating individual subjects and the black line indicating the across-subject median. The dashed vertical line separates complex runs from simple runs.

Figure 5: Cortical distribution of noise ceilings, shown in left lateral, ventral, and right lateral views. Colors indicate the estimated percentage of explainable variance in the mean response across two stimulus repetitions. Gray shading depicts cortical anatomy.

Split	Cortical region	Low-vividness NC	High-vividness NC	$p$
Complex	Postcentral gyrus	$8.21\pm 2.46$ %	$11.55\pm 2.64$ %	0.0039
	Superior temporal gyrus	$12.38\pm 1.20$ %	$13.89\pm 2.16$ %	0.0273
	Fusiform gyrus	$10.57\pm 3.43$ %	$10.42\pm 2.84$ %	0.8203
	Pars orbitalis	$8.75\pm 1.68$ %	$9.22\pm 1.88$ %	0.3594
Simple	Postcentral gyrus	$8.94\pm 2.40$ %	$9.69\pm 2.00$ %	0.0781
	Superior temporal gyrus	$12.08\pm 1.42$ %	$12.06\pm 2.42$ %	1.0000

Appendix

Table 7: Neural response reliability for the 100 lowest- and 100 highest-vividness stimuli selected separately within each participant and split. Values are across-participant mean

\pm

sample standard deviation of noise-ceiling estimates. The complex and simple splits included nine and seven participants, respectively.

p

values were obtained using two-sided paired Wilcoxon signed-rank tests.

Attribute	fMRI accuracy	Chance	Gaussian control	$t$ -test $p$	Wilcoxon $p$
Color	$0.395\pm 0.021$	$0.333$	$0.332\pm 0.031$	$0.00012$	$0.0078$
Position	$0.440\pm 0.144$	$0.200$	$0.200\pm 0.031$	$0.0022$	$0.0078$

Appendix

Table 8: Direct attribute decoding from simple-split fMRI responses and Gaussian-feature controls. fMRI accuracies are means

\pm

sample SD across seven participants; Gaussian values summarize the random-feature control. The

p

values compare participant-level fMRI accuracies with nominal chance and are uncorrected.

Input	BLEU-1	BLEU-4	ROUGE-L F1	Token-F1
Real fMRI	$\mathbf{0.2236\pm 0.0199}$	$\mathbf{0.0759\pm 0.0079}$	$\mathbf{0.2522\pm 0.0137}$	$\mathbf{0.2961\pm 0.0193}$
Gaussian fMRI	$0.2049\pm 0.0075$	$0.0609\pm 0.0025$	$0.2298\pm 0.0050$	$0.2756\pm 0.0033$
Shuffled fMRI	$0.1924\pm 0.0031$	$0.0608\pm 0.0014$	$0.2266\pm 0.0038$	$0.2645\pm 0.0032$

Appendix

Table 9: fMRI-to-text decoding with neural-input controls. Values are means

\pm

sample SD across nine participants. Bold indicates significantly higher performance than both controls after Holm correction.

Metric	Real fMRI	Shuffled-fMRI chance
PixCorr	$\mathbf{0.0562\pm 0.0245}$	$0.0272\pm 0.0072$
SSIM	$\mathbf{0.2910\pm 0.0074}$	$0.2828\pm 0.0013$
CLIP-I 2WC	$\mathbf{65.94\pm 8.20\%}$	$50.02\pm 0.91\%$
Inception 2WC	$\mathbf{65.14\pm 8.86\%}$	$49.82\pm 1.03\%$
CLIP-T 2WC	$\mathbf{67.83\pm 8.81\%}$	$49.90\pm 0.96\%$
SigLIP 2WC	$\mathbf{70.63\pm 10.59\%}$	$49.99\pm 0.22\%$

Appendix

Table 10: MindEye1 reconstruction with real and shuffled fMRI on the complex split. Real-fMRI values are means

\pm

sample SD across nine participants; shuffled-fMRI values are means

\pm

SD across 50 group-level shuffle repetitions. Two-way identification accuracies are reported as percentages. Bold indicates significant real-fMRI performance under a one-sided empirical permutation test.

Panel of Table 11 — Table 11: Representative evaluator errors on original reference images. The evaluator correctly identifies color and spatial position but misclassifies semicircle orientation. Identity answers are abbreviated to the straight-edge direction; the full answer format is “semicircle (straight edge at [direction])”. Reference descriptions are shown for illustration and are not provided to the evaluator.

Split	Speech ROI	Language ROI	Remaining ROI
Complex	$1{,}426\pm 933$	$2{,}191\pm 2{,}153$	$38{,}538\pm 5{,}900$
Simple	$1{,}561\pm 1{,}024$	$2{,}348\pm 2{,}460$	$37{,}537\pm 6{,}344$

Appendix

Table 12: ROI sizes after NC-based vertex selection. Values are the mean

\pm

sample standard deviation of vertex counts across participants (complex:

n=9

; simple:

n=7

). The top 15% were selected separately within each candidate region.

Metric	Speech ROI	Language ROI	Remaining ROI	vs. Speech	vs. Language
PixCorr	$0.046\pm 0.019$	$0.049\pm 0.026$	$0.061\pm 0.024$	$0.0747$	$0.0915$
SSIM	$0.327\pm 0.028$	$0.296\pm 0.011$	$0.291\pm 0.011$	$0.9976$	$0.8892$
CLIP-I 2WC	$0.538\pm 0.064$	$0.613\pm 0.069$	$0.657\pm 0.083$	$0.0011$	$0.0136$
Inception 2WC	$0.537\pm 0.062$	$0.602\pm 0.081$	$0.648\pm 0.092$	$0.0030$	$0.0257$
CLIP-T R@1	$0.013\pm 0.013$	$0.026\pm 0.032$	$0.035\pm 0.039$	$0.0488$	$0.1294$
CLIP-T R@5	$0.057\pm 0.055$	$0.092\pm 0.083$	$0.147\pm 0.105$	$0.0056$	$0.0070$

Appendix

Table 13: ROI-restricted MindEye1 reconstruction performance on the complex split (

n=9

). Values are the mean

\pm

sample standard deviation across participants. The final two columns report Benjamini–Hochberg-adjusted

p

-values from one-sided paired

t

-tests assessing whether the remaining ROI outperformed the respective comparison ROI. Correction was applied jointly across all 20 comparisons; adjusted

p<0.05

indicates significant superiority.

Figure 6: Qualitative comparison of mental imagery generation results for ROI sensitivity analysis. (A) Examples from the complex stimuli split. (B) Examples from the simple stimuli split. For each stimulus, we compare the source image with reconstructions generated by LDM, MindEye1, MindEye2, and MindBridge. The results illustrate how different methods recover semantic content and visual structure across natural images and simple shape stimuli.

	Image-image				Image-text
Method	PixCorr	SSIM	Inception	CLIP-I	CLIP-T	SigLIP	AltCLIP	TIFA
LDM	0.040	0.187	58.30%	59.36%	67.42%	68.30%	68.09%	31.99%
MindEye1	0.061	0.291	64.85%	65.71%	68.06%	70.04%	70.33%	30.91%
MindEye2	0.031	0.265	55.06%	56.75%	55.77%	57.85%	57.06%	24.62%
MindBridge	0.040	0.170	63.69%	64.78%	66.70%	67.82%	68.50%	32.06%

Appendix

Table 14: Complex split baseline results under NSL. Metrics are grouped by source image consistency and spoken description alignment.

	Image-image				Image-text
Method	PixCorr	SSIM	Inception	CLIP-I	CLIP-T	SigLIP	AltCLIP	TIFA
LDM	-0.006	0.636	51.90%	52.31%	55.69%	57.12%	56.87%	48.42%
MindEye1	-0.031	0.963	52.35%	51.10%	51.45%	51.15%	51.84%	44.82%
MindEye2	0.003	0.679	54.39%	53.93%	52.04%	53.07%	52.85%	41.15%
MindBridge	-0.008	0.916	56.84%	55.53%	54.72%	57.58%	54.59%	40.61%

Appendix

Table 15: Simple split baseline results under NSL. Metrics are grouped by source image consistency and spoken description alignment.

Complex split: NS input
Method	PixCorr	SSIM	Inception	CLIP-I
LDM	$0.038\pm 0.014$	$0.185\pm 0.005$	$59.10\pm 5.00\%$	$59.60\pm 4.90\%$
MindEye1	$\boldsymbol{0.056\pm 0.025}$	$\boldsymbol{0.291\pm 0.007}$	$65.10\pm 8.90\%$	$\boldsymbol{65.90\pm 8.20\%}$
MindEye2	$0.034\pm 0.017$	$0.262\pm 0.018$	$55.70\pm 4.70\%$	$57.80\pm 4.30\%$
MindBridge	$0.040\pm 0.006$	$0.170\pm 0.006$	$\boldsymbol{65.20\pm 6.50\%}$	$65.80\pm 7.00\%$
Simple split: NS input

Appendix

Table 16: Participant-level variability in source-image consistency metrics. Values are mean

\pm

sample SD across nine participants for the complex split and seven for the simple split. Inception and CLIP-I scores are reported as percentages. Bold marks the highest mean within each split and input setting. NS denotes

\mathrm{ROI}_{m}\setminus\mathrm{ROI}_{s}

, and NSL denotes

\mathrm{ROI}_{m}\setminus(\mathrm{ROI}_{s}\cup\mathrm{ROI}_{l})

Complex split: NS input
Method	CLIP-T	SigLIP	AltCLIP	TIFA
LDM	$66.80\pm 8.30\%$	$68.50\pm 7.60\%$	$67.30\pm 8.40\%$	$\boldsymbol{32.60\pm 4.40\%}$
MindEye1	$\boldsymbol{67.80\pm 8.80\%}$	$\boldsymbol{70.63\pm 10.59\%}$	$\boldsymbol{69.80\pm 9.60\%}$	$30.30\pm 4.90\%$
MindEye2	$57.90\pm 5.30\%$	$59.10\pm 5.90\%$	$58.80\pm 6.50\%$	$24.30\pm 1.90\%$
MindBridge	$67.70\pm 7.00\%$	$69.20\pm 7.20\%$	$69.60\pm 7.50\%$	$32.30\pm 3.30\%$
Simple split: NS input

Appendix

Table 17: Participant-level variability in spoken-description alignment metrics. All metrics are reported as percentages, with values given as mean

\pm

sample SD across nine participants for the complex split and seven for the simple split. Bold marks the highest mean within each split and input setting. NS and NSL follow Table 16 .

Baseline	Training setting	Primary input ROI	Model fitting and reconstruction
LDM	Single-subject	$\mathrm{ROI}_{m}\setminus\mathrm{ROI}_{s}$	Ridge regression with 5-fold cross-validation over the regularization grid; Stable Diffusion reconstruction with 50 denoising steps, guidance scale 7.5, and img2img strength 0.8.
MindEye1	Single-subject	$\mathrm{ROI}_{m}\setminus\mathrm{ROI}_{s}$	Subject-specific training for 240 epochs with CLIP ViT-L/14 targets; MindEye1 reconstruction and candidate-selection pipeline.
MindEye2	Leave-one-participant-out transfer	$\mathrm{ROI}_{m}\setminus\mathrm{ROI}_{s}$	Joint training on non-target participants for 150 epochs, followed by 15 epochs of target-participant fine-tuning; diffusion-prior and blurry reconstruction branches.
MindBridge	Multi-participant joint training	$\mathrm{ROI}_{m}\setminus\mathrm{ROI}_{s}$	Joint training for 600 epochs with reconstruction and cycle-consistency objectives.

Appendix

Table 18: Summary of baseline configurations. The primary input ROI is

\mathrm{ROI}_{m}\setminus\mathrm{ROI}_{s}

for all four methods. Detailed hyperparameters are provided in the model-specific descriptions above; the additional speech/language-excluded input is evaluated in the ROI sensitivity analysis.

Family	Training identities	Validation identities
Letter	F, G, H, J, K, Q, W, X, Y (9)	C, S (2)
Single digit	0, 1, 2, 3, 4, 6, 7, 9 (8)	5, 8 (2)
Digits	103, 194, 372, 431, 566, 653, 740, 918 (8)	286, 829 (2)
Shape	Circle; cross; diamond; five-pointed star; ring; square; three-by-three dot array; triangle; vertical rectangle; semicircles with the straight edge at the bottom, left, or right (12)	Heart; horizontal rectangle; semicircle with the straight edge at the top (3)
Object	T-shirt; backpack; bicycle; book; bookcase; bottle; bottle cap; bow; bus; car; cat; chair; coffee cup; desk lamp; dog; duck; electric fan; forklift; fountain; hair dryer; hammer; laptop; microwave; mirror; mobile phone; motorcycle; mouse; oven; refrigerator; remote control; scissors; sofa; spoon; table; television; toilet; umbrella; watch (38)	Airplane; bed; chicken; goldfish; golf ball; hatchet; jeans; keyboard; knife; sink; toothbrush (11)

Appendix

Table 19: Identity membership of the additional visual-identity-disjoint split. Counts in parentheses indicate the number of identities. Semicircles are described by the location of their straight edge.

fMRI-Based Visual Decoding

See all 29 papers

Apr 15, 2026

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data

Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the NSD-Imagery benchmark. We propose a latent functional alignment (LFA) approach that maps imagery-evoked activity to the pretrained model's semantic content-enriched conditioning space, by adding a simple alignment module, while keeping the original remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a neural retrieval-based augmentation strategy that selects semantically related NSD perception trials from the same participants. Across four subjects, LFA consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.

Fabrizio Spera, Tommaso Boccato, Michal Olak +2

Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA

Neural Decoding Visual Neuroscience Neuroimaging Latent Space Alignment fMRI-Based Visual Decoding

Ask PDF

May 16, 2026

MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery

To be useful for downstream applications, vision decoding models that are trained to reconstruct seen images from human brain activity must be able to generalize to internally generated visual representations, i.e., mental images. In an analysis of the recently released NSD-Imagery dataset, we demonstrated that while some modern vision decoders can perform quite well on mental image reconstruction, some fail, and that state-of-the-art (SOTA) performance on seen image reconstruction is no guarantee of SOTA performance on mental image reconstruction. Motivated by these findings, we developed MIRAGE, a method explicitly designed to train on vision datasets and cross-decode mental images from brain activity. MIRAGE employs a linear backbone and multi-modal text and image features as input to a diffusion model. Feature metrics and human raters establish MIRAGE as SOTA for mental image reconstruction on the NSD-Imagery benchmark. With ablation analysis we show that mental image reconstruction works best when decoders use image features with relatively few dimensions and include guidance from text-based and both high- and low-level image-based features. Our work indicates that--given the right architecture--existing large-scale datasets using external stimuli are viable training data for decoding mental images, and warrant optimism about the future success and utility of mental image reconstruction.

Reese Kneeland, Cesar Kadir Torrico Villanueva, Jordyn Ojeda +4

University of Minnesota, Minneapolis, Minnesota, United States of America · Medical AI Research Center (MedARC) · Stanford University, Palo Alto, California, United States of America +2

Neural Decoding Image Reconstruction Multimodal Diffusion Models fMRI-Based Visual Decoding

Ask PDF

May 28, 2026

Brain-IT-VQA: From Brain Signals to Answers

Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Building on the Brain Interaction Transformer (Brain-IT), our method decodes language tokens from brain activity and integrates them with a language model to answer visual questions. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.

Roman Beliy, Matias Cosarinsky, Oliver Heinimann +2

Weizmann Institute of Science

Visual Question Answering Neural Decoding Visual Neuroscience Neuroimaging fMRI-Based Visual Decoding Functional Magnetic Resonance Imaging

Ask PDF

Abstract

fMRI-Based Visual Decoding

Explore similar work

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data

MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery

Brain-IT-VQA: From Brain Signals to Answers