Reconstructing dynamic visual experiences from brain activity provides a compelling avenue for exploring the neural mechanisms of human visual perception. While recent progress in fMRI-based image reconstruction has been notable, extending this success to video reconstruction remains a significant challenge. Current fMRI-to-video reconstruction approaches consistently encounter two major shortcomings: (i) inconsistent visual representations of salient objects across frames, leading to appearance mismatches; (ii) poor temporal coherence, resulting in motion misalignment or abrupt frame transitions. To address these limitations, we introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information. At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Leveraging this semantic guidance, SemVideo comprises three key components: a Semantic Alignment Decoder that aligns fMRI signals with CLIP-style embeddings derived from SemMiner, a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture, and a Conditional Video Render that leverages hierarchical semantic guidance for video reconstruction. Experiments conducted on the CC2017 and HCP datasets demonstrate that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.
Figures & tables
Figure 1: Top: While a subject watches an original video stimulus, their brain activity is recorded via fMRI. Middle: Reconstructed results from previous methods, which suffer from two issues: Appearance Mismatch and Motion Misalignment . Bottom: Reconstructed result from SemVideo, achieves both semantic consistency (reconstructing the “kitten”) and motion coherence (matching dynamic actions like “crouching” and “turning”) by leveraging hierarchical semantic descriptions as intermediate targets to guide the fMRI signal decoding process.
Figure 2: Overview of the SemVideo training pipeline. In the first stage, the Semantic Alignment Decoder is trained to map fMRI signals to three levels of semantic targets, denoted as Z(CL) . In the second stage, the Motion Adaptation Decoder is trained to utilize the predicted motion semantics Z^(Cmotion) to refine the latent embedding of each reconstructed frame.
Figure 3: Overview of the SemVideo inference pipeline. fMRI signals are first decoded into Z^(CL) by the SAD. Z^(Cmotion) conditions the MAD to refine frame embeddings E^(x) , which are passed through a VAE decoder, generating a blurry video. E^(x) and Z^(Canchor) guide the SD model to generate anchor frame, combined with the blurry video and Z^(Choli) , is fed into a T2V model, yielding final reconstruction.
Method
Semantic-Level
Pixel-Level
ST-Level
2-way-I ↑
2-way-V ↑
50-way-I ↑
50-way-V ↑
VIFI-score ↑
SSIM ↑
PSNR ↑
Hue-pcc ↑
CLIP ↑
EPE ↓
Nishimoto [ 46 ] (Current Biology’11)
0.742
–
–
–
–
0.119
8.383
0.737
–
–
Wen [ 76 ] (Cerebral Cortex’17)
0.771
0.070
–
0.166
–
0.130
8.301
0.637
–
–
Kupershmidt [ 35 ] (arXiv’22)
0.769
0.768
0.179
–
0.591
0.108
11.030
0.583
0.399
6.344
f-CVGAN [ 71 ] (Cerebral Cortex’22)
0.777
0.721
–
–
0.592
0.108
11.430
0.583
0.399
6.344
Mind-video [ 10 ] (NeurIPS’23)
0.797
0.848
0.174
0.197
0.593
0.177
8.868
0.768
0.409
6.125
Table 1: Quantitative comparison of reconstruction results on the CC2017 dataset. All metrics are averaged across all samples and three subjects, with the best results highlighted in bold and the second-best results underlined. 100 sample sets were constructed using the bootstrap method for hypothesis testing. Colors reflect statistical significance compared to our model. p<0.0001 ( blue ); p<0.01 ( pink ); p<0.05 ( yellow ); p>0.05 ( green )
Figure 4: Qualitative comparison of reconstruction results on the CC2017 dataset between previous methods and our proposed SemVideo . Reconstructions generated by SemVideo are highlighted with red boxes.
Method
Semantic
Pixel
ST-Level
2-way-I ↑
2-way-V ↑
PSNR ↑
Hue-pcc ↑
CLIP ↑
EPE ↓
Nishimoto
0.658
–
11.316
0.645
–
–
Wen
0.702
–
10.197
0.727
–
–
Mind-video
0.779
0.769
9.275
0.793
0.499
9.29
Mind-Animator
0.786
0.778
11.233
0.829
0.511
7.08
Ours
0.815
0.796
9.336
0.858
0.523
4.05
Table 2: Quantitative comparison on the HCP dataset.
Method
Semantic-Level
Pixel-Level
ST-Level
2-way-I ↑
2-way-V ↑
50-way-I ↑
50-way-V ↑
VIFI-score ↑
SSIM ↑
PSNR ↑
Hue-pcc ↑
CLIP ↑
EPE ↓
Ours (full)
0.826
0.860
0.211
0.239
0.590
0.330
9.626
0.841
0.502
4.768
w/o Canchor
0.786
0.808
0.158
0.147
0.534
0.361
9.454
0.835
0.488
4.796
w/o Choli
0.819
0.849
0.205
0.221
0.584
0.299
9.492
0.834
0.490
4.859
w/o Cmotion
0.818
0.846
0.207
0.216
0.583
0.285
9.353
0.741
0.481
4.930
Table 3: Ablation study evaluating the impact of removing different semantic descriptions— Canchor , Choli , and Cmotion —from the decoding targets of the Semantic Alignment Decoder (SAD). All results are reported from subj01 of the CC2017 dataset.
Figure 5: The figure presents two sets of results from subj01 of the CC2017 dataset. On the left, the results of a shuffle test are shown. On the right, the results of an ablation study are displayed, evaluating the impact of the Cmotion and MAD guidance components.
Figure 6: Importance visualization of different ROIs of subject 1 on CC2017, based on the fitted weights from the first layer of the SAD. Weights from each module are averaged and normalized to the [0,1] range for comparison. Please refer to Appendix D for more subjects’ importance visualization.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Prompt templates used in SemMiner and qualitative comparison of captions generated by BLIP2 and SemMiner . SemMiner employs four distinct prompt templates to generate hierarchical semantic descriptions for a given video stimulus. The process begins with the Basic Prompt Pbasic , which establishes a core semantic summary to anchor subsequent generations. This is followed by three specialized prompts in stage 2: (i) Panchor , which produces a detailed description of the initial frame, optimized for guiding text-to-image models; (ii) Pmotion , which focuses on extracting dynamic information such as object motion, direction, speed, and posture changes; and (iii) Pholi , which integrates static and dynamic elements to form a coherent, high-level narrative of the entire video. Compared to BLIP2, which tends to generate short and static descriptions, SemMiner provides richer and more perspective-specific captions.
Figure 8: Video generated from BLIP2 caption(upper) and SemMiner(lower). The middle is the target video.
caption type
BLIP2
Canchor
Cmotion
Choli
number
10.6
32.8
41.1
52.8
Appendix
Table 4: Average number of tokens in the descriptions generated by BLIP2 and SemMiner on CC2017 Datasets.
Pair
BLEU-1
BLEU-4
METEOR
ROUGE-1
ROUGE-L
CLIP-S
Canchor v.s. Cmotion
0.171
0.022
0.186
0.275
0.207
0.655
Canchor v.s. Choli
0.197
0.028
0.222
0.304
0.225
0.665
Cmotion v.s. Choli
0.270
0.067
0.292
0.400
0.293
0.744
Appendix
Table 5: Semantic Similarity Experiments of different caption pairs. Lower values across all metrics indicate greater semantic dissimilarity.
MLLM
Semantic-Level
Pixel-Level
ST-Level
2-way-I ↑
2-way-V ↑
50-way-I ↑
50-way-V ↑
VIFI-score ↑
SSIM ↑
PSNR ↑
Hue-pcc ↑
CLIP ↑
EPE ↓
VideoLlama
0.826
0.860
0.211
0.239
0.590
0.330
9.626
0.841
0.502
4.768
Qwen2.5-vl
0.821
0.863
0.203
0.242
0.589
0.299
9.03
0.858
0.496
4.40
Llava-Video
0.815
0.855
0.207
0.228
0.591
0.309
8.96
0.861
0.493
4.65
Appendix
Table 6: Ablation study on the CC2017 dataset investigating performance variations when using different VLLMs within the SemMiner framework.
Figure 9: More reconstruction results on the CC2017 dataset from three subjects.
Figure 10: More reconstruction results on the HCP dataset from three subjects.
Figure 11: Reconstruction failure cases.
Figure 12: Importance visualization of different ROIs for Subject 2, based on the fitted weights from the first layer of the SFD. Weights from each module are averaged and normalized to the [0,1] range for comparison.
Reconstructing dynamic visual experiences as videos from functional magnetic resonance imaging (fMRI) is pivotal for advancing the understanding of neural processes. However, current fMRI-to-video reconstruction methods are hindered by a semantic gap between noisy fMRI signals and the rich content of videos, stemming from a reliance on incomplete semantic embeddings that neither capture video-specific cues (e.g., actions) nor integrate prior knowledge. To this end, we draw inspiration from the dual-pathway processing mechanism in human brain and introduce CineNeuron, a novel hierarchical framework for semantically enhanced video reconstruction from fMRI signals with two synergistic stages. First, a bottom-up semantic enrichment stage maps fMRI signals to a rich embedding space that comprehensively captures textual semantics, image contents, action concepts, and object categories. Second, a top-down memory integration stage utilizes the proposed Mixture-of-Memories method to dynamically select relevant "memories" from previously seen data and fuse them with the fMRI embedding to refine the video reconstruction. Extensive experimental results on two fMRI-to-video benchmarks demonstrate that CineNeuron surpasses state-of-the-art methods across various metrics.
Decoding visual experience from fMRI is important for understanding human perception and developing brain-computer interfaces. Recent generative decoding methods can reconstruct plausible images and videos, but visual plausibility alone does not ensure that the output is faithfully grounded in brain activity. In this work, we propose Neurons++, a visual-stream fMRI reconstruction framework for both static image and dynamic video reconstruction. Compared with our conference version Neurons, Neurons++ extends the paradigm from video-only reconstruction to visual-stream reconstruction for both images and videos, and replaces parallel task conditioning with serial cortex-inspired decoding. The model first uses image-fMRI pretraining to obtain a stable spatial-semantic initialization, and then performs subject-specific fine-tuning for image and video reconstruction separately. The serial design organizes fMRI-derived representations into explicit spatial, semantic, language-level, and temporal cues. Extensive experiments show that Neurons++ achieves state-of-the-art video reconstruction and competitive image reconstruction across standard benchmarks. Beyond reconstruction fidelity, neural-to-visual analyses indicate that Neurons++ exhibits qualitative directional controllability under neural perturbation, that its fMRI encoder read-out is sensitive to stimulus-relevant cortical features, and that it supports ROI-guided semantic modulation. Code and model weights are available at: https://github.com/xmed-lab/NeuronsPlusPlus.
Haonan Wang, Yuchen Wang, Jingyu Lu +4
The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China
Reconstructing natural images from fMRI requires bridging neural activity with both the structural and semantic representations used by modern generative models. Existing diffusion-based decoders often condition on a single global fMRI embedding, which limits their ability to exploit the hierarchical organization of the visual cortex and makes the contribution of different visual areas difficult to inspect. We propose Hi-DREAM, a brain-inspired hierarchical diffusion framework that structures fMRI conditioning according to early, middle, and late visual Regions of Interest (ROI) streams. A ROI adapter converts these streams into a multi-scale cortical pyramid, and a lightweight ROI-conditioned ControlNet injects the resulting anatomy-aware priors into matched U-Net depths during denoising. Experiments on the Natural Scenes Dataset (NSD) show that Hi-DREAM achieves state-of-the-art high-level semantic reconstruction while retaining strong low-level structure. Further ablation and attribution analyses show that the proposed hierarchy-aware conditioning is effective, and that different ROI streams provide complementary, inspectable contributions to reconstruction.
Guowei Zhang, Yun Zhao, Kai Sun +4
Dept of Data Science and AI, Monash University, Melbourne, Australia · Brain and Mind Centre, University of Sydney, Sydney, Australia · School of Psychological Sciences, Monash University, Melbourne, Australia