SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance

Minghan Yang; Lan Yang; Ke Li; Honggang Zhang; Kaiyue Pang; Yizhe Song

cs.CVFeb 25, 2026

SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance

Authors: Minghan Yang, Lan Yang, Ke Li, Honggang Zhang, Kaiyue Pang, Yizhe Song

Organizations: Beijing University of Posts and Telecommunications · University of Surrey

Abstract

Reconstructing dynamic visual experiences from brain activity provides a compelling avenue for exploring the neural mechanisms of human visual perception. While recent progress in fMRI-based image reconstruction has been notable, extending this success to video reconstruction remains a significant challenge. Current fMRI-to-video reconstruction approaches consistently encounter two major shortcomings: (i) inconsistent visual representations of salient objects across frames, leading to appearance mismatches; (ii) poor temporal coherence, resulting in motion misalignment or abrupt frame transitions. To address these limitations, we introduce SemVideo, a novel fMRI-to-video reconstruction framework guided by hierarchical semantic information. At the core of SemVideo is SemMiner, a hierarchical guidance module that constructs three levels of semantic cues from the original video stimulus: static anchor descriptions, motion-oriented narratives, and holistic summaries. Leveraging this semantic guidance, SemVideo comprises three key components: a Semantic Alignment Decoder that aligns fMRI signals with CLIP-style embeddings derived from SemMiner, a Motion Adaptation Decoder that reconstructs dynamic motion patterns using a novel tripartite attention fusion architecture, and a Conditional Video Render that leverages hierarchical semantic guidance for video reconstruction. Experiments conducted on the CC2017 and HCP datasets demonstrate that SemVideo achieves superior performance in both semantic alignment and temporal consistency, setting a new state-of-the-art in fMRI-to-video reconstruction.

Figures & tables

Figure 1: Top: While a subject watches an original video stimulus, their brain activity is recorded via fMRI. Middle: Reconstructed results from previous methods, which suffer from two issues: Appearance Mismatch and Motion Misalignment . Bottom: Reconstructed result from SemVideo, achieves both semantic consistency (reconstructing the “kitten”) and motion coherence (matching dynamic actions like “crouching” and “turning”) by leveraging hierarchical semantic descriptions as intermediate targets to guide the fMRI signal decoding process.

Figure 2: Overview of the SemVideo training pipeline. In the first stage, the Semantic Alignment Decoder is trained to map fMRI signals to three levels of semantic targets, denoted as $Z(C_{L})$ . In the second stage, the Motion Adaptation Decoder is trained to utilize the predicted motion semantics $\hat{Z}(C_{\text{motion}})$ to refine the latent embedding of each reconstructed frame.

Figure 3: Overview of the SemVideo inference pipeline. fMRI signals are first decoded into $\hat{Z}(C_{\text{L}})$ by the SAD. $\hat{Z}(C_{\text{motion}})$ conditions the MAD to refine frame embeddings $\hat{E}(x)$ , which are passed through a VAE decoder, generating a blurry video. $\hat{E}(x)$ and $\hat{Z}(C_{\text{anchor}})$ guide the SD model to generate anchor frame, combined with the blurry video and $\hat{Z}(C_{\text{holi}})$ , is fed into a T2V model, yielding final reconstruction.

Method	Semantic-Level					Pixel-Level			ST-Level
Method	2-way-I $\uparrow$	2-way-V $\uparrow$	50-way-I $\uparrow$	50-way-V $\uparrow$	VIFI-score $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	Hue-pcc $\uparrow$	CLIP $\uparrow$	EPE $\downarrow$
Nishimoto [ 46 ] (Current Biology’11)	0.742	–	–	–	–	0.119	8.383	0.737	–	–
Wen [ 76 ] (Cerebral Cortex’17)	0.771	0.070	–	0.166	–	0.130	8.301	0.637	–	–
Kupershmidt [ 35 ] (arXiv’22)	0.769	0.768	0.179	–	0.591	0.108	11.030	0.583	0.399	6.344
f-CVGAN [ 71 ] (Cerebral Cortex’22)	0.777	0.721	–	–	0.592	0.108	11.430	0.583	0.399	6.344
Mind-video [ 10 ] (NeurIPS’23)	0.797	0.848	0.174	0.197	0.593	0.177	8.868	0.768	0.409	6.125

Table 1: Quantitative comparison of reconstruction results on the CC2017 dataset. All metrics are averaged across all samples and three subjects, with the best results highlighted in bold and the second-best results underlined. 100 sample sets were constructed using the bootstrap method for hypothesis testing. Colors reflect statistical significance compared to our model.

p<0.0001

( blue );

p<0.01

( pink );

p<0.05

( yellow );

p>0.05

( green )

Figure 4: Qualitative comparison of reconstruction results on the CC2017 dataset between previous methods and our proposed SemVideo . Reconstructions generated by SemVideo are highlighted with red boxes.

Method	Semantic		Pixel		ST-Level
Method	2-way-I $\uparrow$	2-way-V $\uparrow$	PSNR $\uparrow$	Hue-pcc $\uparrow$	CLIP $\uparrow$	EPE $\downarrow$
Nishimoto	0.658	–	11.316	0.645	–	–
Wen	0.702	–	10.197	0.727	–	–
Mind-video	0.779	0.769	9.275	0.793	0.499	9.29
Mind-Animator	0.786	0.778	11.233	0.829	0.511	7.08
Ours	0.815	0.796	9.336	0.858	0.523	4.05

Table 2: Quantitative comparison on the HCP dataset.

Method	Semantic-Level					Pixel-Level			ST-Level
Method	2-way-I $\uparrow$	2-way-V $\uparrow$	50-way-I $\uparrow$	50-way-V $\uparrow$	VIFI-score $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	Hue-pcc $\uparrow$	CLIP $\uparrow$	EPE $\downarrow$
Ours (full)	0.826	0.860	0.211	0.239	0.590	0.330	9.626	0.841	0.502	4.768
w/o $C_{anchor}$	0.786	0.808	0.158	0.147	0.534	0.361	9.454	0.835	0.488	4.796
w/o $C_{holi}$	0.819	0.849	0.205	0.221	0.584	0.299	9.492	0.834	0.490	4.859
w/o $C_{motion}$	0.818	0.846	0.207	0.216	0.583	0.285	9.353	0.741	0.481	4.930

Table 3: Ablation study evaluating the impact of removing different semantic descriptions—

C_{anchor}

C_{holi}

, and

C_{motion}

—from the decoding targets of the Semantic Alignment Decoder (SAD). All results are reported from subj01 of the CC2017 dataset.

Figure 5: The figure presents two sets of results from subj01 of the CC2017 dataset. On the left, the results of a shuffle test are shown. On the right, the results of an ablation study are displayed, evaluating the impact of the $C_{motion}$ and $MAD$ guidance components.

Figure 6: Importance visualization of different ROIs of subject 1 on CC2017, based on the fitted weights from the first layer of the SAD. Weights from each module are averaged and normalized to the $[0,1]$ range for comparison. Please refer to Appendix D for more subjects’ importance visualization.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Prompt templates used in SemMiner and qualitative comparison of captions generated by BLIP2 and SemMiner . SemMiner employs four distinct prompt templates to generate hierarchical semantic descriptions for a given video stimulus. The process begins with the Basic Prompt $P_{basic}$ , which establishes a core semantic summary to anchor subsequent generations. This is followed by three specialized prompts in stage 2: (i) $P_{anchor}$ , which produces a detailed description of the initial frame, optimized for guiding text-to-image models; (ii) $P_{motion}$ , which focuses on extracting dynamic information such as object motion, direction, speed, and posture changes; and (iii) $P_{holi}$ , which integrates static and dynamic elements to form a coherent, high-level narrative of the entire video. Compared to BLIP2, which tends to generate short and static descriptions, SemMiner provides richer and more perspective-specific captions.

Figure 8: Video generated from BLIP2 caption(upper) and SemMiner(lower). The middle is the target video.

caption type	BLIP2	$C_{anchor}$	$C_{motion}$	$C_{holi}$
number	10.6	32.8	41.1	52.8

Appendix

Table 4: Average number of tokens in the descriptions generated by BLIP2 and SemMiner on CC2017 Datasets.

Pair	BLEU-1	BLEU-4	METEOR	ROUGE-1	ROUGE-L	CLIP-S
$C_{anchor}$ v.s. $C_{motion}$	0.171	0.022	0.186	0.275	0.207	0.655
$C_{anchor}$ v.s. $C_{holi}$	0.197	0.028	0.222	0.304	0.225	0.665
$C_{motion}$ v.s. $C_{holi}$	0.270	0.067	0.292	0.400	0.293	0.744

Appendix

Table 5: Semantic Similarity Experiments of different caption pairs. Lower values across all metrics indicate greater semantic dissimilarity.

MLLM	Semantic-Level					Pixel-Level			ST-Level
MLLM	2-way-I $\uparrow$	2-way-V $\uparrow$	50-way-I $\uparrow$	50-way-V $\uparrow$	VIFI-score $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	Hue-pcc $\uparrow$	CLIP $\uparrow$	EPE $\downarrow$
VideoLlama	0.826	0.860	0.211	0.239	0.590	0.330	9.626	0.841	0.502	4.768
Qwen2.5-vl	0.821	0.863	0.203	0.242	0.589	0.299	9.03	0.858	0.496	4.40
Llava-Video	0.815	0.855	0.207	0.228	0.591	0.309	8.96	0.861	0.493	4.65

Appendix

Table 6: Ablation study on the CC2017 dataset investigating performance variations when using different VLLMs within the SemMiner framework.

Figure 9: More reconstruction results on the CC2017 dataset from three subjects.

Figure 10: More reconstruction results on the HCP dataset from three subjects.

Figure 11: Reconstruction failure cases.

Panel of Figure 12 — Figure 12: Importance visualization of different ROIs for Subject 2, based on the fitted weights from the first layer of the SFD. Weights from each module are averaged and normalized to the $[0,1]$ range for comparison.

Explore similar work

Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction
May 14, 2026Yujie Wei, Chenglong Ma, Jianxiong Gao +6Neural Decoding Neuroimaging
1Fudan University · 2Alibaba Group · 3Ant Group
Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability
Oct 9, 2026Haonan Wang, Yuchen Wang, Jingyu Lu +4Neural Decoding Image Reconstruction
The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China
Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping
Nov 14, 2025Guowei Zhang, Yun Zhao, Kai Sun +4Neural Decoding Conditional Image Generation
Dept of Data Science and AI, Monash University, Melbourne, Australia · Brain and Mind Centre, University of Sydney, Sydney, Australia · School of Psychological Sciences, Monash University, Melbourne, Australia

May 14, 2026J/K move · Enter open · S save

Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction

Yujie Wei, Chenglong Ma, Jianxiong Gao, Chenhui Wang+5

1Fudan University · 2Alibaba Group · 3Ant Group

Neural Decoding Neuroimaging

Open paper AskPDF arXiv

Reconstructing dynamic visual experiences as videos from functional magnetic resonance imaging (fMRI) is pivotal for advancing the understanding of neural processes. However, current fMRI-to-video reconstruction methods are hindered by a semantic gap between noisy fMRI signals and the rich content of videos, stemming from a reliance on incomplete semantic embeddings that neither capture video-specific cues (e.g., actions) nor integrate prior knowledge. To this end, we draw inspiration from the dual-pathway processing mechanism in human brain and introduce CineNeuron, a novel hierarchical framework for semantically enhanced video reconstruction from fMRI signals with two synergistic stages. First, a bottom-up semantic enrichment stage maps fMRI signals to a rich embedding space that comprehensively captures textual semantics, image contents, action concepts, and object categories. Second, a top-down memory integration stage utilizes the proposed Mixture-of-Memories method to dynamically select relevant "memories" from previously seen data and fuse them with the fMRI embedding to refine the video reconstruction. Extensive experimental results on two fMRI-to-video benchmarks demonstrate that CineNeuron surpasses state-of-the-art methods across various metrics.

Abstract

Explore similar work

Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping