Modeling Spatiotemporal Neural Frames for High Resolution Brain Dynamic

Wanying Qu; Jianxiong Gao; Wei Wang; Yanwei Fu

eess.IVMar 25, 2026

Modeling Spatiotemporal Neural Frames for High Resolution Brain Dynamic

Authors: Wanying Qu, Jianxiong Gao, Wei Wang, Yanwei Fu

Organizations: Fudan University · Southern University of Science and Technology · Shanghai Innovation Institute

Abstract

Capturing dynamic spatiotemporal neural activity is essential for understanding large-scale brain mechanisms. Functional magnetic resonance imaging (fMRI) provides high-resolution cortical representations that form a strong basis for characterizing fine-grained brain activity patterns. The high acquisition cost of fMRI limits large-scale applications, therefore making high-quality fMRI reconstruction a crucial task. Electroencephalography (EEG) offers millisecond-level temporal cues that complement fMRI. Leveraging this complementarity, we present an EEG-conditioned framework for reconstructing dynamic fMRI as continuous neural sequences with high spatial fidelity and strong temporal coherence at the cortical-vertex level. To address sampling irregularities common in real fMRI acquisitions, we incorporate a null-space intermediate-frame reconstruction, enabling measurement-consistent completion of arbitrary intermediate frames and improving sequence continuity and practical applicability. Experiments on the CineBrain dataset demonstrate superior voxel-wise reconstruction quality and robust temporal consistency across whole-brain and functionally specific regions. The reconstructed fMRI also preserves essential functional information, supporting downstream visual decoding tasks. This work provides a new pathway for estimating high-resolution fMRI dynamics from EEG and advances multimodal neuroimaging toward more dynamic brain activity modeling.

Figures & tables

Figure 2: Architecture and inference workflow of our framework. Top: End-to-end inference pipeline, where EEG features condition a diffusion-transformer-based reconstruction process to produce spatiotemporal fMRI sequences. Bottom: (a) Transformer-based denoising network that jointly models vertex-level tokens with sequence-level EEG cross-attention to reconstruct coherent fMRI trajectories. (b) Null-space sampling module (InterRecon) that enforces measurement consistency during reverse diffusion, enabling intermediate frame reconstruction without retraining.

Frame	Method	Whole Brain			Visual			Visual+Audio
		MSE $\downarrow$	r $\uparrow$	cos $\uparrow$	MSE $\downarrow$	r $\uparrow$	cos $\uparrow$	MSE $\downarrow$	r $\uparrow$	cos $\uparrow$
3	CNN-TC [ 16 ]	0.302	0.808	0.832	0.212	0.823	0.873	0.217	0.836	0.875
	CNN-TAG [ 2 ]	0.297	0.812	0.835	0.207	0.828	0.879	0.212	0.840	0.878
	E2FNet [ 21 ]	0.288	0.818	0.833	0.204	0.831	0.880	0.205	0.844	0.882
	E2FGAN [ 21 ]	0.280	0.819	0.840	0.196	0.832	0.882	0.203	0.846	0.884
	Ours	0.282	0.822	0.847	0.196	0.834	0.886	0.200	0.848	0.887

Table 1: Comparison of fMRI Reconstruction Performance. We compare our method against baseline approaches across different frame lengths and spatial regions (whole brain, visual cortex, and visual–audio cortex). Results are averaged over six subjects.

Method	MSE $\downarrow$	$r$ $\uparrow$	Cos $\uparrow$
Linear	0.280	0.830	0.851
Ours w/o null space	0.272	0.839	0.852
Ours w/ null space	0.250	0.852	0.865

Table 2: Comparison of interpolation and null-space strategies. We compare linear interpolation , our model without null-space sampling , and our full model that incorporates null-space sampling for enhanced temporal and structural consistency.

Figure 3: Comparison of intermediate fMRI frame reconstruction (InterRecon) performance across different frame lengths and methods.

Figure 4: Vertex-level cortical surface visualization of reconstructed fMRI alongside ground truth. The top row shows our reconstructed fMRI (Recon.); the middle row shows the ground-truth fMRI (GT); and the bottom row displays the reconstruction error map; the color scale indicates BOLD signal intensity.

Figure 5: Functional validation through visual decoding. Comparison between video frames decoded from our reconstructed fMRI (Pred.) and the corresponding ground-truth video frames (GT) using the CineSync-f decoder.

Figure S1 : Functional validation through visual decoding. Comparison among the original stimulus video frames Video (GT), the frames decoded from ground-truth fMRI Video (Real fMRI), and the frames decoded from our reconstructed fMRI Video (Pred. fMRI) using the CineSync-f decoder. These three-way visual comparisons further illustrate the preservation of semantic content in the reconstructed fMRI.

Figure S2 : Functional validation through visual decoding. Comparison among the original stimulus video frames Video (GT), the frames decoded from ground-truth fMRI Video (Real fMRI), and the frames decoded from our reconstructed fMRI Video (Pred. fMRI) using the CineSync-f decoder. These three-way visual comparisons further illustrate the preservation of semantic content in the reconstructed fMRI.

Subject	MSE	$\mathbf{r}$	Cos
sub1	0.272 $\pm$ 0.07	0.839 $\pm$ 0.04	0.852 $\pm$ 0.03
sub2	0.189 $\pm$ 0.03	0.865 $\pm$ 0.03	0.890 $\pm$ 0.03
sub3	0.285 $\pm$ 0.03	0.843 $\pm$ 0.06	0.850 $\pm$ 0.06
sub4	0.346 $\pm$ 0.09	0.781 $\pm$ 0.08	0.815 $\pm$ 0.06
sub5	0.288 $\pm$ 0.08	0.812 $\pm$ 0.04	0.839 $\pm$ 0.05
sub6	0.314 $\pm$ 0.149	0.794 $\pm$ 0.06	0.834 $\pm$ 0.06

Table S1 : Subject-wise dynamic fMRI reconstruction performance (3-frame).

Subject	MSE	$\mathbf{r}$	Cos
sub1	0.271 $\pm$ 0.05	0.840 $\pm$ 0.03	0.853 $\pm$ 0.03
sub2	0.188 $\pm$ 0.02	0.865 $\pm$ 0.03	0.890 $\pm$ 0.03
sub3	0.284 $\pm$ 0.09	0.843 $\pm$ 0.04	0.850 $\pm$ 0.04
sub4	0.342 $\pm$ 0.19	0.780 $\pm$ 0.07	0.816 $\pm$ 0.06
sub5	0.275 $\pm$ 0.05	0.819 $\pm$ 0.03	0.846 $\pm$ 0.05
sub6	0.303 $\pm$ 0.11	0.797 $\pm$ 0.04	0.838 $\pm$ 0.04

Table S2 : Subject-wise dynamic fMRI reconstruction performance (10-frame).

Subject	MSE	$\mathbf{r}$	Cos
sub1	0.283 $\pm$ 0.03	0.832 $\pm$ 0.02	0.846 $\pm$ 0.02
sub2	0.196 $\pm$ 0.01	0.857 $\pm$ 0.02	0.883 $\pm$ 0.02
sub3	0.290 $\pm$ 0.07	0.841 $\pm$ 0.03	0.849 $\pm$ 0.04
sub4	0.344 $\pm$ 0.10	0.769 $\pm$ 0.04	0.807 $\pm$ 0.04
sub5	0.280 $\pm$ 0.04	0.814 $\pm$ 0.02	0.842 $\pm$ 0.04
sub6	0.294 $\pm$ 0.104	0.802 $\pm$ 0.03	0.841 $\pm$ 0.04

Table S3 : Subject-wise dynamic fMRI reconstruction performance (30-frame).

	LinearAE Reconstruction	EEGtofMRI
dim	MSE $\downarrow$	MSE $\downarrow$	r $\uparrow$	Cos $\uparrow$
512	0.0045	0.288	0.814	0.830
1024	0.0043	0.282	0.822	0.847
2048	0.0042	0.285	0.820	0.843

Table S4 : Ablation on latent dimensionality.

Explore similar work

DD-INR: Dynamics-Driven Implicit Neural Representation for Accelerated Whole-Brain Functional MRI Reconstruction
Jun 9, 2026Qiaoxin Li, Caini Pan, Pierre-Antoine Comby +2Implicit Neural Representations Accelerated MRI Reconstruction
MIND · MIND, Inria, Palaiseau, France · Neurospin, CEA Paris Saclay, France +3
MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation
Aug 29, 2026Xiangyu Liu, Zeting Yan, Zhitong Yin +2Neuroimaging Representation Learning
Beijing Normal-Hong Kong Baptist University · The University of Hong Kong · Peking University
SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance
Feb 25, 2026Minghan Yang, Lan Yang, Ke Li +3Neural Decoding Temporal Consistency in Video Generation
Beijing University of Posts and Telecommunications · University of Surrey

Jun 9, 2026J/K move · Enter open · S save

DD-INR: Dynamics-Driven Implicit Neural Representation for Accelerated Whole-Brain Functional MRI Reconstruction

Qiaoxin Li, Caini Pan, Pierre-Antoine Comby, Chaithya Giliyar+1

MIND · MIND, Inria, Palaiseau, France · Neurospin, CEA Paris Saclay, France+3

Implicit Neural Representations Accelerated MRI Reconstruction Magnetic Resonance Imaging Neuroimaging

Open paper AskPDF arXiv

Accelerated acquisition of fMRI enables enhanced detection of neurovascular (BOLD) activity in the brain, but image reconstruction becomes challenging with high k-space undersampling: Task-evoked BOLD signals are small in magnitude, which traditional anatomical MRI reconstruction methods fail to recover, as they favor spatial accuracy over temporal fidelity. We present DD-INR, a Dynamics-Driven Implicit Neural Representation framework tailored for accelerated fMRI that benefits from incoherent time-varying sampling and a tailored spatiotemporal prior, outperforming traditional methods, demonstrated in simulation and in-vivo acquisition, both in terms of image quality and retrieval of activation patterns. DD-INR achieves this by splitting the fMRI data into a static background and a temporally varying dynamic component, representing only the dynamics with a dedicated INR, thereby focusing the model's capacity on activation-relevant changes while remaining compact. In general, DD-INR provides a promising framework for accelerated fMRI reconstruction, with the potential to improve the sensitivity and robustness of fMRI studies within practical scan time limits. The source code is available at https://github.com/JoosenLi/DD-INR.

Abstract

Explore similar work

DD-INR: Dynamics-Driven Implicit Neural Representation for Accelerated Whole-Brain Functional MRI Reconstruction

MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation

SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance