BrainCast: A Spatio-Temporal Forecasting Model for Whole-Brain fMRI Time Series Prediction
Authors: Yunlong Gao, Jinbo Yang, Li Xiao, Haiye Huo, Yang Ji, Hao Wang, Aiying Zhang, Yu-Ping Wang
Organizations: Institute of Advanced Technology, University of Science and Technology of China, Hefei 230052, China. · MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China, Hefei 230052, China. · School of Mathematics and Computer Sciences, Nanchang University, Nanchang 330031, China. · Anhui Province Key Laboratory of Biomedical Imaging and Intelligent Processing, HFCNS Institute of Artificial Intelligence, Hefei 230088, China. · School of Data Science, University of Virginia, Charlottesville, VA 22903, USA. · Department of Biomedical Engineering, Tulane University, New Orleans, LA 70118, USA.
Functional magnetic resonance imaging (fMRI) enables noninvasive investigation of brain function, while short clinical scan durations, arising from human and non-human factors, usually lead to reduced data quality and limited statistical power for neuroimaging research. In this paper, we propose BrainCast, a novel spatio-temporal forecasting framework specifically tailored for whole-brain fMRI time series forecasting, to extend informative fMRI time series without additional data acquisition. It formulates fMRI time series forecasting as a multivariate time series prediction task and jointly models temporal dynamics within regions of interest (ROIs) and spatial interactions across ROIs. Specifically, BrainCast integrates a Spatial Interaction Awareness module to characterize inter-ROI dependencies via embedding every ROI time series as a token, a Temporal Feature Refinement module to capture intrinsic neural dynamics within each ROI by enhancing both low- and high-energy temporal components of fMRI time series at the ROI level, and a Spatio-temporal Pattern Alignment module to combine spatial and temporal representations for producing informative whole-brain features. Experimental results on resting-state and task fMRI datasets from the Human Connectome Project demonstrate the superiority of BrainCast over state-of-the-art time series forecasting baselines. Moreover, fMRI time series extended by BrainCast improve downstream cognitive ability prediction, highlighting the clinical and neuroscientific impact brought by whole-brain fMRI time series forecasting in scenarios with restricted scan durations.
Figures & tables
Fig. 1 : The overall framework of BrainCast. (a) fMRI data preprocessing; (b) The Spatial Interaction Awareness (SIA) module for learning between-ROI spatial interactions; (c) The Temporal Feature Refinement (TFR) module for extracting temporal dy- namics within every ROI; and (d) The Spatio-temporal Pattern Alignment (SPA) module for aligning spatio-temporal represent- ations across ROIs and time points.
Fig. 2 : A sliding window partition technique on the fMRI data to generate sequential samples.
Fig. 3 : (a) Illustration of SIAformer, in which self-attention (b) is applied to learning spatial interactions between ROIs.
fMRI paradigm
Frames per run
Run duration (min:sec)
Resting-state
1200
14:33
Working memory
405
5:01
Emotional processing
176
2:16
Language processing
316
3:57
Relational reasoning
232
2:56
Social cognition
274
3:27
TABLE I : Run duration and the number of frames (i.e., time points) per run for each fMRI paradigm.
Dataset
Metric
Model
Ours
iTransformer
DLinear
Amplifier
PatchTST
CrossFormer
FourierGNN
LSTM
RNN
HCP-rs-fMRI
MSE ↓
0.380 (0.004)
0.398 (0.002)
0.529 (0.012)
0.407 (0.006)
0.394 (0.003)
0.417 (0.001)
0.562 (0.005)
0.602 (0.013)
0.592 (0.006)
MAE ↓
0.461 (0.004)
0.478 (0.002)
0.562 (0.009)
0.481 (0.005)
0.473 (0.004)
0.488 (0.003)
0.580 (0.004)
0.601 (0.011)
0.597 (0.008)
R ↑
0.665 (0.002)
0.631 (0.001)
0.446 (0.013)
0.627 (0.006)
0.635 (0.003)
0.628 (0.004)
0.432 (0.002)
0.355 (0.008)
0.379 (0.005)
R 2↑
0.438 (0.003)
0.397 (0.003)
0.199 (0.011)
0.390 (0.003)
0.402 (0.005)
0.396 (0.006)
0.187 (0.003)
0.120 (0.007)
0.143 (0.007)
HCP-t-fMRI
MSE ↓
0.435 (0.004)
0.488 (0.004)
0.633 (0.005)
0.460 (0.009)
0.485 (0.006)
0.480 (0.010)
0.705 (0.007)
0.681 (0.005)
0.663 (0.004)
TABLE II : Performance comparison of different models on HCP-rs-fMRI and HCP-t-fMRI datasets. The mean is reported with the standard deviation shown in parentheses below. The best results are highlighted in bold.
Fig. 4 : Visualization of forecasting results on a randomly selected ROI of one test sample for HCP-rs-fMRI (a) and HCP-t-fMRI (b), respectively. The black dots denote the previous and ground-truth future data, and the blue dots denote the predicted data.
Fig. 5 : Ablation results for the SIA, TFR, and SPA modules in our BrainCast on both HCP-rs-fMRI (a) and HCP-t-fMRI (b), respectively. The dark blue bars correspond to the left y-axis, whereas the dark red bars correspond to the right y-axis.
Fig. 6 : Visualization of the cortical attention maps in the coronal, sagittal, and axial planes, where red indicates higher attention scores (i.e., greater contribution to the model’s prediction) and blue indicates lower contributions.
Fig. 7 : The R results of BrainCast with varying look-back len- gths (i.e., the length of the previous fMRI time series L ) and the fixed future length (i.e., the length of the future fMRI time series T=20 ) on both HCP-rs-fMRI and HCP-t-fMRI.
Method
HCP-rs-fMRI (previous)
HCP-rs-fMRI (prolonged)
BrainNetCNN
5.0155 ± 0.0017
4.8925 ± 0.0303
MLP
4.8778 ± 0.0382
4.7443 ± 0.0221
GCN
4.8473 ± 0.0150
4.7383 ± 0.0470
SVR
4.7698 ± 0.0121
4.6361 ± 0.0598
TABLE III : PMAT prediction performance on HCP-rs-fMRI, measured in terms of RMSE ( mean±standard deviation ).
Department of Biomedical Engineering, Southern University of Science and Technology, China · School of Biomedical Engineering, Shenzhen University, China · Department of Computer Science, University of Warwick, UK