q-bio.NCFeb 4, 2026

BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction

Authors: Xuanhua Yin, Runkai Zhao, Lina Yao, Weidong Cai

Organizations: The University of Sydney, Australia · University of New South Wales, Australia · CSIRO’s Data61

Abstract

Naturalistic fMRI offers a time-resolved window into cortical dynamics during continuous multimodal experience, where future brain states are jointly governed by endogenous neural context and exogenous sensory drive. Forecasting such dynamics, however, remains challenging because fast-changing stimulus streams must be reconciled with the sluggish hemodynamic BOLD response, while cortical activity is organized across functionally heterogeneous brain networks. To address these challenges, we introduce BrainVista, a multimodal autoregressive framework for observed-stimulus conditional brain-state forecasting. BrainVista predicts future fMRI states from the history of brain activity and stimulus tokens temporally aligned to each prediction query, enabling autoregressive prediction without access to future ground-truth fMRI states. Recursive rollout provides a within-run predictive test of brain-state evolution under continuing sensory input. Specifically, we combine Network-wise Tokenizers for functionally structured brain representations, a Spatial Mixer Head for cross-network interaction refinement, and Stimulus-to-Brain (S2B) masking to preserve temporally causal stimulus-brain conditioning while preventing future brain-state leakage and stimulus look-ahead. We evaluate BrainVista on Algonauts 2025, CineBrain, and HAD, three large-scale naturalistic fMRI benchmarks, under long-horizon conditional autoregressive forecasting. BrainVista consistently outperforms baselines, improving pattern correlation by 10.8% and 9.4% relative to the strongest baseline on Algonauts 2025 and CineBrain, respectively. Code is available at https://github.com/xuanhuayin/BrainVista.

Figures & tables

Appendix figures & tables58 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching

    Apr 13, 2026Nicole Rogalla, Yuzhen Qin, Mario Senden +2Functional Magnetic Resonance ImagingProbabilistic Time Series Forecasting

  2. A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli

    Jul 24, 2025Qianyi He, Monica D. Rosenberg, Yuan Chang LeongNeural DecodingNeuroimaging

  3. BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics

    Jun 16, 2026Junfeng Xia, Wenhao Ye, Junxiang Zhang +3Medical Image SynthesisNeuroimaging