When Model Complexity Matters in Cross-Subject Naturalistic fMRI Prediction

Junhao Zhao; Jacob Kang; Colin Kurniawan; David Simberg; Nan Xu

q-bio.NCNov 2, 2026

When Model Complexity Matters in Cross-Subject Naturalistic fMRI Prediction

Authors: Junhao Zhao, Jacob Kang, Colin Kurniawan, David Simberg, Nan Xu

Organizations: Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA · Fischell Department of Bioengineering, University of Maryland, College Park, MD, USA · Department of Computer Science, University of Maryland, College Park, MD, USA · Brain and Behavior Institute, University of Maryland, College Park, MD, USA ·

Abstract

Increasingly complex neural models predict blood-oxygen-level-dependent (BOLD) responses to naturalistic stimuli, but the predictive value of this added complexity remains unclear. We ask when pretrained stimulus representations help and how much downstream complexity is needed, with and without recent measured BOLD history. We compare linear diagnostic references and nonlinear predictors across HCP movie fMRI (154 subjects), BOLD Moments (10 subjects), and Algonauts 2025 (4 subjects). Under a shared offline protocol, models are fitted on training subjects and evaluated on held-out subjects viewing largely familiar stimuli, without refitting. Pretrained visual features generally improve prediction over raw pixels; for linear autoregression, this advantage becomes more pronounced over longer rollouts (recursive predictions without additional measured BOLD). Measured BOLD history provides substantial gains that generally diminish {as rollouts lengthen}. Nonlinear stimulus-only predictors can improve on linear readouts, whereas linear autoregression provides a strong history-conditioned baseline. Controls examine context length, regularization, fitting procedures, and model capacity. Regional response consistency across subjects is associated with held-out prediction accuracy, while less consistent regions tend to gain more from measured history. In paired simulations with fixed linear neural dynamics, slower hemodynamic responses generally sustain high temporal prediction correlations over longer rollouts, illustrating how the observation process can contribute to predictive persistence. Together, these findings support evaluating stimulus representations and downstream complexity separately, using history-conditioned linear prediction as a diagnostic reference under matched inputs and measurement availability.

Figures & tables

Figure 1: Overview of cross-subject naturalistic fMRI prediction and main findings.

Figure 2: Pixels versus V-JEPA visual representations across representative models on HCP four movies (a) and ridge predictors across three datasets (b). Results include all designated folds and neural seeds 0/1/2; complete comparisons are provided in Appendix C.1 .

Figure 3: Cross-subject fMRI prediction across three datasets for eight predictors and history-only refits of RR-AR, BrainVista, MLP, and GRU. Error bars show SD across held-out subjects. Complete ablation results and the rationale for excluding NeuroWorld’s history-only control are provided in Appendix D.2 .

Panel of Figure 4 — Figure 4: Regional prediction and functional connectivity (FC) on HCP at $H=32$ . Top: parcelwise temporal prediction correlations. Bottom: mean predicted FC (lower triangles) and intersubject SD of Fisher- $z$ FC (upper triangles). Annotations report predicted–measured FC similarity (mean $\pm$ SD across held-out subjects). Full three-dataset results: Appendix E.1 .

Figure 5: Regional prediction accuracy and BOLD-history gains (

H=32

) versus training-subject response consistency. Each point represents a cortical parcel, with prediction performance evaluated on held-out subjects and response consistency (x-axis) computed across training subjects. (a) Absolute prediction accuracy for RR-AR. (b) Prediction gain of RR-AR over RR-stim.

Figure 6: Effect of hemodynamic response timing on RR-AR prediction. (a) Audiovisual circuit. (b) Fast, medium, and slow Balloon–Windkessel HRFs. (c) Prediction across refresh intervals and stimulus schedules for the no-HRF neural reference and HRF conditions. Solid lines show history-plus-stimulus predictions; dashed lines show history-only refits. Conditions share neural trajectories; $\mathrm{TR}=1$ s. Error bars: paired-subject bootstrap 95% CIs. Details: Appendix F .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

	HCP four movies	BOLD Moments	Algonauts 2025
Subjects	154	10	4
Subject split	107/15/32	Five two-subject folds	Four LOSO folds
TR	1.00 s	1.75 s	1.49 s
Movie duration	49.4 min	5.1 min	63.1 h
Unique clips	15	102	331
Clip duration	1.1–4.3 min	3 s	$\sim$ 10–12 min

Appendix

Table 2: Dataset designs and cross-subject splits. A stimulus unit denotes a unique movie clip for HCP and BOLD Moments, and a unique movie segment for Algonauts. Unique duration excludes fixation and repeated presentations. BOLD Moments additionally includes 4-s null trials. Full details are provided in Appendix A .

Figure 7: Prediction blocks for

H=1,2,4

and never , illustrated for 12 target frames. Blue markers indicate the measured BOLD history supplied at each block start; supplying measured history again between blocks is termed a refresh . Arrows within red blocks indicate autoregressive rollout. Predicted blocks are concatenated before scoring.

Predictor input	Mapping	Prediction algorithm	Representative work
Stimulus features	Linear	Ridge-regression voxel-wise readout	Wen et al. [ 30 ]
Stimulus features	Nonlinear	Temporal Transformer	TRIBE; TRIBE v2 [ 5 , 6 ]
Stimulus features	Nonlinear	Adaptive layer aggregation and temporal brain encoder	MIRAGE [ 11 ]
BOLD history	Linear	Linear autoregressive models; linearized graph-based predictors	Nozari et al.; Paugam et al. [ 22 , 24 ]
Stimulus features and BOLD history	Nonlinear	Brain tokenizer and autoregressive Transformer	BrainVista [ 33 ]
Stimulus features and BOLD history	Nonlinear	Latent dynamics model and rollout decoder	NeuroWorld [ 4 ]

Appendix

Table 3: Representative downstream prediction approaches, organized by input and prediction algorithm. Linear and nonlinear refer to the downstream predictor, independently of the stimulus encoder. Benchmark adaptations and diagnostic references are described in Section 3.1 .

Model/stage	Max epochs	Batch	Optimizer	LR	WD	Schedule / clip norm
BrainVista tokenizer	50	32	AdamW	$2\cdot 10^{-5}$	.01	Constant / .5
BrainVista predictor	40	32	AdamW	$2\cdot 10^{-4}$	.01	5% warmup, cosine / .5
NeuroWorld LDL	50	16	AdamW	$2\cdot 10^{-4}$	.01	100-update warmup, cosine / .5
NeuroWorld decoder	200	16	AdamW	$10^{-3}$	.01	500-update warmup, cosine / .5
TRIBE v2 scratch	15	8	Adam	$10^{-4}$	0	10% warmup, OneCycle / none
MIRAGE	15	16	AdamW	$10^{-4}$	.01	10% warmup, OneCycle / 1.0

Appendix

Table 4: Declared optimization ceilings and schedules. Every neural stage has patience 10; final prediction stages select by complete-coverage BOLD MSE. Pretraining uses the indicated stage loss. Numbers are not a claim that every stage runs to its ceiling.

Component	Source and scope
Shared input/test protocol	Study design: C=8, dataset K, HRF=3 TR, fixed three-modality slots, full validation MSE, patience10 and temporal ROI corr. These replace model-native evaluation conventions.
BrainVista	Published two-stage structure and nominal training budgets; our implementation supplies dimension reconciliation, mask/filter choices, atlas-label grouping and shared autonomous BOLD-MSE training.
NeuroWorld	Published LDL/decoder structure and nominal budgets; latent widths, SIGReg settings and shared decoder are specified implementation choices; the adapter supplies all C observed frames.
TRIBE v2	Released FmriEncoder class and architecture/config-derived optimizer schedule; scratch initialization, atlas-sized output, shared normalization, common input interval and val-loss selection are comparison adaptations.
Ridge	Explicit affine design, covariance divided by training-row count, intercept unpenalized; adaptive boundary search uses common K-step val MSE.
MIRAGE	Released downstream code and declared width/depth; common cached features, shared ROI readout, no test-subject head fitting, and common validation-MSE selection.

Appendix

Table 5: Configuration provenance and deliberate comparison adaptations. Scientific choices are declared in the shared configuration and, for MIRAGE, its explicit adapter settings, not runtime paper defaults.

Model	HCP	BOLD	Algonauts
RR-AR	19.009	19.009	19.009
RR-AR (rank64)	1.282	1.282	1.282
RR-stim	11.009	11.009	11.009
TRIBE v2 scratch	356.526	153.675	143.430
BrainVista	7.649	7.649	7.649
NeuroWorld	7.613	7.613	7.613

Appendix

Table 6: Downstream model size in millions of allocated parameters or fitted coefficients, excluding frozen stimulus extractors. MLP and GRU are omitted from the table because their parameter counts match dense RR-AR within 1% in each dataset. Rank64 counts the stored factors and output offsets of one prespecified reduced-rank setting; the full rank sweep is shown in Figure 17 .

Figure 8: Complete eight-model pixels versus V-JEPA comparison on HCP, BOLD Moments and Algonauts 2025 Datasets. All designated held-out subjects and neural seeds 0/1/2 are included. Bars show seed SD; RR-AR and RR-stim are deterministic. Dotted connectors lead to the categorical no-refresh endpoint ( never ).

Figure 9: Matched V, A and T inputs across datasets and models. Every displayed neural curve includes all designated held-out subjects and seeds 0/1/2; bars show sample SD of the three subject-equal seed means. Ridge fits are deterministic. All model–input combinations are complete; incomplete seeds or folds are not pooled. The y-axis is absolute temporal ROI correlation; finite H extends to 192, with dotted guides to the categorical never endpoint. BOLD was presented silently, so A/T are source-feature controls. TRIBE and RR-stim are history-free references.

Figure 10: Matched VA, VT and AT controls. All models cover every designated fold and seeds 0/1/2; bars are sample SD of subject-equal seed means. Ridge fits are deterministic. NeuroWorld includes the explicitly documented FP32 recovery fits. BOLD was presented silently.

Figure 11: Generalization across visual encoders on HCP four movies. Each panel compares pixels, V-JEPA, CLIP (ViT-bigG/14), and DINOv2 (ViT-B/14) within one predictor. Curves show temporal ROI correlation across 32 held-out subjects and 1,000 parcels. MLP results are averaged over seeds 0/1/2, with bars indicating sample SD across seed-level means; deterministic ridge fits have no seed error bars. The categorical never endpoint denotes no further measured-history refresh; dotted connectors are visual guides. Feature dimensions and preprocessing are encoder-specific, so this comparison evaluates representation pipelines rather than a dimension-matched encoder ablation.

Figure 12: Context-length controls for

C\in\{1,4,8\}

(rows), with the benchmark training interval fixed. Columns show HCP four movies, BOLD Moments, and Algonauts 2025 using HCP/BOLD fold0 and Algonauts sub-01. All six neural models use seeds 0/1/2; error bars denote sample SD across seeds. Deterministic ridge fits have no seed error bars. Benchmark references use the same held-out fold as the corresponding context-length controls. Both ridge references, all six neural models, and all 14 refresh settings are included. RR-stim does not use measured BOLD context and is therefore unchanged across rows.

Figure 13: History-only, main and stimulus-only comparisons. Each panel shows that column’s main model (history plus stimulus), with history-only ablations for RR-AR, BrainVista, MLP, and GRU, alongside the same three available stimulus-only references: RR-stim, TRIBE v2 and MIRAGE. Neural curves show the mean over seeds 0/1/2 with sample-SD shading; ridge fits are deterministic. BOLD Moments aggregates all five held-out folds, and Algonauts aggregates sub-01, sub-02, sub-03 and sub-05. NeuroWorld is shown with its main inputs only; its history-only control is excluded for the methodological reason described above.

Figure 14: Neural-model training-interval controls for

K=8/32/

benchmark (rows), with

C=8

throughout. Benchmark

K

is 64 for HCP/BOLD Moments and 92 for Algonauts. The six neural models are trained separately at each

K

using seeds 0/1/2. RR-AR and RR-stim are fixed benchmark references repeated across rows.

Figure 15: Same-objective mini-batch optimization control, HCP/BOLD fold0 and Algonauts sub-01, seeds 0/1/2. Top:

(J-J^{*})/(J_{0}-J^{*})

during zero-initialized Adam training; thin lines are individual seeds and the thick line is their mean. The dotted line is the prespecified objective-gap threshold, not a performance-equivalence margin. All nine runs reach 300 epochs without satisfying both numerical convergence criteria. Bottom: absolute temporal ROI correlation of the terminal mini-batch models (mean

\pm

sample SD across three seeds) and the fresh closed-form reference. The near-overlap reflects predictive agreement, not certified convergence. Dotted connectors to never are visual guides to a categorical finite-record condition. No validation/test checkpoint selection or RR initialization is used.

Figure 16: Fixed-grid regularization controls, seed0, one declared fold per dataset. Every tested penalty is shown through H=192 and never, the finite-record no-refresh endpoint.

Figure 17: Reduced-rank controls for the joint RR-AR predictor. All prespecified ranks and the matched dense fit use video, audio, and text features, with C=8 and separately validation-selected ridge penalties. Curves show absolute temporal ROI correlation over all designated folds at H=1–96. Fits are deterministic; seed0 does not denote independent replication. Parameter accounting follows Appendix B.5 .

Figure 18: Ground-truth and predicted functional connectivity at $H=32$ across three datasets, with parcels ordered by Yeo-7 network. In the HCP row, lower triangles show group-average FC in Pearson- $r$ units (left colorbar), and upper triangles show intersubject variability, quantified as the sample SD across subjects of subject-level Fisher- $z$ FC values after averaging across seeds within subject (right colorbar, range 0–0.25). BOLD Moments and Algonauts show mean FC in both triangles; intersubject variability is reported only for HCP.

Dataset	RR-AR	RR-stim	TRIBE v2	BrainVista	NeuroWorld	MIRAGE	MLP	GRU
HCP four movies	$\mathbf{0.73\pm 0.04}$	$0.59\pm 0.05$	$0.56\pm 0.05$	$0.72\pm 0.03$	$0.48\pm 0.04$	$0.58\pm 0.05$	$0.65\pm 0.04$	$0.63\pm 0.05$
BOLD Moments	$\mathbf{0.63\pm 0.05}$	$0.44\pm 0.04$	$0.18\pm 0.03$	$0.47\pm 0.04$	$0.19\pm 0.03$	$0.39\pm 0.03$	$0.38\pm 0.03$	$0.35\pm 0.05$
Algonauts 2025	$0.69\pm 0.01$	$0.60\pm 0.01$	$0.63\pm 0.01$	$\mathbf{0.70\pm 0.01}$	$0.47\pm 0.01$	$0.64\pm 0.01$	$0.69\pm 0.01$	$0.64\pm 0.01$

Appendix

Table 7: Similarity between predicted and ground-truth functional connectivity at

H=32

, measured by Pearson correlation between their off-diagonal upper-triangular entries. Values report the mean

\pm

sample SD across held-out subjects after averaging the three seed-specific correlations within each subject.

Dataset	Subjects	Parcels	RR-AR	RR-stim	History gain
HCP four movies	32	1,000	0.986	0.989	$-0.498$
BOLD Moments	10	1,000	0.809	0.871	$-0.381$
Algonauts 2025	4	1,000	0.969	0.962	$-0.170$

Appendix

Table 8: Across-parcel Spearman associations between training-subject response consistency and held-out prediction at

H=96

. History gain is RR-AR minus RR-stim. These longer-horizon summaries are descriptive spatial associations, not tests over independent parcels.

Figure 19: Comparison of eight models and four history-only controls on simulated neural and BOLD data. Rows show sustained, 5 s ON/25 s OFF, and 20 s ON/20 s OFF stimulation; columns show the no-HRF neural reference and fast, medium, and slow HRFs. Solid lines show full-input predictions (stimulus only or history plus stimulus); dashed lines with open markers show history-only refits. Correlations are averaged over 11 ROIs and then across seeds within subject for neural models. Curves show means across 10 test subjects; error bars show SD across subjects. Never denotes no refresh after the initial context;

\mathrm{TR}=1

s. Values and error-bar portions below zero are clipped.

fMRI Encoding Models

See all 8 papers

Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching
Apr 13, 2026Nicole Rogalla, Yuzhen Qin, Mario Senden +2Functional Magnetic Resonance Imaging Probabilistic Time Series Forecasting
Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands · Department of Cognitive Neuroscience, Maastricht University, Maastricht, the Netherlands · Maastricht Brain Imaging Center, Maastricht University, Maastricht, the Netherlands
A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli
Jul 24, 2025Qianyi He, Monica D. Rosenberg, Yuan Chang LeongNeural Decoding Neuroimaging
Data Science Institute University of Chicago, USA · Department of Psychology, Neuroscience Institute University of Chicago, USA
Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction
Sep 28, 2026Juhyeon Park, Yeonwoo Kim, Peter Yongho Kim +5Medical Imaging Foundation Models fMRI Foundation Models
IPAI, Seoul National University · Microsoft Research · ECE, Seoul National University +1

Apr 13, 2026J/K move · Enter open · S save

Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching

Nicole Rogalla, Yuzhen Qin, Mario Senden, Ahmed El-Gazzar+1

Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands · Department of Cognitive Neuroscience, Maastricht University, Maastricht, the Netherlands · Maastricht Brain Imaging Center, Maastricht University, Maastricht, the Netherlands

Functional Magnetic Resonance Imaging Probabilistic Time Series Forecasting Conditional Generative Modeling Flow-Based Generative Modeling

Open paper AskPDF arXiv

Forecasting neural activity in response to naturalistic stimuli remains a key challenge for understanding brain dynamics and enabling downstream neurotechnological applications. Here, we introduce a generative forecasting framework for modeling neural dynamics based on autoregressive flow matching (AFM). Building on recent advances in transport-based generative modeling, our approach probabilistically predicts neural responses at scale from multimodal sensory input. Specifically, we learn the conditional distribution of future neural activity given past neural dynamics and concurrent sensory input, explicitly modeling neural activity as a temporally evolving process in which future states depend on recent neural history. We evaluate our framework on the Algonauts project 2025 challenge functional magnetic resonance imaging dataset using subject-specific models. AFM significantly outperforms both a non-autoregressive flow-matching baseline and the official challenge general linear model baseline in predicting short-term parcel-wise blood oxygenation level-dependent (BOLD) activity, demonstrating improved generalization and widespread cortical prediction performance. Ablation analyses show that access to past BOLD dynamics is a dominant driver of performance, while autoregressive factorization yields consistent, modest gains under short-horizon, context-rich conditions. Together, these findings position autoregressive flow-based generative modeling as an effective approach for short-term probabilistic forecasting of neural dynamics with promising applications in closed-loop neurotechnology.

Abstract

fMRI Encoding Models

Explore similar work

Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching

A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction