MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching

Bui Van Hung; Hoang-Son Vo; Mai Duc Minh Huy; Dang Thanh Vu; Soo-Hyung Kim

cs.AIOct 11, 2026

MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching

Authors: Bui Van Hung, Hoang-Son Vo, Mai Duc Minh Huy, Dang Thanh Vu, Soo-Hyung Kim

Organizations: Chonnam National University, Gwangju, South Korea · Ho Chi Minh University of Science, Ho Chi Minh City, Vietnam · AISEED, Inc., South Korea

Abstract

Current fMRI encoding models frame prediction as deterministic regression, yet identical stimuli evoke a distribution of neural responses (arising from physiological noise, hemodynamic variability, and cognitive fluctuations) that a single point estimate cannot capture. This work introduces MultiFlow, a generative fMRI encoding framework based on Conditional Flow Matching (CFM) and a formulation termed Stimulus-Informed Source Flow Matching (SISFM). SISFM generalises CFM by conditioning the source distribution on the stimulus: a learnable temporal convolution shapes the source mean with an HRF-length receptive field, while a variance head shrinks its scale, shortening the expected transport path. SISFM is instantiated with a Diffusion Transformer (DiT-X) decoder parameterising the velocity field, with sub-layer Attention Residuals, heterogeneous fusion of ten frozen vision, audio, language, and audiovisual backbones, and per-subject Yeo-network heads. On the Algonauts 2025 dataset, a single MultiFlow checkpoint recovers most of the benefit of large deterministic ensembles: it trails the 1000-model TRIBE ensemble in-distribution by 0.003 PCC (0.3163 vs. 0.3195), and the strongest in- and out-of-distribution ensembles (VIBE-20, MedARC-50) by 0.010 and 0.009 PCC, using a single trained model rather than an ensemble of 20-1000. The same checkpoint also provides stimulus-conditioned uncertainty estimates and sample-based diversity.

Figures & tables

Figure 1: Overview of the MultiFlow pipeline. Time-aligned visual frames , audio samples , and transcripts pass through ten frozen foundation-model backbones; the fused representation drives a generative model that synthesises whole-brain fMRI activity over the Schaefer-1000 parcellation. Rather than a single point estimate, MultiFlow models the stimulus-conditioned BOLD distribution via Stimulus-Informed Source Flow Matching (SISFM).

Figure 2: Three views of fMRI encoding (schematic). (a) Deterministic regression maps the stimulus $\mathbf{x}$ to a single point estimate $\hat{\mathbf{y}}$ (red), collapsing the target BOLD distribution onto its conditional mean. (b) Standard CFM draws source samples $\mathbf{z}_{0}$ (orange) from a fixed Gaussian $p_{0}=\mathcal{N}(\mathbf{0},\mathbf{I})$ and transports each to $p_{\text{data}}\!\approx\!p(\mathbf{y}\mid\mathbf{x})$ along a long ODE path. (c) MultiFlow (ours) draws the same samples from a stimulus-conditioned source $p_{0}(\mathbf{z}_{0}\mid\mathbf{x})=\mathcal{N}\!\bigl(\boldsymbol{\mu}_{\phi}(\mathbf{x}),\sigma_{\phi}(\mathbf{x})^{2}\mathbf{I}\bigr)$ whose $\boldsymbol{\mu}_{\phi}$ and $\sigma_{\phi}$ are learned to place the source closer to the target, shortening the expected transport path. In (b,c) the curves are independent samples from a single model; the panels differ only in the source distribution.

Figure 3: MultiFlow overview. Ten frozen foundation backbones feed a multi-token fusion module and a RoPE temporal encoder. The encoded context conditions both the SISFM source generator (producing a stimulus-aligned starting distribution) and the DiT-X velocity decoder. Per-subject Yeo-network heads project the latent state to a 1000-parcel BOLD prediction.

Figure 4: SISFM source generator. A Conv1D–ReLU–Conv1D–Sigmoid stage produces a non-negative driver, convolved by a depthwise Conv1D with HRF-length kernel to yield

\boldsymbol{\mu}_{\phi}

. A pooled MLP–Softplus head produces

\sigma_{\phi}

Single-run, matched training compute (1 trained model, same features/data/splits)
Method	#Trained models	#Source draws	Sub-01	Sub-02	Sub-03	Sub-05	Mean
Ridge [ Gifford et al.(2025b)Gifford, Bersch, St-Laurent, Pinsard, Boyle, Bellec, Oliva, Roig, and Cichy ]	–	–	0.2152	0.2031	0.2189	0.1759	0.2033
TRIBE [ d’Ascoli et al.(2025)d’Ascoli, Doyle, Banville, Caucheteux, El Karoui, and King ]	1	–	0.2827	0.2689	0.2817	0.2344	0.2670
VIBE [ Schwetlick et al.(2025) ]	1	–	0.3201	0.3042	0.3093	0.2595	0.2983
MedARC [ Villanueva et al.(2025) ]	1	–	0.3271	0.3097	0.3209	0.2642	0.3055
Regression DiT-X (ours, det.)	1	–	0.2887	0.2734	0.2807	0.2331	0.2690

Table 1: In-distribution performance (Friends S7). Per-subject and mean per-parcel PCC (

\uparrow

) over Schaefer-1000. #Trained models counts independently trained checkpoints; #Source draws counts samples averaged from the SISFM source of a single checkpoint (source-averaged inference, Section 3.6 ), which adds no training cost. “Matched training compute” refers to the training budget: all single-run rows share MultiFlow’s features, data, splits, and frozen backbones, whose inference cost dominates. Regression DiT-X is identical to MultiFlow except that the SISFM source and variance head are removed and MSE is used, so its gap to MultiFlow isolates the generative formulation. MultiFlow uses

K=5

draws (

\tau=0.2

); bold is best.

Single-run, matched training compute (1 trained model, same features/data/splits)
Method	#Trained models	#Source draws	Sub-01	Sub-02	Sub-03	Sub-05	Mean
Ridge [ Gifford et al.(2025b)Gifford, Bersch, St-Laurent, Pinsard, Boyle, Bellec, Oliva, Roig, and Cichy ]	–	–	0.0986	0.0859	0.1021	0.0715	0.0895
TRIBE [ d’Ascoli et al.(2025)d’Ascoli, Doyle, Banville, Caucheteux, El Karoui, and King ]	1	–	0.1875	0.1726	0.1838	0.1381	0.1705
VIBE [ Schwetlick et al.(2025) ]	1	–	0.2172	0.1921	0.2135	0.1646	0.1968
MedARC [ Villanueva et al.(2025) ]	1	–	0.2282	0.1958	0.2275	0.1702	0.2054
Regression DiT-X (ours, det.)	1	–	0.1372	0.1291	0.1463	0.1147	0.1318

Table 2: Out-of-distribution performance (6 held-out movies). Per-subject and mean per-parcel PCC (

\uparrow

) over Schaefer-1000. Columns as in Table 1 : #Trained models counts independently trained checkpoints, #Source draws counts averaged samples from a single checkpoint. Regression DiT-X is the architecture-matched MSE control (see Table 1 ). MultiFlow uses

K=5

draws (

\tau=0.2

). Bold denotes the best result.

	Hold-out (test) subject (model trained on remaining 3)
Method	Sub-01 train: 02, 03, 05	Sub-02 train: 01, 03, 05	Sub-03 train: 01, 02, 05	Sub-05 train: 01, 02, 03
TRIBE [ d’Ascoli et al.(2025)d’Ascoli, Doyle, Banville, Caucheteux, El Karoui, and King ]	0.1625	0.1584	0.1691	0.1242
VIBE [ Schwetlick et al.(2025) ]	0.2376	0.2442	0.2499	0.2336
MedARC [ Villanueva et al.(2025) ]	0.2007	0.2040	0.2232	0.1855
MultiFlow (ours)	0.2388	0.2507	0.2502	0.2442

Table 3: Cross-subject generalisation (leave-one-subject-out). Column indicates the held-out test subject. PCC is averaged over three clips (Friends S6, Figures , Life ). Baselines are single-run reproductions (no ensembling, same features/pipeline as MultiFlow). Bold denotes the best result.

Figure 5: Predicted BOLD timeseries across the cortical hierarchy (sub-01, Friends s06e01a). Top: per-parcel PCC on the fsaverage5 inflated surface (LH lateral, RH medial), six regions annotated. Bottom: predicted (red) vs. measured (black) BOLD averaged over each region’s parcels across $\sim$ 656 s; parcel-mean R on the full S6 holdout is given next to each track.

Table 9

ODE steps $N$	5	10	20	30
CFM + Gaussian	0.2520	0.2815	0.2941	0.2973
CFM + SISFM	0.3094	0.3182	0.3205	0.3213
$\Delta$ (SISFM $-$ Gauss)	$+$ 0.0574	$+$ 0.0367	$+$ 0.0264	$+$ 0.0240

Table 7: Step-count robustness (S6 PCC,

\uparrow

). The SISFM source preserves accuracy under step-count reduction better than a Gaussian source, the behaviour predicted by its shorter expected transport path.

Neuroimaging

Neuroimaging uses brain measurements and computational methods to map anatomy, activity, connectivity, and disease.

See all 235 papers

Oct 11, 2026

StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching

The inherent variability of fMRI responses across trials encourages a generative view of the stimulus-to-response mapping. Prior generative encoders often operate in compressed latent spaces and condition predominantly on semantic features, which provide limited low-level cues for early visual cortex. We propose StimFlow, which synthesizes fMRI directly in native voxel space using conditional flow matching with a linear path. This design aligns training and inference distributions, enabling competitive performance with a single sample and reducing reliance on averaging. To match the visual hierarchy, StimFlow combines structural DINOv2 features with semantic CLIP features, which prove complementary along the cortical processing stream. With a single sample, StimFlow achieves a per-voxel Pearson correlation of 0.371, matching baselines that average five draws. Generation in voxel space allows direct ROI analysis and concept localization without a decoder. Furthermore, it recovers over half of full-data performance from just one hour of scans, supporting efficient cross-subject adaptation.

Bui Van Hung, Dang Thanh Vu, Hoang-Son Vo +3

Chonnam National University, Gwangju, South Korea · AISEED, Inc., South Korea · Ho Chi Minh University of Science, Ho Chi Minh City, Vietnam +1

Neuroimaging Flow-Based Generative Modeling

Ask PDF

May 26, 2026

FM-fMRI: Event Conditioned Flow Matching for Rest-to-Task fMRI Time-Series Synthesis

Task-based fMRI provides a direct readout of task-evoked neural dynamics, but it is expensive and difficult to acquire at scale, motivating rest-to-task synthesis from widely available resting-state fMRI (rsfMRI). We propose FM-fMRI, an event-conditioned flow-matching model that learns a continuous-time conditional vector field to generate task ROI time series from a subject's rsfMRI and the task event information. The formulation enables fast ODE-based sampling and flexible conditioning over heterogeneous event schedules. Rather than optimizing for pointwise reconstruction, we evaluated generated signals using complementary criteria that probe temporal and spectral structure, subject and group-level connectome consistency, and distributional alignment. On the public Human Connectome Project and internal BioPoint autism cohort, FM-fMRI achieves the strongest spectral and connectivity agreement and improved distribution-level matching over conditional diffusion, generative adversarial networks (GANs), and variational autoencoders (VAEs) baselines. Furthermore, we augment the BioPoint cohort by synthesizing task-fMRI ROI time series with our method, improving downstream autism classification and demonstrating practical utility in data-limited clinical settings. The code will be available on GitHub.

Peiyu Duan, Jiyao Wang, Nicha C. Dvornek +4

Department of Biomedical Engineering, Yale University, New Haven, CT, USA · Department of Radiology & Biomedical Imaging, Yale University, New Haven, CT, USA · Department of Electrical Engineering, Yale University, New Haven, CT, USA

Flow Matching Functional Magnetic Resonance Imaging Synthetic Data Generation Magnetic Resonance Imaging Medical Image Synthesis Conditional Generative Modeling

Ask PDF

May 28, 2026

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

Recent progress in task-optimized neural networks has established encoding models as a powerful tool for predicting brain responses to naturalistic stimuli, yet most existing approaches rely on unimodal representations. The emergence of omni-modal foundation models and rich multimodal neural datasets enables encoding models that jointly integrate visual, auditory, and linguistic information across subjects. We introduce MIRAGE, a brain encoding framework for predicting whole-brain fMRI responses to naturalistic audiovisual stimuli. MIRAGE achieves state-of-the-art performance via a native multimodal backbone and adaptive feature gating across layers. These representations are then combined with a transformer-based brain encoder and a subject-specific linear head over the cortical parcels. Controlled comparisons show that natively multimodal features consistently outperform post-hoc aggregation of independent unimodal features, across architectural levels and backbones. Beyond predictive accuracy, the learned attention weights are directly inspectable to interpret the modality-specific gating profile over the backbone, and each modality traces a distinct anatomical pattern across cortex. Together, these results propose adaptive layer-wise aggregation of natively multimodal features as a generalizable, interpretable, and accurate approach for whole-brain encoding.

Abdulkadir Gokce, Badr AlKhamissi, Martin Schrimpf

NeuroAI Lab, EPFL

Neuroimaging Neural Encoding Multimodal Fusion Multimodal Learning Multimodal Understanding Gated Multimodal Fusion

Ask PDF

Abstract

Neuroimaging

Explore similar work

StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching

FM-fMRI: Event Conditioned Flow Matching for Rest-to-Task fMRI Time-Series Synthesis

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding