StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching

Bui Van Hung; Dang Thanh Vu; Hoang-Son Vo; Mai Duc Minh Huy; Mai Tien Dung; Soo-Hyung Kim

cs.CVOct 11, 2026

StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching

Authors: Bui Van Hung, Dang Thanh Vu, Hoang-Son Vo, Mai Duc Minh Huy, Mai Tien Dung, Soo-Hyung Kim

Organizations: Chonnam National University, Gwangju, South Korea · AISEED, Inc., South Korea · Ho Chi Minh University of Science, Ho Chi Minh City, Vietnam · MIREA – Russian Technological University, Moscow, Russia

Abstract

The inherent variability of fMRI responses across trials encourages a generative view of the stimulus-to-response mapping. Prior generative encoders often operate in compressed latent spaces and condition predominantly on semantic features, which provide limited low-level cues for early visual cortex. We propose StimFlow, which synthesizes fMRI directly in native voxel space using conditional flow matching with a linear path. This design aligns training and inference distributions, enabling competitive performance with a single sample and reducing reliance on averaging. To match the visual hierarchy, StimFlow combines structural DINOv2 features with semantic CLIP features, which prove complementary along the cortical processing stream. With a single sample, StimFlow achieves a per-voxel Pearson correlation of 0.371, matching baselines that average five draws. Generation in voxel space allows direct ROI analysis and concept localization without a decoder. Furthermore, it recovers over half of full-data performance from just one hour of scans, supporting efficient cross-subject adaptation.

Figures & tables

Figure 1: Overview of the StimFlow training and inference framework. (a) Training Process : Conditioning vectors $\mathbf{c}$ (combining CLIP semantics and timestep information) are derived from the input image, while structural context from DINOv2 is cross-attended. Voxel tokens are patchified and processed through $12$ DiT-1D layers to regress the conditional velocity field $\mathbf{u}_{t}$ directly in native voxel space using a masked MSE loss. (b) Inference Process : Starting from Gaussian noise $\mathbf{x}_{0}$ , the velocity field is integrated over time using an ODE Euler solver to synthesize the final fMRI response.

Regression baselines
	Voxel-Level		Semantic-Level (via decoding)
Method	MSE $\downarrow$	Pearson $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Linear Regression	0.394	0.334	90.1%	87.2%	0.728	0.432
Transformer Encoder	0.387	0.337	89.8%	85.5%	0.759	0.440
DiT-1D (MSE)	0.352	0.355	91.0%	89.0%	0.720	0.412
Generative, single sample ( $T{=}1$ )

Table 1: Visual-to-fMRI synthesis on NSD. StimFlow is the multi-subject model (shared trunk with per-subject adapters, trained jointly on subjects 1, 2, 5, 7); results are averaged over subjects (per-subject Pearson std

\approx 0.05

; per-subject values in Appendix S6 ). Voxel-level metrics are computed per voxel across images and averaged over trials; semantic-level metrics are obtained by decoding the synthesized fMRI with a frozen MindEye2 model. Methods are grouped into deterministic regression baselines, single-sample generative (

T{=}1

), and multi-sample generative (

T{=}5

). DiT-1D (MSE) is an ablative control sharing StimFlow’s backbone and DINOv2

+

CLIP conditioning but trained with a deterministic MSE loss instead of flow matching, isolating the contribution of the generative objective.

	$T{=}1$		$T{=}5$
Feature backbone	MSE $\downarrow$	Pearson $\uparrow$	MSE $\downarrow$	Pearson $\uparrow$
ResNet-50	0.354	0.348	0.346	0.364
ConvNeXt	0.355	0.345	0.347	0.361
ViT-B/16	0.347	0.363	0.339	0.379
DINOv2 only	0.354	0.351	0.346	0.368
CLIP only	0.361	0.338	0.354	0.351

Table 2: Feature backbone ablation, averaged over subjects 1, 2, 5, 7. The default StimFlow conditions on DINOv2 (structural, via cross-attention) and CLIP (semantic, via AdaLN). Each alternative backbone replaces both feature streams. “DINOv2 only” and “CLIP only” retain a single original stream.

Figure 2: Per-stream feature complementarity , averaged over subjects 1, 2, 5, 7. (a) Per-voxel Pearson correlation of the DINOv2-only (structural) and CLIP-only (semantic) conditioning variants, by NSD functional stream (early

\rightarrow

higher). (b) Their gap (

r_{\mathrm{DINOv2}}-r_{\mathrm{CLIP}}

): on average the structural stream leads in early-to-mid visual cortex and the advantage reverses in the higher streams (Spearman

\rho=-0.86

p<0.05

across the seven streams), suggesting a structure-to-semantics gradient.

Figure 3: Per-stream fidelity of the multi-subject model. Per-voxel Pearson correlation of multi-subject StimFlow (shared trunk + per-subject adapters), mean

\pm

std over subjects 1/2/5/7, overlaid on the per-stream noise ceiling (grey); percentages give the captured fraction of the ceiling. Fidelity remains a substantial fraction of the ceiling across the visual hierarchy, with no region showing a pronounced drop.

Held-out	Train		Full	Few-shot budget
subject	set	Trials	(ref.)	1 h	2 h	3 h	4 h
Sub1	2,5,7	$T{=}1$	0.375 (100%)	0.182 (48%)	0.189 (50%)	0.206 (55%)	0.210 (56%)
Sub1	2,5,7	$T{=}5$	0.391 (100%)	0.215 (55%)	0.225 (58%)	0.249 (64%)	0.251 (64%)
Sub2	1,5,7	$T{=}1$	0.384 (100%)	0.193 (50%)	0.208 (54%)	0.227 (59%)	0.228 (59%)
Sub2	1,5,7	$T{=}5$	0.398 (100%)	0.234 (59%)	0.255 (64%)	0.268 (67%)	0.271 (68%)
Sub5	1,2,7	$T{=}1$	0.423 (100%)	0.260 (61%)	0.271 (64%)	0.281 (66%)	0.290 (68%)

Table 3: Few-shot novel-subject adaptation with increasing data budget under a leave-one-subject-out protocol. The shared trunk is trained on three subjects and adapted to the held-out subject (only its per-subject adapters + LoRA) using

1

–

4

h of that subject’s stimulus data. We report per-voxel Pearson correlation (per voxel, across images) for both single-sample (

T{=}1

) and multi-sample (

T{=}5

) synthesis. The Full column is the multi-subject StimFlow model trained on all four subjects with full data (

{\sim}40

h each); parenthesised values give the fraction of that full-data reference recovered.

Figure 4: Qualitative reconstruction comparison (subject 1). Each column is one test image. Rows, top to bottom: the seen stimulus (GT), the MindEye2 reconstruction from ground-truth fMRI, and the reconstructions from StimFlow-synthesized fMRI at $T{=}1$ and $T{=}5$ . Reconstructions from synthesized fMRI recover the same semantic content as those from real fMRI.

Figure 5: Concept-selective region localization (subject 1). For each concept (rows), a montage of example stimuli is shown beside the selectivity map (mean response over the concept’s images minus the average response across all test images) rendered on the cortical flatmap, from StimFlow’s synthesized responses ( synth ) and the ground-truth responses ( GT ). White contours mark functional ROIs (V1–hV4, FFA/OFA, PPA/OPA, EBA, VWFA). Each concept is scaled to its own range. The synthesized maps localize the same concept-selective regions as the ground truth.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value	MSE $\downarrow$	Pearson $\uparrow$
Patch width $P$	$16$ ( $N{=}1024$ )	0.341	0.368
	$32$ (default)	0.342	0.371
	$64$ ( $N{=}256$ )	0.349	0.357
Heads $H$	$4$ ( $d{=}128$ )	0.344	0.367
	$8$ (default)	0.342	0.371
	$16$ ( $d{=}32$ )	0.343	0.369

Appendix

Table S1 : Low-impact architecture sweeps (

T{=}1

, averaged over subjects 1/2/5/7). Each block varies one hyperparameter with the rest at default; the default row (MSE

0.342

, Pearson

0.371

) is the same model, repeated as the anchor for each axis.

N

is the token count,

d

the per-head dimension.

Depth $L_{d}$	Params (M)	MSE $\downarrow$	Pearson $\uparrow$
$6$	39.4	0.351	0.352
$12$ (default)	74.1	0.342	0.371
$18$	108.8	0.341	0.372
$24$	143.4	0.342	0.371

Appendix

Table S2 : Trunk-depth sweep (

T{=}1

, averaged over subjects 1/2/5/7). Default in bold.

Panel of Figure 11 — Figure S1 : Inference solver vs. step count ( $T{=}1$ , averaged over subjects 1/2/5/7). Euler fidelity (blue) plateaus by $50$ steps (default, dashed line). Heun (red star) reaches the same plateau in $25$ steps, but requires the same $50$ NFEs as Euler- $50$ ( $2$ NFEs/step), with no gain at equal compute.

Adapted blocks	Rank	Trainable (M)	Pearson $\uparrow$
none (adapters only)	–	0.559	0.204
last $1$	$4$	0.592	0.226
last $1$	$8$	0.625	0.234
last $1$	$16$	0.690	0.235
last $2$	$8$	0.690	0.236

Appendix

Table S3 : LoRA rank and adapted-block sweep for few-shot adaptation (

1

h budget,

T{=}5

, averaged over the four held-out subjects). “Trainable” counts the per-subject adapters (

0.56

M) plus the LoRA residual. Default in bold.

Subject	$N_{\text{voxels}}$	$T{=}1$ Pearson [95% CI]	$T{=}5$ Pearson [95% CI]
1	15 724	$0.375\;[0.372,\;0.378]$	$0.391\;[0.388,\;0.394]$
2	14 278	$0.384\;[0.380,\;0.388]$	$0.398\;[0.394,\;0.402]$
5	13 039	$0.423\;[0.419,\;0.427]$	$0.437\;[0.434,\;0.441]$
7	12 682	$0.301\;[0.297,\;0.304]$	$0.320\;[0.317,\;0.323]$
Average	–	$0.371\;\pm 0.051$	$0.387\;\pm 0.049$

Appendix

Table S4 : Per-subject voxel-level Pearson correlation (mean over voxels, computed across images) with bootstrap 95% CI (

B{=}10{,}000

). Default multi-subject StimFlow model.

Figure S2 : Per-image, across-voxel Pearson under two normalization regimes. Per-image, across-voxel Pearson (SynBrain’s reported metric) for SynBrain and for a stimulus-agnostic mean-only predictor, under the mean-retained regime (constant division, per-voxel mean kept) and under standard z-scoring. MindSimulator (published) and StimFlow are shown for reference. The mean-only predictor scores high when the mean is retained and collapses under z-scoring, showing that the inflated mean-retained Pearson reflects the static per-voxel mean rather than stimulus-driven fidelity. The mean-only bars are computed on subject 1 (the subject with the mean-retained signal on hand); SynBrain bars are from our reproduction (mean-retained

=

originally reported

0.687

; z-scored

=0.340

Neuroimaging

Neuroimaging uses brain measurements and computational methods to map anatomy, activity, connectivity, and disease.

See all 235 papers

MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching
Oct 11, 2026Bui Van Hung, Hoang-Son Vo, Mai Duc Minh Huy +2Neuroimaging Flow-Based Generative Modeling
Chonnam National University, Gwangju, South Korea · Ho Chi Minh University of Science, Ho Chi Minh City, Vietnam · AISEED, Inc., South Korea
FM-fMRI: Event Conditioned Flow Matching for Rest-to-Task fMRI Time-Series Synthesis
May 26, 2026Peiyu Duan, Jiyao Wang, Nicha C. Dvornek +4Flow Matching Functional Magnetic Resonance Imaging
Department of Biomedical Engineering, Yale University, New Haven, CT, USA · Department of Radiology & Biomedical Imaging, Yale University, New Haven, CT, USA · Department of Electrical Engineering, Yale University, New Haven, CT, USA
Flow Matching with In-Context Priors for Out-of-Distribution Brain Dynamics
Jun 10, 2026Sam Gijsen, Michał Łukomski, Marc-André Schulz +1Diffusion Transformer Time Series Generation
Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany · Charité – Universitätsmedizin Berlin, Department of Psychiatry and Psychotherapy, Germany +1

Oct 11, 2026J/K move · Enter open · S save

MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching

Bui Van Hung, Hoang-Son Vo, Mai Duc Minh Huy, Dang Thanh Vu+1

Chonnam National University, Gwangju, South Korea · Ho Chi Minh University of Science, Ho Chi Minh City, Vietnam · AISEED, Inc., South Korea

Neuroimaging Flow-Based Generative Modeling Conditional Flow Matching Uncertainty Quantification

Open paper AskPDF arXiv

Current fMRI encoding models frame prediction as deterministic regression, yet identical stimuli evoke a distribution of neural responses (arising from physiological noise, hemodynamic variability, and cognitive fluctuations) that a single point estimate cannot capture. This work introduces MultiFlow, a generative fMRI encoding framework based on Conditional Flow Matching (CFM) and a formulation termed Stimulus-Informed Source Flow Matching (SISFM). SISFM generalises CFM by conditioning the source distribution on the stimulus: a learnable temporal convolution shapes the source mean with an HRF-length receptive field, while a variance head shrinks its scale, shortening the expected transport path. SISFM is instantiated with a Diffusion Transformer (DiT-X) decoder parameterising the velocity field, with sub-layer Attention Residuals, heterogeneous fusion of ten frozen vision, audio, language, and audiovisual backbones, and per-subject Yeo-network heads. On the Algonauts 2025 dataset, a single MultiFlow checkpoint recovers most of the benefit of large deterministic ensembles: it trails the 1000-model TRIBE ensemble in-distribution by 0.003 PCC (0.3163 vs. 0.3195), and the strongest in- and out-of-distribution ensembles (VIBE-20, MedARC-50) by 0.010 and 0.009 PCC, using a single trained model rather than an ensemble of 20-1000. The same checkpoint also provides stimulus-conditioned uncertainty estimates and sample-based diversity.

Abstract

Neuroimaging

Explore similar work

MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching

FM-fMRI: Event Conditioned Flow Matching for Rest-to-Task fMRI Time-Series Synthesis

Flow Matching with In-Context Priors for Out-of-Distribution Brain Dynamics