NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity

Weijian Mai; Mu Nan; Yu Zhu; Jiahang Cao; Rui Zhang; Yuqin Dai; Chunfeng Song; Andrew F. Luo

cs.LGApr 10, 2026

NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity

Authors: Weijian Mai, Mu Nan, Yu Zhu, Jiahang Cao, Rui Zhang, Yuqin Dai, Chunfeng Song, Andrew F. Luo,

Organizations: Shanghai Artificial Intelligence Laboratory · University of Hong Kong · Shenzhen Loop Area Institute · Tsinghua University ·

Abstract

Visual encoding and decoding models act as gateways to understanding the neural mechanisms underlying human visual perception. Typically, visual encoding models that predict brain activity from stimuli and decoding models that reproduce stimuli from brain activity are treated as distinct tasks, requiring separate models and training procedures. This separation is inefficient and fails to model the consistency between encoding and decoding processes. To address this limitation, we propose NeuroFlow, the first unified framework that jointly models visual encoding and decoding from neural activity within a single flow model. NeuroFlow introduces two key components: (1) NeuroVAE is designed as a variational backbone to model neural variability and establish a compact, semantically structured latent space for bidirectional modeling across visual and neural modalities. (2) Cross-modal Flow Matching (XFM) bypasses the typical paradigm of noise-to-data diffusion guided by a specific modality condition, instead learning a reversibly consistent flow model between visual and neural latent distributions. For the first time, visual encoding and decoding are reformulated as a time-dependent, reversible process within a shared latent space for unified modeling. Empirical results demonstrate that NeuroFlow achieves superior overall performance in visual encoding and decoding tasks with higher computational efficiency compared to any isolated methods. We further analyze principal factors that steer the model toward encoding-decoding consistency and, through brain functional analyses, demonstrate that NeuroFlow captures consistent activation patterns underlying neural variability. NeuroFlow marks a major step toward unified visual encoding and decoding from neural activity, providing mechanistic insights that inform future bidirectional visual brain-computer interfaces.

Figures & tables

Figure 2: Visual encoding-decoding frameworks.

Figure 3: Cross-modal alignment pipelines.

Figure 4: Architecture overview. Stage-1 (A): NeuroVAE introduces probabilistic learning to model neural variability and constrains the latent space with visual semantics for consistent image-to-fMRI synthesis. Stage-2 (B): XFM unifies encoding and decoding processes by learning a time-dependent, reversible flow between empirical visual and neural latent distributions. Stage-3 (C): Encoding and decoding are performed within a single model at inference, where reversing the temporal direction naturally transitions between the two processes.

Figure 5: Qualitative visual encoding and decoding performance comparisons. Left: NeuroFlow achieves superior decoding quality in semantic fidelity and visual structure. Right: NeuroFlow suppresses irrelevant cortical activity while enhancing category-specific regions, capturing consistent activation patterns underlying neural variability to support image synthesis consistent with visual stimuli. Positive (red) and negative (blue) values indicate increased and reduced activations, with color intensity reflecting the absolute magnitude of deviation.

Method	Type	Decoding				Encoding ( $\to$ Decoding)				Retrieval
Method	Type	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Raw $\uparrow$	Syn $\uparrow$
MindSimulator [ 3 ]	E	-	-	-	-	93.1%	91.2%	.689	.391	-	-
SynBrain [ 39 ]	E	-	-	-	-	95.7%	94.3%	.639	.362	84.8%	92.5%
BrainDiffuser [ 43 ]	D	91.3%	90.9%	.728	.421	-	-	-	-	18.8%	-
MindEye [ 51 ]	D	94.6%	93.3%	.622	.343	-	-	-	-	90.0%	-
MindEye2 [ 52 ]	D	95.4%	93.0%	.618	.344	-	-	-	-	98.8%	-

Table 1: Quantitative visual encoding and decoding performance comparisons.

Method	Type	Pretrain	Architecture	Param
SynBrain [ 39 ]	E	$\boldsymbol{\times}$	VAE+Transformer	690M
MindEye [ 51 ]	D	$\boldsymbol{\times}$	MLP+DP	1.00B
MindEye2 [ 52 ]	D	$\boldsymbol{\checkmark}$	Linear+MLP+DP	2.60B
NeuroFlow	E&D	$\boldsymbol{\times}$	VAE+XFM	660M

Table 2: Model efficiency comparisons.

Figure 6: Empirical visualizations. (A) Ablation study : removing key objectives leads to degraded visual fidelity and semantic coherence. (B) Flow trajectory : Encoding trajectory reveals a suppression of early visual responses and a transition toward functional regions (i.e., FFA and EBA). Decoding trajectory evolves from an initial structural sketch, not Gaussian noises , to a realistic and high-fidelity image. (C-D) Brain functional analysis : category-selective fMRI activations and voxel-wise evaluation derived from raw and synthetic fMRI, computed on the whole test set, showing that NeuroFlow suppresses early visual activity and emphasizes higher-order functional regions.

Method	Decoding				Encoding ( $\to$ Decoding)				Retrieval
Method	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Raw $\uparrow$	Syn $\uparrow$
w/o $z_{c}$	94.7%	94.0%	.683	.378	97.2%	96.6%	.636	.373	84.1%	90.4%
w/o $\mathcal{L}_{kl}$	89.0%	79.6%	.807	.475	60.7%	57.1%	.956	.606	75.1%	11.5%
w/o $\mathcal{L}_{cyc}$	94.9%	93.7%	.687	.383	96.8%	95.5%	.658	.382	81.4%	88.9%
w/o $\mathcal{L}_{clip}$ - $\mathcal{L}_{cyc}$	57.6%	58.8%	.961	.674	52.6%	51.3%	.983	.700	0.3%	0.5%
w/o $\mathcal{L}_{\text{XFM}}$	87.7%	83.7%	.788	.497	60.3%	58.1%	.953	.644	86.4%	14.1%

Table 3: Quantitative performance comparisons between NeuroFlow and its ablated variants on Subject 1.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: High-order functional regions in the visual cortex.

Schedule	Decoding				Encoding ( $\rightarrow$ Decoding)				Retrieval
Schedule	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Raw $\uparrow$	Syn $\uparrow$
Linear	95.4%	94.4%	.682	.377	98.5%	98.4%	.618	.359	86.4%	96.0%
Cosine*	95.9%	95.0%	.675	.370	98.5%	98.7%	.600	.347	86.4%	96.4%

Appendix

Table 4: Ablation experiments on linear and cosine interpolation schedules.

Step	Decoding				Encoding ( $\rightarrow$ Decoding)				Retrieval
Step	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Raw $\uparrow$	Syn $\uparrow$
10	95.0%	94.3%	.682	.385	98.0%	98.3%	.617	.356	86.4%	96.0%
20*	95.9%	95.0%	.675	.370	98.5%	98.7%	.600	.347	86.4%	96.4%
30	95.7%	95.3%	.674	.372	98.5%	98.7%	.597	.345	86.4%	96.4%

Appendix

Table 5: Ablation experiments on the effect of different sampling steps.

Method	Decoding				Encoding $\rightarrow$ Decoding				Retrieval
Method	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Raw $\uparrow$	Syn $\uparrow$
NeuroVAE	87.7%	83.7%	.788	.497	60.3%	58.1%	.953	.644	86.4%	14.1%
NeuroVAE+MSE	85.2%	81.4%	.832	.536	51.0%	52.0%	.978	.671	86.4%	2.8%
NeuroVAE+LRs	86.8%	83.0%	.794	.496	90.3%	87.0%	.783	.493	86.4%	90.8%
NeuroVAE+XFM*	95.9%	95.0%	.675	.370	98.5%	98.7%	.600	.347	86.4%	96.4%

Appendix

Table 6: Quantitative comparison between NeuroFlow (NeuroVAE+XFM*) and baseline models.

Method	Decoding				Encoding $\rightarrow$ Decoding				Retrieval
Method	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Raw $\uparrow$	Syn $\uparrow$
Sub1	95.9%	95.0%	.675	.370	98.5%	98.7%	.600	.347	86.4%	96.4%
Sub2	95.4%	93.3%	.677	.362	98.5%	98.5%	.595	.344	81.3%	97.8%
Sub5	97.0%	95.9%	.645	.345	98.8%	99.0%	.569	.329	84.7%	98.3%
Sub7	93.9%	92.7%	.699	.378	98.5%	98.5%	.598	.344	69.8%	95.5%

Appendix

Table 7: Quantitative subject-specific visual encoding and decoding results.

Figure 8: Qualitative comparisons between NeuroFlow (NeuroVAE-XFM) and baseline models.

Figure 9: Qualitative subject-specific visual encoding and decoding results.

Figure 10: Subject-specific fMRI activations and corresponding image reconstructions.

Figure 11: Subject-specific category-selective fMRI activations: Faces, Bodies, Places, and Food.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 184 papers

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability
Oct 9, 2026Haonan Wang, Yuchen Wang, Jingyu Lu +4fMRI-Based Visual Decoding Neural Decoding
The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China
BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language
Jun 29, 2026Haitao Wu, Qirui Zhang, Zhouheng Yao +8Cross-Modal Representation Learning Unified Multimodal Models
School of Artificial Intelligence, Tianjin University, Tianjin, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China.
MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
May 23, 2026Zexuan Chen, Sichao Liu, Runhao Lu +4EEG-Based Visual Decoding Cross-Modal Alignment
KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2

Oct 9, 2026J/K move · Enter open · S save

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

Haonan Wang, Yuchen Wang, Jingyu Lu, Qixiang Zhang+3

The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China

fMRI-Based Visual Decoding Neural Decoding Image Reconstruction Neuroimaging Visual Neuroscience

Open paper AskPDF arXiv

Decoding visual experience from fMRI is important for understanding human perception and developing brain-computer interfaces. Recent generative decoding methods can reconstruct plausible images and videos, but visual plausibility alone does not ensure that the output is faithfully grounded in brain activity. In this work, we propose Neurons++, a visual-stream fMRI reconstruction framework for both static image and dynamic video reconstruction. Compared with our conference version Neurons, Neurons++ extends the paradigm from video-only reconstruction to visual-stream reconstruction for both images and videos, and replaces parallel task conditioning with serial cortex-inspired decoding. The model first uses image-fMRI pretraining to obtain a stable spatial-semantic initialization, and then performs subject-specific fine-tuning for image and video reconstruction separately. The serial design organizes fMRI-derived representations into explicit spatial, semantic, language-level, and temporal cues. Extensive experiments show that Neurons++ achieves state-of-the-art video reconstruction and competitive image reconstruction across standard benchmarks. Beyond reconstruction fidelity, neural-to-visual analyses indicate that Neurons++ exhibits qualitative directional controllability under neural perturbation, that its fMRI encoder read-out is sensitive to stimulus-relevant cortical features, and that it supports ROI-guided semantic modulation. Code and model weights are available at: https://github.com/xmed-lab/NeuronsPlusPlus.

Abstract

Neural Decoding

Explore similar work

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding