cs.LGApr 10, 2026

NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity

Authors: Weijian Mai, Mu Nan, Yu Zhu, Jiahang Cao, Rui Zhang, Yuqin Dai, Chunfeng Song, Andrew F. Luo, 

Organizations: Shanghai Artificial Intelligence Laboratory · University of Hong Kong · Shenzhen Loop Area Institute · Tsinghua University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Visual encoding and decoding models act as gateways to understanding the neural mechanisms underlying human visual perception. Typically, visual encoding models that predict brain activity from stimuli and decoding models that reproduce stimuli from brain activity are treated as distinct tasks, requiring separate models and training procedures. This separation is inefficient and fails to model the consistency between encoding and decoding processes. To address this limitation, we propose NeuroFlow, the first unified framework that jointly models visual encoding and decoding from neural activity within a single flow model. NeuroFlow introduces two key components: (1) NeuroVAE is designed as a variational backbone to model neural variability and establish a compact, semantically structured latent space for bidirectional modeling across visual and neural modalities. (2) Cross-modal Flow Matching (XFM) bypasses the typical paradigm of noise-to-data diffusion guided by a specific modality condition, instead learning a reversibly consistent flow model between visual and neural latent distributions. For the first time, visual encoding and decoding are reformulated as a time-dependent, reversible process within a shared latent space for unified modeling. Empirical results demonstrate that NeuroFlow achieves superior overall performance in visual encoding and decoding tasks with higher computational efficiency compared to any isolated methods. We further analyze principal factors that steer the model toward encoding-decoding consistency and, through brain functional analyses, demonstrate that NeuroFlow captures consistent activation patterns underlying neural variability. NeuroFlow marks a major step toward unified visual encoding and decoding from neural activity, providing mechanistic insights that inform future bidirectional visual brain-computer interfaces.

Figures & tables

Figure 2: Visual encoding-decoding frameworks.
Figure 3: Cross-modal alignment pipelines.
Figure 4: Architecture overview. Stage-1 (A): NeuroVAE introduces probabilistic learning to model neural variability and constrains the latent space with visual semantics for consistent image-to-fMRI synthesis. Stage-2 (B): XFM unifies encoding and decoding processes by learning a time-dependent, reversible flow between empirical visual and neural latent distributions. Stage-3 (C): Encoding and decoding are performed within a single model at inference, where reversing the temporal direction naturally transitions between the two processes.
Figure 5: Qualitative visual encoding and decoding performance comparisons. Left: NeuroFlow achieves superior decoding quality in semantic fidelity and visual structure. Right: NeuroFlow suppresses irrelevant cortical activity while enhancing category-specific regions, capturing consistent activation patterns underlying neural variability to support image synthesis consistent with visual stimuli. Positive (red) and negative (blue) values indicate increased and reduced activations, with color intensity reflecting the absolute magnitude of deviation.
MethodTypeDecodingEncoding ( →\to Decoding)Retrieval
Incep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowIncep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowRaw ↑\uparrowSyn ↑\uparrow
MindSimulator [ 3 ]E----93.1%91.2%.689.391--
SynBrain [ 39 ]E----95.7%94.3%.639.36284.8%92.5%
BrainDiffuser [ 43 ]D91.3%90.9%.728.421----18.8%-
MindEye [ 51 ]D94.6%93.3%.622.343----90.0%-
MindEye2 [ 52 ]D95.4%93.0%.618.344----98.8%-
Table 1: Quantitative visual encoding and decoding performance comparisons.
MethodTypePretrainArchitectureParam
SynBrain [ 39 ]E×\boldsymbol{\times}VAE+Transformer690M
MindEye [ 51 ]D×\boldsymbol{\times}MLP+DP1.00B
MindEye2 [ 52 ]D✓\boldsymbol{\checkmark}Linear+MLP+DP2.60B
NeuroFlowE&D×\boldsymbol{\times}VAE+XFM660M
Table 2: Model efficiency comparisons.
Figure 6: Empirical visualizations. (A) Ablation study : removing key objectives leads to degraded visual fidelity and semantic coherence. (B) Flow trajectory : Encoding trajectory reveals a suppression of early visual responses and a transition toward functional regions (i.e., FFA and EBA). Decoding trajectory evolves from an initial structural sketch, not Gaussian noises , to a realistic and high-fidelity image. (C-D) Brain functional analysis : category-selective fMRI activations and voxel-wise evaluation derived from raw and synthetic fMRI, computed on the whole test set, showing that NeuroFlow suppresses early visual activity and emphasizes higher-order functional regions.
MethodDecodingEncoding ( →\to Decoding)Retrieval
Incep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowIncep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowRaw ↑\uparrowSyn ↑\uparrow
w/o zcz_{c}94.7%94.0%.683.37897.2%96.6%.636.37384.1%90.4%
w/o Lkl\mathcal{L}_{kl}89.0%79.6%.807.47560.7%57.1%.956.60675.1%11.5%
w/o Lcyc\mathcal{L}_{cyc}94.9%93.7%.687.38396.8%95.5%.658.38281.4%88.9%
w/o Lclip\mathcal{L}_{clip} - Lcyc\mathcal{L}_{cyc}57.6%58.8%.961.67452.6%51.3%.983.7000.3%0.5%
w/o LXFM\mathcal{L}_{\text{XFM}}87.7%83.7%.788.49760.3%58.1%.953.64486.4%14.1%
Table 3: Quantitative performance comparisons between NeuroFlow and its ablated variants on Subject 1.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: High-order functional regions in the visual cortex.
ScheduleDecodingEncoding ( →\rightarrow Decoding)Retrieval
Incep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowIncep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowRaw ↑\uparrowSyn ↑\uparrow
Linear95.4%94.4%.682.37798.5%98.4%.618.35986.4%96.0%
Cosine*95.9%95.0%.675.37098.5%98.7%.600.34786.4%96.4%
Appendix
Table 4: Ablation experiments on linear and cosine interpolation schedules.
StepDecodingEncoding ( →\rightarrow Decoding)Retrieval
Incep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowIncep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowRaw ↑\uparrowSyn ↑\uparrow
1095.0%94.3%.682.38598.0%98.3%.617.35686.4%96.0%
20*95.9%95.0%.675.37098.5%98.7%.600.34786.4%96.4%
3095.7%95.3%.674.37298.5%98.7%.597.34586.4%96.4%
Appendix
Table 5: Ablation experiments on the effect of different sampling steps.
MethodDecodingEncoding →\rightarrow DecodingRetrieval
Incep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowIncep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowRaw ↑\uparrowSyn ↑\uparrow
NeuroVAE87.7%83.7%.788.49760.3%58.1%.953.64486.4%14.1%
NeuroVAE+MSE85.2%81.4%.832.53651.0%52.0%.978.67186.4%2.8%
NeuroVAE+LRs86.8%83.0%.794.49690.3%87.0%.783.49386.4%90.8%
NeuroVAE+XFM*95.9%95.0%.675.37098.5%98.7%.600.34786.4%96.4%
Appendix
Table 6: Quantitative comparison between NeuroFlow (NeuroVAE+XFM*) and baseline models.
MethodDecodingEncoding →\rightarrow DecodingRetrieval
Incep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowIncep ↑\uparrowCLIP ↑\uparrowEff ↓\downarrowSwAV ↓\downarrowRaw ↑\uparrowSyn ↑\uparrow
Sub195.9%95.0%.675.37098.5%98.7%.600.34786.4%96.4%
Sub295.4%93.3%.677.36298.5%98.5%.595.34481.3%97.8%
Sub597.0%95.9%.645.34598.8%99.0%.569.32984.7%98.3%
Sub793.9%92.7%.699.37898.5%98.5%.598.34469.8%95.5%
Appendix
Table 7: Quantitative subject-specific visual encoding and decoding results.
Figure 8: Qualitative comparisons between NeuroFlow (NeuroVAE-XFM) and baseline models.
Figure 9: Qualitative subject-specific visual encoding and decoding results.
Figure 10: Subject-specific fMRI activations and corresponding image reconstructions.
Figure 11: Subject-specific category-selective fMRI activations: Faces, Bodies, Places, and Food.
More in this topic

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 184 papers

Explore similar work

CardsList
Oct 9, 2026

Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

Decoding visual experience from fMRI is important for understanding human perception and developing brain-computer interfaces. Recent generative decoding methods can reconstruct plausible images and videos, but visual plausibility alone does not ensure that the output is faithfully grounded in brain activity. In this work, we propose Neurons++, a visual-stream fMRI reconstruction framework for both static image and dynamic video reconstruction. Compared with our conference version Neurons, Neurons++ extends the paradigm from video-only reconstruction to visual-stream reconstruction for both images and videos, and replaces parallel task conditioning with serial cortex-inspired decoding. The model first uses image-fMRI pretraining to obtain a stable spatial-semantic initialization, and then performs subject-specific fine-tuning for image and video reconstruction separately. The serial design organizes fMRI-derived representations into explicit spatial, semantic, language-level, and temporal cues. Extensive experiments show that Neurons++ achieves state-of-the-art video reconstruction and competitive image reconstruction across standard benchmarks. Beyond reconstruction fidelity, neural-to-visual analyses indicate that Neurons++ exhibits qualitative directional controllability under neural perturbation, that its fMRI encoder read-out is sensitive to stimulus-relevant cortical features, and that it supports ROI-guided semantic modulation. Code and model weights are available at: https://github.com/xmed-lab/NeuronsPlusPlus.

Haonan Wang, Yuchen Wang, Jingyu Lu +4

The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China

fMRI-Based Visual DecodingNeural DecodingImage ReconstructionNeuroimagingVisual Neuroscience
AskPDF
Jun 29, 2026

BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at GitHub.

Haitao Wu, Qirui Zhang, Zhouheng Yao +8

School of Artificial Intelligence, Tianjin University, Tianjin, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China.

Cross-Modal Representation LearningUnified Multimodal ModelsNeural DecodingMultimodal Generation
AskPDF
May 23, 2026

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. We introduce a tri-modal contrastive framework for EEG-based visual decoding that aligns EEG, visual, and textual representations within a unified latent space. Our approach follows a two-stage design. First, we pre-train an EEG encoder via masked reconstruction on unlabeled trials, learning spatio-temporal regularities that transfer robustly to downstream tasks. Second, we jointly align EEG, image, and LLM-generated textual descriptions through contrastive learning, where text supervision acts as a semantic regularizer that injects linguistic structure into the shared space without overwhelming the primary EEG-image signal. The encoder integrates subject-specific adaptation, graph-attention over channels, and temporal-spatial convolutional embeddings. On the Things-EEG2 200-way zero-shot benchmark, our framework achieves 54.1% Top-1 and 83.4% Top-5 accuracy, substantially exceeding the strongest prior baseline (32.4% / 64.0%), with paired Wilcoxon tests confirming significance (p < 0.01) over all in-subject baselines. We validate generalization on Things-MEG. Analysis reveals that compact embedding geometries (CN-CLIP) outperform much larger backbones, and that decoding aligns with established neurophysiology of visual processing. This work is a critical step towards robust, semantically-grounded visual decoding from non-invasive temporal neural signals. The source code is publicly available in https://github.com/anon-eeg/eeg_image_decoding.

Zexuan Chen, Sichao Liu, Runhao Lu +4

KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2

EEG-Based Visual DecodingCross-Modal AlignmentMultimodal Contrastive Learning
AskPDF