Autoregressive Visual Decoding from EEG Signals

Sicheng Dai; Hongwang Xiao; Shan Yu; Qiwei Ye

cs.LGFeb 26, 2026

Autoregressive Visual Decoding from EEG Signals

Authors: Sicheng Dai, Hongwang Xiao, Shan Yu, Qiwei Ye

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artifcial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology · Beijing Academy of Artificial Intelligence ·

Abstract

Electroencephalogram (EEG) signals have become a popular medium for decoding visual information due to their cost-effectiveness and high temporal resolution. However, current approaches face significant challenges in bridging the modality gap between EEG and image data. These methods typically rely on complex adaptation processes involving multiple stages, making it hard to maintain consistency and manage compounding errors. Furthermore, the computational overhead imposed by large-scale diffusion models limit their practicality in real-world brain-computer interface (BCI) applications. In this work, we present AVDE, a lightweight and efficient framework for visual decoding from EEG signals. First, we leverage LaBraM, a pre-trained EEG model, and fine-tune it via contrastive learning to align EEG and image representations. Second, we adopt an autoregressive generative framework based on a "next-scale prediction" strategy: images are encoded into multi-scale token maps using a pre-trained VQ-VAE, and a transformer is trained to autoregressively predict finer-scale tokens starting from EEG embeddings as the coarsest representation. This design enables coherent generation while preserving a direct connection between the input EEG signals and the reconstructed images. Experiments on two datasets show that AVDE outperforms previous state-of-the-art methods in both image retrieval and reconstruction tasks, while using only 10% of the parameters. In addition, visualization of intermediate outputs shows that the generative process of AVDE reflects the hierarchical nature of human visual perception. These results highlight the potential of autoregressive models as efficient and interpretable tools for practical BCI applications.

Figures & tables

Figure 1: A typical unCLIP framework employed in previous EEG-based visual decoding works Li et al. (2024) ; Zhang et al. (2025) ; Xiao et al. (2025) ; Scotti et al. (2023) . Despite its flexibility, the framework comprises multiple stages (five in this case), each introducing potential sources of error that can accumulate and degrade overall performance. Furthermore, the computational and memory demands of its components present significant challenges for practical implementation in BCIs.

Figure 2: AVDE involves two training stages. Stage 1: A pre-trained EEG encoder is fine-tuned using contrastive learning to more effectively capture visual information embedded in EEG signals. This adaptation aims to provide a more informative initialization for the subsequent visual reconstruction process. Stage 2: A visual autoregressive transformer is trained using the next-scale prediction objective (Equation 7 ). Specifically, the model takes the sequence $([s],R_{1},R_{2},\dots,R_{K-1})$ as input and predicts the corresponding sequence $(R_{1},R_{2},R_{3},\dots,R_{K})$ . Training is guided by a standard cross-entropy loss.

Method	Low-level		High-level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	AlexNet(2) $\uparrow$	AlexNet(5) $\uparrow$	Inception $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$
Li et al. (2024)	0.160	0.345	0.776	0.866	0.734	0.786	0.582
GeoCap	0.148	0.380	0.813	0.877	0.712	0.791	0.582
CognitionCapturer	0.175	0.366	0.760	0.610	0.721	0.744	0.577
AVDE (Ours)	0.188	0.396	0.817	0.889	0.765	0.795	0.557

Table 2: Quantitative assessments of EEG-based visual reconstruction quality on Subject-08.

Figure 3: Qualitative Comparison of Visual Reconstruction Performance. Selected reconstruction results from subject-08 demonstrate that the visual stimuli reconstructed by our method preserve finer-grained features, suggesting improved fidelity and detail compared to alternative approaches.

Method	Params (M)	Steps	FLOPs (G)	Inference Time (ms)	Memory Usage (MB)
Li et al. (2024)	3818.1	4	8738.6	310.4	4826.73
AVDE (Ours)	425.3	10	1350.5	91.2	1809.63

Table 3: Comparison with state-of-the-art method on inference efficiency. All metrics are evaluated using PyTorch’s built-in profiler on a single NVIDIA A100 GPU. The batch size is fixed at 1, and results are averaged over 200 runs to ensure stability and reliability.

Method	Low-level		High-level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	AlexNet(2) $\uparrow$	AlexNet(5) $\uparrow$	Inception $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$
LaBraM+VAR	0.147	0.366	0.766	0.835	0.724	0.747	0.586
ATM+VAR	0.141	0.351	0.752	0.821	0.711	0.731	0.601
EEGNet+VAR	0.132	0.323	0.733	0.803	0.687	0.712	0.627
NICE+VAR	0.136	0.341	0.742	0.812	0.701	0.719	0.613
LaBraM+ Li et al. (2024)	0.138	0.346	0.746	0.817	0.707	0.726	0.606

Table 4: Impact of using different EEG encoders or generative framework on the reconstruction performance. The results are averaged over all subjects.

Figure 4: Intermediate reconstructions generated by AVDE across 10 progressive scales. Each row corresponds to a distinct EEG-evoked reconstruction instance, and each column represents the cumulative output up to a given scale. This process reflects the hierarchical nature of human visual perception, drawing parallels to the function of successive cortical visual areas (e.g., V1, V2/V4, and IT).

Figure 5: Analysis of similarities between intermediate scales and brain regions. (a) The mean channel embeddings from five brain regions are compared with the intermediate image embeddings. Cosine similarity is used as the measure. (b) Since the generative process is cumulative, the similarities generally increase as more scales are involved. (c) Stepwise increase captures the incremental information contributed by each scale. The step increase for occipital regions peaks at early scales and gradually diminishes thereafter. The temporal and parietal regions exhibit relatively sustained step increases across early and middle scales, followed by a decline in later scales. The frontal and central regions show low step increases initially, which progressively rise and peak at late scales.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters		Values
Temporal Encoder	Iput channels	{1,8,8}
	Output channels	{8,8,8}
	Kernel size	{15,3,3}
	Stride	{8,1,1}
	Padding	{7,1,1}
Transformer encoder layers		12

Appendix

Table 5: Hyperparameters for the LaBraM-based EEG encoder.

Hyperparameters	Values
Number of layers	16
Hidden size	1024
MLP size	4096
Number of heads	16
Number of scales/steps	10
Size of scales	(1, 2, 3, 4, 5, 6, 8, 10, 13, 16)

Appendix

Table 6: Hyperparameters for the visual autoregressive transformer.

Subject	PixCorr $\uparrow$	SSIM $\uparrow$	AlexNet(2) $\uparrow$	AlexNet(5) $\uparrow$	Inception $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$
Subject	Low-level		High-level
Sub-01	0.106	0.283	0.577	0.664	0.556	0.539	0.626
Sub-02	0.096	0.277	0.564	0.643	0.547	0.527	0.639
Sub-03	0.095	0.264	0.541	0.617	0.525	0.501	0.651
Sub-04	0.080	0.249	0.505	0.578	0.480	0.479	0.677
Sub-05	0.086	0.253	0.528	0.592	0.504	0.482	0.664

Appendix

Table 8: Quantitative assessments of EEG-based visual reconstruction quality on EEG-ImageNet.

Table 9: Reconstruction quality on EEG-ImageNet with baselines.

Table 11: Reconstruction quality on THINGS-MEG with baselines.

Subject	PixCorr $\uparrow$	SSIM $\uparrow$	AlexNet(2) $\uparrow$	AlexNet(5) $\uparrow$	Inception $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$
Subject	Low-level		High-level
Sub-01	0.138	0.362	0.750	0.837	0.725	0.721	0.609
Sub-02	0.120	0.388	0.754	0.811	0.710	0.737	0.604
Sub-03	0.168	0.379	0.763	0.827	0.704	0.746	0.603
Sub-04	0.126	0.361	0.748	0.835	0.731	0.765	0.593
Sub-05	0.145	0.373	0.743	0.814	0.691	0.732	0.586

Appendix

Table 13: Quantitative assessments of EEG-based visual reconstruction quality on all subjects.

Table 14: Reconstruction quality on THINGS-EEG with baselines.

Figure 6: Subject-08’s Best, Medium, and Worst reconstructions selected based on PixCorr.

Figure 7: Subject-08’s Best, Medium, and Worst reconstructions selected based on SwAV.

Figure 8: Subject-10’s Best, Medium, and Worst reconstructions selected based on PixCorr.

Figure 9: Subject-10’s Best, Medium, and Worst reconstructions selected based on SwAV.

Figure 10: Subject-03’s Best, Medium, and Worst reconstructions selected based on PixCorr.

Figure 11: Subject-03’s Best, Medium, and Worst reconstructions selected based on SwAV.

EEG-Based Visual Decoding

EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.

See all 44 papers

Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstruction
Apr 24, 2026Yongxiang Lian, Yueyang Cang, Pingge Hu +2Diffusion Model Guidance EEG-Based Visual Decoding
Department of Automation Tsinghua University Haidian District, 100084, Beijing, China · China Academy of Information and Communications Technology Haidian District, 100084, Beijing, China
MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
May 23, 2026Zexuan Chen, Sichao Liu, Runhao Lu +4EEG-Based Visual Decoding Cross-Modal Alignment
KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2
Aligning What EEG Can See: Structural Representations for Brain-Vision Matching
Mar 7, 2026Jingyi Tang, Shuai Jiang, Fei Su +1EEG-Based Visual Decoding Cross-Modal Alignment
Beijing University of Posts and Telecommunications, Beijing, 100876, China

Apr 24, 2026J/K move · Enter open · S save

Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstruction

Yongxiang Lian, Yueyang Cang, Pingge Hu, Yuchen He+1

Department of Automation Tsinghua University Haidian District, 100084, Beijing, China · China Academy of Information and Communications Technology Haidian District, 100084, Beijing, China

Diffusion Model Guidance EEG-Based Visual Decoding Image Reconstruction

Open paper AskPDF arXiv

Objective: Decoding visual information from electroencephalography (EEG) is an important problem in neuroscience and brain-computer interface (BCI) research. Existing methods are largely restricted to natural images and categorical representations, with limited capacity to capture structural features and to differentiate objective perception from subjective cognition. We propose a Structure-Guided Diffusion Model (SGDM) that incorporates explicit structural information for EEG-based visual reconstruction. Approach: SGDM is evaluated on the Kilogram abstract visual object dataset and the THINGS natural image dataset using a two-stage generative mechanism. The framework combines a structurally supervised variational autoencoder with a spatiotemporal EEG encoder aligned to a visual embedding space via contrastive learning. Structural information is integrated into a diffusion model through ControlNet to guide image generation from EEG features. Results: SGDM outperforms existing methods on both abstract and natural image datasets. Reconstructed images achieve higher fidelity in low-level visual features and semantic representations, indicating improved decoding accuracy and strong generalization across diverse visual domains. Spatiotemporal analysis of EEG signals further reveals hierarchical structural encoding patterns, consistent with the neural dynamics of visual cognition. Significance: These findings validate the effectiveness of SGDM in capturing explicit structural geometry and generating images with high fidelity to individual cognitive representations. By enabling decoding of complex visual content from EEG signals, the framework extends neural decoding beyond low-dimensional or categorical outputs. This supports BCIs with increased degrees of freedom for intention decoding and more flexible brain-to-machine communication.

Abstract

EEG-Based Visual Decoding

Explore similar work

Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstruction

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

Aligning What EEG Can See: Structural Representations for Brain-Vision Matching