ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding

Kieren Yu; Chang Huang; Ziyang Liu; Kaishun Wu

cs.CVOct 20, 2026

ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding

Authors: Kieren Yu, Chang Huang, Ziyang Liu, Kaishun Wu

Abstract

EEG visual decoding remains difficult because scalp recordings are noisy, spatially coarse, and variable across trials and subjects. Most systems align each trial with a dense image embedding, although that embedding mixes semantic content with instance-specific texture, pose, and background. This target can demand details absent from EEG and provides no explicit account of the visual factors supporting a prediction. We propose ProtoEEG, a prototype-mediated framework that reformulates EEG--image alignment as soft composition over a shared learnable visual codebook. A trainable EEG encoder and a frozen visual teacher produce assignment distributions over the same $K=64$ prototypes. Their weighted reconstructions form a structured representation for retrieval and image generation. Prototype contrast supplies the main cross-modal training signal, distribution alignment coordinates codebook use across modalities, an instance anchor preserves fine-grained discrimination, and diversity regularization prevents redundant prototypes. The shared assignment profile also makes the learned bottleneck directly testable through prototype retention, random selection, and identity shuffling. Across five seeds, ProtoEEG reaches 64.0% Top-1 and 91.5% Top-5 on THINGS-EEG2 200-way retrieval. After retraining on EEG-ImageNet, it obtains 44.8% Top-1 and 73.8% Top-5 in 80-way retrieval. ProtoEEG also ranks first on all seven reconstruction metrics. Removing prototype contrast lowers Top-1 by 56.9 percentage points, while shuffling prototype identities reduces it to 0.5%. Prototype analyses further show distributed assignments and broad codebook use rather than one-prototype routing. These results support structured alignment targets as a practical alternative to increasing encoder or teacher capacity alone.

Figures & tables

Figure 1: Dense alignment matches EEG to instance-specific image embeddings and can treat related concepts as negatives, as shown on the left. ProtoEEG instead maps both modalities to soft assignments over $K=64$ shared visual prototypes, shown on the right.

Figure 2: Overview of ProtoEEG. A frozen visual teacher and a trainable EEG encoder produce soft assignments over a shared prototype codebook. The diagram denotes $\hat{z}_{v}$ and $\hat{z}_{e}$ by $z_{\mathrm{img}}$ and $z_{\mathrm{eeg}}$ . These embeddings support retrieval and image generation. Training uses prototype contrast $\mathcal{L}_{\mathrm{nce}}$ , distribution alignment $\mathcal{L}_{\mathrm{kl}}$ , and diversity regularization $\mathcal{L}_{\mathrm{div}}$ ; the instance anchor $\mathcal{L}_{\mathrm{inst}}$ .

(a) Intra-subject: train and test on one subject
	CLIP family								RN50 family
	NICE		ATM		AVDE		ProtoEEG		UBP		ATS		NeuroBridge		ProtoEEG
Subject	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5
Sub-01	20.1	47.9	23.2	51.2	25.0	55.2	39.3	71.8	41.2	70.5	53.0	79.0	50.0	77.6	62.9	89.7
Sub-02	19.2	36.9	18.8	43.2	24.1	51.0	32.7	63.5	51.2	80.9	62.0	87.5	63.2	90.6	61.8	94.1
Sub-03	21.2	53.8	27.3	57.0	27.5	58.6	35.5	66.6	51.2	82.0	61.5	89.0	61.6	91.1	64.9	93.6

Table 1: THINGS-EEG2 200-way retrieval accuracy (%). ProtoEEG reports five-seed means. Bold marks best within each visual-encoder family. ProtoEEG consistently outperforms dense alignment baselines in both intra-subject and inter-subject settings, with CLIP gains uniform across subjects while RN50 improvements are more variable.

Method	Metric	Sub-01	Sub-02	Sub-03	Sub-04	Sub-05	Sub-06	Sub-07	Sub-08	Avg
ATM	Top-1	30.6	24.5	21.3	15.1	20.5	18.7	14.6	9.1	19.3
ATM	Top-5	63.3	62.4	64.1	42.8	55.7	69.9	67.6	35.3	57.6
AVDE	Top-1	30.8	27.0	22.7	15.4	21.8	32.9	30.1	14.4	24.4
AVDE	Top-5	63.4	62.8	64.3	43.5	56.6	70.3	69.5	38.8	58.7
ProtoEEG	Top-1	50.2	16.9	56.5	25.3	17.7	57.4	57.3	77.1	44.8
ProtoEEG	Top-5	88.6	51.2	82.3	48.6	41.4	93.6	85.5	98.8	73.8

Table 2: EEG-ImageNet 80-way accuracy (%). ProtoEEG reports five-seed means.

Figure 3: Qualitative reconstruction comparison on twelve THINGS-EEG2 test concepts.

Method	Low-level			High-level
	PixCorr $\uparrow$	SSIM $\uparrow$	AlexNet(2) $\uparrow$	AlexNet(5) $\uparrow$	Inception $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$
ATM	0.130	0.339	0.740	0.822	0.685	0.716	0.599
AVDE	0.147	0.366	0.766	0.835	0.724	0.747	0.586
ProtoEEG	0.154	0.390	0.779	0.855	0.743	0.768	0.506

Table 3: Reconstruction performance on THINGS-EEG2. ProtoEEG reports five-seed means over ten subjects.

Figure 4: Prototype diagnostics over five seeds and ten subjects. Panels a–f show prototype coverage, assignment entropy, temporal windows, activating images, top- $k$ retention, and channel groups.

Configuration	Top-1	Top-5	$\Delta$ T1
Loss ablation
Full Loss	64.0	91.5	—
w/o $\mathcal{L}_{\mathrm{nce}}$	7.1	17.7	$-56.9$
w/o $\mathcal{L}_{\mathrm{kl}}$	33.7	58.5	$-30.3$
w/o $\mathcal{L}_{\mathrm{inst}}$	45.5	65.4	$-18.5$
w/o $\mathcal{L}_{\mathrm{div}}$	51.6	77.1	$-12.4$

Table 4: ProtoEEG diagnostics: five-seed means over all ten THINGS-EEG2 subjects.

EEG-Based Visual Decoding

EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.

See all 44 papers

May 23, 2026

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. We introduce a tri-modal contrastive framework for EEG-based visual decoding that aligns EEG, visual, and textual representations within a unified latent space. Our approach follows a two-stage design. First, we pre-train an EEG encoder via masked reconstruction on unlabeled trials, learning spatio-temporal regularities that transfer robustly to downstream tasks. Second, we jointly align EEG, image, and LLM-generated textual descriptions through contrastive learning, where text supervision acts as a semantic regularizer that injects linguistic structure into the shared space without overwhelming the primary EEG-image signal. The encoder integrates subject-specific adaptation, graph-attention over channels, and temporal-spatial convolutional embeddings. On the Things-EEG2 200-way zero-shot benchmark, our framework achieves 54.1% Top-1 and 83.4% Top-5 accuracy, substantially exceeding the strongest prior baseline (32.4% / 64.0%), with paired Wilcoxon tests confirming significance (p < 0.01) over all in-subject baselines. We validate generalization on Things-MEG. Analysis reveals that compact embedding geometries (CN-CLIP) outperform much larger backbones, and that decoding aligns with established neurophysiology of visual processing. This work is a critical step towards robust, semantically-grounded visual decoding from non-invasive temporal neural signals. The source code is publicly available in https://github.com/anon-eeg/eeg_image_decoding.

Zexuan Chen, Sichao Liu, Runhao Lu +4

KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2

EEG-Based Visual Decoding Cross-Modal Alignment Multimodal Contrastive Learning

Ask PDF

Sep 4, 2026

ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding

Electroencephalogram (EEG) visual decoding aims to recover visual semantics from non-invasive neural time-series signals, for which robust alignment between noisy neural responses and stable semantic representations is key to achieving high-performance decoding. Despite recent advances in contrastive learning, robust EEG decoding remains challenging because existing methods rely on fixed visual or textual anchors whose semantic relations may become misaligned with EEG representations that vary across trials, subjects, and learning stages. Our empirical evidence shows that this instability appears across both standard EEG decoding protocols and more challenging robustness settings, including strict cross-subject transfer and realistic personalized continual adaptation. We provide a formal analysis showing that fixed semantic supervision can bias optimization when EEG-specific relations evolve, and that structure-agnostic perturbations may distort semantically important EEG components. To address these issues, we propose Progressive Contrastive Alignment (ProCA), a unified and model-agnostic framework for adaptive neural-semantic alignment. ProCA progressively refines class-level contrastive supervision from frozen vision-language priors to EEG-aware semantic relations, and introduces structure-consistent interpolation to constrain feature mixing according to channel-wise and temporal importance. Across subject-dependent, subject-independent, strict cross-subject transfer, and continual adaptation settings, ProCA achieves average relative Top-1/Top-5 gains of 7.4%/3.9%, 10.0%/4.6%, 28.1%/17.8%, and 16.8%/11.6%, respectively.

Kanglei Zhou, Chunyan Lan, Dongyang Li +2

Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · School of Life Sciences, Tsinghua University, Beijing, China · Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University, Beijing, China

EEG-Based Visual Decoding Contrastive Learning Cross-Modal Alignment Cross-Subject EEG Decoding

Ask PDF

Mar 13, 2026

CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment

Visual stimuli reconstruction from EEG remains challenging due to fidelity loss and representation shift. We propose CognitionCapturerPro, an enhanced framework that integrates EEG with multi-modal priors (images, text, depth, and edges) via collaborative training. Our core contributions include an uncertainty-weighted similarity scoring mechanism to quantify modality-specific fidelity and a fusion encoder for integrating shared representations. By employing a simplified alignment module and a pre-trained diffusion model, our method significantly outperforms the original CognitionCapturer on the THINGS-EEG dataset, improving Top-1 and Top-5 retrieval accuracy by 25.9% and 10.6%, respectively. Code is available at: https://github.com/XiaoZhangYES/CognitionCapturerPro.

Kaifan Zhang, Lihuo He, Junjie Ke +4

Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China. · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China

EEG-Based Visual Decoding Uncertainty-Aware Multimodal Fusion Cross-Modal Alignment

Ask PDF

Abstract

EEG-Based Visual Decoding

Explore similar work

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding

CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment