ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding

Kieren Yu; Chang Huang; Ziyang Liu; Kaishun Wu

cs.CVOct 20, 2026

ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding

Authors: Kieren Yu, Chang Huang, Ziyang Liu, Kaishun Wu

Abstract

EEG visual decoding remains difficult because scalp recordings are noisy, spatially coarse, and variable across trials and subjects. Most systems align each trial with a dense image embedding, although that embedding mixes semantic content with instance-specific texture, pose, and background. This target can demand details absent from EEG and provides no explicit account of the visual factors supporting a prediction. We propose ProtoEEG, a prototype-mediated framework that reformulates EEG--image alignment as soft composition over a shared learnable visual codebook. A trainable EEG encoder and a frozen visual teacher produce assignment distributions over the same $K=64$ prototypes. Their weighted reconstructions form a structured representation for retrieval and image generation. Prototype contrast supplies the main cross-modal training signal, distribution alignment coordinates codebook use across modalities, an instance anchor preserves fine-grained discrimination, and diversity regularization prevents redundant prototypes. The shared assignment profile also makes the learned bottleneck directly testable through prototype retention, random selection, and identity shuffling. Across five seeds, ProtoEEG reaches 64.0% Top-1 and 91.5% Top-5 on THINGS-EEG2 200-way retrieval. After retraining on EEG-ImageNet, it obtains 44.8% Top-1 and 73.8% Top-5 in 80-way retrieval. ProtoEEG also ranks first on all seven reconstruction metrics. Removing prototype contrast lowers Top-1 by 56.9 percentage points, while shuffling prototype identities reduces it to 0.5%. Prototype analyses further show distributed assignments and broad codebook use rather than one-prototype routing. These results support structured alignment targets as a practical alternative to increasing encoder or teacher capacity alone.

Figures & tables

Figure 1: Dense alignment matches EEG to instance-specific image embeddings and can treat related concepts as negatives, as shown on the left. ProtoEEG instead maps both modalities to soft assignments over $K=64$ shared visual prototypes, shown on the right.

Figure 2: Overview of ProtoEEG. A frozen visual teacher and a trainable EEG encoder produce soft assignments over a shared prototype codebook. The diagram denotes $\hat{z}_{v}$ and $\hat{z}_{e}$ by $z_{\mathrm{img}}$ and $z_{\mathrm{eeg}}$ . These embeddings support retrieval and image generation. Training uses prototype contrast $\mathcal{L}_{\mathrm{nce}}$ , distribution alignment $\mathcal{L}_{\mathrm{kl}}$ , and diversity regularization $\mathcal{L}_{\mathrm{div}}$ ; the instance anchor $\mathcal{L}_{\mathrm{inst}}$ .

(a) Intra-subject: train and test on one subject
	CLIP family								RN50 family
	NICE		ATM		AVDE		ProtoEEG		UBP		ATS		NeuroBridge		ProtoEEG
Subject	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5	T1	T5
Sub-01	20.1	47.9	23.2	51.2	25.0	55.2	39.3	71.8	41.2	70.5	53.0	79.0	50.0	77.6	62.9	89.7
Sub-02	19.2	36.9	18.8	43.2	24.1	51.0	32.7	63.5	51.2	80.9	62.0	87.5	63.2	90.6	61.8	94.1
Sub-03	21.2	53.8	27.3	57.0	27.5	58.6	35.5	66.6	51.2	82.0	61.5	89.0	61.6	91.1	64.9	93.6

Table 1: THINGS-EEG2 200-way retrieval accuracy (%). ProtoEEG reports five-seed means. Bold marks best within each visual-encoder family. ProtoEEG consistently outperforms dense alignment baselines in both intra-subject and inter-subject settings, with CLIP gains uniform across subjects while RN50 improvements are more variable.

Method	Metric	Sub-01	Sub-02	Sub-03	Sub-04	Sub-05	Sub-06	Sub-07	Sub-08	Avg
ATM	Top-1	30.6	24.5	21.3	15.1	20.5	18.7	14.6	9.1	19.3
ATM	Top-5	63.3	62.4	64.1	42.8	55.7	69.9	67.6	35.3	57.6
AVDE	Top-1	30.8	27.0	22.7	15.4	21.8	32.9	30.1	14.4	24.4
AVDE	Top-5	63.4	62.8	64.3	43.5	56.6	70.3	69.5	38.8	58.7
ProtoEEG	Top-1	50.2	16.9	56.5	25.3	17.7	57.4	57.3	77.1	44.8
ProtoEEG	Top-5	88.6	51.2	82.3	48.6	41.4	93.6	85.5	98.8	73.8

Table 2: EEG-ImageNet 80-way accuracy (%). ProtoEEG reports five-seed means.

Figure 3: Qualitative reconstruction comparison on twelve THINGS-EEG2 test concepts.

Method	Low-level			High-level
	PixCorr $\uparrow$	SSIM $\uparrow$	AlexNet(2) $\uparrow$	AlexNet(5) $\uparrow$	Inception $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$
ATM	0.130	0.339	0.740	0.822	0.685	0.716	0.599
AVDE	0.147	0.366	0.766	0.835	0.724	0.747	0.586
ProtoEEG	0.154	0.390	0.779	0.855	0.743	0.768	0.506

Table 3: Reconstruction performance on THINGS-EEG2. ProtoEEG reports five-seed means over ten subjects.

Figure 4: Prototype diagnostics over five seeds and ten subjects. Panels a–f show prototype coverage, assignment entropy, temporal windows, activating images, top- $k$ retention, and channel groups.

Configuration	Top-1	Top-5	$\Delta$ T1
Loss ablation
Full Loss	64.0	91.5	—
w/o $\mathcal{L}_{\mathrm{nce}}$	7.1	17.7	$-56.9$
w/o $\mathcal{L}_{\mathrm{kl}}$	33.7	58.5	$-30.3$
w/o $\mathcal{L}_{\mathrm{inst}}$	45.5	65.4	$-18.5$
w/o $\mathcal{L}_{\mathrm{div}}$	51.6	77.1	$-12.4$

Table 4: ProtoEEG diagnostics: five-seed means over all ten THINGS-EEG2 subjects.

EEG-Based Visual Decoding

EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.

See all 44 papers

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
May 23, 2026Zexuan Chen, Sichao Liu, Runhao Lu +4EEG-Based Visual Decoding Cross-Modal Alignment
KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2
ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding
Sep 4, 2026Kanglei Zhou, Chunyan Lan, Dongyang Li +2EEG-Based Visual Decoding Contrastive Learning
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · School of Life Sciences, Tsinghua University, Beijing, China · Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University, Beijing, China
CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment
Mar 13, 2026Kaifan Zhang, Lihuo He, Junjie Ke +4EEG-Based Visual Decoding Uncertainty-Aware Multimodal Fusion
Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China. · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China

May 23, 2026J/K move · Enter open · S save

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

Zexuan Chen, Sichao Liu, Runhao Lu, Huichao Qi+3

KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland+2

EEG-Based Visual Decoding Cross-Modal Alignment Multimodal Contrastive Learning

Open paper AskPDF arXiv

Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. We introduce a tri-modal contrastive framework for EEG-based visual decoding that aligns EEG, visual, and textual representations within a unified latent space. Our approach follows a two-stage design. First, we pre-train an EEG encoder via masked reconstruction on unlabeled trials, learning spatio-temporal regularities that transfer robustly to downstream tasks. Second, we jointly align EEG, image, and LLM-generated textual descriptions through contrastive learning, where text supervision acts as a semantic regularizer that injects linguistic structure into the shared space without overwhelming the primary EEG-image signal. The encoder integrates subject-specific adaptation, graph-attention over channels, and temporal-spatial convolutional embeddings. On the Things-EEG2 200-way zero-shot benchmark, our framework achieves 54.1% Top-1 and 83.4% Top-5 accuracy, substantially exceeding the strongest prior baseline (32.4% / 64.0%), with paired Wilcoxon tests confirming significance (p < 0.01) over all in-subject baselines. We validate generalization on Things-MEG. Analysis reveals that compact embedding geometries (CN-CLIP) outperform much larger backbones, and that decoding aligns with established neurophysiology of visual processing. This work is a critical step towards robust, semantically-grounded visual decoding from non-invasive temporal neural signals. The source code is publicly available in https://github.com/anon-eeg/eeg_image_decoding.

Abstract

EEG-Based Visual Decoding

Explore similar work

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding

CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment