Replaying the Movie in Your Mind Decoding Dynamic Visual Perception and Recall from EEG

Zikai Fang; Tian-Yi Zhou; Xuan-Hao Liu; Yixi Li; Xiaoyu Xing; Yutong Pei; Bao-Liang Lu; Wei-Long Zheng

cs.CVOct 9, 2026

Replaying the Movie in Your Mind Decoding Dynamic Visual Perception and Recall from EEG

Authors: Zikai Fang, Tian-Yi Zhou, Xuan-Hao Liu, Yixi Li, Xiaoyu Xing, Yutong Pei, Bao-Liang Lu, Wei-Long Zheng

Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, China

Abstract

Decoding dynamic visual content from EEG has advanced rapidly for visual perception, yet it remains unclear which aspects of neural representations are preserved when people internally replay dynamic scenes. We introduce PAIR, an EEG benchmark collected from 20 participants using an interleaved, clip-wise paradigm that pairs naturalistic video viewing with immediate visual recall. This design enables a comparison between externally driven perception and internally generated recall for the same stimulus content within a fixed trial sequence. We benchmark multiple decoding models on hierarchical semantic classification at both fine-grained and coarse-grained levels, meta-feature decoding tasks covering motion, color, object number, face, and human presence, and a state-decoding task that discriminates perception from recall. Within-subject validation results show numerically similar performance for several model--task combinations, alongside higher perception accuracy in two coarse-grained comparisons after multiple-comparison correction. In contrast, perception and recall are highly separable at the state level under the present validation protocol. State accuracy remains high after alpha-band ablation, although this does not exclude ocular or other task-related contributions.

Figures & tables

Fig. 1: Model structure of GCN-LSTM.

Fig. 2: The hierarchical structure of the dataset, consisting of 4 major categories and 20 subcategories.

Fig. 3: The experimental paradigm. Each block contains 50 trials with a prepare-watch-recall structure.

Raw EEG Signals
Model	20-c	4-c	Fast/slow	Color	Number	Face	Human
Uniform chance	5	25	50	20	33.3	50	50
ShallowNet	$10.50\pm 1.67$	$38.27\pm 3.63$	$60.90\pm 3.50$	$27.60\pm 2.66$	$45.50\pm 3.72$	$62.40\pm 3.05$	$59.93\pm 3.27$
DeepNet	$11.10\pm 1.23$	$38.87\pm 3.39$	$62.20\pm 2.45$	$29.73\pm 2.76$	$46.43\pm 2.77$	$62.93\pm 2.92$	$61.03\pm 4.15$
EEGNet	$10.47\pm 1.42$	$37.53\pm 3.41$	$61.53\pm 3.19$	$29.40\pm 3.42$	$45.93\pm 3.31$	$63.00\pm 3.01$	$60.27\pm 3.50$
Conformer	$10.20\pm 1.48$	$37.17\pm 3.48$	$\mathbf{62.47\pm 4.33}$	$29.60\pm 3.40$	$\mathbf{49.13\pm 2.75}$	$\mathbf{65.20\pm 3.73}$	$\mathbf{62.33\pm 3.18}$

Table 1: Perception validation accuracy (%; mean

\pm

SD across 20 participants after averaging three sessions each). Neural-network scores are best validation accuracies; SVM uses its fixed configuration. Highest means within each feature group are in bold .

Raw EEG Signals (Recall)
Model	20-c	4-c	Fast/slow	Color	Number	Face	Human
Uniform chance	5	25	50	20	33.3	50	50
ShallowNet	$9.40\pm 1.67$	$33.97\pm 3.23$	$61.03\pm 3.27$	$27.70\pm 2.57$	$44.70\pm 3.28$	$62.83\pm 3.16$	$60.20\pm 3.73$
DeepNet	$9.80\pm 2.13$	$34.47\pm 2.81$	$\mathbf{62.43\pm 4.15}$	$28.97\pm 2.64$	$46.20\pm 3.71$	$63.40\pm 3.32$	$60.93\pm 2.69$
EEGNet	$9.63\pm 1.58$	$34.10\pm 3.30$	$59.43\pm 3.32$	$28.13\pm 2.97$	$45.67\pm 2.64$	$62.10\pm 4.05$	$60.40\pm 2.66$
Conformer	$10.20\pm 1.69$	$35.83\pm 2.31$	$61.63\pm 2.89$	$29.30\pm 2.09$	$47.83\pm 3.11$	$64.63\pm 3.09$	$\mathbf{61.53\pm 3.25}$

Table 2: Recall validation accuracy (%; mean

\pm

SD across 20 participants). Aggregation and score selection follow Table 1 . Highest means within each feature group are in bold .

Fig. 4: Perception (blue) and recall (orange) across semantic and meta-feature tasks. Bars show mean best validation accuracy; error bars show SD across 20 participants after averaging sessions. Similar bars do not establish statistical equivalence.

Feature	Method	Full Band	w/o Alpha	$\Delta$
–	Chance Level	50.00	50.00	0.00
Raw EEG	DeepNet	$84.77\pm 7.87$	$79.23\pm 10.22$	5.53
	SVM	$88.10\pm 4.80$	$86.73\pm 5.11$	1.37
	EEGNet	$94.88\pm 3.39$	$92.53\pm 3.84$	2.35
	Conformer	$95.07\pm 3.48$	$93.88\pm 3.69$	1.18
	GCN-LSTM	$95.52\pm 3.33$	$95.57\pm 1.76$	-0.05

Table 3: State validation accuracy (%; mean

\pm

SD across 20 participants), with full-band inputs and alpha ablation. Score selection follows Table 1 .

\Delta

is the full-minus-ablated mean difference in percentage points, computed before rounding.

Fig. 5: Mean sensor-level validation accuracy for the ‘Human’ meta-feature during perception and recall, using a shared single-channel Conformer and a common color scale. Values are averaged across sessions and participants; no spatial significance is implied.

EEG-Based Visual Decoding

EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.

See all 44 papers

What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment
Jun 24, 2026Salini Yadav, Taveena Lotey, Pravendra Singh +1Representation Learning EEG-Based Visual Decoding
Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India
Neuroscience-inspired Staged Representation Learning with Disentangled Coarse- and Fine-Grained Semantics for EEG Visual Decoding
May 16, 2026Xiang Gao, Hui Tian, Yanming Zhu +2Disentangled Representation Learning Cross-Modal Representation Learning
School of Information and Communication Technology, Griffith University, Parklands Drive, Southport, 4222, QLD, Australia
Brain-to-Image Generation: Reconstructing Visual Stimuli from EEG using Generative Adversarial Networks
Sep 13, 2026Harshit GoyalEEG-Based Visual Decoding Image Reconstruction
BITS Pilani, India

Jun 24, 2026J/K move · Enter open · S save

What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

Salini Yadav, Taveena Lotey, Pravendra Singh, Partha Pratim Roy

Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India

Representation Learning EEG-Based Visual Decoding Cross-Modal Alignment Multi-View Learning

Open paper AskPDF arXiv

Zero-shot visual decoding from electroencephalography (EEG) aims to infer visual semantics from non-invasive neural recordings, but remains challenging due to the low signal-to-noise ratio, non-stationarity, and limited spatial resolution of EEG. Existing EEG-vision alignment methods often rely on holistic EEG embeddings, which can obscure the complementary temporal, spectral, and spatial structure underlying visual perception. We introduce a unified multiview EEG representation learning framework for aligning brain responses with visual semantic embeddings. Our method builds an EEG encoder that jointly models three complementary views: input-conditioned state-space temporal dynamics, learnable wavelet-based spectral decomposition for sample-adaptive frequency modeling, and attention-modulated graph learning for structured electrode interactions. The resulting multiview EEG embeddings are fused and aligned with pretrained visual representations in a shared semantic space using contrastive learning with EEG-specific regularization, enabling 200-way zero-shot visual classification. Experiments on THINGS-EEG benchmark show that our method achieves state-of-the-art performance, with 54.8% Top-1 and 85.6% Top-5 accuracy in the within-subject setting and 15.3% Top-1 and 45.4% Top-5 accuracy in the cross-subject setting. We further present the first systematic cross-session EEG-image decoding evaluation, achieving 40.8% Top-1 and 78.0% Top-5 accuracy. These results suggest that explicitly modeling multiview neural structure improves both semantic alignment and generalization in EEG-based visual decoding.

Abstract

EEG-Based Visual Decoding

Explore similar work

What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

Neuroscience-inspired Staged Representation Learning with Disentangled Coarse- and Fine-Grained Semantics for EEG Visual Decoding

Brain-to-Image Generation: Reconstructing Visual Stimuli from EEG using Generative Adversarial Networks