EEGFaceSem: An EEG Benchmark with Paired Generative Latents for Semantic Visual Modeling
Organizations: University of Helsinki Helsinki, Finland · University of Macau Macau, China · LUT University Lahti, Finland · University of Copenhagen Copenhagen, Denmark
Abstract
Electroencephalography (EEG) is widely used to measure brain activity for brain-computer interface applications for its non-invasive nature and high temporal resolution. However, progress in modeling the neural responses to visual semantics has been limited by a reliance on fixed image collections that lack the fine-grained control needed to isolate fine-grained changes in visual features and their relation to perception. In this work, we present EEGFaceSem, a large-scale dataset pairing high-quality EEG with photorealistic face images synthesized from a generative model. We provide the generative latent vector for each stimulus, enabling research into the continuous mapping between a visual latent space and neural representations. Each participant's brain responses were recorded while they attended to specific semantic features (e.g., smiling, hair color) in a rapid serial visual presentation paradigm. The dataset contains EEG signals from 30 participants, totaling 67,200 time-locked EEG recordings. We present benchmarks for single-trial semantic classification under single-subject, cross-subject, and subject-adapted settings, as well as for cross-subject voting that aggregates per-stimulus predictions across participants. We further demonstrate the dataset's unique utility with two complementary applications: deriving semantic directions in the latent space for brain-supervised image editing, and conditioning a generative model on brain-decoded labels for brain-supervised image generation. Our dataset paves the way for advancing brain-computer interfaces (BCI) for semantic-level visual saliency detection and brain-supervised generative modeling. Our dataset and code are openly released at OSF (https://osf.io/2guk3/) and at HuggingFace https://huggingface.co/datasets/yefllower/EEGFaceSem.
Figures & tables
| Dataset | #Subject | Visual stimulus content | Standardized | Semantic | Latent Representation | Modality |
|---|---|---|---|---|---|---|
| Grootswagers, Tijl, et al. [ 13 ] | 50 | Object images | no | yes | no | EEG |
| Guger, Christoph, et al. [ 14 ] | 10 | Characters | yes | yes | no | EEG |
| Gifford, Alessandro T., et al. [ 10 ] | 10 | Object images | no | yes | no | EEG |
| Wakeman, Daniel G., and Richard N. Henson. [ 38 ] | 19 | Face images | no | yes | no | EEG+MEG+fMRI |
| Rybář, Milan, Riccardo Poli, and Ian Daly. [ 36 ] | 12 | Object images | no | no | no | EEG+fNIRS |
| Ours | 30 | Generated faces | yes | yes | yes | EEG |
| subject | channel | task | blocks per task | target per block | non-target per block | sampling rate | total trials |
|---|---|---|---|---|---|---|---|
| 30 | 32 | 8 | 4 | 20 | 50 | 1000 | 67200 |
| female | male | blond | darkhaired | smiles | nosmile | old | young |
| 7460 | 7416 | 7354 | 7248 | 7365 | 7553 | 7493 | 7459 |
| Contrast | Score | Positive group | Negative group | AUC |
|---|---|---|---|---|
| Female vs Male | ||||
| NoSmile vs Smiling | ||||
| Dark vs Blond hair | ||||
| Old vs Young |
| Task | LDA | LR | MLP | EEGNet | EEGPT |
|---|---|---|---|---|---|
| Female | |||||
| Male | |||||
| Blond | |||||
| Darkhaired | |||||
| Smiles | |||||
| Nosmile |
| Task | LDA | LR | MLP | EEGNet | EEGPT |
|---|---|---|---|---|---|
| Female | |||||
| Male | |||||
| Blond | |||||
| Darkhaired | |||||
| Smiles | |||||
| Nosmile |
| Task | MLP | EEGNet | EEGPT |
|---|---|---|---|
| Female | |||
| Male | |||
| Blond | |||
| Darkhaired | |||
| Smiles | |||
| Nosmile |
| Task | LDA | LR | MLP | EEGNet | EEGPT |
|---|---|---|---|---|---|
| Female | 0.7960 0.0228 | 0.9736 0.0084 | 0.9801 0.0065 | 0.9830 0.0058 | 0.9720 0.0079 |
| Male | 0.7137 0.0257 | 0.9543 0.0118 | 0.9605 0.0091 | 0.9695 0.0100 | 0.9328 0.0126 |
| Blond | 0.8641 0.0205 | 0.9862 0.0073 | 0.9921 0.0043 | 0.9881 0.0069 | 0.9765 0.0095 |
| Darkhaired | 0.7795 0.0262 | 0.9644 0.0107 | 0.9654 0.0103 | 0.9668 0.0101 | 0.9407 0.0140 |
| Smiles | 0.7386 0.0208 | 0.9253 0.0117 | 0.9442 0.0102 | 0.9608 0.0084 | 0.8942 0.0137 |
| Nosmile | 0.6811 0.0206 | 0.9104 0.0130 | 0.9115 0.0123 | 0.9334 0.0112 | 0.8696 0.0155 |
| Feature window | |||
|---|---|---|---|
| s | 0.4900 | 0.5825 | 0.7206 |
| [0.4820, 0.4984] | [0.5681, 0.5969] | [0.7005, 0.7411] | |
| s | 0.4987 | 0.5242 | 0.6598 |
| [0.4917, 0.5063] | [0.5172, 0.5311] | [0.6435, 0.6760] |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Brain | Explicit | Random | ||||
|---|---|---|---|---|---|---|
| Task | ||||||
| female | 0.750 | 0.90 | 0.800 | 0.90 | 0.550 | 0.62 |
| male | 0.725 | 0.78 | 0.925 | 0.93 | 0.500 | 0.53 |
| blond | 0.875 | 0.97 | 0.800 | 0.97 | 0.500 | 0.50 |
| darkhaired | 0.650 | 0.72 | 0.700 | 0.80 | 0.575 | 0.57 |
| smiles | 0.800 | 0.93 | 0.900 | 0.95 | 0.500 | 0.50 |
| Gender | Smile | Blond | Young | |
|---|---|---|---|---|
| Gender | – | |||
| Smile | – | |||
| Blond | – | |||
| Young | – |
| Task | Full window | Truncated window | Neighbors non-target |
|---|---|---|---|
| Female | 0.7888 | 0.7194 | 0.6359 |
| Male | 0.7296 | 0.6672 | 0.6225 |
| Blond | 0.7744 | 0.7092 | 0.6444 |
| Darkhaired | 0.7395 | 0.6920 | 0.6335 |
| Smiles | 0.7306 | 0.6660 | 0.5969 |
| Nosmile | 0.7076 | 0.6302 | 0.5704 |
EEG-Based Visual Decoding
EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.
See all 53 papers