Brain2Face: A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars

Camilo Andrés Martínez Martínez; Olaf Dünkel; Kartik Teotia; Maurice Rekrut; Christian Theobalt; Marc Habermann

cs.CVOct 12, 2026

Brain2Face: A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars

Authors: Camilo Andrés Martínez Martínez, Olaf Dünkel, Kartik Teotia, Maurice Rekrut, Christian Theobalt, Marc Habermann

Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus · German Research Center for Artificial Intelligence (DFKI)

Abstract

Driving digital head avatars is classically done by hand-crafted artist design or by image-based facial performance capture. In this work, we ask whether spatio-temporal tracking of the humans's head geometry and appearance can be performed solely from brain signals, i.e., Electroencephalography (EEG). Since prior work mostly focused on activity and emotion recognition, this task is relatively unexplored without even having publicly available high-quality datasets and benchmarks. Thus, we contribute a dataset featuring paired EEG and human head geometry as well as appearance annotations. We synchronize 64-channel EEG with per-frame expression latent codes of a personalized Gaussian head avatar. Second, we introduce a baseline to the task of expression recovery of a head avatar from EEG signals. To this end, we propose a dedicated EEG-to-expression model that takes a temporal sequence of EEG signals as input and regresses the latent expression parameters of the 3D Gaussian head avatar. Lastly, we validate our capture and algorithmic design choices through a benchmark protocol that we introduce to this task. We believe this work paves the way towards better understanding of the relation between EEG signals and facial surface as well as appearance changes.

Figures & tables

Panel of Figure 2 — Fig. 2: Overview of the Brain2Face dataset. Our dataset comprises paired EEG and multi-view video data covering multiple facial expressions, views, and subjects.

Property	Value
Subjects	6
Sessions per subject	2
Total capture time	4 h 15 min
Expression types per session	75
Expression data	74 min 15 s
EEG channels	64

Tab. 1: Brain2Face dataset statistics. Paired EEG and multi-view face recordings across all subjects and sessions.

Model	LPIPS $\downarrow$	Blendshape MAE $\downarrow$	Latent MSE $\downarrow$	Mesh MAE [mm] $\downarrow$	Lmk Wing [mm] $\downarrow$
BENDR ( Kostas et al., 2021 )	0.0607	0.0259	0.242	0.906	2.737
BIOT ( Yang et al., 2023 )	0.0671	0.0570	0.247	1.106	2.768
CBraMod ( Wang et al., 2025 )	0.0624	0.0266	0.253	0.914	2.791
CodeBrain ( Ma et al., 2026 )	0.0587	0.0249	0.231	0.748	2.563
EEGPT ( Wang et al., 2024 )	0.0659	0.0567	0.247	0.877	2.754
LaBraM ( Jiang et al., 2024 )	0.0817	0.0476	0.236	1.308	2.751

Tab. 2: Frozen EEG foundation model comparison. Results on test sequences with seen expressions for frozen pretrained EEG encoders with a nonlinear MLP decoder, evaluated on Subject-03. REVE [scratch] trains the same encoder and decoder from random initialization, with the encoder unfrozen, with the same data and compute.

Pretrained frozen
Decoder depth	LPIPS $\downarrow$	Blendshape MAE $\downarrow$	Latent MSE $\downarrow$	Mesh MAE [mm] $\downarrow$	Lmk Wing [mm] $\downarrow$
$L=1$	0.0650	0.0470	0.178	1.034	2.445
$L=2$	0.0643	0.0434	0.188	1.074	2.447
$L=3$	0.0624	0.0449	0.168	1.003	2.395
$L=4$	0.0608	0.0424	0.168	0.989	2.386
Pretrained unfrozen

Tab. 3: REVE initialization and decoder-depth comparison. REVE with

L=1

–

4

decoder blocks on seen expression types held out in the test session, evaluated on Subject-03, comparing frozen and unfrozen EEG encoders.

Seen expressions (S; held-out session for each expression)
Initialization	LPIPS $\downarrow$	Blendshape MAE $\downarrow$	Latent MSE $\downarrow$	Mesh MAE [mm] $\downarrow$	Lmk Wing [mm] $\downarrow$
Scratch	0.0736	0.0463	0.270	1.237	2.820
Pretrained frozen	0.0701	0.0453	0.235	1.163	2.611
Pretrained unfrozen	0.0651	0.0412	0.214	1.046	2.430
Unseen expressions (U; absent from training)
Scratch	0.0791	0.0581	0.305	1.305	2.897

Tab. 4: Effect of foundation-model initialization. Average results over all six subjects, evaluating held-out test sequences with expressions seen or unseen during training.

Seen expressions (S; held-out session for each expression)
Variant (zeroed)	LPIPS $\downarrow$	Blendshape MAE $\downarrow$	Latent MSE $\downarrow$	Mesh MAE [mm] $\downarrow$	Lmk Wing [mm] $\downarrow$
All channels (0)	0.0651	0.0412	0.214	1.046	2.430
Bad channels (4–19)	0.0668	0.0398	0.219	1.039	2.426
Motor region (20)	0.0649	0.0408	0.216	1.051	2.439
Frontal (15)	0.0676	0.0433	0.221	1.082	2.537
Occipital (10)	0.0669	0.0427	0.221	1.077	2.486

Tab. 5: Channel-masking ablations. Average subject results using pretrained-unfrozen REVE-base and the

L=2

decoder. Numbers in parentheses give the number of zeroed channels; Sec. A.5 lists the electrodes.

Panel of Figure 8 — Fig. 3: Qualitative reconstruction of seen and unseen expression types. Each row shows one subject, with Ours and Ground truth paired for each expression group. Ground truth denotes the fitted Gaussian-avatar render at the matching frame.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

		Test sequences
	Training (T)	Seen expressions (S)	Unseen expressions (U)
Session 1	$b_{1},b_{2}$	$b_{3}$	$b_{4}$
Session 2	$b_{1},b_{3}$	$b_{2}$	$b_{4}$
Segments (%)	655 (73.5)	119 (13.4)	117 (13.1)

Appendix

Tab. 6: Cross-session split. Cells list the expression groups assigned to each role in each session. Counts give retained five-second segments and their percentage of the 891 segments across six subjects. Each subject has a separate model.

Index	Group	Expression name
1	$b_{1}$	lips_puffed
2	$b_{1}$	raise_left_eyebrow
3	$b_{1}$	relaxed_mouth_open
4	$b_{1}$	left_side_sneer
5	$b_{1}$	show_all_teeth
6	$b_{1}$	jaw_open_pull_lips_in

Appendix

Tab. 7: Full expression repertoire. The 75 expression labels and their fixed group assignments. Indices follow this group order and do not indicate recording order.

EEG-Based Visual Decoding

EEG-based visual decoding maps brain signals recorded during visual perception to image identities, attributes, or reconstructed scenes.

See all 44 papers

See the Emotion: A Facial Emoji Proxy Modeling for EEG Emotion Recognition
Jul 3, 2026Jingjing Hu, Guo Dan, Haofan Cheng +4Facial Expression Synthesis Multimodal Model Interpretability
Hefei University of Technology · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · The Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology +3
MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment
Jul 1, 2026Zhenhang Li, Xin Zhou, Hao Deng +1EEG Decoding Cross-Modal Alignment
Binghamton University · Massachusetts General Hospital
emg2face: Expressive Facial Animation with High-Density Surface EMG
Oct 7, 2026Ganidhu Abey, Wendy Greening, Ashika Kamboj +5Facial Expression Synthesis Human Avatar Animation
University of British Columbia · Google · Imperial College London +1

Jul 3, 2026J/K move · Enter open · S save

See the Emotion: A Facial Emoji Proxy Modeling for EEG Emotion Recognition

Jingjing Hu, Guo Dan, Haofan Cheng, Ying Zeng+3

Hefei University of Technology · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · The Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology+3

Facial Expression Synthesis Multimodal Model Interpretability EEG Emotion Recognition

Open paper AskPDF arXiv

Despite the high accuracy of EEG-based emotion recognition, existing models remain opaque "black boxes", lacking semantic grounding between abstract neural features and human-interpretable states. In this paper, we reframe EEG explainability as a cross-modal generation task, shifting the paradigm from feature attribution to behavioral visualization. We introduce Facial Emoji Proxy Modeling, a novel framework that translates high-dimensional EEG signals into identity-anonymized facial emojis. Guided by the neuroscientific inspiration of neural-facial association, this approach grounds neural representations in the manifold of observable facial dynamics. Technically, our framework integrates FMENet, a specialized backbone modeling expression-relevant spatial synergies, and the Facial Emoji Learning Branch (FELB), which treats emoji reconstruction as a structured semantic regularizer. Extensive experiments on EAV and MMER benchmarks demonstrate that our method achieves state-of-the-art accuracy among EEG-only models. Crucially, it generates semantically faithful facial animations that provide a transparent, privacy-preserving window into the brain's emotional evolution, effectively allowing users to "see the emotion" directly from neural signals. Code is available at https://github.com/xian-sh/SeeEmotion

Abstract

EEG-Based Visual Decoding

Explore similar work

See the Emotion: A Facial Emoji Proxy Modeling for EEG Emotion Recognition

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment

emg2face: Expressive Facial Animation with High-Density Surface EMG