Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation

Chenqi Li; Yu Liu; Shuo Zhang; Timothy Denison; Tingting Zhu

cs.LGMar 4, 2026

Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation

Authors: Chenqi Li, Yu Liu, Shuo Zhang, Timothy Denison, Tingting Zhu

Organizations: Department of Engineering Science, University of Oxford.

Abstract

Pretraining for electroencephalogram (EEG) foundation models has predominantly relied on self-supervised masked reconstruction, a paradigm largely adapted from and inspired by the success of vision and language foundation models. However, unlike images and text, EEG datasets are notoriously expensive to collect and characterized by low signal-to-noise ratio. These challenges introduce difficulties in scaling the EEG foundation models and capturing the underlying neural semantics through reconstruction. In this work, we ask the question: can we stand on the shoulders of well-established foundation models from well-represented modalities to bootstrap the pretraining of EEG foundation models? We first demonstrate that mainstream foundation models, such as those from vision and time series, transfer surprisingly well to EEG domain. To this end, we propose the Multi-Teacher Distillation Pretraining (MTDP) framework for pretraining EEG foundation models via a two-stage multi-teacher distillation. In the first stage, we introduce a learnable gating network to fuse representations from diverse teachers (e.g., DINOv3 and Chronos) via a masked latent denoising objective. In the second stage, we distill the fused representation into an EEG foundation model. Extensive evaluations across 9 downstream tasks and 12 datasets demonstrate that our MTDP-based EEG foundation model outperforms its self-supervised counterparts while requiring only 25% of the pretraining data.

Figures & tables

Figure 1: Comparison of EEG Foundation Model Pretraining. a) The conventional self-supervised pretraining where EEG foundation model reconstructs missing patches in the temporal, frequency or latent-domain. b) The proposed framework to bootstrap EEG foundation model pretraining by standing on the shoulders of well-established foundation models from well-represented modalities.

Figure 2: Linear probing performance of CBraMod and DINOv3 on EEG downstream tasks. Balanced Accuracy (%).

Figure 3: Overview of Two-Stage Multi-Teacher Distillation Pretraining (MTDP). Stage 1: Teacher Representation Fusion. A learnable gating network is introduced to weigh and fuse representations from frozen teacher models. The gate is trained via a masked latent denoising objective. Stage 2: Knowledge Distillation. The fused teacher representation acts as the target to pretrain the student EEG foundation model. The distillation loss is minimized to align the student representations with the fused representations.

Task	Dataset	Rate	# Channels	Duration	# Samples	Label
Motor Imagery Classification	PhysioNet-MI	160 $\mathrm{Hz}$	64	4s	9,837	4-class
	SHU-MI	250 $\mathrm{Hz}$	32	4s	11,988	2-class
	BCIC-IV-2a	250 $\mathrm{Hz}$	22	4s	5088	4-class
Emotion Recognition	FACED	250 $\mathrm{Hz}$	32	10s	10,332	9-class
	SEED-V	1000 $\mathrm{Hz}$	62	1s	115,001	5-class
Sleep Staging	ISRUC	200 $\mathrm{Hz}$	6	30s	90,187	5-class

Table 1: Overview of Downstream Datasets

Methods	Pretrain Dataset (Hours)	FACED			Physionet-MI			SHU-MI
Methods	Pretrain Dataset (Hours)	B. Acc.	AUC-PR	AUROC	B. Acc.	AUC-PR	AUROC	B. Acc.	Kappa	F1-W
BIOT ( Yang et al., 2023 )	122	51.18	44.76	51.36	61.53	48.75	61.58	61.79	67.70	66.09
LaBraM ( Jiang et al., 2024 )	2500	52.73	46.98	52.88	61.73	49.12	61.77	61.66	67.61	66.04
CBraMod 25% ( Wang et al., 2025 )	2250	55.34	49.54	55.63	61.83	49.12	61.91	62.75	68.84	68.86
CBraMod 100% ( Wang et al., 2025 )	9000	55.09	50.41	56.18	64.17	52.22	64.27	63.70	71.39	69.88
CBraMod-MTDP 25% (Ours)	2250	56.57	50.74	56.51	62.74	50.31	62.68	63.78	71.58	70.52

Table 2: Comparison of EEG foundation model performance on 12 downstream tasks. “25%” and “100%” indicate pretraining dataset size. Metrics acronyms: Balanced Accuracy (B. Acc.), Weighted F1 (F1-W), Cohen’s Kappa (Kappa).

Component	Set 1	Set 2	Set 3	Set 4
Chronos	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
DINOv3	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Two-Stage	$\times$	$\times$	$\times$	$\checkmark$
FACED	51.13	54.40	56.39	56.57
PhysioNetMI	62.57	62.62	62.56	62.74
SHU-MI	62.67	62.27	62.11	63.78

Table 3: Ablation of different distillation methods

Figure 4: Linear probing performance of CBraMod and CBraMod-MTDP on EEG downstream tasks. Balanced Accuracy (%).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Settings
Data
Channels	16
Time points	6000
Patch dimension	200
Sequence length	30 $\mathrm{s}$
CBraMod Model Architecture

Appendix

Table 4: Pretraining Hyperparameters

Figure 5: Loss curve of stage 1 and stage 2 pretraining

Hyperparameters	Settings
Epochs	50
Batch size	64
Dropout	0.1
Optimizer	AdamW
Bakbone Learning rate	{5e-5, 1e-4, 5e-4}
Multi-LR	{Yes, No}

Appendix

Table 5: Fine-tuning Hyperparameters

Methods	Pretrain Dataset (Hours)	FACED			Physionet-MI			SHU-MI
Methods	Pretrain Dataset (Hours)	B. Acc.	AUC-PR	AUROC	B. Acc.	AUC-PR	AUROC	B. Acc.	Kappa	F1-W
EEGNet	0	40.90	33.42	41.24	58.14	44.68	57.96	58.89	63.11	62.83
EEGConformer	0	45.59	38.58	45.14	60.49	47.36	60.62	59.00	63.70	63.51
SPaRCNet	0	46.73	39.78	47.29	59.32	45.64	59.37	59.78	65.10	64.31
ContraWR	0	48.87	42.31	48.84	58.92	45.27	59.18	58.73	63.15	62.73
CNN-Transformer	0	46.97	40.17	47.20	60.53	47.25	60.41	59.75	64.12	63.23

Appendix

Table 6: Comparison of EEG foundation model performance on 12 downstream tasks (Part 1/2). “25%” and “100%” indicate pretraining dataset size. Metrics acronyms: Balanced Accuracy (B. Acc.), Weighted F1 (F1-W), Cohen’s Kappa (Kappa).

Methods	Pretrain Dataset (Hours)	ISRUC			TUEV			CHB-MIT
Methods	Pretrain Dataset (Hours)	B. Acc.	AUC-PR	AUROC	B. Acc.	AUC-PR	AUROC	B. Acc.	Kappa	F1-W
EEGNet	0	71.54	70.40	75.13	38.76	35.77	65.39	56.58	19.14	80.48
EEGConformer	0	74.00	71.43	76.34	40.74	39.67	69.83	59.76	22.09	82.26
SPaRCNet	0	74.87	70.97	76.24	41.61	42.33	70.24	58.76	12.47	81.43
ContraWR	0	74.02	71.78	76.10	43.84	39.12	68.93	63.44	22.64	80.97
CNN-Transformer	0	73.63	71.29	77.19	40.87	38.15	68.54	63.89	24.79	86.62

Appendix

Table 7: Comparison of EEG foundation model performance on 12 downstream tasks (Part 2/2). “25%” and “100%” indicate pretraining dataset size. Metrics acronyms: Balanced Accuracy (B. Acc.), Weighted F1 (F1-W), Cohen’s Kappa (Kappa).

EEG Foundation Models

Electroencephalography (EEG) foundation models learn reusable representations of brain signals for transfer across people, tasks, and recording setups.

See all 96 papers

EEG-JEPA: Structured Latent Prediction for EEG Foundation Models
Jul 31, 2026Jinhao Li, Zhiyuan Ma, Xueqiao Han +8Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture
Tsinghua Laboratory of Brain and Intelligence, Tsinghua University · School of Basic Medical Sciences, Tsinghua Medicine, Tsinghua University · School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University +2
MSBraM: A Multi-scale Self-supervised Brain Foundation Model for Hierarchical EEG Dynamics Learning
Jul 23, 2026Tao Zhou, Jing Han, Lingyu Shu +1Electroencephalography Self-Supervised Pre-Training
Hunan University Changsha, China
CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding
Jul 10, 2026Gabriel Mahuas, Victoria Shevchenko, Ugo Tanielian +2Contrastive Learning Hybrid CNN-Transformer Architectures
Sigma Nova Paris, France · Goethe University Frankfurt Frankfurt, Germany

Jul 31, 2026J/K move · Enter open · S save

EEG-JEPA: Structured Latent Prediction for EEG Foundation Models

Jinhao Li, Zhiyuan Ma, Xueqiao Han, Zhongye Xia+7

Tsinghua Laboratory of Brain and Intelligence, Tsinghua University · School of Basic Medical Sciences, Tsinghua Medicine, Tsinghua University · School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University+2

Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture Predictive Representation Learning EEG Foundation Models

Open paper AskPDF arXiv

Electroencephalography (EEG) foundation models aim to learn reusable representations from large-scale unlabeled recordings. A common pretraining strategy is masked waveform reconstruction, but applying supervision directly to noisy EEG may encourage models to recover predictable background activity, acquisition effects, and artifacts rather than neural structure that transfers across tasks. This raises a central question: what should an EEG foundation model predict to learn transferable representations? We introduce EEG-JEPA a structured latent-prediction framework for EEG foundation modeling. Rather than reconstructing masked voltage samples, a masked context encoder and predictor infer contextual latent states produced by an exponential-moving-average target encoder that observes the complete input. EEG-JEPA organizes target design along three complementary dimensions: target content specifies what representation is predicted, target support specifies where prediction occurs over structured electrode--time regions through Neurotopology-Aware Multi-scale Electrode-Temporal Masking (N-MET), and target depth specifies at which encoder layers supervision is applied. Together, these designs shift EEG pretraining from recovering missing measurements to inferring latent states from structured electrode--time context. We evaluate EEG-JEPA through controlled objective comparisons, frozen multitask transfer, and full fine-tuning. Under the same backbone, pretraining corpus, and training duration, EEG-JEPA improves the 14-task frozen macro balanced accuracy from 40.49% to 50.42% over CBraMod-style masked waveform reconstruction. Multi-source continuation further raises this result to 52.94%, the highest average among the EEG foundation models evaluated on EEG-FM-Bench. Under protocol-matched full fine-tuning, EEG-JEPA also improves the nine-task average balanced accuracy from 68.98% to 70.65%.

Abstract

EEG Foundation Models

Explore similar work

EEG-JEPA: Structured Latent Prediction for EEG Foundation Models

MSBraM: A Multi-scale Self-supervised Brain Foundation Model for Hierarchical EEG Dynamics Learning

CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding