Electroencephalography (EEG) provides a non-invasive window into neural dynamics at high temporal resolution and plays a pivotal role in clinical neuroscience research. Despite this potential, prevailing computational approaches to EEG analysis remain largely confined to task-specific classification objectives or coarse-grained pattern recognition, offering limited support for clinically meaningful interpretation. To address these limitations, we introduce NeuroNarrator, the first generalist EEG-to-text foundation model designed to translate electrophysiological segments into precise clinical narratives. A cornerstone of this framework is the curation of NeuroCorpus-160K, the first harmonized large-scale resource pairing over 160,000 EEG segments with structured, clinically grounded natural-language descriptions. Our architecture first aligns temporal EEG waveforms with spatial topographic maps via a rigorous contrastive objective, establishing spectro-spatially grounded representations. Building on this grounding, we condition a Large Language Model through a state-space-inspired formulation that integrates historical temporal and spectral context to support coherent clinical narrative generation. This approach establishes a principled bridge between continuous signal dynamics and discrete clinical language, enabling interpretable narrative generation that facilitates expert interpretation and supports clinical reporting workflows. Extensive evaluations across diverse benchmarks and zero-shot transfer tasks highlight NeuroNarrator's capacity to integrate temporal, spectral, and spatial dynamics, positioning it as a foundational framework for time-frequency-aware, open-ended clinical interpretation of electrophysiological data.
Figures & tables
Figure 1: Illustrative example of segment-level, clinically grounded EEG interpretation. In contrast to coarse recording-level interpretaion, this sample demonstrates the generation of a fine-grained clinical narrative for a 10-second segment. The generated text systematically synthesizes four dimensions of electrophysiological analysis: (i) Event & Clinical Labels, identifying specific morphological abnormalities (e.g., spike-and-slow-wave complexes); (ii) Spatial Energy Distribution, capturing prominent signal and energy variations in specific anatomical regions (e.g., right fronto-temporal); (iii) Frequency-Domain Features, identifying the dominant spectral bands; and (iv) Temporal Context, characterizing the non-stationary evolution of brain states relative to preceding segments.
Figure 2: NeuroNarrator architecture for spectro–spatially grounded and temporally coherent EEG-to-text generation. (a) Dual-stream spectro–spatial grounding encodes each EEG segment using a pretrained EEG encoder operating on multichannel waveforms and a frozen vision encoder processing the corresponding scalp topographic map. Modality-specific features are projected into a shared latent space and aligned via a contrastive objective, enforcing correspondence between spectral dynamics and spatial energy distributions. (b) State-space–inspired generative modeling conditions text generation on both the aligned spectro–spatial embedding of the current segment and a short trajectory of preceding EEG segments, serving as a proxy for latent brain-state evolution. These continuous embeddings are injected as soft prompt tokens, replacing designated placeholder positions in the language-model prompt alongside task instructions, enabling the synthesis of clinically grounded narratives that preserve waveform morphology, dominant frequency structure, spatial localization, and temporal dynamics.
Figure 3: Overview of NeuroCorpus-160K construction. (a) Distribution of the aggregated datasets across major clinical domains. (b) The unified data processing workflow, which transforms raw recordings into clinically grounded narratives via three stages: signal preprocessing, structured feature extraction, and LLM-driven description refinement.
Dataset
Task
Train
Test
Samples
Hours
Samples
Hours
AD-65 Miltiadous et al. (2023)
Alzheimer’s Disease Detection
5,460
15.17
1,382
3.84
ADHD-Child Nasrabadi et al. (2020)
ADHD Detection
1,147
3.19
330
0.92
Grasp-and-Lift Luciw et al. (2014)
Hand Manipulation Event Detection
2,531
7.03
894
2.48
HMC Alvarez-Estevez and Rijsman (2021)
Sleep Stage Classification
25,000
69.44
1,000
2.78
MDD-64 Mumtaz (2016)
Major Depressive Disorder Detection
5,738
15.94
1,466
4.07
Table 1: Constituent datasets and split statistics for NeuroCorpus-160K. Summary of the 16 public EEG datasets integrated into NeuroCorpus-160K, including the associated task and the sizes of the training and held-out test splits, reported as the number of non-overlapping 10-s segments (Samples) and total recording duration (Hours).
Component
Structured Template
LLM Refined Descriptions
Event & Clinical Labels
This EEG segment from a 19-year-old female subject is recorded during a mental arithmetic task with cognitive load.
In this EEG segment from a 19-year-old female subject recorded during a mental arithmetic task with cognitive load, the observed patterns reflect task-related dynamics across frequency bands and scalp regions.
Frequency-Domain Features(Conditioned on {xt−k}k=1N,xt )
The delta frequency band (0–4 Hz) exhibits the highest power and shows a moderate decrease. Theta activity (4–8 Hz) shows a noticeable increase, while the alpha band (8–12 Hz) presents a significant decrease. Stable readings are observed in both the beta (12–30 Hz) and gamma (30–50 Hz) bands, while the high-gamma band (50–75 Hz) remains consistently low.
The EEG power is predominantly concentrated in the fronto-central and parietal regions, with the C4 channel maintaining the highest energy levels.
Table 2: Example of the transformation from structured quantitative scaffolds to clinically grounded narratives. This comparison demonstrates the synthesis of rigid feature templates (left) into a fluent, coherent description (right). Note that while the Frequency-Domain refinement is conditioned on the spectral history of preceding segments ( {xt−k}k=1N ) alongside the current segment ( xt ) to infer temporal trends, only the current segment’s features are listed here for conciseness.
Dataset
Topographic Map Retrieval
EEG Signal Retrieval
R@1
R@5
R@10
MeanR
R@1
R@5
R@10
MeanR
AD-65 Miltiadous et al. (2023)
88.35
98.91
99.71
1.38
92.04
98.99
99.78
1.22
ADHD-Child Nasrabadi et al. (2020)
87.27
96.06
98.48
2.58
88.79
96.97
98.18
1.87
Grasp_and_Lift Luciw et al. (2014)
74.72
93.96
97.32
2.19
77.63
95.19
97.87
1.88
MDD-64 Mumtaz (2016)
85.95
97.82
99.11
1.50
87.65
98.29
99.05
1.51
Motor Imagery Schalk et al. (2004)
71.45
91.94
95.89
2.94
75.82
93.70
96.47
2.54
Table 3: Quantitative assessment of bidirectional spectro-spatial retrieval. The table reports cross-modal retrieval performance across 16 heterogeneous datasets, evaluating the alignment accuracy between temporal EEG embeddings and spatial topographic map embeddings. Metrics include Recall@K ( R@1,R@5,R@10 ; higher is better) and Mean Rank ( MeanR ; lower is better).
Figure 4: Visualization of the learned spectro-spatial manifold via t-SNE Maaten and Hinton (2008) projection. The plots depict the latent distribution of EEG segments (triangles) and corresponding topographic maps (circles) sampled from the held-out NeuroCorpus-160K evaluation split. Left: In the absence of contrastive alignment, the representations exhibit a distinct modality gap, with temporal and spatial features forming fragmented, disjoint clusters. Right: Following contrastive optimization, the embedding space demonstrates a rigorous topological correspondence; matched EEG and topographic map pairs are tightly co-located within dataset-specific clusters.
Figure 5: Statistical validation of the learned spectro-spatial metric space. Left: Cosine similarity matrix. The pronounced diagonal dominance confirms a rigorous one-to-one correspondence between temporal EEG embeddings and spatial topographic map embeddings, effectively minimizing off-diagonal ambiguity. Right: Distribution of cosine similarity scores for matched versus mismatched pairs.
Figure 6: Multi-dimensional clinical evaluation of generated narratives. We assess the physiological fidelity of NeuroNarrator on the held-out NeuroCorpus-160K test set using a structured GPT-4.1 adjudicator. To capture the complexity of expert interpretation, each generated narrative is scored against the ground truth along five complementary axes: (i) clinical event or condition identification, (ii) dominant brain region localization, (iii) dominant frequency band classification, (iv) temporal trend description, and (v) detection of salient non-dominant spectral changes, with each axis contributing one point to a maximum score of 5.
Figure 7: Alignment of human and automated scoring distributions. Comparison of clinical fidelity scores assigned by human experts versus the GPT-4.1 adjudicator on a stratified subset of 1,500 narratives.
Dataset
BERTScore
Fact-F1
ROUGE-L
AD-65 Miltiadous et al. (2023)
0.750 ± 0.045
0.665 ± 0.237
0.419 ± 0.079
ADHD-Child Nasrabadi et al. (2020)
0.743 ± 0.049
0.706 ± 0.218
0.411 ± 0.084
Grasp and Lift Luciw et al. (2014)
0.728 ± 0.044
0.737 ± 0.251
0.365 ± 0.078
MDD-64 Mumtaz (2016)
0.732 ± 0.044
0.747 ± 0.209
0.386 ± 0.079
Motor Imagery Schalk et al. (2004)
0.718 ± 0.044
0.692 ± 0.247
0.353 ± 0.076
PD-31 Rockhill et al. (2021)
0.741 ± 0.053
0.683 ± 0.235
0.397 ± 0.091
Table 4: Quantitative assessment of linguistic and semantic fidelity on NeuroCorpus-160K. Performance of NeuroNarrator across all held-out benchmark datasets is evaluated using complementary automated metrics: BERTScore, Rule-based Fact-F1 and ROUGE-L.
Methods
TUAB
TUEV
TUSL
SEED
HMC
Workload
Non-LLM Models
EEGNet Lawhern et al. (2018)
79.3 ± 0.3
35.7 ± 1.0
36.1 ± 3.8
47.1 ± 0.8
62.2 ± 0.8
61.1 ± 5.5
EEGConformer Song et al. (2022)
80.1 ± 0.3
32.1 ± 1.0
44.9 ± 2.1
44.9 ± 0.8
49.5 ± 0.8
75.5 ± 4.0
SPaRCNet Jing et al. (2023)
79.8 ± 1.1
31.4 ± 3.6
40.0 ± 3.2
48.6 ± 2.4
66.0 ± 1.7
73.2 ± 4.9
ContraWR Yang et al. (2023b)
82.3 ± 0.3
42.2 ± 1.4
39.9 ± 1.4
56.4 ± 2.8
69.8 ± 0.8
63.3 ± 3.1
CNN-Transformer Peh et al. (2022)
82.2 ± 0.6
36.0 ± 4.7
35.5 ± 3.1
56.2 ± 1.2
69.9 ± 0.5
58.3 ± 7.7
Table 5: Benchmarking Open-Vocabulary Interpretation Against Specialized Closed-Set Baselines. We report Balanced Accuracy across six representative subsets of NeuroCorpus-160K. The evaluation contrasts NeuroNarrator (an open-vocabulary generative model) against two categories of discriminative baselines: conventional non-LLM deep learning models and NeuroLM, a closed-set LLM baseline.
Figure 8: Cross-domain zero-shot generalization and architectural ablation analysis. (a) Zero-shot classification performance on three external datasets (Depression-122, Siena, and SEED-IV) strictly excluded from NeuroCorpus-160K. (b) Ablation study on NeuroCorpus-160K quantifying the contribution of specific modules to clinical narrative fidelity.
Electroencephalography (EEG) foundation models learn reusable representations of brain signals for transfer across people, tasks, and recording setups.
Clinical EEG interpretation requires reasoning over full EEG sessions and integrating signal patterns with clinical context. Existing EEG foundation models are largely designed for short-window decoding and do not incorporate clinical context. We introduce CLEF, a clinically grounded long-context EEG foundation model. CLEF represents EEG sessions as 3D multitaper spectrogram tokens, enabling tractable Transformer modeling at session scale, and aligns embeddings with neurologist reports and structured EHR data through contrastive objectives. We evaluate CLEF on a new 234-task benchmark spanning disease phenotypes, medication exposures, and EEG findings, with more than 260k EEG sessions from over 108k patients. CLEF outperforms prior EEG foundation models on 229 of 234 tasks, improving mean AUROC from 0.65 to 0.74. Reconstruction-only pretraining surpasses prior EEG foundation models, while report and EHR alignment yields further gains. Held-out concept and external-cohort experiments suggest that these representations transfer beyond observed alignment targets. These results support session-scale, clinically grounded representation learning as a promising foundation-model paradigm for clinical EEG.
Peng Cao, Ali Mirzazadeh, Jong Woo Lee +2
MIT CSAIL · Brigham and Women’s Hospital, Harvard Medical School · Massachusetts General Hospital, Harvard Medical School
Electroencephalography (EEG)-to-language models offer a promising approach for translating complex neural signals into human-readable descriptions. However, generating plausible language does not necessarily demonstrate that a model has learned to interpret the underlying EEG signal. A direct way to assess such interpretation is to probe objectively measurable properties of the signal itself. We therefore construct an EEG-language dataset of 160,781 segments annotated with signal-level question answering (QA) pairs, where each question targets one of five property families and requires a quantitative comparison involving spectral, spatial, temporal, or lateral signal properties. Each segment is also paired with a feature-grounded EEG caption. We further propose a Dual-Domain EEG Encoder that jointly models temporal waveforms and spectral-spatial representations, and introduce EEG-Captioner, a unified EEG-to-language framework trained with both signal-level QA and EEG caption supervision. Experiments show that the proposed encoder achieves the highest overall macro-averaged balanced accuracy (Macro-BA) among the compared EEG encoders, while EEG-Captioner achieves a Macro-BA of 88.81% on signal-level QA and improves caption factual consistency from 82.06% to 83.46% over caption-only language adaptation, demonstrating the complementary benefits of signal-level QA and caption supervision.
Leyuan Jin, Jiahao Mei, Yixuan Yuan +3
X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China
EEG foundation models such as BIOT, LaBraM, and EEGMamba have achieved remarkable performance in neural signal decoding, but their black-box nature limits clinical trust and neuroscientific validation. We propose a unified attribution framework for interpreting EEG foundation models across heterogeneous architectures. The framework integrates gradient-, perturbation-, and activation-based explanation methods to analyze model behavior in spatial, temporal, and frequency dimensions. Spatially, it identifies critical EEG channels and visualizes their distributions using topographic maps. Temporally, it highlights decision-relevant signal segments through attribution heatmaps. In the frequency domain, it quantifies the contributions of canonical EEG rhythms via spectral perturbation analysis. To assess explanation reliability, we introduce a population-level evaluation combining Area Over the Perturbation Curve (AOPC) and cross-method consistency analysis. The framework further leverages Large Language Models (LLMs) to transform structured attribution outputs into natural-language reports, bridging low-level neural representations and high-level semantic reasoning. Experiments on benchmark datasets, including Mumtaz2016 and TUAB, demonstrate that the generated explanations are consistent with established neurophysiological markers, validating meaningful neural representations while exposing potential dependencies on artifacts and spurious patterns. The proposed framework provides a standardized approach for evaluating the interpretability, reliability, and physiological plausibility of EEG foundation models.