eess.SPFeb 24, 2026
SaveNeuroNarrator: A Generalist EEG-to-Text Foundation Model for Clinical Interpretation via Spectro-Spatial Grounding and Temporal State-Space Reasoning
Organizations: Department of Systems Engineering, Stevens Institute of Technology, 1 Castle Point Terrace, Hoboken, 07030, New Jersey, USA.
Abstract
Electroencephalography (EEG) provides a non-invasive window into neural dynamics at high temporal resolution and plays a pivotal role in clinical neuroscience research. Despite this potential, prevailing computational approaches to EEG analysis remain largely confined to task-specific classification objectives or coarse-grained pattern recognition, offering limited support for clinically meaningful interpretation. To address these limitations, we introduce NeuroNarrator, the first generalist EEG-to-text foundation model designed to translate electrophysiological segments into precise clinical narratives. A cornerstone of this framework is the curation of NeuroCorpus-160K, the first harmonized large-scale resource pairing over 160,000 EEG segments with structured, clinically grounded natural-language descriptions. Our architecture first aligns temporal EEG waveforms with spatial topographic maps via a rigorous contrastive objective, establishing spectro-spatially grounded representations. Building on this grounding, we condition a Large Language Model through a state-space-inspired formulation that integrates historical temporal and spectral context to support coherent clinical narrative generation. This approach establishes a principled bridge between continuous signal dynamics and discrete clinical language, enabling interpretable narrative generation that facilitates expert interpretation and supports clinical reporting workflows. Extensive evaluations across diverse benchmarks and zero-shot transfer tasks highlight NeuroNarrator's capacity to integrate temporal, spectral, and spatial dynamics, positioning it as a foundational framework for time-frequency-aware, open-ended clinical interpretation of electrophysiological data.
Figures & tables
Figure 1: Illustrative example of segment-level, clinically grounded EEG interpretation. In contrast to coarse recording-level interpretaion, this sample demonstrates the generation of a fine-grained clinical narrative for a 10-second segment. The generated text systematically synthesizes four dimensions of electrophysiological analysis: (i) Event & Clinical Labels, identifying specific morphological abnormalities (e.g., spike-and-slow-wave complexes); (ii) Spatial Energy Distribution, capturing prominent signal and energy variations in specific anatomical regions (e.g., right fronto-temporal); (iii) Frequency-Domain Features, identifying the dominant spectral bands; and (iv) Temporal Context, characterizing the non-stationary evolution of brain states relative to preceding segments.
Figure 2: NeuroNarrator architecture for spectro–spatially grounded and temporally coherent EEG-to-text generation. (a) Dual-stream spectro–spatial grounding encodes each EEG segment using a pretrained EEG encoder operating on multichannel waveforms and a frozen vision encoder processing the corresponding scalp topographic map. Modality-specific features are projected into a shared latent space and aligned via a contrastive objective, enforcing correspondence between spectral dynamics and spatial energy distributions. (b) State-space–inspired generative modeling conditions text generation on both the aligned spectro–spatial embedding of the current segment and a short trajectory of preceding EEG segments, serving as a proxy for latent brain-state evolution. These continuous embeddings are injected as soft prompt tokens, replacing designated placeholder positions in the language-model prompt alongside task instructions, enabling the synthesis of clinically grounded narratives that preserve waveform morphology, dominant frequency structure, spatial localization, and temporal dynamics.
Figure 3: Overview of NeuroCorpus-160K construction. (a) Distribution of the aggregated datasets across major clinical domains. (b) The unified data processing workflow, which transforms raw recordings into clinically grounded narratives via three stages: signal preprocessing, structured feature extraction, and LLM-driven description refinement.
| Dataset | Task | Train | Test | ||
| Samples | Hours | Samples | Hours | ||
| AD-65 Miltiadous et al. (2023) | Alzheimer’s Disease Detection | 5,460 | 15.17 | 1,382 | 3.84 |
| ADHD-Child Nasrabadi et al. (2020) | ADHD Detection | 1,147 | 3.19 | 330 | 0.92 |
| Grasp-and-Lift Luciw et al. (2014) | Hand Manipulation Event Detection | 2,531 | 7.03 | 894 | 2.48 |
| HMC Alvarez-Estevez and Rijsman (2021) | Sleep Stage Classification | 25,000 | 69.44 | 1,000 | 2.78 |
| MDD-64 Mumtaz (2016) | Major Depressive Disorder Detection | 5,738 | 15.94 | 1,466 | 4.07 |
Table 1: Constituent datasets and split statistics for NeuroCorpus-160K. Summary of the 16 public EEG datasets integrated into NeuroCorpus-160K, including the associated task and the sizes of the training and held-out test splits, reported as the number of non-overlapping 10-s segments (Samples) and total recording duration (Hours).
| Component | Structured Template | LLM Refined Descriptions |
|---|---|---|
| Event & Clinical Labels | This EEG segment from a 19-year-old female subject is recorded during a mental arithmetic task with cognitive load. | In this EEG segment from a 19-year-old female subject recorded during a mental arithmetic task with cognitive load, the observed patterns reflect task-related dynamics across frequency bands and scalp regions. |
| Frequency-Domain Features(Conditioned on ) | Frequency band powers: delta (0–4 Hz): 39.8676 ; theta (4–8 Hz): 11.5915 ; alpha (8–12 Hz): 5.7254 ; beta (12–30 Hz): 5.2747 ; gamma (30–50 Hz): 1.1490 ; high-gamma (50–75 Hz): 0.0810 . | The delta frequency band (0–4 Hz) exhibits the highest power and shows a moderate decrease. Theta activity (4–8 Hz) shows a noticeable increase, while the alpha band (8–12 Hz) presents a significant decrease. Stable readings are observed in both the beta (12–30 Hz) and gamma (30–50 Hz) bands, while the high-gamma band (50–75 Hz) remains consistently low. |
| Spatial Energy Distribution | Highest-energy channel is C4 (103.2591 ). High-energy channels: FZ, C4, PZ. Medium-energy channels: F3, F4, F8, C3, CZ, P3, P4, T4, T6. Low-energy channels: F7, O1, O2, T3, T5. | The EEG power is predominantly concentrated in the fronto-central and parietal regions, with the C4 channel maintaining the highest energy levels. |
Table 2: Example of the transformation from structured quantitative scaffolds to clinically grounded narratives. This comparison demonstrates the synthesis of rigid feature templates (left) into a fluent, coherent description (right). Note that while the Frequency-Domain refinement is conditioned on the spectral history of preceding segments ( ) alongside the current segment ( ) to infer temporal trends, only the current segment’s features are listed here for conciseness.
| Dataset | Topographic Map Retrieval | EEG Signal Retrieval | ||||||
|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | MeanR | R@1 | R@5 | R@10 | MeanR | |
| AD-65 Miltiadous et al. (2023) | 88.35 | 98.91 | 99.71 | 1.38 | 92.04 | 98.99 | 99.78 | 1.22 |
| ADHD-Child Nasrabadi et al. (2020) | 87.27 | 96.06 | 98.48 | 2.58 | 88.79 | 96.97 | 98.18 | 1.87 |
| Grasp_and_Lift Luciw et al. (2014) | 74.72 | 93.96 | 97.32 | 2.19 | 77.63 | 95.19 | 97.87 | 1.88 |
| MDD-64 Mumtaz (2016) | 85.95 | 97.82 | 99.11 | 1.50 | 87.65 | 98.29 | 99.05 | 1.51 |
| Motor Imagery Schalk et al. (2004) | 71.45 | 91.94 | 95.89 | 2.94 | 75.82 | 93.70 | 96.47 | 2.54 |
Table 3: Quantitative assessment of bidirectional spectro-spatial retrieval. The table reports cross-modal retrieval performance across 16 heterogeneous datasets, evaluating the alignment accuracy between temporal EEG embeddings and spatial topographic map embeddings. Metrics include Recall@K ( ; higher is better) and Mean Rank ( ; lower is better).
Figure 4: Visualization of the learned spectro-spatial manifold via t-SNE Maaten and Hinton (2008) projection. The plots depict the latent distribution of EEG segments (triangles) and corresponding topographic maps (circles) sampled from the held-out NeuroCorpus-160K evaluation split. Left: In the absence of contrastive alignment, the representations exhibit a distinct modality gap, with temporal and spatial features forming fragmented, disjoint clusters. Right: Following contrastive optimization, the embedding space demonstrates a rigorous topological correspondence; matched EEG and topographic map pairs are tightly co-located within dataset-specific clusters.
Figure 5: Statistical validation of the learned spectro-spatial metric space. Left: Cosine similarity matrix. The pronounced diagonal dominance confirms a rigorous one-to-one correspondence between temporal EEG embeddings and spatial topographic map embeddings, effectively minimizing off-diagonal ambiguity. Right: Distribution of cosine similarity scores for matched versus mismatched pairs.
Figure 6: Multi-dimensional clinical evaluation of generated narratives. We assess the physiological fidelity of NeuroNarrator on the held-out NeuroCorpus-160K test set using a structured GPT-4.1 adjudicator. To capture the complexity of expert interpretation, each generated narrative is scored against the ground truth along five complementary axes: (i) clinical event or condition identification, (ii) dominant brain region localization, (iii) dominant frequency band classification, (iv) temporal trend description, and (v) detection of salient non-dominant spectral changes, with each axis contributing one point to a maximum score of 5.
Figure 7: Alignment of human and automated scoring distributions. Comparison of clinical fidelity scores assigned by human experts versus the GPT-4.1 adjudicator on a stratified subset of 1,500 narratives.
| Dataset | BERTScore | Fact-F1 | ROUGE-L |
|---|---|---|---|
| AD-65 Miltiadous et al. (2023) | 0.750 0.045 | 0.665 0.237 | 0.419 0.079 |
| ADHD-Child Nasrabadi et al. (2020) | 0.743 0.049 | 0.706 0.218 | 0.411 0.084 |
| Grasp and Lift Luciw et al. (2014) | 0.728 0.044 | 0.737 0.251 | 0.365 0.078 |
| MDD-64 Mumtaz (2016) | 0.732 0.044 | 0.747 0.209 | 0.386 0.079 |
| Motor Imagery Schalk et al. (2004) | 0.718 0.044 | 0.692 0.247 | 0.353 0.076 |
| PD-31 Rockhill et al. (2021) | 0.741 0.053 | 0.683 0.235 | 0.397 0.091 |
Table 4: Quantitative assessment of linguistic and semantic fidelity on NeuroCorpus-160K. Performance of NeuroNarrator across all held-out benchmark datasets is evaluated using complementary automated metrics: BERTScore, Rule-based Fact-F1 and ROUGE-L.
| Methods | TUAB | TUEV | TUSL | SEED | HMC | Workload |
|---|---|---|---|---|---|---|
| Non-LLM Models | ||||||
| EEGNet Lawhern et al. (2018) | 79.3 0.3 | 35.7 1.0 | 36.1 3.8 | 47.1 0.8 | 62.2 0.8 | 61.1 5.5 |
| EEGConformer Song et al. (2022) | 80.1 0.3 | 32.1 1.0 | 44.9 2.1 | 44.9 0.8 | 49.5 0.8 | 75.5 4.0 |
| SPaRCNet Jing et al. (2023) | 79.8 1.1 | 31.4 3.6 | 40.0 3.2 | 48.6 2.4 | 66.0 1.7 | 73.2 4.9 |
| ContraWR Yang et al. (2023b) | 82.3 0.3 | 42.2 1.4 | 39.9 1.4 | 56.4 2.8 | 69.8 0.8 | 63.3 3.1 |
| CNN-Transformer Peh et al. (2022) | 82.2 0.6 | 36.0 4.7 | 35.5 3.1 | 56.2 1.2 | 69.9 0.5 | 58.3 7.7 |
Table 5: Benchmarking Open-Vocabulary Interpretation Against Specialized Closed-Set Baselines. We report Balanced Accuracy across six representative subsets of NeuroCorpus-160K. The evaluation contrasts NeuroNarrator (an open-vocabulary generative model) against two categories of discriminative baselines: conventional non-LLM deep learning models and NeuroLM, a closed-set LLM baseline.
Figure 8: Cross-domain zero-shot generalization and architectural ablation analysis. (a) Zero-shot classification performance on three external datasets (Depression-122, Siena, and SEED-IV) strictly excluded from NeuroCorpus-160K. (b) Ablation study on NeuroCorpus-160K quantifying the contribution of specific modules to clinical narrative fidelity.
More in this topic
EEG Foundation Models
Electroencephalography (EEG) foundation models learn reusable representations of brain signals for transfer across people, tasks, and recording setups.
See all 100 papers