Recasting Visual Language Models as EEG Foundation Models for Clinical Neurodiagnostics
Organizations: Massachusetts General Hospital, Boston, MA, 02114, USA. · Binghamton University, State University of New York, Binghamton, NY, 13902, USA. · Harvard Medical School, Boston, MA, 02115, USA. · Brigham and Women’s Hospital, Boston, 02115, MA, USA.
Abstract
Historically automated electroencephalography (EEG) systems was done using task-specific supervised models trained on purposely annotated data. A more recent modeling paradigm relies on creating a unified representation, or a 'foundation model', for multiple modality-specific tasks through large-scale domain-specific pretraining on the unannotated EEG signal. The quality of this unified representation directly depends on the availability of the modality-specific datasets. At the same time, generalist Vision Language Models (VLMs) exhibit broad few-shot learning capacities as well as good fine-tuning adaptability to different visual and language tasks. Can VLMs learn how to read and interpret EEG visually, emulating to a degree the way human clinicians inspect the EEG patterns? In this work we make a case that they can: we show that generalist VLMs without large-scale EEG pretraining perform competitively across a collection of clinical neuro-diagnostic tasks, can adapt to different tasks in a few-shot learning setting better than specialist foundation models, and exhibit better performance with model scaling when modality-specific models fail to exhibit.
Figures & tables
| TUEV | TUAB | |||||
| Model | TUEH | EEG Pre. | Canon | Bench | Canon | Bench |
| Qwen (2b) | 0 | 0 | 0.573 | 0.650 | - | - |
| Qwen (4b) | 0 | 0 | 0.601 | 0.715 | 0.805 | 0.783 |
| Qwen (8b) | 0 | 0 | 0.634 | 0.750 | - | - |
| Qwen (32b) | 0 | 0 | 0.653 | 0.746 | - | - |
| BIOT (2023) | 1 | 1 | 0.528 | 0.570 | 0.795 | 0.795 |
| Dataset (partition) | Task | Original channels | Window (s) | Train | Validation | Test | Classes | Subjects |
| TUAB (official) | Normal–abnormal | 21 | 10 | 300,187 | 72,323 | 36,945 | 2 | 1,660/416/253 |
| TUAB (EEG-FM) | Normal–abnormal | 21 | 15 | 247,976 | 12,315 | 12,277 | 2 | 2,076/101/152 |
| TUEV (official) | Six-class event classification | 22 | 5 | 61,697 | 22,235 | 29,421 | 6 | 232/58/80 |
| TUEV (EEG-FM) | Six-class event classification | 22 | 5 | 87,834 | 12,473 | 13,046 | 6 | 328/20/22 |
| vEpiSet | IED detection | 29 | 4 | 20,306 | 2,430 | 2,713 | 2 | 67/8/9 |
| SPaRCNet | Six-class event classification | 16 | 10 | 384 * | 2,473 | 3,083 | 6 | 276/75/94 |
Electroencephalography
Electroencephalography (EEG) studies brain-generated electrical activity to understand cognition and support diagnosis, monitoring, and brain-computer interfaces.
See all 200 papers