Organizations: School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Language, Intelligence, and Machine Center, Shenzhen Loop Area Institute, Shenzhen, China · School of Computing, National University of Singapore, Singapore · College of Systems Engineering, National University of Defense Technology, Changsha, China · School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, China
Electroencephalography (EEG) is widely used in clinical assessment, brain-computer interfaces, and cognitive-state analysis, yet EEG modeling remains fragmented across task paradigms. Recent multimodal large language models (MLLMs) provide a promising interface for flexible multimodal reasoning, but their ability to understand EEG inputs remains limited. In this work, we propose OneEEG, a unified EEG-MLLM framework for multi-task EEG understanding. By training on our curated OneEEG-Instruct, a multi-task EEG instruction-tuning dataset, OneEEG jointly models numerical EEG representations and visual EEG representations, and introduces a task-aware Mixture of Projectors (MoP) to align heterogeneous EEG features across different task families with the LLM embedding space. We further establish OneEEG-Bench, a benchmark for evaluating current MLLMs and our method on multi-task EEG understanding. Experiments show that existing MLLMs have limited ability to interpret EEG inputs, while OneEEG achieves stronger performance by integrating numerical and visual EEG representations with task-aware EEG-to-LLM projection. Extensive ablations and analyses further reveal the effects of visual EEG renderings, MoP variants, router behavior, and feature dispersion across EEG tasks.
Figures & tables
Fig. 1: Motivation of MoP. Colored points denote EEG features from different task families or datasets. A shared EEG-to-LLM projector forces heterogeneous EEG features through a single projector, which may cause cross-task conflicts as indicated by the red dashed paths. MoP mitigates this bottleneck by aligning task-dependent EEG features through task-aware projection.
Fig. 2: Construction pipeline of OneEEG-Instruct and OneEEG-Bench. Raw EEG datasets are first converted into EEG signal frames with metadata, followed by preprocessing, subject-level splitting, and EEG image rendering. Question-answer annotations are then generated from the processed signals, images, and metadata, forming label-constrained MCQ and yes/no QA samples.
Dataset
Family
Setting
Train/Eval
Format
BCIC-IV-2a [ 40 ]
Motor
Std.
4.6k / 320
MCQ+Y/N
EEG Motor [ 41 ]
Motor
Std.
17.7k / 320
MCQ+Y/N
EEGMAT [ 42 ]
State
Std.
3.8k / 240
MCQ+Y/N
SEED-IV [ 43 ]
State
Std.
60.1k / 360
MCQ+Y/N
TUAB [ 44 , 45 ]
Clinical
Std.
50.0k / 300
Y/N
TUEP [ 44 , 45 ]
Clinical
Std.
50.9k / 300
Y/N
TABLE I: Dataset composition of OneEEG-Instruct and OneEEG-Bench.
Fig. 3: Example of the closed-set QA annotation design in OneEEG. Task-oriented QA focuses on dataset-specific EEG tasks, while basic attribute QA focuses on general signal-level properties.
Fig. 4: Overview of OneEEG. OneEEG integrates numerical EEG representations, visual EEG representations, and text instructions into a unified MLLM input. OneEEG uses both numerical and visual streams for standard-setting datasets and the visual stream only for non-standard-setting datasets. The left panel details MoP: basic EEG attribute questions are assigned to a fixed basic projector, while task-oriented questions are handled by a router branch that selects one projector from the projector bank.
Fig. 5: Two-stage training strategy of OneEEG. In Stage 1, projector alignment trains MoP while keeping the signal encoder, image encoder, visual projector, and the LLM backbone frozen. In Stage 2, multimodal instruction tuning further trains MoP, the visual projector, and LoRA parameters of the LLM, while the signal and image encoders remain frozen. Snowflakes denote frozen modules, and flames denote trainable modules.
Model
Invalid ↓
Basic
Standard channel setting
Non-standard channel setting
Overall
Macro
Motor
State
Clinical
Event
Sleep
Vigilance
Closed-source MLLMs
Gemini-3 Flash [ 53 ]
5.94
26.77
41.09
52.50
51.50
52.72
53.66
52.97
47.02
47.32
Gemini-3 Pro [ 26 ]
2.62
38.75
31.72
50.67
51.83
48.04
71.25
69.73
49.27
51.71
GPT-5.5 [ 25 ]
0.02
46.31
36.56
49.67
52.83
51.30
58.71
75.68
50.39
53.01
Open-source MLLMs
TABLE II: Main evaluation results on OneEEG-Bench. Invalid denotes the invalid-output rate, while all remaining columns report accuracy.
Fig. 6: Per-dataset evaluation results on OneEEG-Bench. Each axis reports the accuracy computed over all samples from the corresponding dataset, with larger values indicating better performance. For readability, only representative baselines are shown in the radar plot.
Variant
Projector
Route Strategy
Visual Prefix
Task Family
Overall
Macro
Basic
Motor
State
Clinical
Event
Shared
Shared
–
–
67.96
44.69
47.83
63.00
59.13
56.42
56.52
Shared + Prefix
Shared
–
✓
67.78
41.88
49.33
65.17
61.09
57.06
57.05
MoP-Free
MoP
Free
–
67.78
39.84
43.33
56.83
58.48
53.33
53.25
MoP-Free + Prefix
MoP
Free
✓
69.63
43.91
50.83
65.50
58.59
57.39
57.69
MoP
MoP
Fixed
–
71.11
39.22
43.00
64.17
55.00
54.06
54.50
TABLE III: Ablation study on the core design choices of OneEEG under the Phase 1 setting. All metrics follow the same definitions as in the main evaluation, restricted to the Phase 1 evaluation set.
Model
Input
Basic
Standard channel setting
Non-standard channel setting
Overall
Macro
Motor
State
Clinical
Event
Sleep
Vigilance
GPT-5.5
Waveform
46.31
36.56
49.67
52.83
51.30
58.71
75.68
50.39
53.01
Spectrogram
28.41
36.72
49.33
50.00
32.83
60.10
55.68
42.49
44.72
Waveform+Spectrogram
45.65
33.44
50.67
52.00
50.65
55.57
75.14
49.22
51.87
Qwen2.5-VL-7B
Waveform
29.39
32.19
52.00
50.33
25.43
65.33
84.32
42.73
48.43
Spectrogram
26.60
31.09
41.17
49.33
20.98
63.76
76.22
38.86
44.16
TABLE IV: Ablation on EEG visual representations. We compare waveform images, spectrogram images, and their combination using GPT-5.5 and Qwen2.5-VL-7B. All metrics follow the same definitions as in the main evaluation.
Fig. 7: t-SNE visualization of EEG representations before projection (a) and after projection (b), (c). Each point denotes an evaluation sample, and colors indicate task families. MoP produces more structured family-level grouping after projection.
Fig. 8: Comparison of MoP router variants under the Phase 1 setting. Hard top-1 routing and gated top-1 routing with noise achieve comparable Overall and Macro performance.
Fig. 9: MoP router projector usage under hard top-1 routing. Rows denote projector slots and columns denote datasets, with each cell showing the selection ratio.
Electroencephalography (EEG) studies brain-generated electrical activity to understand cognition and support diagnosis, monitoring, and brain-computer interfaces.
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California · Alfred E. Mann Department of Biomedical Engineering, University of Southern California +1
State Key Laboratory of Brain-machine Intelligence, Zhejiang University · College of Computer Science and Technology, Zhejiang University · MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University