EEGProbe: Do Vision-Language Models Really Understand EEG? A Benchmark for Signal-Grounded EEG Signals Evaluation
Organizations: Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, 819-0395, Japan · Department of Civil Engineering, Sichuan University of Science and Engineering, Zigong, 643002, China · Faculty of Arts and Science, Kyushu University, Fukuoka, 819-0395, Japan
Abstract
Vision-language models (VLMs) have shown strong multimodal capabilities across scientific and biomedical domains, yet their ability to understand electroencephalography (EEG) signals remains unclear. We introduce EEGProbe, a signal-grounded benchmark for evaluating whether general-purpose VLMs can extract and reason from minimally processed, channel-wise EEG without task-specific feature engineering or EEG-specific adaptation. EEGProbe contains 620 EEG-centered questions across 13 subtasks and three capability categories: EEG Signal Classification, EEG Cross-modal Alignment, and EEG Neurophysiological Reasoning. To reduce shortcut-based answering, EEGProbe preserves the temporal structure of the recordings where possible, minimizes benchmark-induced processing, and applies signal-necessity testing followed by expert verification. We evaluate 11 representative open- and closed-source VLMs under a unified zero-shot protocol. The strongest model, GPT-5.6 Sol, achieves an overall score of only 41.63%, with cross-modal EEG alignment emerging as a major bottleneck. We further examine model behavior from a neurophysiological perspective and analyze how temporal, spectral, spatial, and physiological EEG structure relates to model successes and failures. Inference-time neurophysiological guidance can reshape model predictions by helping models recognize the neurophysiological patterns present in the input and connect them with relevant neurological knowledge, narrowing the gap between general multimodal competence and EEG understanding. EEGProbe provides a testbed for current VLMs and a basis for developing future multimodal models that can interpret EEG evidence more reliably. The EEGProbe benchmark will be publicly available at https://huggingface.co/datasets/2122ZHU/EEGProbe.
Figures & tables
| Question Format | Response Requirement | Representative EEG Use |
|---|---|---|
| Multiple-choice | Models select the correct answer from four candidate options. | EEG Signal Classification and Cross-modal Alignment. |
| Closed-form | Models provide a constrained response, such as a category, numerical value, or short phrase. | Seizure detection, ERP recognition, temporal reasoning, and data quality assessment. |
| Open-ended response | Models provide a free-form interpretation grounded in the given EEG signal. | Frequency-band analysis and brain-region interpretation. |
| Category | Abbr. | Subtask | # Instances | Question Format |
| EEG Signal Classification | MI | Motor Imagery Classification | 50 | Multiple-choice |
| ER | Emotion Recognition | 50 | Multiple-choice | |
| SSC | Sleep Stage Classification | 50 | Multiple-choice | |
| SD | Seizure Detection | 50 | Closed-form | |
| ERP | Event-Related Potential Recognition | 50 | Closed-form | |
| EEG Cross-Modal Alignment | EIM | EEG–Image Matching | 50 | Multiple-choice |
| EEG Signal Classification | EEG Cross-modal Alignment | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | MI | ERP | ER | SSC | SD | EAM | EIM | ETM | EVM |
| GPT-5.6 Sol | 34.00 | 56.00 | 28.00 | 34.00 | 64.00 | 22.00 | 24.00 | 18.00 | 24.00 |
| Gemini-3.8-Flash † | 24.00 | 62.00 | 26.00 | 34.00 | 58.00 | 24.00 | 36.00 | 26.00 | 22.00 |
| DeepSeek-V4.1-Flash † | 26.00 | 40.00 | 28.00 | 36.00 | 50.00 | 28.00 | 24.00 | 34.00 | 20.00 |
| MiniMax-M3 †,§ | 24.00 | 48.00 | 24.00 | 28.00 | 50.00 | 20.00 | 30.00 | 20.00 | 30.00 |
| GPT-5.6 Luna † | 20.00 | 52.00 | 20.00 | 28.00 | 56.00 | 28.00 | 36.00 | 26.00 | 16.00 |
| Method | Correct | Accuracy |
|---|---|---|
| Baseline | 4/20 | 20.0% |
| Baseline + guidance | 8/20 | 40.0% |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Criterion | Weight |
|---|---|
| Correctly identifies Signal A as having stronger posterior alpha activity. | 0.30 |
| Reports posterior regions or supporting channels consistent with the evidence, such as P3, P4, and P8. | 0.25 |
| Characterizes the effect as strong and correctly describes its posterior specificity. | 0.25 |
| Relates the pattern conservatively to alpha blocking or the Berger effect. | 0.20 |
| Total | 1.00 |
| Criterion | Weight |
|---|---|
| Correctly identifies a late positive pattern with relative centroparietal prominence. | 0.25 |
| Reports positive polarity and a compatible peak latency near 371 ms ( 50 ms). | 0.20 |
| Uses sample-specific evidence such as P3, Pz, and CPz and the approximately 8.7 V centroparietal dominance. | 0.25 |
| Provides a conservative P3-like target-evaluation or context-updating interpretation. | 0.20 |
| Avoids definitive claims about component identity, source localization, or cognitive state. | 0.10 |
| Total | 1.00 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,055 papers