Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction

Electroencephalography (EEG) analysis extends beyond assigning predefined labels to recordings; it requires workflows connecting natural-language instructions, signal processing, quantitative evidence, and scientific interpretation. We term this capability comprehensive EEG understanding. Existing evaluations, however, primarily target isolated decoding tasks or system-specific demonstrations, leaving the competence of large language models (LLMs) insufficiently quantified. We introduce \benchmarkname{}, a unified benchmark for comprehensive, instruction-conditioned EEG understanding. It comprises four subsets---Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration---covering 17 datasets, \numcases{} tasks, and over \numinstances{} real-data instances. Given an instruction and EEG recordings with optional physiological signals, a system must perform the analysis and produce a scientifically grounded report and, when required, artifacts. Outputs are assessed through numerical, categorical, set, sequence, semantic, and artifact validation. We evaluate \nummodels{} representative LLMs across more than 100K executions under two paradigms: autonomous code execution with CodeAct and structured agentic analysis with BrainAgent. Results vary substantially across models, subsets, difficulty levels, and execution paradigms, showing that EEG competence depends on the model and its operationalization. \benchmarkname{} provides a reproducible testbed for advancing LLM-based EEG understanding. The code and benchmark will be released soon, with evaluation results continuously updated.

Foundation models (FMs) promise to extract unified representations that generalize across downstream tasks. They have emerged across fields, including electroencephalography (EEG), but it is less clear how effective they are in this particular field. Published evaluations differ in datasets, in the EEG-specific preprocessing that might influence reported results, and in the reported metrics, frequently obscuring the clinical relevance in EEG. We introduce NeuroAtlas, the largest EEG benchmark to date: 42 datasets and 260k hours covering clinical EEG (epilepsy, sleep medicine, brain age estimation) and brain-computer interfaces, and include multiple datasets per task along with bespoke clinical evaluation metrics. Besides evaluating EEG-FMs with respect to supervised baselines, we present results from generic time-series FMs. We report three findings. First, EEG-specific FMs do not consistently outperform time-series FMs, which have neither EEG-focused architectures nor been pretrained on EEG. Second, standard machine learning metrics are insufficient to assess clinical utility: thus, we thoroughly evaluate more appropriate measures such as the quality of event-level decision-making, hypnogram-derived features, and the brain-age gap in the domains of epilepsy, sleep, and brain age, respectively. Third, model rankings and performance can vary substantially within domains. We conclude that pretrained models perform largely on par, with only narrow advantages for a few, and that current models do not yet deliver on the promise of an out-of-the-box unified EEG model. NeuroAtlas exposes this gap and provides the datasets and metrics for the next generation of unified EEG FMs.

Electroencephalography (EEG) classification has shown significant promise in applications ranging from neurodegenerative disease detection to brain-computer interfaces (BCIs). Whilst deep learning methods have increasingly dominated the research landscape in recent years; these models remain largely opaque 'black boxes' with limited interpretability. In this paper, we revisit the utility of feature-based classification via a systematic pipeline designed to integrate per-electrode signal statistics with cross-brain connectivity metrics. The proposed "KnowEEG" pipeline extracts 783 per-electrode statistical features alongside between-electrode connectivity statistics, utilizing a balanced sampling strategy between high-dimensional per electrode features and connectivity. This feature-based pipeline achieves performance comparable to or exceeding state-of-the-art deep learning models across five diverse classification tasks: emotion detection, mental workload classification, eyes open/closed detection, abnormal EEG classification, and event detection. This performance is achieved without the need for Graphics Processing Unit (GPU) resources. Furthermore, the pipeline provides inherent explainability through feature importance rankings. We demonstrate via the eyes open/closed task that this explainability identifies discriminative biomarkers in the occipital region that are consistent with established neuroscience literature. These results suggest that despite significant advances in deep learning, structured feature engineering remains a high performance and interpretable method for EEG classification.