EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

Huyu Wu; Weining Weng; Yuchen Liu; Yiqiang Chen; Yang Gu

cs.LGSep 10, 2026

EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

Authors: Huyu Wu, Weining Weng, Yuchen Liu, Yiqiang Chen, Yang Gu

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence accumulation, multi-step reasoning, and coordinated use of specialized signal-processing tools. Although large language models (LLMs) have recently shown promise as autonomous agents for EEG analysis, existing EEG agentic evaluations remain fragmented, covering limited tasks over narrow temporal horizons with inconsistent protocols, and providing no comprehensive assessment of agents' reasoning, tool-use, and workflow construction capabilities. To address this gap, we propose EEGAgentBench, a unified benchmark for systematically evaluating LLM agents on short- and long-horizon EEG analysis. EEGAgentBench spans six representative EEG applications ranging from knowledge question answering to sleep staging. It encompasses signal durations from 2 seconds to nearly 23 hours, with prediction targets ranging from class labels to event intervals and epoch-level sequences. This design supports unified evaluation across knowledge reasoning, short-horizon interpretation, long-horizon event detection, and sequential understanding. The benchmark further provides 10 deterministic EEG analysis tools that expose only task-relevant signal measurements. Agents must therefore select tools autonomously, accumulate evidence iteratively, and construct multi-step workflows. For evaluation, we benchmark 29 frontier LLMs from 15 model families. Results demonstrate that EEGAgentBench effectively distinguishes agent capabilities beyond model scale and inference cost, while revealing substantial limitations of current LLM agents in long-horizon EEG analysis, particularly in sustained evidence accumulation and multi-step reasoning.

Figures & tables

Figure 2: Overview of the six EEGAgentBench tasks and the agent–environment interaction.

Setting	Task	Data source	$N$	Signal scale	Output	Main metrics
Knowledge	T1 EEG Knowledge QA	MedMCQA-EEG-strict	38	No signal	One label (A/B/C/D)	Accuracy
Short horizon	T2 Artifact Identification	EEGdenoiseNet	300	2 s, 1 channel	One label (clean/ocular/muscle)	Accuracy, Macro-F1
Short horizon	T3 Normal-vs-Epileptic	Bonn	188	23.6 s, 1 channel	One label (normal/epileptic)	Accuracy, Macro-F1
Short horizon	T4 Dementia Cohort	ds004504	69	5.1–21.5 min, 19 channels	One label (AD/FTD/HC)	Accuracy, Macro-F1
Long horizon	T5 Seizure Detection	CHB-MIT	280	12.4 min–4.0 h, 18–32 EEG channels	Event interval list	Dice-S, Event-F1
Long horizon	T6 Sleep Staging	Sleep-EDFx	197	5.7–23.0 h, 5–7 PSG channels	30-s epoch label sequence	Macro-F1, Cohen’s kappa

Table 1: Overview of the six EEGAgentBench tasks. PSG denotes polysomnography.

Panel of Table 2 — Table 2: Main results on EEGAgentBench. Acc. denotes accuracy, Event-F1 denotes the subject-averaged SzCORE-style event-based F1 [ 20 ] , and Dice-S denotes Dice@seizure. Overall is the unweighted mean of T1 accuracy, Macro-F1 for T2–T4 and T6, and Event-F1 for T5. Best values are shown in blue bold , and second-best values are italicized .

Figure 3: Dataset construction and filtering process.

Figure 4: Tool-use behavior across the five signal tasks. (a) Mean tool-use rate, averaged equally across 29 models. (b) Per-model median tool calls per trajectory for fixed Top-5 and Bottom-5 groups ranked by Overall.

Figure 5: Task-level performance distributions across 29 models using the six primary metrics that form Overall. Horizontal jitter only separates overlapping points. Boxes show medians and interquartile ranges; whiskers extend to 1.5 IQR, and labeled points mark task leaders. Dashed lines indicate uniform-random baselines for T1–T4; no random baseline is shown for T5 or T6.

Figure 6: Estimated cost versus Overall for 19 models. The dashed line marks the cost–performance Pareto frontier.

Figure 7: Long-horizon EEG error analysis. (a) Event-miss rate on seizure-positive records versus over-alarm rate on seizure-free records for T5. (b–c) Hypnograms from five agents on 7.4-hour and 21.9-hour Sleep Cassette records; numbers denote recording-level Macro-F1. (d) Per-model Macro-F1 across seven recording-duration bins for T6. Thin lines denote model means, the band denotes the interquartile range across models, and dark line denotes median.

Parameter	Value	Role
Minimum overlap	0 s	A prediction matches a reference event if they overlap for any positive duration
Tolerance before start	30 s	Extend each reference event 30 s before its start
Tolerance after end	60 s	Extend each reference event 60 s after its end
Minimum gap between events	90 s	Merge events whose gap is strictly shorter than 90 s
Maximum event duration	300 s	Split events longer than 300 s into consecutive events
Time resolution	10 Hz	Quantize interval boundaries to 0.1 s

Table 3: Default parameters of the SzCORE-style event-based F1 used for T5.

Model	Family	Parameters	Context
Qwen3-1.7B	Qwen	1.7B	32K
Qwen3-4B	Qwen	4B	32K
Qwen3-8B	Qwen	8B	32K
ERNIE-4.5-21B-A3B	ERNIE	21B-A3B	128K
Qwen3.6-27B	Qwen	27B	256K
Qwen3.5-35B-A3B	Qwen	35B-A3B	256K

Table 4: Parameter counts and context windows of the 29 evaluated models.

Model	Prompt tokens (M)	Completion tokens (M)	All tokens (M)
LongCat-2	589.8	8.9	598.7
DeepSeek-V4-Pro	405.3	4.5	409.8
Claude-Opus-4.6	390.6	17.7	408.3
GLM-5.2	381.4	26.2	407.6
Grok-4.5	389.9	5.8	395.7
Hy3	320.0	25.7	345.7

Table 5: Token usage for the full evaluation, in millions of tokens.

Model	Input price	Output price
GPT-5.5	5.0	30.0
GPT-5.6-sol	5.0	30.0
Claude-Opus-4.6	5.0	25.0
GPT-5.4	2.5	15.0
Qwen3.7-Max	2.5	7.5
Grok-4.5	2.0	6.0

Table 6: Public input and output prices used for cost estimation, in USD per million tokens.

Figure 8: SNR dose–response on T2 artifact identification. Left: ocular (EOG) artifacts. Right: muscle (EMG) artifacts. SNR decreases from

+2

-7

dB so that the artifact grows stronger from left to right; the vertical axis is per-class accuracy at each SNR level. Four models illustrate balanced identification, over-calling, artifact-type confusion, and failure to detect artifacts. The band and center line show the interquartile range and median across all 29 models, and the dashed line marks the uniform-random level of

1/3

. The reference column on the far left shows clean-class accuracy for the four highlighted models.

Figure 9: T5 case (GPT-5.6-sol, CHB-MIT chb15_40 ): reference and predicted seizure intervals over the one-hour recording. The model detects the second and third seizures but misses the first.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding
Aug 4, 2026Yangxuan Zhou, Sha Zhao, Yuning Chen +4Electroencephalography LLM Evaluation
State Key Laboratory of Brain-machine Intelligence, Zhejiang University · College of Computer Science and Technology, Zhejiang University · MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University
CogEEGAgent: Toward Autonomous Cognitive EEG Analysis with Grounded Execution and Selection-Aware Verification
Jul 27, 2026Dengzhe Hou, Lingyu Jiang, Fangzhou Lin +1Electroencephalography Selective Inference
1Graduate School of Information Sciences, Tohoku University · 2Unprecedented-scale Data Analytics Center, Tohoku University · 3Texas A&M University +1
What the LLM Should Not Say: Boundary-Aware Context Grounding for A Seven-Channel EEG Agent
Jun 25, 2026Zhiyuan Xu, Yueqing Dai, Junling Li +1Electroencephalography LLM Grounding
Shanghai Pulse Element Intelligent Technology Co., Ltd. (NeuraDock)

Aug 4, 2026J/K move · Enter open · S save

BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding

Yangxuan Zhou, Sha Zhao, Yuning Chen, Chen Wu+3

State Key Laboratory of Brain-machine Intelligence, Zhejiang University · College of Computer Science and Technology, Zhejiang University · MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University

Electroencephalography LLM Evaluation

Open paper AskPDF arXiv

Electroencephalography (EEG) analysis extends beyond assigning predefined labels to recordings; it requires workflows connecting natural-language instructions, signal processing, quantitative evidence, and scientific interpretation. We term this capability comprehensive EEG understanding. Existing evaluations, however, primarily target isolated decoding tasks or system-specific demonstrations, leaving the competence of large language models (LLMs) insufficiently quantified. We introduce \benchmarkname{}, a unified benchmark for comprehensive, instruction-conditioned EEG understanding. It comprises four subsets---Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration---covering 17 datasets, \numcases{} tasks, and over \numinstances{} real-data instances. Given an instruction and EEG recordings with optional physiological signals, a system must perform the analysis and produce a scientifically grounded report and, when required, artifacts. Outputs are assessed through numerical, categorical, set, sequence, semantic, and artifact validation. We evaluate \nummodels{} representative LLMs across more than 100K executions under two paradigms: autonomous code execution with CodeAct and structured agentic analysis with BrainAgent. Results vary substantially across models, subsets, difficulty levels, and execution paradigms, showing that EEG competence depends on the model and its operationalization. \benchmarkname{} provides a reproducible testbed for advancing LLM-based EEG understanding. The code and benchmark will be released soon, with evaluation results continuously updated.

Abstract

LLM Agent Evaluation

Explore similar work

BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding

CogEEGAgent: Toward Autonomous Cognitive EEG Analysis with Grounded Execution and Selection-Aware Verification

What the LLM Should Not Say: Boundary-Aware Context Grounding for A Seven-Channel EEG Agent