ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms

Haitao Li; Chenglin Li; Zhengyao Ding; Ziyu Li; Yiheng Mao; Zhengxing Huang

cs.LGSep 27, 2026

ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms

Authors: Haitao Li, Chenglin Li, Zhengyao Ding, Ziyu Li, Yiheng Mao, Zhengxing Huang

Organizations: Zhejiang University · Shanghai Innovation Institute

Abstract

Multimodal large language models (MLLMs) can now interpret a standard ten-second, twelve-lead electrocardiogram (ECG) with clinically grounded, reward-verified reasoning. Real cardiac monitoring is different. Ambulatory (Holter) and telemetry recordings span hours to days and are read as they stream in, and their clinically decisive findings are paroxysmal, brief episodes buried in an otherwise unremarkable trace. Such a recording cannot be held in one context at diagnostic resolution, and its future has not yet happened, so a reader must work online, deciding what to measure now, committing evidence to memory as it passes, and reporting events as they occur. We recast long-duration ECG interpretation as a long-horizon, online (streaming, causal) sequential decision process and introduce ECG-Scroll. As a benchmark, long ambulatory recordings are streamed to an agent chunk by chunk, and it must localize, quantify, and promptly flag paroxysmal events without access to future signal; because the underlying signal is retained, every answer is checkable against objective ground truth, giving rule-based rather than judge-based rewards, and the streaming formulation adds a metric batch evaluation cannot express, the detection latency between an event's onset and the moment the agent records it. As an agent environment, it is a fixed, gym-style interaction layer that exercises three competencies single-glance ECG models never touch: Memory, Tool use through signal-grounded measurement rather than reading pixels, and Planning of what to measure now and when to commit. We release 390 whole-recording instances spanning 2,536 hours of two-lead ambulatory ECG and evaluate a signal-threshold rule agent alongside off-the-shelf LLM agents online, characterizing how they use memory, tools, and planning and where the benchmark's head-room lies.

Figures & tables

Figure 1: ECG-Scroll at a glance. A long ambulatory recording streams left to right, and a monotonic cursor ( now ) advances one chunk at a time, splitting it into the observable past and the not-yet-arrived future. The agent observes only the past: it measures the elapsed signal with signal-grounded tools and writes cursor-timestamped findings to memory.

Figure 2: The AF core. (a) Whole-recording durations. (b) AF episodes per recording (log axis). (c) Individual AF episode durations (log axis). (d) Per-recording AF burden, the fraction of time in AF.

Task	Source	Answer	Verifier / Metric ( $\uparrow$ )	$N$	Mean dur.
Episode detection	afdb, ltafdb	interval set	temporal $F_{1}$ (IoU $\geq\tau$ )	107	20.5 h
AF burden	afdb, ltafdb	scalar	rel. error score	107	20.5 h
Change detection	afdb	change-point	hit@ $\pm 10$ s ( $\|\hat{t}-t^{\star}\|\leq\delta$ )	23	10.2 h
Ischemia detection	edb	interval set	temporal $F_{1}$ (IoU $\geq 0.75$ )	85	2.0 h
PVC burden	mitdb, ltdb	scalar	rel. error score	55	3.1 h
Rare-event search	mitdb, ltdb	interval	hit@ $\pm 15$ s (midpoint tol.)	13	0.5 h

Table 1: The ECG-Scroll task suite. Each task is a whole-recording streaming instance with an objective, rule-based verifier.

N

is the number of whole-recording instances and Mean dur. their average length.

Action / tool	Target	Output / effect
Control actions (drive the causal stream)
skim(duration)	cover / plan	advance the cursor by duration s; return the newly-elapsed span’s lossy profile (peak feature + coarse sub-bins)
measure(tool,t0,t1)	localize	run one tool the model chooses over an elapsed sub-window without advancing time; picks among the signal (and, under +dx , diagnostic) tools below
write(entry)	commit	append a cursor-timestamped finding to memory; the answer is assembled from these when the stream ends
Signal tools (numbers only, no label; clinician’s reading order)
get_rate_rhythm	rate, AF, pauses	hr_bpm, rr_mean/sd_ms, rr_irregularity (CV), rr_min/max ( per-beat: rr_intervals_ms)

Table 2: The ECG-Scroll agent’s actions and tools. Control actions drive the causal stream. Signal tools, which the model chooses among when it calls measure , are organized in a clinician’s reading order and return numbers only. Diagnostic tools instead return a ready-made classification; we evaluate agents with and without them.

		Episode	AF burden	Change	Ischemia	PVC burden	Rare-event
Agent	Tier	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 10s $\uparrow$ )	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 15s $\uparrow$ )
Rule Agent	–dx	0.02	0.26	0.17	0.07	0.34	0.00
	+dx	0.52	0.83	0.39	0.80	0.25	0.85
Qwen3-8B	–dx	0.03	0.17	0.13	0.00	0.47	0.00
	+dx	0.03	0.24	0.17	0.00	0.58	0.46
Qwen3-30B	–dx	0.03	0.17	0.17	0.00	0.47	0.00

Table 3: Planning agent: the model drives its own scan. Each model is run without ( --dx ) and with ( +dx ) the diagnostic tools, against the Rule Agent reference. Metrics are accuracies in

[0,1]

, higher is better.

		Episode	AF burden	Change	Ischemia	PVC burden	Rare-event
Agent	Tier	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 10s $\uparrow$ )	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 15s $\uparrow$ )
Rule Agent	–dx	0.02	0.26	0.17	0.07	0.34	0.00
	+dx	0.52	0.83	0.39	0.80	0.25	0.85
Qwen3-8B	–dx	0.02	0.30	0.09	0.04	0.23	0.00
	+dx	0.50	0.96	0.35	0.80	0.45	0.77
Qwen3-30B	–dx	0.01	0.31	0.04	0.05	0.41	0.00

Table 4: Judge agent: the environment walks the record chunk by chunk and the model only judges which elapsed chunks are events, with planning removed.

Figure 3: Planning agent behavior aggregated across all six tasks with the diagnostic tools (task-equal-weighted). Left: mean skim, measure, and write actions per recording. Right: fraction of recordings ending with no committed finding.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Task and dataset coverage. (a) Instance counts across the five task types and five databases. (b) Fraction of each stream occupied by target events, from AF episodes and transient ischemic ST-episodes down to near-instantaneous rare-event targets (log axis).

Electrocardiogram Interpretation

See all 34 papers

Aug 19, 2026

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

While multimodal large language models (MLLMs) excel in medical applications, most of them favor static images or short-term signals. In the critical field of dynamic electrocardiograms (ECG), models struggle with complex temporal reasoning and diagnostic report generation due to a lack of high-quality datasets and benchmarks. To address this, we introduce (i) Holtercare-23K, a large-scale multimodal dynamic ECG dataset comprising 22,980 QA pairs derived from 788 clinical Holter records and featuring a novel signal-video-text tri-modal alignment. Based on this dataset, we present (ii) Holtercare-Bench, a multimodal benchmark that evaluates models on temporal localization, clinical diagnosis, and global summarization. Zero-shot evaluations of leading MLLMs reveal a significant performance gap in processing ultra-long pathological sequences. However, fine-tuning representative models yields substantial improvements. This work illuminates the limitations of current MLLMs in electrophysiology and provides a foundational benchmark for long-term medical MLLMs. Our project is available at https://github.com/ZJU4HealthCare/Holtercare-Bench.

Yihan Xie, Hanwen Cui, Runze Ye +8

Zhejiang University · Beijing Institute of Technology · University of Electronic Science and Technology of China

Electrocardiography Electrocardiogram Interpretation Multimodal Model Evaluation Multimodal Large Language Models

Ask PDF

Mar 15, 2026

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce ECG-Reasoning-Benchmark, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.

Jungwoo Oh, Hyunseung Chung, Junhee Lee +6

KAIST · Ajou University School of Medicine · Yonsei University College of Medicine +1

Electrocardiogram Interpretation Clinical Reasoning Multimodal Large Language Models Multimodal Model Evaluation Electrocardiography

Ask PDF

Feb 17, 2026

CAMEL: An ECG Language Model for Forecasting Cardiac Events

Electrocardiograms (ECG) are electrical recordings of the heart that are critical for diagnosing cardiovascular conditions. ECG language models (ELMs) have recently emerged as a promising framework for ECG classification accompanied by report generation. However, current models cannot forecast future cardiac events despite the immense clinical value for planning earlier intervention. To address this gap, we propose CAMEL, the first ELM that is capable of inference over longer signal durations which enables its forecasting capability. Our key insight is a specialized ECG encoder which enables cross-understanding of ECG signals with text. We train CAMEL using established LLM training procedures, combining LoRA adaptation with a curriculum learning pipeline. Our curriculum includes ECG classification, metrics calculations, and multi-turn conversations to elicit reasoning. CAMEL demonstrates strong zero-shot performance across 6 tasks and 9 datasets, including ECGForecastBench, a new benchmark that we introduce for forecasting arrhythmias. CAMEL is on par with or surpasses ELMs and fully supervised baselines both in- and out-of-distribution, achieving SOTA results on ECGBench (+7.0% absolute average gain) as well as ECGForecastBench (+12.4% over fully supervised models and +21.1% over zero-shot ELMs).

Neelay Velingker, Alaia Solko-Breslin, Mayank Keoliya +9

University of Pennsylvania

Electrocardiogram Foundation Models Electrocardiogram Interpretation Clinical Outcome Prediction Electrocardiogram Classification Forecasting Benchmarks

Ask PDF

Abstract

Electrocardiogram Interpretation

Explore similar work

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

CAMEL: An ECG Language Model for Forecasting Cardiac Events