ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms

Haitao Li; Chenglin Li; Zhengyao Ding; Ziyu Li; Yiheng Mao; Zhengxing Huang

cs.LGSep 27, 2026

ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms

Authors: Haitao Li, Chenglin Li, Zhengyao Ding, Ziyu Li, Yiheng Mao, Zhengxing Huang

Organizations: Zhejiang University · Shanghai Innovation Institute

Abstract

Multimodal large language models (MLLMs) can now interpret a standard ten-second, twelve-lead electrocardiogram (ECG) with clinically grounded, reward-verified reasoning. Real cardiac monitoring is different. Ambulatory (Holter) and telemetry recordings span hours to days and are read as they stream in, and their clinically decisive findings are paroxysmal, brief episodes buried in an otherwise unremarkable trace. Such a recording cannot be held in one context at diagnostic resolution, and its future has not yet happened, so a reader must work online, deciding what to measure now, committing evidence to memory as it passes, and reporting events as they occur. We recast long-duration ECG interpretation as a long-horizon, online (streaming, causal) sequential decision process and introduce ECG-Scroll. As a benchmark, long ambulatory recordings are streamed to an agent chunk by chunk, and it must localize, quantify, and promptly flag paroxysmal events without access to future signal; because the underlying signal is retained, every answer is checkable against objective ground truth, giving rule-based rather than judge-based rewards, and the streaming formulation adds a metric batch evaluation cannot express, the detection latency between an event's onset and the moment the agent records it. As an agent environment, it is a fixed, gym-style interaction layer that exercises three competencies single-glance ECG models never touch: Memory, Tool use through signal-grounded measurement rather than reading pixels, and Planning of what to measure now and when to commit. We release 390 whole-recording instances spanning 2,536 hours of two-lead ambulatory ECG and evaluate a signal-threshold rule agent alongside off-the-shelf LLM agents online, characterizing how they use memory, tools, and planning and where the benchmark's head-room lies.

Figures & tables

Figure 1: ECG-Scroll at a glance. A long ambulatory recording streams left to right, and a monotonic cursor ( now ) advances one chunk at a time, splitting it into the observable past and the not-yet-arrived future. The agent observes only the past: it measures the elapsed signal with signal-grounded tools and writes cursor-timestamped findings to memory.

Figure 2: The AF core. (a) Whole-recording durations. (b) AF episodes per recording (log axis). (c) Individual AF episode durations (log axis). (d) Per-recording AF burden, the fraction of time in AF.

Task	Source	Answer	Verifier / Metric ( $\uparrow$ )	$N$	Mean dur.
Episode detection	afdb, ltafdb	interval set	temporal $F_{1}$ (IoU $\geq\tau$ )	107	20.5 h
AF burden	afdb, ltafdb	scalar	rel. error score	107	20.5 h
Change detection	afdb	change-point	hit@ $\pm 10$ s ( $\|\hat{t}-t^{\star}\|\leq\delta$ )	23	10.2 h
Ischemia detection	edb	interval set	temporal $F_{1}$ (IoU $\geq 0.75$ )	85	2.0 h
PVC burden	mitdb, ltdb	scalar	rel. error score	55	3.1 h
Rare-event search	mitdb, ltdb	interval	hit@ $\pm 15$ s (midpoint tol.)	13	0.5 h

Table 1: The ECG-Scroll task suite. Each task is a whole-recording streaming instance with an objective, rule-based verifier.

N

is the number of whole-recording instances and Mean dur. their average length.

Action / tool	Target	Output / effect
Control actions (drive the causal stream)
skim(duration)	cover / plan	advance the cursor by duration s; return the newly-elapsed span’s lossy profile (peak feature + coarse sub-bins)
measure(tool,t0,t1)	localize	run one tool the model chooses over an elapsed sub-window without advancing time; picks among the signal (and, under +dx , diagnostic) tools below
write(entry)	commit	append a cursor-timestamped finding to memory; the answer is assembled from these when the stream ends
Signal tools (numbers only, no label; clinician’s reading order)
get_rate_rhythm	rate, AF, pauses	hr_bpm, rr_mean/sd_ms, rr_irregularity (CV), rr_min/max ( per-beat: rr_intervals_ms)

Table 2: The ECG-Scroll agent’s actions and tools. Control actions drive the causal stream. Signal tools, which the model chooses among when it calls measure , are organized in a clinician’s reading order and return numbers only. Diagnostic tools instead return a ready-made classification; we evaluate agents with and without them.

		Episode	AF burden	Change	Ischemia	PVC burden	Rare-event
Agent	Tier	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 10s $\uparrow$ )	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 15s $\uparrow$ )
Rule Agent	–dx	0.02	0.26	0.17	0.07	0.34	0.00
	+dx	0.52	0.83	0.39	0.80	0.25	0.85
Qwen3-8B	–dx	0.03	0.17	0.13	0.00	0.47	0.00
	+dx	0.03	0.24	0.17	0.00	0.58	0.46
Qwen3-30B	–dx	0.03	0.17	0.17	0.00	0.47	0.00

Table 3: Planning agent: the model drives its own scan. Each model is run without ( --dx ) and with ( +dx ) the diagnostic tools, against the Rule Agent reference. Metrics are accuracies in

[0,1]

, higher is better.

		Episode	AF burden	Change	Ischemia	PVC burden	Rare-event
Agent	Tier	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 10s $\uparrow$ )	( $F_{1}\uparrow$ )	(score $\uparrow$ )	(hit@ $\pm$ 15s $\uparrow$ )
Rule Agent	–dx	0.02	0.26	0.17	0.07	0.34	0.00
	+dx	0.52	0.83	0.39	0.80	0.25	0.85
Qwen3-8B	–dx	0.02	0.30	0.09	0.04	0.23	0.00
	+dx	0.50	0.96	0.35	0.80	0.45	0.77
Qwen3-30B	–dx	0.01	0.31	0.04	0.05	0.41	0.00

Table 4: Judge agent: the environment walks the record chunk by chunk and the model only judges which elapsed chunks are events, with planning removed.

Figure 3: Planning agent behavior aggregated across all six tasks with the diagnostic tools (task-equal-weighted). Left: mean skim, measure, and write actions per recording. Right: fraction of recordings ending with no committed finding.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Task and dataset coverage. (a) Instance counts across the five task types and five databases. (b) Fraction of each stream occupied by target events, from AF episodes and transient ischemic ST-episodes down to near-instantaneous rare-event targets (log axis).

Electrocardiogram Interpretation

See all 34 papers

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis
Aug 19, 2026Yihan Xie, Hanwen Cui, Runze Ye +8Electrocardiography Electrocardiogram Interpretation
Zhejiang University · Beijing Institute of Technology · University of Electronic Science and Technology of China
ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
Mar 15, 2026Jungwoo Oh, Hyunseung Chung, Junhee Lee +6Electrocardiogram Interpretation Clinical Reasoning
KAIST · Ajou University School of Medicine · Yonsei University College of Medicine +1
CAMEL: An ECG Language Model for Forecasting Cardiac Events
Feb 17, 2026Neelay Velingker, Alaia Solko-Breslin, Mayank Keoliya +9Electrocardiogram Foundation Models Electrocardiogram Interpretation
University of Pennsylvania

Aug 19, 2026J/K move · Enter open · S save

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

Yihan Xie, Hanwen Cui, Runze Ye, Juekai Lin+7

Zhejiang University · Beijing Institute of Technology · University of Electronic Science and Technology of China

Electrocardiography Electrocardiogram Interpretation Multimodal Model Evaluation Multimodal Large Language Models

Open paper AskPDF arXiv

While multimodal large language models (MLLMs) excel in medical applications, most of them favor static images or short-term signals. In the critical field of dynamic electrocardiograms (ECG), models struggle with complex temporal reasoning and diagnostic report generation due to a lack of high-quality datasets and benchmarks. To address this, we introduce (i) Holtercare-23K, a large-scale multimodal dynamic ECG dataset comprising 22,980 QA pairs derived from 788 clinical Holter records and featuring a novel signal-video-text tri-modal alignment. Based on this dataset, we present (ii) Holtercare-Bench, a multimodal benchmark that evaluates models on temporal localization, clinical diagnosis, and global summarization. Zero-shot evaluations of leading MLLMs reveal a significant performance gap in processing ultra-long pathological sequences. However, fine-tuning representative models yields substantial improvements. This work illuminates the limitations of current MLLMs in electrophysiology and provides a foundational benchmark for long-term medical MLLMs. Our project is available at https://github.com/ZJU4HealthCare/Holtercare-Bench.

Abstract

Electrocardiogram Interpretation

Explore similar work

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

CAMEL: An ECG Language Model for Forecasting Cardiac Events