Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

Yihan Xie; Hanwen Cui; Runze Ye; Juekai Lin; Haoyang Wang; Jinhao Mao; Bo Zhang; Wenqiao Zhang

cs.LGAug 19, 2026

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

Authors: Yihan Xie, Hanwen Cui, Runze Ye, Juekai Lin, Haoyang Wang, Jinhao Mao, Bo Zhang, Wenqiao Zhang,

Organizations: Zhejiang University · Beijing Institute of Technology · University of Electronic Science and Technology of China

Abstract

While multimodal large language models (MLLMs) excel in medical applications, most of them favor static images or short-term signals. In the critical field of dynamic electrocardiograms (ECG), models struggle with complex temporal reasoning and diagnostic report generation due to a lack of high-quality datasets and benchmarks. To address this, we introduce (i) Holtercare-23K, a large-scale multimodal dynamic ECG dataset comprising 22,980 QA pairs derived from 788 clinical Holter records and featuring a novel signal-video-text tri-modal alignment. Based on this dataset, we present (ii) Holtercare-Bench, a multimodal benchmark that evaluates models on temporal localization, clinical diagnosis, and global summarization. Zero-shot evaluations of leading MLLMs reveal a significant performance gap in processing ultra-long pathological sequences. However, fine-tuning representative models yields substantial improvements. This work illuminates the limitations of current MLLMs in electrophysiology and provides a foundational benchmark for long-term medical MLLMs. Our project is available at https://github.com/ZJU4HealthCare/Holtercare-Bench.

Figures & tables

Figure 1. Overview of our proposed Holtercare-23K dataset.

Figure 2. The multimodal generation and data construction framework of HolterAgent.

Figure 3. Statistical distribution of Holtercare-23K. (a) Density estimation of recording durations. (b) Log-scale counts of beat-level annotations. (c) Frequencies of the top 10 rhythm-level events.

Figure 4. Zero-shot performance of four representative models across diverse metrics.

Figure 5. Performance comparison of Phi-4-mini-3.8B and Qwen3-VL-8B before and after fine-tuning. Metrics include Accuracy , Score MAE , and Score GPT . Tasks lacking these metrics are evaluated by F1-Bio .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Description	Count
Valid QRS Complexes
N	Normal Beat	71,429,269
Af	Atrial Fibrillation	4,822,970
S	Atrial Premature Beat (APB)	1,042,095
P	Unclassified Pacing	840,298
V	Ventricular Premature Beat (VPB)	779,227

Appendix

Table 7. Frequency of beat-level annotations across Holtercare-23K.

Event	Count	Event	Count
Premature Atrial Contraction (PAC)	888	Sinus Arrest	6
Premature Ventricular Contraction (PVC)	775	Atrial Flutter (AFL)	6
PAC Couplets	396	Atrial Escape Rhythm	4
Atrial Tachycardia (AT)	339	Wandering Atrial Pacemaker (WAP)	4
PVC Couplets	152	Ventricular Escape Rhythm	3
Ventricular Tachycardia (VT)	94	Atrial Undersensing	2

Appendix

Table 8. Frequency of rhythm-level annotations across Holtercare-23K.

Panel of Figure 6 — Figure 6. Prompt for QA pairs generation in Evidence Reasoning tasks of Open-QA .

Panel of Figure 7 — Figure 7. Prompt for report evaluation in General Summary tasks of Report Generation .

Model	Modality	Event Timing		Diagnosis		Evidence Reasoning
Model	Modality	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$
Generalist Models
GPT-5-mini		37.50	10.83	8.70	1.96	96.67	38.48
Claude-4.5-Haiku		11.68	1.25	6.98	0.55	57.53	8.97
Phi-4-mini-3.8B		53.57	13.22	53.85	15.73	96.49	55.56
Phi-4-mini-3.8B FT		95.00	77.39	98.46	87.05	93.70	72.64

Appendix

Table 9. Supplementary performance comparison of evaluated models on Open-QA tasks from Holtercare-Bench, evaluated by additional BLEU n-gram overlap metrics.

Model	Modality	Statistical Overview		General Summary
Model	Modality	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$
Generalist Models
GPT-5-mini		44.64	3.65	60.47	6.95
Claude-4.5-Haiku		15.92	1.15	27.60	1.00
Phi-4-mini-3.8B		56.76	10.99	52.43	6.03
Phi-4-mini-3.8B FT		97.67	68.03	94.83	71.94

Appendix

Table 10. Supplementary performance comparison of evaluated models on Report Generation tasks from Holtercare-Bench, evaluated by additional BLEU n-gram overlap metrics.

Model	Modality	Presence	Event Counting	Event Timing	HR Extremum Timing	Diagnosis
Generalist Models
InternVL-3.5-8B		58.01	29.66	47.26	31.85	39.37
MiniCPM-V4.5-8B		54.90	51.38	28.05	35.99	26.38
Gemini-3.0-Flash		67.32	14.98	62.50	53.18	41.73
Qwen3-VL-8B		60.95	31.80	28.66	34.71	32.28
Qwen3-VL-8B FT		95.10	83.79	97.23	71.97	94.88

Appendix

Table 11. Supplementary performance comparison of video-capable models evaluated under the text modality on Closed-QA tasks from Holtercare-Bench, evaluated by accuracy.

Model	Modality	Presence	Event Counting	Event Timing	HR Extremum Timing	Diagnosis
Phi-4-mini-3.8B		$1.95\times 10^{-5}$	$3.99\times 10^{-16}$	$6.33\times 10^{-20}$	$3.09\times 10^{-8}$	$1.76\times 10^{-11}$
Qwen3-VL-8B		$6.17\times 10^{-41}$	$5.26\times 10^{-32}$	$2.18\times 10^{-52}$	$6.57\times 10^{-18}$	$3.55\times 10^{-35}$
Qwen3-VL-8B		$7.08\times 10^{-44}$	$3.14\times 10^{-36}$	$6.14\times 10^{-47}$	$1.08\times 10^{-17}$	$1.30\times 10^{-28}$

Appendix

Table 12. Statistical significance analysis of improvements from zero-shot baselines to fine-tuned models on Closed-QA tasks from Holtercare-Bench. The reported

p

-values are derived from McNemar tests.

Electrocardiography

Electrocardiography studies the heart’s electrical signals to measure cardiac function and support diagnosis, monitoring, and research.

See all 107 papers

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
Mar 15, 2026Jungwoo Oh, Hyunseung Chung, Junhee Lee +6Electrocardiogram Interpretation Clinical Reasoning
KAIST · Ajou University School of Medicine · Yonsei University College of Medicine +1
ELF: A Family of Encoder-Free ECG-Language Models
Jan 5, 2026William Han, Tony Chen, Chaojing Duan +6Healthcare Electrocardiography
Carnegie Mellon University, USA · Allegheny Health Network, USA · University of California, Los Angeles, USA +1
ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms
Sep 27, 2026Haitao Li, Chenglin Li, Zhengyao Ding +3Electrocardiogram Interpretation Electrocardiography
Zhejiang University · Shanghai Innovation Institute

Mar 15, 2026J/K move · Enter open · S save

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

Jungwoo Oh, Hyunseung Chung, Junhee Lee, Min-Gyu Kim+5

KAIST · Ajou University School of Medicine · Yonsei University College of Medicine+1

Electrocardiogram Interpretation Clinical Reasoning Multimodal Large Language Models Multimodal Model Evaluation Electrocardiography

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce ECG-Reasoning-Benchmark, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.

Abstract

Electrocardiography

Explore similar work

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

ELF: A Family of Encoder-Free ECG-Language Models

ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms