ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

Jungwoo Oh; Hyunseung Chung; Junhee Lee; Min-Gyu Kim; Hangyul Yoon; Ki Seong Lee; Youngchae Lee; Muhan Yeo

cs.LGMar 15, 2026

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

Authors: Jungwoo Oh, Hyunseung Chung, Junhee Lee, Min-Gyu Kim, Hangyul Yoon, Ki Seong Lee, Youngchae Lee, Muhan Yeo, +1 more

Organizations: KAIST · Ajou University School of Medicine · Yonsei University College of Medicine · Seoul National University Bundang Hospital

Abstract

While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce ECG-Reasoning-Benchmark, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.

Figures & tables

Figure 1: End-to-end schematic of the Automated ECG Analysis Pipeline. The workflow illustrates a systematic progression: from the initial delineation of fundamental waveforms in the raw signal to the quantification of physiological features, which are then mapped into discrete clinical findings and ultimately resulting in a definitive diagnostic conclusion.

Figure 2: An illustrative example of the 4-step recursive evaluation loop for diagnosing Complete Left Bundle Branch Block. The process begins with an Initial Diagnostic Question, followed by the verification of necessary findings. The figure depicts the first loop, which verifies the presence of “Prolonged QRS duration” grounded in the wave segment and measurement range, and the fourth loop, which confirms the “Notched R wave in lateral leads” with specific lead and wave grounding.

Source Dataset	Model	IDA (%)	Completion (%)	Depth (0-4)	GT-RDA (%)
PTB-XL	PULSE (7B) Liu et al. (2024)	$80.93$	$5.45$	$1.24$	$35.18$
	GEM (7B) Lan et al. (2025)	$84.37$	$6.13$	$1.34$	$65.66$
	ECG-R1-SFT (8B) Jin et al. (2026)	$85.12$	$5.45$	$1.80$	$21.76$
	ECG-R1-RL (8B) Jin et al. (2026)	$\bf 85.41$	$5.90$	$1.79$	$22.70$
	OpenTSLM (3B) Langer et al. (2025)	$54.77$	$0.65$	$0.17$	$37.58$
	Hulu-Med (7B) Jiang et al. (2025)	$55.87$	$4.96$	$1.20$	$86.87$

Table 1: Quantitative benchmark results of the evaluated models on ECG-Reasoning-Benchmark across both PTB-XL and MIMIC-IV-ECG source datasets. Models are grouped into ECG-specific, medical domain, general domain, and proprietary categories, with the highest score for each metric highlighted in bold within its respective source dataset.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Metric	P Wave		QRS Complex		T Wave
Method	Metric	P Onset	P Offset	QRS Onset	QRS Offset	T Onset	T Offset
ecgpuwave Goldberger et al. (2000)	Recall	$0.872_{\pm 0.013}$	$0.872_{\pm 0.013}$	$0.993_{\pm 0.003}$	$0.992_{\pm 0.003}$	$0.890_{\pm 0.008}$	$0.848_{\pm 0.010}$
ecgpuwave Goldberger et al. (2000)	Precision	$0.818_{\pm 0.029}$	$0.821_{\pm 0.030}$	$0.998_{\pm 0.002}$	$0.997_{\pm 0.002}$	$0.983_{\pm 0.008}$	$0.911_{\pm 0.001}$
NeuroKit2 Makowski et al. (2021)	Recall	$0.969_{\pm 0.005}$	$0.967_{\pm 0.006}$	$0.861_{\pm 0.011}$	$0.796_{\pm 0.017}$	$0.781_{\pm 0.018}$	$0.722_{\pm 0.011}$
NeuroKit2 Makowski et al. (2021)	Precision	$0.762_{\pm 0.040}$	$0.762_{\pm 0.038}$	$0.854_{\pm 0.012}$	$0.789_{\pm 0.018}$	$0.908_{\pm 0.018}$	$0.836_{\pm 0.024}$
UNet3+ Joung et al. (2024)	Recall	$0.972_{\pm 0.004}$	$0.972_{\pm 0.004}$	$0.999_{\pm 0.001}$	$0.999_{\pm 0.001}$	$0.983_{\pm 0.006}$	$0.981_{\pm 0.006}$

Appendix

Table 2: Quantitative performances for the ECG segmentation task. The evaluation was conducted on the LUDB dataset Kalyakulina et al. (2020) . Mean and 95% confidence interval are shown across 3 different train, validation, and test splits. We followed the recommendations of the Association for Medical Instrumentation Association for the Advancement of Medical Instrumentation (1999) , where it is considered that an onset or an offset is detected correctly if their deviation from the GT annotations does not exceed in the tolerance of 150 ms.

Figure 3: Comparison of P wave detection before (top) and after (bottom) post-processing. Shaded backgrounds indicate GT annotations, while colored signals show predictions. Our pipeline correctly identifies the unannotated P waves (circled), which are penalized as False Positives due to the missing GT.

Figure 4: Prevalence of diagnostic labels in the PTB-XL (left) and MIMIC-IV-ECG (right) datasets. Diagnoses directly covered by the 17 core logic diagrams of ECG-Reasoning-Benchmark are highlighted in coral, while conditions that can be indirectly derived from these core diagnoses are highlighted in light coral.

Source Dataset		PTB-XL	MIMIC-IV-ECG
# of Unique ECGs		$2,868$	$3,316$
# of Cases (Multi-turn)	$+$	$1,377$	$1,655$
# of Cases (Multi-turn)	$-$	$1,707$	$1,704$
# of Total QA pairs		$24,097$	$27,025$
Avg. reasoning turns		$7.81$	$8.05$

Appendix

Table 3: Overall statistics of ECG-Reasoning-Benchmark derived from the PTB-XL and MIMIC-IV-ECG datasets

Dx		Paths	# Cases	# Per Reasoning Path	# Cases	# Per Reasoning Path
		# Reasoning	PTB-XL		MIMIC-IV-ECG
1AVB	$+$	1	100	100	100	100
1AVB	$-$	2	100	11 \| 89	100	50 \| 50
2AVB	$+$	1	3	3	49	49
2AVB	$-$	2	100	50 \| 50	100	50 \| 50
3AVB	$+$	2	9	0 \| 9	100	50 \| 50

Appendix

Table 4: Detailed case distribution for the 17 core diagnoses in ECG-Reasoning-Benchmark . The table outlines the total number of positive (

+

) and negative (

-

) cases, the number of distinct reasoning paths for each outcome, and the specific stratification of cases across these paths for both PTB-XL and MIMIC-IV-ECG source datasets.

Figure 5: System prompt provided to the evaluated models

Figure 6: Logic diagram for First Degree AV Block (1AVB)

Figure 7: Logic diagram for Second Degree AV Block (2AVB)

Figure 8: Logic diagram for Third Degree AV Block (3AVB)

Figure 9: Logic diagram for Complete Left Bundle Branch Block (CLBBB)

Figure 10: Logic diagram for Complete Right Bundle Branch Block (CRBBB)

Figure 11: Logic diagram for Left Anterior Fascicular Block (LAFB)

Figure 12: Logic diagram for Left Posterior Fascicular Block (LPFB)

Figure 13: Logic diagram for Left Ventricular Hypertrophy (LVH)

Figure 14: Logic diagram for Right Ventricular Hypertrophy (RVH)

Figure 15: Logic diagram for Premature Atrial Complex (PAC) and Premature Ventricular Complex (PVC)

Figure 16: Logic diagram for Anterior Myocardial Infarction (AMI)

Figure 17: Logic diagram for Inferior Myocardial Infarction (IMI)

Figure 18: Logic diagram for Lateral Myocardial Infarction (LMI)

Figure 19: Logic diagram for Anterior Ischemia (ISCAN)

Figure 20: Logic diagram for Inferior Ischemia (ISCIN)

Figure 21: Logic diagram for Lateral Ischemia (ISCLA)

Explore similar work

How Well Do Multimodal Models Reason on ECG Signals?
Feb 27, 2026Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu +11Multimodal Reasoning Multimodal Model Evaluation
University of Illinois Urbana Champaign · Google Inc · Rush University +4
Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classification
May 17, 2026Yang Wu, Xiaoyan Yuan, Hau-San Wong +1RL for Language Model Reasoning Electrocardiogram Classification
1City University of Hong Kong · 2Beijing Institute of Technology · 3Shenzhen MSU-BIT University
Diagnosing as Cardiologists Do: ECG Agents with Doctor-Grounded Priors for Clinical Reasoning Across Diseases and Populations
Aug 10, 2026Hongxiang Gao, He-yang Xu, Yuwen Li +6Electrocardiography Electrocardiogram Classification
State Key Laboratory of Digital Medical Engineering, Southeast University · School of Instrument Science and Engineering, Southeast University

Feb 27, 2026J/K move · Enter open · S save

How Well Do Multimodal Models Reason on ECG Signals?

Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu, Patrick Langer+10

University of Illinois Urbana Champaign · Google Inc · Rush University+4

Multimodal Reasoning Multimodal Model Evaluation Medical Reasoning Multimodal Large Language Models Clinical Reasoning Electrocardiography

Open paper AskPDF arXiv

While multimodal large language models offer a promising solution to the "black box" nature of health AI by generating interpretable reasoning traces, verifying the validity of these traces remains a critical challenge. Existing evaluation methods are either unscalable, relying on manual clinician review, or superficial, utilizing proxy metrics (e.g. QA) that fail to capture the semantic correctness of clinical logic. In this work, we introduce a reproducible framework for evaluating reasoning in ECG signals. We propose decomposing reasoning into two distinct, components: (i) Perception, the accurate identification of patterns within the raw signal, and (ii) Deduction, the logical application of domain knowledge to those patterns. To evaluate Perception, we employ an agentic framework that generates code to empirically verify the temporal structures described in the reasoning trace. To evaluate Deduction, we measure the alignment of the model's logic against a structured database of established clinical criteria in a retrieval-based approach. This dual-verification method enables the scalable assessment of "true" reasoning capabilities.

Abstract

Explore similar work

How Well Do Multimodal Models Reason on ECG Signals?

Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classification

Diagnosing as Cardiologists Do: ECG Agents with Doctor-Grounded Priors for Clinical Reasoning Across Diseases and Populations