RAG-TRACE: Tracing Measurement Evidence and Diagnostic Completeness in AI-Generated ECG Reports

Elyas Irankhah; Shahed Salehzehi; Parisa Hosseini

cs.CLNov 1, 2026

RAG-TRACE: Tracing Measurement Evidence and Diagnostic Completeness in AI-Generated ECG Reports

Authors: Elyas Irankhah, Shahed Salehzehi, Parisa Hosseini

Organizations: Department of Emergency Medicine, Yale University. · Department of Biomedical Engineering, Hakim Sabzevari University. · Neuroscience Research Center, Mashhad University of Medical Sciences.

Abstract

Purpose: Retrieval-augmented generation (RAG) can introduce electrocardiogram (ECG) criteria that lack patient-specific measurement support. This study evaluates the reliability of retrieved diagnostic criteria in AI-generated ECG reports. Methods: We analyzed 1,083 human-validated PTB-XL patient records using a library of 47 criteria cards, including 33 without numeric verification rules. Language models generated structured reports with and without retrieval. Criterion applicability was evaluated using Marquette 12SL measurements, with independent assessment using Uni-G and comparison against expert-validated diagnostic labels. Results: Keyword retrieval introduced untestable or measurement-disconfirmed criteria in 46.8% of GPT-4.1-mini reports. A deterministic verification rule withdrew 647 criterion applications, including 639 without numeric checks, while preserving 268 of 269 applications independently supported by Uni-G. Restricting retrieval to 14 testable criteria reduced the initial unsupported-criterion rate to 13.8%. Under matched instructions and an identical output schema, retrieval increased unsupported-criterion reports from 17.4% to 31.1%, while hypertrophy recall remained unchanged at 62.6%. Conclusion: These findings identify a fundamental limitation of retrieval-based clinical report generation: retrieved criteria do not necessarily provide valid patient-specific evidence. Measurement-based verification can reduce unsupported criterion applications, but diagnostic validity requires independent assessment.

Figures & tables

Criterion	Threshold or description	Cards
Original numeric criteria
QRS duration	$\geq 120$ ms	1
PR interval	$\geq 200$ ms	1
QRS axis	$\leq-30^{\circ}$	1
Sokolow–Lyon voltage	$\geq 3.5$ mV	1
Bazett QTc	$\geq 450$ ms (men); $\geq 460$ ms (women)	1

Table 1: Composition of the ECG criteria library. Each numeric threshold is an operational predicate used for verification. It is not sufficient, by itself, for a clinical diagnosis.

Figure 1: Overview of the RAG-TRACE framework, showing dataset preparation, ECG criteria library construction, evidence retrieval strategies, and structured inputs for language model-based ECG report generation.

Figure 2: RAG-TRACE framework for ECG report generation and measurement-based verification using Marquette 12SL and Uni-G.

Figure 3: Distribution of diagnostic superclass combinations among 1,083 PTB-XL patients. Bars represent combination frequencies, with normal-only recordings forming the largest group ( $n=483$ ).

Class	AUROC	AUPRC	F1
Normal	0.949	0.923	0.865
Myocardial infarction	0.940	0.839	0.745
ST/T change	0.934	0.793	0.750
Conduction disturbance	0.931	0.847	0.765
Hypertrophy	0.918	0.680	0.633
Macro average	0.935	0.816	0.751

Table 2: Calibrated superclass model on the PTB-XL test fold (

n=2{,}157

	gpt-4.1-mini		Qwen3-8B
	Keyword	After 12SL	Keyword	After 12SL
Expert diagnosis absent	0.101	0.101	0.051	0.051
Uni-G rejects a criterion	0.053	0.046	0.208	0.047
Release score	0.468	0	0.654	0
Mean 12SL-supported criteria kept	0.294	0.294	0.306	0.306

Table 3: Keyword retrieval and the same note after the 12SL withdrawal. Expert diagnosis absent: an affirmed superclass is missing from the PTB-XL label. Uni-G rejects: an applied criterion fails the second algorithm. Release: an applied card has no check, or the 12SL predicate is false. The last row is a mean count, not a record rate.

	gpt-4.1-mini	Qwen3-8B
Uni-G supports, and the edit keeps	268/269	274/282
12SL false, and Uni-G rejects	7/8	203/211
No numeric check, withdrawn	639/639	880/880
12SL predicate false, withdrawn	8/8	211/211

Table 4: Keyword-note applications judged by Uni-G. Uni-G was not an input to the 12SL edit. The second row is the subset of the failed 12SL predicates that Uni-G also rejected. The unchecked row is separate.

	gpt-4.1-mini		Qwen3-8B
Note	Mismatch	Unstated	Mismatch	Unstated
No retrieval	0.075	0.344	0.016	0.344
Keyword retrieval	0.249	0.289	0.341	0.311
12SL withdrawal	0.246	0.289	0.228	0.314
Both algorithms	0.222	0.293	0.196	0.318

Table 5: Statement mismatch against stored cardiologist codes for the original eight checks, and the mean number of those findings left unstated.

	Control	Keyword	After 12SL
14 numeric cards
Expert diagnosis absent	0.075	0.098	0.098
Uni-G rejects a criterion	0	0.079	0.055
Release score	0	0.138	0
Mean 12SL-supported criteria kept	0	0.386	0.386
Identical schema, full library

Table 6: Controlled retrieval on the same 1,083 patients. The upper block uses the 14 cards that have a numeric check, and its control column is the original note. The lower block is the identical-schema comparison on the full library. Both arms use one instruction and one JSON schema, and both may emit a criterion claim. Only the keyword arm receives retrieved cards.

Arm	$n$	No cards	Keyword	Difference
gpt-4.1-mini, full library	1,083	0.198	0.230	0.032 (0.016, 0.049)
gpt-4.1-mini, 14 numeric cards	1,083	0.198	0.252	0.054 (0.038, 0.072)
gpt-4.1-mini, matched prompt	1,083	0.212	0.221	0.008 ( $-0.001$ , 0.018)
gpt-4.1-mini, constrained identifier	1,083	0.198	0.140	$-0.057$ ( $-0.077$ , $-0.038$ )
Qwen3-8B	1,082	0.217	0.218	0.001 ( $-0.009$ , 0.011)
Qwen3-14B	1,083	0.227	0.178	$-0.049$ ( $-0.062$ , $-0.037$ )

Table 7: Exploratory label-aware expert-absent rates. A counted diagnosis is a saved superclass label that the sentence does not negate. The no-card column for the numeric-only and constrained arms is the original no-card note, joined by electrocardiogram identifier. Intervals are 95% bootstrap confidence intervals.

Figure 4: Diagnosis agreement for the same models. A, class recall with no retrieval. B, class recall with keyword retrieval. C, notes with a false diagnosis. D, micro-F1. In C and D, blue is no retrieval and green is keyword retrieval. Sample sizes are 1,083 except Qwen3-8B (1,082), MedGemma (850), and DeepSeek (112). DeepSeek is a selected subset.

Figure 5: Precision, recall, micro-F1, and false-diagnosis notes for the six models with a full paired sample. A, no retrieval. B, keyword retrieval. Cell values are percentages. Blue darkens as precision, recall, and F1 rise. Red darkens as the false-diagnosis rate rises. The row label gives the number of patients. MedGemma is the 850 patients with all three notes. DeepSeek is omitted.

Figure 6: The same four measures for those six models. Each axis rises when agreement improves. The fourth axis is the percentage of notes with no false diagnosis. The radial scale starts at 55. A, no retrieval. B, keyword retrieval. MedGemma is the 850 patients with all three notes.

Arm	Precision	Recall	F1	False-diagnosis notes	Omissions
No cards	0.841	0.770	0.804	0.175	312
14-card constraint	0.844	0.752	0.795	0.167	337

Table 8: Different-schema diagnosis agreement for gpt-4.1-mini,

n=1{,}083

. Both arms use one instruction. The schema admits a criterion claim only when retrieved cards are present. The identical-schema comparison is Table 6 . Precision, recall, and F1 are micro averages against the stored PTB-XL superclass codes. A false-diagnosis note affirms at least one superclass that is absent from that code.

Figure 7: Diagnosis micro-F1 change under keyword retrieval. Each row is keyword minus no retrieval on the same patients. The dot is the estimate, the line is the 95% bootstrap interval, and the gray shape is the bootstrap distribution. The interval uses 10,000 patient resamples. MedGemma is the 904 patients who have both notes. The three-note table uses 850.

Medical Report Generation

Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.

See all 86 papers

ECG-LENS: Lead-Aware Clinical Context Enriched ECG Report Generation and Evaluation
Aug 6, 2026Akanta Das, Tasinul Islam Ahon, Ahmed Mahir Sultan Rumi +3Medical Report Generation Electrocardiography
Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Samsung Research America, USA · National Health Service, UK
When Retrieval Hurts: Measuring and Explaining Retrieval-Induced Hallucination in Chest X-ray Report Generation
Sep 22, 2026Emmanuel Idoko, Abdusshakur Olabisi, Shiloh Oni +1Hallucination in Language Models RAG Robustness
University of Lagos · Machine Learning Collective · NITHUB, University of Lagos +2
ATRIA: Adaptive Traceable ECG Reporting with Iterative Agents
Jun 23, 2026Donggyun Hong, Kyuhwan Lee, Junmyung Kwon +1Medical Report Generation Clinical Decision Support
MedicalAI Co., Ltd. · South Korea

Aug 6, 2026J/K move · Enter open · S save

ECG-LENS: Lead-Aware Clinical Context Enriched ECG Report Generation and Evaluation

Akanta Das, Tasinul Islam Ahon, Ahmed Mahir Sultan Rumi, Md Mahbubur Rahman+2

Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Samsung Research America, USA · National Health Service, UK

Medical Report Generation Electrocardiography Electrocardiogram Interpretation

Open paper AskPDF arXiv

Electrocardiography (ECG) is one of the most widely used non-invasive tools for diagnosing cardiovascular disease, but transforming multi-lead ECG recordings into reliable clinical reports remains challenging. Automating ECG report generation could reduce clinicians' interpretive workload, improve diagnostic efficiency, and expand access to cardiac assessment in underserved communities. Unlike image-based report-generation tasks, ECG interpretation requires the analysis of subtle temporal morphologies, followed by coherent diagnostic reasoning expressed in dense clinical terminology. Existing systems predominantly focus on classification, while current report-generation methods often produce outputs that remain inadequate for practical clinical use. To address these challenges, we propose ECG-LENS, an end-to-end ECG report-generation framework that jointly integrates multi-lead signal modeling, diagnosis-aware representations, and clinically grounded text generation. ECG-LENS combines lead-wise encoders that preserve localized waveform morphology with a global encoder that captures inter-lead dependencies. To guide report generation, we fuse signal representations with clinically enriched textual prompts that condition a GPT-2 decoder. We further introduce an ECG-specific report-preprocessing strategy that helps the model focus on clinically meaningful findings. Finally, because lexical metrics may under- or overestimate report quality, we propose F1-ECGBERT, a BERT-based, ECG-specific metric that measures agreement between diagnostic labels extracted from generated and reference reports. In-domain experiments on PTB-XL and cross-domain evaluation on MIMIC-IV-ECG show that ECG-LENS consistently outperforms state-of-the-art methods, with absolute gains of 4.0%, 6.3%, and 11.5% in METEOR, ROUGE-L, and F1-ECGBERT, respectively, over the strongest baselines.

Abstract

Medical Report Generation

Explore similar work

ECG-LENS: Lead-Aware Clinical Context Enriched ECG Report Generation and Evaluation

When Retrieval Hurts: Measuring and Explaining Retrieval-Induced Hallucination in Chest X-ray Report Generation

ATRIA: Adaptive Traceable ECG Reporting with Iterative Agents