cs.AIFeb 27, 2026
SaveHow Well Do Multimodal Models Reason on ECG Signals?
Organizations: University of Illinois Urbana Champaign · Google Inc · Rush University · ETH Zurich · St. Christopher’s Hospital for Children · Stanford University · University of Washington
Abstract
While multimodal large language models offer a promising solution to the "black box" nature of health AI by generating interpretable reasoning traces, verifying the validity of these traces remains a critical challenge. Existing evaluation methods are either unscalable, relying on manual clinician review, or superficial, utilizing proxy metrics (e.g. QA) that fail to capture the semantic correctness of clinical logic. In this work, we introduce a reproducible framework for evaluating reasoning in ECG signals. We propose decomposing reasoning into two distinct, components: (i) Perception, the accurate identification of patterns within the raw signal, and (ii) Deduction, the logical application of domain knowledge to those patterns. To evaluate Perception, we employ an agentic framework that generates code to empirically verify the temporal structures described in the reasoning trace. To evaluate Deduction, we measure the alignment of the model's logic against a structured database of established clinical criteria in a retrieval-based approach. This dual-verification method enables the scalable assessment of "true" reasoning capabilities.
Figures & tables
| Method | Applicable to Health Data | Assesses Reasoning Text | Inspects Non-Text Modality | Allows Multiple Valid Reasonings | Reproducible |
| Step-wise Decomposition | ✓ | ✓ | ✓ | ||
| Question Answering | ✓ | ✓ | |||
| Zero-shot Classification | ✓ | ✓ | |||
| n-gram Match | ✓ | ✓ | ✓ | ||
| Expert Assessment | ✓ | ✓ | ✓ | ✓ | |
| ECG_ReasonEval (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 1: Comparison of Reasoning Evaluations. Current methods typically force a trade-off between scalability and depth. Objective proxies (e.g. QA) measure final answer accuracy but ignore the reasoning trace, while Expert Assessment is neither reproducible nor scalable. Furthermore, rigid methods like Step-wise (standard in math) fail in health because they assume a single correct derivation chain. In contrast, clinical diagnosis allows for multiple valid reasoning paths : different experts often cite distinct but equally valid sets of features to support the same diagnosis.
Figure 1: ECG_ReasonEval Framework. We decompose reasoning evaluation into two independent axes: (i) Perception , verifying if reasoning is grounded in the signal, and (ii) Deduction , evaluating if the logic aligns with clinical consensus. For Perception, we employ a flexible data science agent that dynamically writes and executes Python code to verify specific findings. For Deduction, we query a medical criteria database with the reasoning trace and check if the retrieved criteria are tagged with the same label as the original signal’s ground truth.
Figure 2: The Perception Pipeline. First, our data science agent extracts discrete, verifiable findings from the reasoning trace. It then generates executable Python code to empirically verify these claims against the raw ECG signal. This process is augmented by a segmentation tool, which lowers the code generation complexity by providing pre-computed wave delineations. The pipeline outputs a boolean verification status for each finding, determining if the reasoning describes the signal accurately.
| Perception Assessment | ↑ ↑ Supporting Assessment | ↓ ↓ Adversarial Assessment | ||
| Global Acc | Acc@ Thresh50% | Acc@ Thresh100% | Acc@ Thresh100% | |
| ECG_ReasonEval | ||||
| Ablations | ||||
| w/o Parsing Findings | – | |||
| w/o Unified Agent | ||||
| w/o DL Segment Tool | ||||
Table 2: Assessment of ECG_ReasonEval Perception. We evaluate our Perception pipeline’s reliability using two protocols: Supporting Assessment , which measures the verification rate of ground-truth cardiologist notes, and Adversarial Assessment , which tests robustness against hallucination by inverting clinical descriptors (e.g. Wide Narrow QRS). The strong performance gap between these settings confirms that ECG_ReasonEval effectively distinguishes between present and absent signal features.
Figure 3: Causes of Perception Failure. With our team of physicians, we manually examined the failed reasoning traces in the val set to categorize error sources. The results suggest Perception is highly reliable and capable of auditing errors in human annotations.
Figure 4: The Deduction Pipeline. The input reasoning trace is first censored of its final diagnostic label, embedded using the Gemini model, and used to query our database of diagnostic criteria. We retrieve the top- most semantically similar articles and calculate Precision@ against the ground truth. A high score indicates that the reasoning trace accurately maps to the correct pathology according to established medical standards, mirroring how physicians cross-reference diagnostic criteria.
| Deduction Assessment | Physician A | Physician B | Physician C | ||||||
| ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | ↑ ↑ Precis | |
| @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 | |
| ECG_ReasonEval | 0.83 | 0.76 | 0.69 | 0.79 | 0.78 | 0.71 | 0.79 | 0.75 | 0.70 |
| Ablations | |||||||||
| w/o Quote Extract | 0.73 | 0.66 | 0.63 | 0.83 | 0.70 | 0.65 | 0.72 | 0.70 | 0.64 |
| w/o Struct. Extract | 0.77 | 0.65 | 0.59 | 0.74 | 0.73 | 0.65 | 0.77 | 0.73 | 0.66 |
Table 3: Assessment of ECG_ReasonEval Deduction. We evaluate the alignment of our retrieved medical references used during Deduction against diagnostic criteria independently authored by three physicians. The full ECG_ReasonEval pipeline consistently performs well with high Precision@ , confirming that our approach effectively captures nuanced “clinical logic”.
Figure 5: Summary of Baselines with ECG_ReasonEval on Perception and Deduction. We see all models perform much worse than the physician performance across both perception and deduction. While the TSLMs (OpenTSLM and QoQMed) that were trained with explicit time-series adapters often do better in perception to capture time-series features correctly, frontier VLMs like Claude Opus 4.5 Plot and Gemini 3.1 Plot are able to harness their world knowledge to align with the broader clinical literature, achieving stronger deduction performance. Most notably, Gemini establishes the strongest overall baseline across both perception and deduction metrics.
Figure 6: Sample output of Claude (Plot). Claude often produces correct diagnoses (high Deduction score), but with incorrect time-series reasoning (low Perception score). In this example, ”deep QS complexes” are indicative of myocardial infarction, but do not appear in the ECG. This implies a post-hoc reasoning, in which the model first predicts the label then hallucinates justification. This is further implied by the structure of reasoning text, which first states the diagnosis then causally generates reasoning text.
Figure 7: Sample output of OpenTSLM. While OpenTSLM generally has good Perception, their low Deduction performance implies that it suffers from being unable to identify the most salient features for precise diagnosis. In this example, although OpenTSLM correctly perceived the inverted t-waves, it failed to realize that these pathological features, in conjunction with the higher QRS voltage could indicate left ventricular hypertrophy, a disease with significant health implications.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | MIMIC-IV | ECG-QA Diagnosis | ECG-QA Rhythm | ECG-QA Form | ||||||||||||||||||||||||
| Perception | Deduction | Orig | Perception | Deduction | Orig | Perception | Deduction | Orig | Perception | Deduction | Orig | |||||||||||||||||
| Mac@50 | Mac@100 | Micro | P@1 | P@5 | P@10 | Acc | Mac@50 | Mac@100 | Micro | P@1 | P@5 | P@10 | Acc | Mac@50 | Mac@100 | Micro | P@1 | P@5 | P@10 | Acc | Mac@50 | Mac@100 | Micro | P@1 | P@5 | P@10 | Acc | |
| OpenTSLM | 0.63 | 0.07 | 0.52 | 0.12 | 0.12 | 0.13 | 0.01 | 0.79 | 0.07 | 0.62 | 0.04 | 0.07 | 0.07 | 0.10 | 0.68 | 0.09 | 0.56 | 0.20 | 0.21 | 0.20 | 0.15 | 0.74 | 0.30 | 0.60 | 0.14 | 0.12 | 0.12 | 0.12 |
| QoQ-Med | 0.55 | 0.03 | 0.47 | 0.11 | 0.12 | 0.12 | 0.14 | 0.50 | 0.00 | 0.46 | 0.09 | 0.08 | 0.09 | 0.08 | 0.79 | 0.25 | 0.69 | 0.21 | 0.19 | 0.18 | 0.14 | 0.75 | 0.03 | 0.59 | 0.12 | 0.15 | 0.15 | 0.13 |
| Claude Array | 0.83 | 0.09 | 0.66 | 0.15 | 0.15 | 0.16 | 0.16 | 0.61 | 0.02 | 0.55 | 0.17 | 0.17 | 0.18 | 0.18 | 0.73 | 0.02 | 0.61 | 0.22 | 0.22 | 0.20 | 0.15 | 0.59 | 0.04 | 0.53 | 0.05 | 0.09 | 0.12 | 0.14 |
| Claude Plot | 0.61 | 0.04 | 0.56 | 0.22 | 0.21 | 0.21 | 0.22 | 0.71 | 0.05 | 0.59 | 0.22 | 0.20 | 0.20 | 0.22 | 0.66 | 0.03 | 0.57 | 0.25 | 0.25 | 0.24 | 0.20 | 0.56 | 0.06 | 0.52 | 0.18 | 0.21 | 0.22 | 0.21 |
Table 4: Comprehensive Evaluation on MIMIC-IV and ECG-QA. We compare models across four datasets. Metrics are grouped by Perception (labeling accuracy), Deduction (retrieval quality), and Original (downstream accuracy). All models are now evaluated across all available tasks.
Figure 8: Correlation with Prediction Accuracy. Prior approaches have focused on benchmarking only on Final Classification Accuracy, without explicitly benchmarking the reasoning capability. Deduction is highly correlated to final accuracy, but Perception is not. This discrepancy indicates that high final accuracy does not imply that the model is reasoning correctly on the specific signal features that are present, leading to issues in trustworthiness.
| Antonyms | Notes |
| Wide Narrow | e.g. Wide QRS |
| Elevation Depression | e.g. ST Elevation |
| Regular Irregular | e.g. Regular rhythm |
| Upright Inverted | e.g. Inverted P wave |
| Prolonged Shortened | e.g. Prolonged PR |
| Left Right | e.g. Left axis deviation |
Table 5: Antonym Mappings used during Adversarial Assessment.
| ICD-10 Code | Diagnosis |
| I21.3; I21.3 | ST segment elevation (STEMI) myocardial infarction; Acute ST elevation myocardial infarction |
| I48.91 | Atrial fibrillation |
| I48.92; I48.4 | Atrial flutter; Typical atrial flutter |
| I45.81 | Long QT syndrome |
| I47.1 | Supraventricular tachycardia |
| I49.3 | Ventricular premature depolarization (PVCs) |
Table 6: ICD-10 Codes and Corresponding Diagnoses used in MIMIC-IV evaluation.
Figure 9: MIMIC-IV cardiologist note flagged as being incorrect by ECG_ReasonEval , and verified to be incorrect by our cardiologists. Team’s note: I don’t agree with the ST changes in the limb leads, I think that’s just artifact. I think it’s ok to say non-specific ST changes in V5 and V6 but it’s submillimeter in many areas so would also be fine to leave it out. The QRS morphology does not look like an incomplete RBBB even though the report says that.
Figure 10: MIMIC-IV cardiologist note flagged as being incorrect by ECG_ReasonEval , and verified to be incorrect by our cardiologists. Team’s note: I wouldn’t call minor ST abnormalities. It’s just artifact, or maybe it thinks the TWI are abnormal in V1 and V2, but that can be a normal variant.
Figure 11: MIMIC-IV cardiologist note flagged as being incorrect by ECG_ReasonEval , and verified to be incorrect by our cardiologists. Team’s note: PR interval looks normal.
Figure 12: MIMIC-IV cardiologist note flagged as being incorrect by ECG_ReasonEval , and verified to be incorrect by our cardiologists. Team’s Note: Agreed, does not look like LVH, also don’t see left axis deviation or strain pattern which would be expected with LVH.
Figure 13: Number of findings per trace.
Figure 14: Verified Findings per Trace.
Figure 15: Verbatim System Prompt Specification for the Perception Verification Agent.
Figure 16: System Prompt for the Debugging Agent. This agent is triggered when the initial code execution fails, tasked with either repairing the Python function or flagging the claim as unverifiable.
Figure 17: Example of our Deduction Pipeline. This example is how we validate our deduction, where we embed the Physician’s reasoning, then compare it against our database to find the Top- most similar diagnostic criteria. Here, for Incomplete Right Bundle Block, we are able to consistently get at least 3 database entries on IRBBB based on the phsyician’s definition, despite each definition being so differnetly framed. In particular, Physician B is very short, and doesnt even describe what RBBB is in it, just that IRBBB is not RBBB, but yet, we are still able to identify IRBBB articles. During inference, instead of a physician reasoning being used as input, the evaluated model’s generated reasoning is used and compared to our database.
Figure 18: System Prompt Specification for the Medical Knowledge Extraction Agent. This agent distills unstructured medical literature into structured diagnostic clusters for cross-verification against perception-level findings.
Figure 19: System Prompt Specification for the ECG Data Standardization Agent. This agent enforces a controlled vocabulary and lead-expansion logic to ensure extraction results are machine-verifiable against numeric features.