SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity

Md Ashik Khan; Md Nahid Siddique

cs.CLSep 12, 2026

SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity

Authors: Md Ashik Khan, Md Nahid Siddique

Organizations: Department of Computer Science and Engineering Indian Institute of Technology Kharagpur Kharagpur, India · Knight Foundation School of Computing and Information Sciences Florida International University Miami, FL, USA

Abstract

Multimodal clinical AI typically assumes that the waveform, report, metadata, and downstream predictions attached to a record belong to the same patient. In practice, linkage failures can silently assemble individually plausible but cross-patient components, creating a safety problem that standard predictive models are not designed to detect. We study this problem as multimodal record integrity triage: given an assembled record, should its modalities be trusted to belong together? We introduce SHIFT-M3, a lightweight text-based pre-fusion screen that measures alignment-based consistency between two separately produced ECG text views: an LLM-generated interpretation and a clinical report summary. On 784,680 MEETI ECG records, SHIFT-M3 achieves 97.6% TPR@5% FPR for full text-view swaps (AUROC 0.996), 90.3% for partial swaps (AUROC 0.974), and 97.7% for label-matched hard negatives (AUROC 0.996) with only 573,569 parameters. Compared with same-dataset lexical baselines, the gains are largest on partial swaps and hard negatives, suggesting that the model is learning more than surface overlap. We also introduce the CMST (Conflict-type Multimodal Stress Test) evaluation taxonomy, a three-seed stability study, a loss ablation, a temporal-tolerance sweep, and a shared-token masking control. The main remaining failure mode is longitudinal ambiguity: at the default operating point, same-patient cross-visit pairs still produce 87.0% Type-II false positives.

Figures & tables

Figure 1: Method overview. On the left, SHIFT-M3 encodes the interpretation and report with separate modality-specific streams and produces a pre-fusion conflict score before any downstream multimodal model is invoked. On the right, the CMST taxonomy decomposes evaluation into full text-view swaps, partial swaps, hard negatives, and same-patient temporal controls, making it possible to separate cross-patient mismatch detection from longitudinal tolerance.

Split	Records	Subjects	Multi-Visit
Train	629,386	128,477	82,414
Validation	76,742	16,059	10,342
Test	78,552	16,061	10,245
Total	784,680	160,597	103,001

Table 1: Patient-level split in the MEETI dataset pipeline. Splitting by subject prevents leakage of repeated visits across train, validation, and test.

Model	Type-I AUROC	Type-III AUROC	Hard Neg AUROC	Type-II FPR@5%
MiniLM (frozen)	0.729	0.527	0.731	9.9%
Bio_ClinicalBERT (frozen)	0.555	0.500	0.556	7.3%
SHIFT-M3	0.996	0.974	0.996	87.0%

Table 2: Frozen pretrained encoder baselines versus SHIFT-M3 on the held-out test set. Both encoders score pairs via cosine distance between independently encoded embeddings without fine-tuning. Lower Type-II FPR in frozen models reflects weak conflict sensitivity across all pairs, not better longitudinal tolerance. Best value per column in bold; higher is better for AUROC, lower is better for Type-II FPR.

Model	T-I	T-III	HN
Word cosine	84.4	24.4	84.1
Char cosine	45.7	13.6	46.0
LogReg	89.1	32.5	89.2
SHIFT-M3	97.6	90.3	97.7

Table 3: Held-out CMST performance. SHIFT-M3 improves substantially over same-dataset lexical baselines on full text-view swaps, partial swaps, and hard negatives. Type-II is a false-positive rate, so lower is better. Best value per column in bold.

Figure 4: ROC curves across the three cross-patient stressors. SHIFT-M3 (blue) consistently hugs the upper-left corner, with the largest margin on partial swaps (center panel, AUROC 0.974 vs. 0.824). Each curve is annotated with its AUROC. The dashed diagonal represents random chance.

Seed	Type-I TPR@5	Type-III TPR@5	Hard Neg TPR@5	Type-II FPR@5	Type-I AUROC	Type-III AUROC	Hard Neg AUROC
42	97.55	89.40	97.83	86.42	0.9962	0.9725	0.9964
43	97.66	91.41	97.81	87.86	0.9963	0.9770	0.9961
44	97.47	90.02	97.57	86.56	0.9961	0.9737	0.9960

Table 4: Per-seed held-out metrics from the three-run stability analysis. The main detection metrics vary little across seeds, indicating that the reported gains are not artifacts of a single run.

Metric	Mean	Std
Type-I TPR@5	97.56	0.09
Type-III TPR@5	90.28	1.03
Hard Neg TPR@5	97.73	0.15
Type-II FPR@5	86.95	0.79
Type-I AUROC	0.9962	0.0001
Type-III AUROC	0.9744	0.0023

Table 5: Three-seed stability summary. Small standard deviations on Type-I, Type-III, and hard-negative metrics indicate stable optimization.

Variant	Type-I TPR@5	Type-III TPR@5	Hard Neg TPR@5	Type-II FPR@5	Type-I AUROC	Type-III AUROC	Hard Neg AUROC
Full	97.55	89.40	97.83	86.42	0.9962	0.9725	0.9964
No task loss	97.55	89.40	97.83	86.42	0.9962	0.9725	0.9964
No contrastive loss	34.28	14.01	34.12	13.37	0.8445	0.6634	0.8443
No temporal loss	97.76	91.03	97.98	87.86	0.9964	0.9760	0.9967
No swap loss	96.73	84.40	97.04	84.79	0.9945	0.9575	0.9949

Table 6: Loss ablation. The contrastive and swap losses provide most of the integrity signal, whereas the temporal term mainly moderates false positives on same-patient longitudinal pairs. The task-loss variant matches Full because the task and identity streams share no parameters.

Clean-pair FPR budget	Type-I TPR@thr	Type-III TPR@thr	Hard Neg TPR@thr	Type-II FPR@thr
1%	96.6%	63.0%	96.8%	77.2%
2%	97.0%	69.0%	97.3%	80.2%
5%	97.7%	73.7%	97.8%	82.8%
10%	98.5%	76.6%	98.4%	85.4%
20%	99.7%	84.3%	99.5%	90.3%

Table 7: Operating-point frontier across clean-pair FPR budgets. Type-III results use the text-splice variant. No threshold setting resolves longitudinal ambiguity: Type-II FPR remains elevated across the full range.

Config	$\lambda_{temp}$	Margin	Type-I TPR@5	Type-III TPR@5	Hard Neg TPR@5	Type-II FPR@5	Type-I AUROC	Type-III AUROC
No temporal	0.0	0.5	97.76	91.03	97.98	87.86	0.9964	0.9760
Low temporal	0.25	0.5	97.66	90.64	97.89	87.67	0.9963	0.9747
Default	0.5	0.5	97.6	90.3	97.7	87.0	0.996	0.974
Compromise	1.0	0.5	97.21	89.13	97.32	84.13	0.995	0.970
Margin 0.6	1.0	0.6	97.39	88.62	97.58	84.23	0.9957	0.9693
Aggressive	2.0	0.5	92.79	77.44	93.06	73.07	0.984	0.936

Table 8: Completed temporal-weight sweep runs. Larger temporal weight lowers Type-II false positives, but the gain comes at the cost of weaker cross-patient detection.

Setting	Thr.	T-I TPR@5	T-I AUC	Clean mean
Original	0.131	97.6	0.996	0.067
Masked shared tokens	0.452	83.3	0.960	0.212

Table 9: Compact control summaries shown side by side. Left: shared-token masking control. Right: audit tables exported at the main 5% clean-pair threshold.

(a) False Positive: Clean Pair Flagged as Swap
LLM Interp.	The ECG image analysis begins with the global features revealing a heart rate of 40 bpm , which suggests bradycardia … PR interval measurements vary significantly … large fluctuations in P and QRS amplitudes … conclusion stands with a sinus rhythm without prominent abnormalities , aligning with the representation of a normal ECG .
Clin. Report	Sinus rhythm, Normal ECG
(b) False Negative: Full Text-view Swap Flagged as Clean Match
Case A LLM	…overall heart rate is 81 bpm, suggesting a normal sinus rhythm . However… indicates the presence of premature ventricular contractions (PVCs) , aligning with the finding of a borderline ECG .
Case B Report	Sinus rhythm with PVC(s) , Borderline ECG

Table 10: Representative false positive and false negative cases at the 5% FPR threshold. Terms contributing to model conflict are highlighted in red ; terms consistent across both views are highlighted in teal .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	Adam
$\beta_{1}$ / $\beta_{2}$ / $\epsilon$	0.9 / 0.999 / $10^{-8}$
Learning rate	$3\times 10^{-4}$
Batch size	128
Epochs	10
Embedding dim	128

Appendix

Table 11: Hyperparameter configuration used for all reported experiments.

Multimodal Anomaly Detection

Multimodal anomaly detection identifies unusual patterns by combining complementary signals such as images, depth, video, text, and sensor data.

See all 97 papers

How Well Do Multimodal Models Reason on ECG Signals?
Feb 27, 2026Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu +11Electrocardiogram Interpretation Multimodal Model Evaluation
University of Illinois Urbana Champaign · Google Inc · Rush University +4
Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI
Aug 2, 2026Quang Bui, Shlok Jaiswal, Samuel Paik-Heintz +14Missing-Modality Learning Multimodal Robustness
MIT Critical Data, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States · American International School Vienna, Vienna, Austria · Neuqua Valley High School, Naperville, Illinois, United States +18
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Feb 27, 2026Kejing Yin, Haizhou Xu, Wenfang Yao +5Multimodal Fusion Multimodal Model Evaluation
Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University

Feb 27, 2026J/K move · Enter open · S save

How Well Do Multimodal Models Reason on ECG Signals?

Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu, Patrick Langer+10

University of Illinois Urbana Champaign · Google Inc · Rush University+4

Electrocardiogram Interpretation Multimodal Model Evaluation Multimodal Reasoning Medical Reasoning Multimodal Large Language Models Clinical Reasoning+1

Open paper AskPDF arXiv

While multimodal large language models offer a promising solution to the "black box" nature of health AI by generating interpretable reasoning traces, verifying the validity of these traces remains a critical challenge. Existing evaluation methods are either unscalable, relying on manual clinician review, or superficial, utilizing proxy metrics (e.g. QA) that fail to capture the semantic correctness of clinical logic. In this work, we introduce a reproducible framework for evaluating reasoning in ECG signals. We propose decomposing reasoning into two distinct, components: (i) Perception, the accurate identification of patterns within the raw signal, and (ii) Deduction, the logical application of domain knowledge to those patterns. To evaluate Perception, we employ an agentic framework that generates code to empirically verify the temporal structures described in the reasoning trace. To evaluate Deduction, we measure the alignment of the model's logic against a structured database of established clinical criteria in a retrieval-based approach. This dual-verification method enables the scalable assessment of "true" reasoning capabilities.

Abstract

Multimodal Anomaly Detection

Explore similar work

How Well Do Multimodal Models Reason on ECG Signals?

Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI

When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion