cs.CLAug 30, 2026
SaveSUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
Organizations: Beijing University of Technology · Beijing Institute of Technology · Department of Rheumatology and Immunology, China-Japan Friendship Hospital · Dongbei University of Finance and Economics
Abstract
Current evaluations of large language models (LLMs) primarily focus on factual knowledge retrieval, overlooking the fundamental challenge of navigating the complex, non-bijective mappings between clinical indicators and diagnoses. Existing benchmarks fail to assess whether large language models truly possess the reasoning capability required for diagnostic ambiguity scenarios, where identical clinical presentations may correspond to different etiologies, and diagnostic convergence scenarios, where heterogeneous symptoms ultimately indicate the same disease. To address this issue, we propose SUP-MIMIC, a multi-task framework utilizing MIMIC-IV-v3.1 that comprises Basic Assessment (BA), Diagnostic Divergence Task (DDT), and Diagnostic Convergence Task (DCT). Specifically, DDT is designed to evaluate the model's "one-to-many" disambiguation capability among phenotypically similar cases, while DCT assesses the model's ability to identify "many-to-one" diagnostic patterns across different pathophysiological pathways. Comprehensive evaluation of state-of-the-art LLMs reveals substantial performance degradation on DDT and DCT compared to baseline tasks, exposing a systemic reliance on statistical shortcuts over genuine causal reasoning. Our findings further highlight a conservative bias toward "healthy" predictions, implying non-trivial risks for missed diagnoses in realistic medical settings. This work establishes a rigorous methodology for quantifying clinical reasoning robustness and provides a roadmap for enhancing the safety of language models in clinical medicine.
Figures & tables
Figure 1: Illustration of two adversarial diagnostic contrasts. In Diagnostic Divergence, Patient A and Patient B share similar clinical profiles, yet the model correctly identifies bacterial pneumonia in Patient A but fails to recognize SLE in Patient B despite decisive laboratory differences. In Diagnostic Convergence, both Patient C and Patient D have bacterial pneumonia, yet the model only recognizes the classic presentation in Patient C and misses the same diagnosis in Patient D.
Figure 2: Overview of the SUP-MIMIC construction pipeline. Multi-label ICU records are converted into binary patient-diagnosis verification labels. The full 448-dimensional feature space is used as LLM input, while diagnosis-specific top- features are used for similarity estimation and adversarial pair mining across BA, DDT, and DCT.
Figure 3: Diagram of the two-stage expert validation.
Figure 4: Accuracy heatmap for 17 clinical categories. Categories defined by distinctive single-system biomarker profiles (top rows) maintain stable accuracy, whereas multi-system categories (bottom rows) exhibit significant DCT collapse. Significance markers denote within-category paired -tests against BA ( , , ).
| Model | ADR | SRS | Sick R | Healthy R | |||||
|---|---|---|---|---|---|---|---|---|---|
| Open-Source General LLMs | |||||||||
| Llama3.3-70B | 0.3572 | 0.5806 | 0.3659 | 0.4346 | 0.2038 | 0.2214 | 0.1646 | 0.3638 | 0.7964 |
| Qwen2.5-7B | 0.2986 | 0.4995 | 0.3115 | 0.3699 | 0.2134 | 0.2934 | 0.1178 | 0.3025 | 0.7066 |
| Qwen2.5-14B | 0.3263 | 0.5795 | 0.2825 | 0.3961 | 0.2256 | 0.3026 | 0.1320 | 0.3232 | 0.7606 |
| Qwen2.5-32B | 0.6163 | 0.3462 | 0.5081 | 0.2694 | 0.3428 | 0.2595 | 0.4450 | 0.8240 | |
| Mistral-7B | 0.2639 | 0.4927 | 0.1561 | 0.3042 | 0.3028 | 0.2802 | 0.0732 | 0.2266 | 0.6924 |
Table 1: Main results on SUP-MIMIC. All metrics are reported as means over five runs; standard deviations are omitted for brevity. The best result in each column is bolded and the second-best is underlined .
Figure 5: Distribution of four failure modes across 13 models. (a) Overall proportions. (b) Per-model breakdown grouped by model category.
Figure 6: Performance degradation under adversarial evaluation. (a,b) Correlation between BA accuracy and task accuracy for DDT and DCT respectively. (c) Per-model degradation scores; positive values indicate performance loss. (d) Distribution of degradation scores for general-purpose versus medical models. Blue circles denote general-purpose LLMs; orange crosses denote medical LLMs; the grey dashed line marks zero degradation.
Figure 7: The Qwen2.5 series demonstrates scalability under BA, DDT, and DCT algorithms; line graphs show accuracy at 7B, 14B, and 32B resolutions; bar charts show the relative performance improvement from 7B to 32B resolution.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Trade-off between diagnostic accuracy and variance as a function of the number of important features (K), with K = 20 achieving the best balance.
| Important | Available | Feature | Model | Accuracy | Std. Dev. |
|---|---|---|---|---|---|
| Features | Features | Utilization | Accuracy | Variance | (Accuracy) |
| 5 | 8 | 1.78% | 0.2862 | 0.00153 | 0.0391 |
| 10 | 124 | 27.67% | 0.2954 | 0.00246 | 0.0496 |
| 15 | 150 | 33.48% | 0.2943 | 0.00521 | 0.0722 |
| 20 | 230 | 51.33% | 0.3563 | 0.00407 | 0.0638 |
| 25 | 257 | 57.36% | 0.3092 | 0.01384 | 0.1177 |
Table 2: Comparison of model performance and feature utilization across different numbers of important features. Accuracy, variance, and standard deviation are reported for the overall (ALL) setting.
Figure 9: Diagnostic Performance Disparity Across Evaluated LLMs. The dumbbell plot in it illustrates the performance trade-offs between Disease Recall (blue circles) and Healthy Recall (orange circles). Models on the vertical axis are sorted in descending order of their Disease Recall proficiency. The horizontal grey lines represent the performance discrepancy within each model, with adjacent numerical annotations indicating the exact absolute gap.
Figure 10: AUC-ROC by model and task type. Near-identical AUC across BA, CDT, and DCT confirms that confidence-based discrimination is task-independent.
Figure 11: Sick Recall at default threshold ( ) vs EER-optimized threshold. EER calibration improves Sick Recall by 10–15 percentage points across all models.
Figure 12: System prompt used for clinical reasoning and diagnostic evaluation.
Figure 13: Feature presentation format provided to the LLMs during inference.
| Case | Disease | Model Output Explanation | Error Model |
|---|---|---|---|
| DCT | Cerebrovascular Disease | The recorded variable itself constitutes explicit positive evidence, indicating that the patient has cerebrovascular disease. In addition, the patient is of extremely advanced age (92 years) and has multiple comorbidities, including diabetes mellitus, congestive heart failure, renal disease, and sepsis, all of which are well-established risk factors for cerebrovascular events. On admission, the neurological status was profoundly impaired (GCS = 3) , suggesting a possible acute central nervous system event. Furthermore, the patient exhibited severe coagulation dysfunction (INR up to 8.31, PT 92 s, platelet count 16–48 /L, fibrinogen approximately 90 mg/dL) . In the context of sepsis , these abnormalities substantially increase the risk of intracranial hemorrhage or hemorrhagic transformation following ischemic stroke. At the same time, concurrent hypoxia and shock also raise the possibility of ischemic brain injury. Although imaging evidence is lacking, in a high-risk ICU setting, and to avoid underdiagnosis, the decision is ultimately made as "Yes," based on the confirmed positive history and multiple strongly supportive clinical indicators. | claude-sonnet-4 (Error Type: Biomarker Omission) |
| DCT | Cervical Spine Degeneration | The patient is an 85-year-old male. Advanced age is one of the most important risk factors for degenerative changes of the cervical spine, including intervertebral disc degeneration, uncovertebral or facet joint osteoarthritis, osteophyte formation, and spinal canal or neural foraminal stenosis. Epidemiological studies indicate that the imaging detection rate of such changes in this age group is extremely high and nearly ubiquitous . The patient is currently in a critical condition (SOFA score 20, GCS 3), complicated by sepsis and acute kidney injury (AKI) . In this clinical context, it is not feasible to reliably assess or exclude cervical degenerative changes based on symptoms and physical examination alone . Following the principle of maximizing sensitivity and avoiding missed diagnoses, in the absence of definitive imaging evidence to the contrary, the presence of degenerative changes should be favored. In addition, the patient has comorbidities such as peripheral vascular disease, chronic pulmonary disease, and renal disease, which further suggest a high long-term risk of degenerative changes. There is no evidence supporting alternative specific etiologies such as inflammatory spinal disease or rheumatologic disorders. Although current laboratory abnormalities (infection, hepatic and renal dysfunction, and coagulation disturbances) explain the overall critical illness state, they do not directly conflict with or negate cervical spine degeneration. Therefore, despite the absence of direct imaging labels, given the extremely high age-related prior probability and the inability to adequately evaluate the condition in the current clinical setting. | deepseekv3 (Error Type: Feature Misweighting) |
| DDT | Sleep Disorders | The patient is a 78-year-old male in a critical ICU/emergency condition, complicated by sepsis and acute kidney injury (AKI), with a high overall disease severity (APSIII 74, SAPSII 52, SOFA 4). Such a critically ill physiological state and environment are inherently prone to disturbances of the sleep–wake cycle and marked sleep fragmentation. In addition, the patient is of advanced age and has a history of dementia and cerebrovascular disease, all of which are important risk factors for sleep disorders, including insomnia, circadian rhythm disturbances, and sleep-related breathing disorders. A GCS score of only 6 indicates significantly impaired consciousness and strongly suggests the likelihood of ICU interventions such as sedation and mechanical ventilation. Together with the ICU environment—characterized by noise, frequent nursing interventions, and disrupted light–dark cycles—these factors further contribute to the disruption of normal sleep architecture. (Though the model correctly identified related risk factors and impaired consciousness, it incorrectly conflated transient critical-care sleep disruption with a formal diagnosis of chronic or clinically established sleep disorders.) The patient also exhibits pronounced systemic inflammation and physiological stress responses, including leukocytosis (peak WBC 18.3), fever (Tmax 38°C), tachycardia (peak HR 151), metabolic alkalosis, hypernatremia (maximum Na 150), and anemia (lowest Hb 7.5). All of these factors are closely associated with delirium and sleep disturbances. At present, there is no evidence that clearly rules out sleep disorders. In this high-risk critical care setting, to avoid underdiagnosis, the presence of sleep disorders is therefore favored. | Gemini-2.5 (Error Type: Feature Comorbidity Conflation) |
| BA | Coagulation Defect | The patient is in a state of severe sepsis/shock (sepsis = 1, SOFA = 15, peak lactate 13.8 mmol/L, AKI = 1), under which acquired coagulopathy is highly likely, including disseminated intravascular coagulation (DIC) or liver-related coagulation abnormalities. Multiple coagulation and platelet parameters show clear abnormalities, providing strong evidence of a coagulation defect: 1)Marked thrombocytopenia: Platelet count on ICU admission was 61 × 10 9 /L, with a nadir of 34 × 10 9 /L, consistent with moderate-to-severe thrombocytopenia . 2)Significant coagulation dysfunction: PT up to 26.6 s, INR up to 2.6 , and PTT up to 52.5 s , all indicating prolonged coagulation pathways. 3)Reduced fibrinogen: Fibrinogen decreased to 132–155 mg/dL , suggesting consumptive hypofibrinogenemia. 4)Severe hepatic dysfunction: AST peaked at 3404 U/L, ALT at 357 U/L, total bilirubin up to 23.7 mg/dL, with a MELD score of 25, indicating impaired synthesis of coagulation factors . 5)Severe anemia: Hemoglobin decreased to a nadir of 4.8 g/dL , suggesting possible bleeding or consumption of clotting factors. It overemphasizes isolated abnormal coagulation markers without jointly considering the dynamic recovery trend and the absence of definitive evidence for active DIC or persistent coagulation failure. Although partial recovery of some parameters was observed after treatment, including improvement in fibrinogen and platelet count (Fbg 477 mg/dL, PLT 496 × 10 9 /L) and a reduction of INR to 1.4, a prior classic triad of "thrombocytopenia + prolonged PT/INR/PTT + hypofibrinogenemia" was clearly present. In the context of sepsis, this constellation is highly consistent with DIC or acquired coagulation defect. An isolated low INR value at certain time points (0.7) is insufficient to negate previously sustained abnormalities. Taken together, given the high-risk clinical context and the strong supporting evidence above, the patient is judged to have a coagulation defect. | GLM-4.7 (Error Type: Combinatorial Neglect) |
| DDT | Altered Mental Status | A comprehensive assessment of whether the patient has altered mental status is as follows: 1)Direct evidence of consciousness level: The most direct indicator of current consciousness is a GCS score of 15, suggesting that the patient is awake, alert, and generally oriented. This represents strong negative evidence for altered mental status. In critically ill settings, significant mental status changes (e.g., delirium, lethargy, confusion, or disorientation) are typically accompanied by a decreased GCS score or abnormal sedation-related assessments; however, no such findings are present here. 2)Presence of risk factors for altered mental status without corresponding objective findings: The patient has sepsis (SIRS = 3, WBC elevated to 13.9) , transient hypotension (minimum SBP 73.5) , marked hypoxemia (PaO2/FiO2 as low as 74, SpO2 down to 82%) , acid–base fluctuations (pH 7.28–7.54), and transient lactate elevation (peak 4.5) . All of these factors may increase the risk of delirium or metabolic encephalopathy. However, there is currently no objective evidence of corresponding impairment in consciousness. 3)Metabolic and electrolyte abnormalities are insufficient to support definite encephalopathy: Sodium nadir is 131, indicating only mild hyponatremia. Glucose ranges from 77 to 195, without significant hypoglycemia or hyperosmolar state. PaCO2 peak is 48, not consistent with significant hypercapnia. BUN peaks at 32, providing insufficient evidence for uremic encephalopathy. Total bilirubin peaks at 1.6 and MELD is approximately 17.5, which does not support definite hepatic encephalopathy. Variations in calcium and magnesium are also insufficient to independently explain significant cognitive impairment. 4)Overall assessment: Although the patient has multiple high-risk factors for delirium or encephalopathy, the most robust and direct evidence of mental status (GCS = 15) remains dominant. There are also no clear positive psychiatric or objective neurological findings indicating altered consciousness. Therefore, the current assessment leans toward "no clear evidence of altered mental status." However, given the critical illness context, continued monitoring for potential delirium or encephalopathy is still warranted. | GPT-3.5 (Error Type: Comorbidity Conflation) |
| BA | Arterial Embolism and Thrombosis | The patient was in a state of severe sepsis/septic shock (SOFA 17, SIRS 4) , with extremely severe metabolic acidosis and hyperlactatemia occurring early in the course (peak lactate 18.2 mmol/L, lowest pH 7.14), accompanied by acute kidney injury (initial Cr 4.3, subsequently improved) and marked "shock liver" manifestations (peak AST/ALT 4296/2270), suggesting severe systemic hypoperfusion and tissue hypoxia. At the same time, there is clear evidence of DIC/consumptive coagulopathy : INR peaked at 4.3; PT/PTT were significantly prolonged; fibrinogen decreased to 117–187 mg/dL; and platelet count dropped to a nadir of 64 × 10 9 /L. Severe sepsis complicated by DIC commonly leads to arterial and microcirculatory thrombosis, which can subsequently cause ischemic injury to organs such as the liver and kidneys. The current laboratory abnormalities and organ dysfunction are highly consistent with this pathophysiological process. Although direct imaging evidence is lacking, in a high-risk ICU setting, the condition should be managed as positive in order to avoid missed diagnosis. In addition, diagnosis_long_title1 = 418 is highly likely to correspond to "arterial embolism and thrombosis," providing further support. At present, there is no clear evidence against the diagnosis (such as normal coagulation function or imaging definitively excluding thrombosis). Overall assessment: the findings support the presence of arterial embolism and thrombosis. | GPT-4o (Error Type: Comorbidity Conflation) |
Table 3: Representative diagnostic error attribution cases from different LLMs. Green text indicates clinically reasonable and well-supported evidence; red text indicates misused, misinterpreted clinical evidence or flawed reasoning; highlighted background indicates missing evidence or unaddressed analytical gaps.
More in this topic
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 419 papers