Auditable Clinical Support with Decision Agents
Organizations: Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA. · Department of Anesthesiology, Medical Faculty, RWTH Aachen University, Aachen, Germany. · Department of Intensive Care and Intermediate Care, Medical Faculty, RWTH Aachen University, Aachen, Germany. · Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA. ·
Abstract
Clinicians reviewing a diagnostic work-up will ask what was considered at each step and why the assessment changed. Agents powered by free text can reach physician-level diagnostic accuracy, but their written reasoning does not always reflect the factors that went into the decision. Here, we present Decision2Care, a decision agent that works up the acutely ill patient without producing any text. A decision model assigns a probability to each permitted EHR action, and fixed rules carry out the most probable one. We evaluated Decision2Care on the records of 546 emergency admissions across 12 diseases. Decision2Care reached a diagnostic accuracy of 86.5%, compared with 86.1% for the MIRA-v2 agent, and required a median of 2.2 s per work-up, compared with 92-100 s. Each work-up tracked which options Decision2Care was considering at each step. In 10 of the 12 diseases, the differential diagnosis changed most at the test on which the guideline bases the diagnosis. Where Decision2Care was most confident after a single work-up, all but four of its diagnoses were correct (98.5%). We traced each of these four errors back to a single imaging report. Three of them were corrected when that report was removed. The same kind of decision was used to select the treatments, and Decision2Care's drug orders were judged guideline-adherent more often than MIRA-v2's. In emergency patients, its prediction of death was comparable to that of three intensivists on the same cases. In five of the eight intensive care cohorts, it exceeded the best of the three 24-h severity scores within 2-6 h of records. Our results show that decision agents can carry out clinical work-ups while recording the options, probabilities, and execution rules behind each action for audit.
Figures & tables
| Measure | n | Decision2Care | MIRA-v2 workflow | ||
| This work | GPT-OSS-120B | GLM-5 | Claude Sonnet 5 | ||
| Computational efficiency | |||||
| Median time per work-up, s a | — | 2.2 (1.7–2.7) | 92.7 (74.3–116.6) | 92.4 (75.5–121.8) | 100.1 (87.5–119.1) |
| Cost per case, 1 work-up, US$ b | 546 | 0.0024 | 0.0376 | 0.1266 | 0.1460 |
| Total cost, 1 work-up, US$ b | 546 | 1.31 | 20.5 | 69.1 | 79.7 |
| Total cost, 5 work-ups, US$ c | 546 | — | 102.6 | 345.5 | 398.5 |
| Code | ICD-10-CM title | Answers | Share of errors (%) |
|---|---|---|---|
| K80.20 | Calculus of gallbladder without cholecystitis without obstruction | 98 | 68.1 |
| K85.10 | Biliary acute pancreatitis without necrosis or infection | 20 | 13.9 |
| K83.1 | Obstruction of bile duct | 10 | 6.9 |
| K71.2 | Toxic liver disease with acute hepatitis | 5 | 3.5 |
| K66.1 | Hemoperitoneum | 5 | 3.5 |
| K80.71 | Calculus of gallbladder and bile duct without cholecystitis with obstruction | 5 | 3.5 |
| Disease | Physicians, correct | Physicians, accuracy, % | Decision2Care, admissions | Decision2Care, accuracy, % | Difference, points |
|---|---|---|---|---|---|
| Appendicitis | 38 of 43 | 88.4 | 148 | 99.9 | |
| Cholecystitis | 36 of 45 | 80.0 | 129 | 77.7 | |
| Diverticulitis | 36 of 44 | 81.8 | 54 | 88.1 | |
| Pancreatitis | 33 of 42 | 78.6 | 52 | 91.5 | |
| Pneumonia | 15 of 26 | 57.7 | 21 | 45.7 | |
| Pulmonary embolism | 40 of 45 | 88.9 | 15 | 92.0 |
| Variant | Accuracy (%) | Change (points) | 95% half- width | (Holm) | |
|---|---|---|---|---|---|
| ICD options in a seeded random order (seed 1) | 87.7 | +1.25 | 1.39 | 0.080 | 0.798 |
| ICD options in a seeded random order (seed 2) | 86.8 | +0.33 | 1.17 | 0.579 | 1.000 |
| ICD options in a seeded random order (seed 3) | 86.6 | +0.15 | 1.45 | 0.843 | 1.000 |
| Every ICD Choice asked under 5 random orders, probabilities averaged | 87.5 | +1.06 | 1.04 | 0.046 | 0.510 |
| ICD instructions paraphrased (set 1), routing unchanged | 86.1 | -0.40 | 1.21 | 0.513 | 1.000 |
| ICD instructions paraphrased (set 2), routing unchanged | 87.0 | +0.51 | 0.98 | 0.307 | 1.000 |
| Disease | n | Jev 1.13.0 (5 runs) | PPLX decider v1.2 (5 runs) | Qwen3.8-Flash-Next readout (5 runs) |
|---|---|---|---|---|
| Appendicitis | 148 | 99.9 | 98.6 | 98.8 |
| Cholecystitis | 129 | 77.7 | 80.6 | 87.8 |
| Diverticulitis | 54 | 88.1 | 90.7 | 92.6 |
| Pancreatitis | 52 | 91.5 | 85.8 | 82.7 |
| Pneumonia | 21 | 45.7 | 41.9 | 52.4 |
| Pulmonary embolism | 15 | 92.0 | 86.7 | 70.7 |
| Disease | Guideline | Tests counted |
|---|---|---|
| Appendicitis | WSES Jerusalem guidelines 2020 (Di Saverio et al., World J Emerg Surg 15:27) [ 21 ] | ultrasound, CT or MRI of the abdomen |
| Cholecystitis | WSES 2020 (Pisano et al., World J Emerg Surg 15:61) [ 26 ] | ultrasound, MRI or HIDA scan of the abdomen |
| Bowel obstruction | WSES position paper on small bowel obstruction 2021 (Amara et al., World J Emerg Surg, doi:10.1186/s13017-021-00379-8) [ 23 ] | CT or radiograph of the abdomen |
| Diverticulitis | ASCRS 2020 (Hall et al., Dis Colon Rectum 63:728) [ 22 ] | ultrasound, CT or MRI of the abdomen |
| Pancreatitis | WSES 2019 (Leppäniemi et al., World J Emerg Surg 14:27) [ 29 ] | ultrasound or CT of the abdomen; lipase or amylase |
| Pneumonia | ATS/IDSA 2019 (Metlay et al., Am J Respir Crit Care Med 200:e45) [ 60 ] | CT or radiograph of the chest |
| Disease | Case- runs | Decisive step, n (%) | At a guideline test, n (%) | Received (%) | Change at a guideline test | Change at other actions |
|---|---|---|---|---|---|---|
| Appendicitis | 740 | 179 (24) | 104 (58) | 98 | +0.11 (155) | +0.03 (238) |
| Cholecystitis | 645 | 279 (43) | 171 (61) | 85 | +0.15 (119) | +0.04 (376) |
| Bowel obstruction | 335 | 119 (36) | 114 (96) | 99 | +0.15 (84) | -0.01 (221) |
| Diverticulitis | 270 | 176 (65) | 135 (77) | 96 | +0.43 (53) | +0.04 (105) |
| Pancreatitis | 260 | 102 (39) | 97 (95) | 71 | +0.24 (59) | +0.01 (148) |
| Pneumonia | 105 | 37 (35) | 28 (76) | 79 | +0.12 (19) | +0.00 (123) |
| Domain | Disease | Decision2Care (this work) | MIRA-v2 + GPT-OSS-120B | MIRA-v2 + GLM-5 | MIRA-v2 + Claude Sonnet 5 |
|---|---|---|---|---|---|
| Physical examination | Overall | 96.4 (94.5, 97.7) | 74.8 (71.0, 78.3) | 100.0 (99.3, 100.0) | 100.0 (99.3, 100.0) |
| Physical examination | Appendicitis | 98.6 (95.2, 99.6) | 90.4 (84.6, 94.2) | 100.0 (97.5, 100.0) | 100.0 (97.5, 100.0) |
| Physical examination | Cholecystitis | 95.8 (90.8, 98.1) | 49.9 (41.4, 58.4) | 100.0 (97.1, 100.0) | 100.0 (97.1, 100.0) |
| Physical examination | Diverticulitis | 98.1 (90.2, 99.7) | 85.9 (74.3, 92.8) | 100.0 (93.4, 100.0) | 100.0 (93.4, 100.0) |
| Physical examination | Pancreatitis | 90.0 (78.9, 95.6) | 51.2 (38.0, 64.2) | 100.0 (93.1, 100.0) | 100.0 (93.1, 100.0) |
| Physical examination | Pneumonia | 100.0 (84.5, 100.0) | 81.0 (60.0, 92.3) | 100.0 (84.5, 100.0) | 100.0 (84.5, 100.0) |
| System | Rule | Retained, n (%) | Retained accuracy (%) | Retained errors / all |
|---|---|---|---|---|
| Decision2Care | confidence 0.70 (Supplementary Fig. 6 b) | 269 (49.3) | 98.5 [96.2, 99.4] | 4 / 71 |
| Decision2Care | confidence 0.70, each of the 5 runs | 269–273 (49.3–50.0) | 97.8–98.5 | 4–6 / 71–77 |
| Decision2Care | confidence 0.80, set on the 325 other admissions | 148 (27.1) | 99.3 [96.3, 99.9] | 1 / 71 |
| MIRA-v2 + GPT-OSS-120B | ConsistencyDx 0.90, the published threshold (Supplementary Fig. 6 d, f, h) | 300 (54.9) | 96.0 [93.1, 97.7] | 12 / 69 |
| MIRA-v2 + GPT-OSS-120B | ConsistencyDx 0.95 | 205 (37.5) | 97.6 [94.4, 99.0] | 5 / 69 |
| MIRA-v2 + GLM-5 | ConsistencyDx 0.90, the published threshold (Supplementary Fig. 6 d, f, h) | 343 (62.8) | 98.0 [95.8, 99.0] | 7 / 53 |
| Condition | Accuracy, all cases (%) | Coverage (%) | Retained accuracy (%) | Retained errors | AUROC | Mean confidence |
|---|---|---|---|---|---|---|
| As recorded | 86.5 (85.9–87.0) | 49.7 (49.3–50.0) | 98.4 (97.8–98.5) | 4.4 (4.0–6.0) | 0.834 (0.831–0.839) | 0.650 (0.649–0.652) |
| History withheld | 59.4 (58.6–60.3) | 35.1 (34.2–36.4) | 97.5 (97.3–98.0) | 4.8 (4.0–5.0) | 0.910 (0.908–0.912) | 0.538 (0.535–0.542) |
| Decisive results made normal | 22.2 (22.0–22.5) | 4.2 (3.7–4.6) | 84.3 (80.0–87.0) | 3.6 (3.0–5.0) | 0.569 (0.555–0.580) | 0.379 (0.376–0.383) |
| Disease group | Treatment | n | Decision2Care | MIRA-v2 + GPT-OSS-120B | MIRA-v2 + GLM-5 | MIRA-v2 + Claude Sonnet 5 | Documented care |
|---|---|---|---|---|---|---|---|
| Appendicitis | Antibiotics | 145 | 99.3 [97.2, 100.0] | 89.2 [87.1, 91.3] | 93.5 [91.0, 95.9] | 100.0 [100.0, 100.0] | 84.7*** [79.0, 90.3] |
| Appendicitis | Supportive care | 145 | 100.0 [100.0, 100.0] | 89.1 [87.0, 91.2] | 69.1*** [65.3, 73.1] | 90.9 [88.6, 93.1] | 100.0 [100.0, 100.0] |
| Appendicitis | Appendectomy | 134 | 100.0 [100.0, 100.0] | 97.5 [96.0, 98.8] | 100.0 [100.0, 100.0] | 99.9 [99.6, 100.0] | – |
| Cholecystitis | Antibiotics | 91 | 97.8 [94.5, 100.0] | 89.5 [86.2, 92.6] | 97.4 [95.1, 99.2] | 98.4 [95.9, 100.0] | 93.3 [87.8, 97.8] |
| Cholecystitis | Supportive care | 91 | 100.0 [100.0, 100.0] | 89.2 [85.8, 92.5] | 72.8*** [66.8, 78.4] | 99.1 [98.0, 100.0] | 98.9 [96.6, 100.0] |
| Cholecystitis | Cholecystectomy | 83 | 100.0 [100.0, 100.0] | 66.5*** [61.2, 71.2] | 100.0 [100.0, 100.0] | 99.5 [98.8, 100.0] | – |
| Disease | Category | n | Decision2Care | MIRA-v2 + GPT-OSS-120B | MIRA-v2 + GLM-5 | MIRA-v2 + Claude Sonnet 5 | Documented care |
|---|---|---|---|---|---|---|---|
| All, per patient | mean over the categories | 355 | 88.2 [86.0, 90.3] | 65.6*** [63.9, 67.3] | 58.7*** [57.0, 60.4] | 79.7*** [77.6, 81.7] | 81.4*** [78.8, 83.9] |
| Urinary tract infection | antibiotic | 10 | 70.0 [40.0, 100.0] | 69.0 [47.0, 89.0] | 90.0 [70.0, 100.0] | 80.0 [58.0, 100.0] | 90.0 [70.0, 100.0] |
| Cholecystitis | analgesic | 91 | 100.0 [100.0, 100.0] | 89.2 [85.9, 92.5] | 70.0*** [64.1, 75.6] | 98.7 [97.4, 99.8] | 98.9 [96.6, 100.0] |
| Cholecystitis | antibiotic | 91 | 91.2 [84.6, 96.7] | 94.6* [92.6, 96.5] | 91.8 [88.2, 95.2] | 98.1 [95.4, 100.0] | 93.3 [87.8, 97.8] |
| Cholecystitis | intravenous fluids | 91 | 100.0 [100.0, 100.0] | 13.7*** [10.1, 17.6] | 4.5*** [2.3, 7.0] | 71.8*** [65.7, 77.7] | 97.8 [94.4, 100.0] |
| Diverticulitis | analgesic | 49 | 28.6 [16.3, 40.8] | 7.8** [2.4, 13.9] | 7.8*** [4.1, 12.2] | 5.3** [1.2, 10.2] | 36.7 [24.5, 51.0] |
| Measure | Decision2Care | MIRA-v2 + GPT-OSS-120B | MIRA-v2 + GLM-5 | MIRA-v2 + Claude Sonnet 5 |
|---|---|---|---|---|
| Drug classes against those given on the first day or in the emergency department (435 admissions with care recorded; mean per admission) | ||||
| F1 | 40.3 [39.1, 41.6] | 34.7*** [33.4, 36.1] | 35.2*** [33.8, 36.7] | 40.2 [39.0, 41.5] |
| Precision: classes ordered that were given (%) | 46.8 [45.2, 48.4] | 49.5** [47.6, 51.5] | 57.0*** [54.7, 59.4] | 54.2*** [52.3, 56.1] |
| Recall: classes given that were ordered (%) | 37.2 [35.9, 38.5] | 30.1*** [28.8, 31.3] | 27.4*** [26.2, 28.7] | 33.6*** [32.4, 34.8] |
| Classes ordered per admission (given: 8.9) | 6.90 [6.70, 7.11] | 5.61*** [5.37, 5.85] | 4.37*** [4.15, 4.59] | 5.56*** [5.35, 5.78] |
| Antibiotic decision as given (%) | 81.8 [78.0, 85.3] | 79.3 [76.4, 82.2] | 83.5* [80.3, 86.6] | 84.7 [81.6, 88.0] |
| Disease | Database | Patients (30/90 d) | Deaths (30/90 d) | AUROC, 30 d, mean (range) | AUROC, 90 d, mean (range) |
|---|---|---|---|---|---|
| Intracerebral haemorrhage | MIMIC-III | 316 / 317 | 131 / 140 | 0.864 (0.862–0.865) | 0.854 (0.851–0.857) |
| Intracerebral haemorrhage | MIMIC-IV | 652 / 653 | 220 / 242 | 0.895 (0.893–0.899) | 0.879 (0.877–0.880) |
| Traumatic brain injury | MIMIC-III | 1,528 | 271 / 319 | 0.871 (0.870–0.873) | 0.855 (0.855–0.856) |
| Traumatic brain injury | MIMIC-IV | 2,481 | 334 / 421 | 0.891 (0.889–0.893) | 0.856 (0.854–0.858) |
| Sepsis | MIMIC-III | 614 / 615 | 164 / 225 | 0.779 (0.774–0.783) | 0.762 (0.759–0.766) |
| Sepsis | MIMIC-IV | 335 / 336 | 81 / 119 | 0.768 (0.757–0.776) | 0.717 (0.709–0.726) |
| Sens., 30 d | Spec., 30 d | AUROC, 30 d (agent minus intensivist) | Unsure, 30 d | Sens., 90 d | Spec., 90 d | AUROC, 90 d (agent minus intensivist) | Unsure, 90 d | |
|---|---|---|---|---|---|---|---|---|
| Intensivist 1 | 0.63 | 0.60 | 0.720 (-0.15 to +0.13) | 51 (71%) | 0.65 | 0.53 | 0.655 (-0.17 to +0.12) | 55 (76%) |
| Intensivist 2 | 0.70 | 0.76 | 0.755 (-0.15 to +0.05) | 37 (51%) | 0.68 | 0.78 | 0.741 (-0.23 to -0.01) | 37 (51%) |
| Intensivist 3 | 0.63 | 0.49 | 0.670 (-0.06 to +0.15) | 15 (21%) | 0.62 | 0.50 | 0.637 (-0.13 to +0.11) | 15 (21%) |
| Mean of the 3 | 0.65 | 0.61 | 0.727 (-0.12 to +0.08) | 103 (48%) | 0.65 | 0.60 | 0.696 (-0.19 to +0.05) | 107 (50%) |
| Agent, 1 run | 0.707 (0.565–0.836) | 0.628 (0.496–0.757) | ||||||
| Weighted back to the cohorts | ||||||||
| Reviews | Essential record unread | Step unnecessary | Prediction at least defensible | Prediction plausible | Plausibility (mean) | Interval agreed | |
|---|---|---|---|---|---|---|---|
| Intensivist 1 | 72 | 5 (7%) | 55 (76%) | 67 (93%) | 44 (61%) | 3.90 | 50 (69%) |
| Intensivist 2 | 72 | 12 (17%) | 27 (38%) | 50 (69%) | 30 (42%) | 3.21 | 40 (56%) |
| Intensivist 3 | 72 | 3 (4%) | 40 (56%) | 65 (90%) | 51 (71%) | 4.10 | 53 (74%) |
| Intracerebral haemorrhage | 30 | 0 (0%) | 16 (53%) | 27 (90%) | 19 (63%) | 3.97 | 21 (70%) |
| Traumatic brain injury | 30 | 0 (0%) | 8 (27%) | 25 (83%) | 18 (60%) | 3.87 | 18 (60%) |
| Sepsis | 30 | 8 (27%) | 16 (53%) | 26 (87%) | 15 (50%) | 3.60 | 18 (60%) |
| Disease | Patients (deaths) | 1 h | 2 h | 3 h | 4 h | 5 h | 6 h | 7 h | 8 h | 9 h | 10 h | 11 h | 12 h | First hour |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Intracerebral haemorrhage | 774 (196) | 0.678 | 0.787 | 0.825 | 0.839 | 0.845 | 0.854 | 0.842 | 0.862 | 0.863 | 0.871 | 0.875 | 0.874 | 2 |
| Traumatic brain injury | 2,790 (358) | 0.724 | 0.790 | 0.819 | 0.835 | 0.847 | 0.854 | 0.857 | 0.865 | 0.868 | 0.866 | 0.871 | 0.872 | 2 |
| Sepsis | 851 (170) | 0.611 | 0.659 | 0.698 | 0.688 | 0.698 | 0.720 | 0.701 | 0.724 | 0.726 | 0.744 | 0.734 | 0.750 | none |
| Acute kidney injury | 531 (159) | 0.692 | 0.766 | 0.763 | 0.737 | 0.736 | 0.738 | 0.742 | 0.735 | 0.702 | 0.711 | 0.733 | 0.721 | 2 |
| Pulmonary embolism | 326 (29) | 0.702 | 0.707 | 0.751 | 0.710 | 0.784 | 0.814 | 0.796 | 0.820 | 0.814 | 0.818 | 0.829 | 0.801 | none |
| ARDS candidates | 92 (24) | 0.624 | 0.704 | 0.722 | 0.706 | 0.748 | 0.742 | 0.760 | 0.723 | 0.724 | 0.723 | 0.762 | 0.737 | 3 |
| Disease | Score | Score at 24 h, AUROC (95% CI) | Prediction at 2 h | Difference at 2 h (95% CI) | Prediction at 12 h (95% CI) | Difference at 12 h (95% CI) |
|---|---|---|---|---|---|---|
| Intracerebral haemorrhage | APS III | 0.631 (0.582–0.679) | 0.787 | +0.156 (+0.097, +0.214) | 0.874 (0.846–0.899) | +0.244 (+0.190, +0.296) |
| SAPS II | 0.713 (0.672–0.751) | +0.074 (+0.022, +0.127) | +0.161 (+0.120, +0.203) | |||
| OASIS | 0.728 (0.687–0.766) | +0.059 (+0.011, +0.109) | +0.146 (+0.105, +0.188) | |||
| Traumatic brain injury | APS III | 0.701 (0.670–0.733) | 0.790 | +0.089 (+0.053, +0.126) | 0.872 (0.854–0.891) | +0.171 (+0.139, +0.205) |
| SAPS II | 0.756 (0.730–0.781) | +0.034 (+0.004, +0.066) | +0.116 (+0.087, +0.147) | |||
| OASIS | 0.763 (0.738–0.789) | +0.027 (-0.001, +0.058) | +0.109 (+0.084, +0.135) |