Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label
Organizations: Department of Mathematics and Statistics, Missouri University of Science and Technology · Department of Internal Medicine, Division of Epidemiology, University of Utah School of Medicine
Abstract
Background. Many medical AI models are trained to reproduce a single reference label and hand the most likely label to the next step in a clinical workflow. A classifier can select that label correctly while providing an inadequate summary for estimating downstream risk. Methods. We use Jeffrey's rule of conditioning to specify when probabilities over intermediate states, such as diagnoses or imaging categories, can be carried forward to an outcome risk. We separate two errors of reporting only the most likely label, discarding the probability of the other states and applying the risk of patients who have the leading diagnosis to a patient who only probably has it. We also give conditions under which the label and the full distribution lead to the same decision, and we propose guardrails for device developers, regulatory reviewers, and health insurers. Results. In a hypothetical chest-pain differential, reporting only the benign leading diagnosis greatly understates the patient's risk of death or cardiac arrest, so a patient who warrants evaluation could be discharged or denied imaging. In a hypothetical BI-RADS example, the most likely category is "probably benign," yet the propagated malignancy risk exceeds the ceiling that defines that category. Conclusions. A predicted label is not an observed state, and a calibrated classifier is not automatically a valid outcome model. Clinical and insurance coverage decisions should rest on the patient's outcome risk, judged against a threshold justified by the consequences of acting and not acting. Developers should state what each model output means and check that every step between the model and the decision preserves the patient's risk.
Figures & tables
| Primary etiology | Illustrative | |
|---|---|---|
| Gastroesophageal disease (GD) | ||
| Musculoskeletal pain (MP) | ||
| Acute coronary syndrome (ACS) | ||
| Pulmonary embolism (PE) | ||
| Aortic dissection (AD) |
| Category | BI-RADS risk interpretation | Illustrative |
|---|---|---|
| 3 | Probably benign, | |
| 4A | Low suspicion, to |
| Guardrail | What the developer produces | What a reviewer checks |
|---|---|---|
| 1. State what the numbers mean | A specification of the state partition, reference-label protocol, outcome with its time horizon and care policy, target population, and whether the output is a posterior (under which prevalence) or a likelihood. | That state probability , state risk , and label risk are kept distinct; that labels cannot silently become recorded facts; that the classifier and risk table use compatible definitions. |
| 2. Select for the distribution, not the label | Model and calibration-map selection by proper scoring rules (log loss, multiclass Brier score) and full-vector calibration [ 10 , 16 ] , on data disjoint from training. | Per-class reliability diagnostics; evidence that accuracy or top-class confidence was not the sole selection criterion. |
| 3. Validate the downstream risk | A risk table estimated under the same protocol and population, the label-only comparator , held-out outcome performance of , and a test of whether patient data add prognostic information within states. | Comparison of with , the modal-state lookup, and a direct outcome model where feasible; subgroups; sampling uncertainty; transport to the deployment population. |
| 4. Export a justified summary | An output record carrying the full vector, the propagated scalar, or a partial vector with its omitted mass, never renormalized, plus version identifiers for every component. | That each reduction (argmax, top- , rounding) preserves what its intended task requires. |
| 5. Audit compression against the decision margin | The distribution of , the disagreement rate in each direction, and the fraction of cases the agreement condition settles. | How inconclusive cases are handled; that failure of a sufficient condition is not read as evidence of error, nor its success as clinical safety. |
| 6. Test users and revalidate after changes | Human-factors evidence that users distinguish state probability from outcome risk, and a change-control and outcome-monitoring plan. | Comprehension and decisions rather than visual preference; fallback use and delays; repeated validation when the classifier, table, population, or interface changes. |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Definition | Plain-language reading |
|---|---|---|
| Probability over patient encounters in the specified setting | The population being described | |
| , | Patient information supplied to the model | What the model sees |
| , | Intermediate state, one of mutually exclusive states | The diagnosis or imaging category |
| , | Outcome event and its indicator | What the patient is at risk of |
| The correct probability of state for this patient | ||
| , | Model’s reported probability of state ; | What the model says |
Clinical Decision Support
Clinical Decision Support uses data, medical knowledge, and AI to help clinicians assess patients and choose tests, diagnoses, and treatments.
See all 256 papers