When Should AI Decide, Defer, and Deviate? Responsible Automation for Deceased Donor Kidney Allocation

Madeleine Pollack; Thomas Daillak; Mehmet Toner; Korkut Uygun; Heidi Yeh; Swati Gupta

cs.AIOct 28, 2026

When Should AI Decide, Defer, and Deviate? Responsible Automation for Deceased Donor Kidney Allocation

Authors: Madeleine Pollack, Thomas Daillak, Mehmet Toner, Korkut Uygun, Heidi Yeh, Swati Gupta

Organizations: Massachusetts Institute of Technology, Cambridge, MA 02139 · Massachusetts General Hospital, Boston, MA 02114 · Harvard Medical School, Boston, MA 02115

Abstract

We ask when an AI trained on historical decisions should be allowed to recommend decisions, defer to human experts, or in other cases, flag concerning patterns in data and deviate from past practice. This can be challenging because past decisions record what was done rather than what should have been done, and where data is sparse, a model's predictions may be too uncertain to act on. We provide a framework for addressing this problem using a case study in deceased-donor kidney allocation. Nearly 30% of kidneys recovered from deceased donors in the United States are never transplanted, largely because they are hard to place through the standard waitlist sequence. One can use AI to flag kidneys at risk of delayed or failed placement for out-of-sequence (OOS) allocation, but this comes at the cost of bypassing candidates higher on the waitlist. Thus, OOS allocation should only be used when the organ is demonstrably at high risk of discard on the standard sequence. Using Organ Procurement and Transplantation Network match-run data from 2016-2019 and 2023-2025 (excluding COVID-19-disrupted years), we build the Arbitrator framework, which combines various interpretable models and acts only when the uncertainty in their predictions cannot change the preferred action. Our Arbitrator resolves 93.5% of organ placement decisions using explicit medical criteria learned from data, and defers the remaining 6.5% to a human or a more complex model. We show that the Arbitrator correctly identifies 70.1% of historically discarded low-KDPI kidneys as OOS candidates, projecting that up to 43.9% of historically discarded kidneys could be flagged for expedited placement and potentially transplanted. We then exploit the interpretability of the Arbitrator to create an agentic audit comparing the learned rules to published clinical literature in order to flag decision-making patterns that are contentious.

Figures & tables

Panel of Figure 1 — Figure 1: Our uncertainty-aware, auditable Arbitrator model. Each interpretable model certifies an action only when its confidence interval lies entirely on one side of $\tau_{\beta}$ . Organs with certified, non-conflicting actions are resolved; all others are deferred. An auditor (expert or language model) checks each resolved rule against clinical references and recommends deciding, deviating, or further deferring. XGBoost handles deferred organs.

Figure 2: ML models identify more non-EM kidneys while recommending fewer EM kidneys for OOS allocation. Each point corresponds to one value of $\beta$ and shows low-KDPI non-EM kidneys identified for potential OOS placement (y-axis) against low-KDPI EM kidneys unnecessarily routed OOS (x-axis). Stars mark $\beta=0.5$ .

Figure 3: CART and SparseRF provide complementary certification. Test kidney certification level at $\beta=0.5$ by CART and SparseRF.

Donor features	Rule(s)	CART 99% CI	SparseRF 99% CI	Pathway
KDPI $>90.5\%$ ; GS $>20\%$ ; vascular changes $\geq 10\%$	C164, F153_400	0.032–0.067 (OOS)	0.023–0.057 (OOS)	Dual-certified OOS
KDPI $\leq 41.5\%$ ; Cr $\leq 2.595$ mg/dL; IF $<10\%$ ; GS $\leq 10\%$ ; lipase $>1290.5$ U/L; no kidney biopsy; HCV NAT negative	C19, F25_13, F81_9, F164_278, F206_243	0.918–0.939 (OS)	0.888–0.932 (OS)	Dual-certified OS
$78.5\%<$ KDPI $\leq 90.5\%$ ; GS $\leq 10\%$ ; recovery year through 2019; Cr $\leq 2.295$ mg/dL; continued smoking not recorded; BUN $>16.5$ mg/dL	C130	0.499–0.663 (OOS)	0.000–1.000 (defer)	Single-certified OOS
$27.5\%<$ KDPI $\leq 43.5\%$ ; $2.595<$ Cr $\leq 3.765$ mg/dL; sodium $>142.5$ mEq/L	C49, F25_59	0.557–0.786 (defer)	0.788–0.882 (OS)	Single-certified OS
KDPI $\leq 41.5\%$ ; Cr $\leq 2.595$ mg/dL; IF $\geq 10\%$	C21	0.486–0.710 (defer)	0.000–1.000 (defer)	Deferred

Table 1: Interpretable Arbitrator rules and certification pathways at

\beta=0.5

. A model certifies on-sequence (OS) or out-of-sequence (OOS) placement when its entire 99% confidence interval lies above or below the decision threshold

\tau_{\beta}=2/3

, respectively; otherwise it defers. C denotes a CART leaf and F a SparseRF rule. Additional certification profiles and selected audited rules are presented in the Appendix. Abbreviations: Cr, terminal creatinine; GS, glomerulosclerosis; IF, interstitial fibrosis; BUN, blood urea nitrogen; HCV NAT, hepatitis C nucleic acid test.

Certification Pathway	Accuracy	FPR	FNR	Avg. cost
Any
XGBoost	0.815	0.223	0.165	0.131
Arbitrator	0.802	0.252	0.170	0.143
Dual certified
XGBoost	0.853	0.208	0.116	0.108
Arbitrator	0.849	0.226	0.113	0.114

Table 2: Performance by certification pathway at

\beta=0.5

. FPR and FNR denote false positive and false negative rates, respectively. The positive class corresponds to EM organs. For deferred kidneys, only XGBoost predictions are available. Certification pathway “Any” includes all 21,574 test kidneys; its Arbitrator metrics include XGBoost decisions for deferred kidneys.

Figure 4: Arbitration pathway varies across KDPI. Certification pathways for 21,574 held-out kidneys at $\beta=0.5$ , grouped by KDPI.

Rule	Condensed description	Conf. Interval
C36	$41.5\%<\mathrm{KDPI}\leq 60.5\%$ ; $1.675<\mathrm{Cr}\leq 2.595$ ; IF $<10\%$ ; GS $\leq 10\%$ ; no coronary angiogram; recovery after 2019	$[.377,.645]$
C51	KDPI $\leq 43.5\%$ ; Cr $>3.765$ ; cause of death not recorded as anoxia	$[.158,.534]$

Table 3: Rules audited by the LLM agent. In both rules, the recommendation from the Arbitrator is to place the organs out-of-sequence (OOS); however, the audit proposed keeping them on-sequence (OS), based on [ 24 ; 42 ; 30 ] . KDPI: Kidney Donor Profile Index; Cr: terminal creatinine (mg/dL); IF: interstitial fibrosis; GS: glomerulosclerosis .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Recorded OOS indicators by match-run year. The numerator is the number of distinct donors with code 861, 862, 863 or 887 in either PTR refusal field during that calendar year; the denominator is all distinct donors with kidney PTR match-run records that year, before study exclusions. The asterisk marks partial 2025 data, January 1–June 30.

Stage	Kidneys removed	Kidneys remaining
Kidney observations in linked study inputs	—	159,961
Donor has an observable OOS placement pathway	16,056	143,905
All donor offers removed by eligibility filtering	58	143,847
Disposition outside retained categories or unknown	26	143,821
Eligible prediction cohort	—	143,821

Appendix

Table S1: Construction of the eligible pooled kidney dataset from the linked study inputs. Each row removes kidneys from the preceding row; offer-record removal counts are reported separately in the text.

Panel of Figure 10 — Figure S2 : Empirical offer-level acceptance and discard statistics. (A) Cumulative eligible final acceptance among eventually accepted kidneys by original PTR sequence. (B) Empirical probability of explicit organ discard among kidneys not accepted by sequence $k$ . Both panels use identical KDPI bands: 0–20%, 20–40%, 40–60%, 60–85%, and 85–100% (labeled 85%+). Bands include their lower endpoint and exclude their upper endpoint.

Year or epoch	Kidneys	Accepted (%)	EM (%)
2016	18,420	77.78	70.55
2017	18,988	78.74	71.79
2018	19,588	79.85	72.45
2019	21,874	79.93	72.36
2020*	23,322	78.03	70.65
2021*	25,252	73.06	64.98

Appendix

Table S3: Retained kidney acceptance rates by donor recovery year or epoch. Accepted denotes organ-assigned eligible final acceptance at any original sequence; EM denotes acceptance by original sequence 100. Asterisks identify rows from the excluded 2020–2022 match-run cohort. Cohorts contain minimal, but positive overlap, so their totals must not be pooled.

Figure S3 : Test-set calibration of the five probability models (21,574 kidneys). Kidneys are grouped into ten equal-width bins of predicted early-acceptance probability; each point plots the bin’s mean prediction against its observed early-acceptance rate.

Figure S4 : SparseRF rule-budget comparison for $K\in\{8,16,32,64,128,256\}$ on the same fitted forest and cohort. (A) Average cost and interpretable resolution at $\beta=0.5$ . (B) Selected rules and source trees by budget. (C) Interpretable resolution and (D) cost difference relative to XGBoost across $\beta$ .

		Resolution policy		Average cost
Pathway	Kidneys (%)	Single	Dual	Single	Dual
Dual	17,121 (79.36)	Resolved	Resolved	0.1136	0.1136
CART-only	1,689 (7.83)	Resolved	Deferred	0.2866	0.2238
SparseRF-only	1,371 (6.35)	Resolved	Deferred	0.2666	0.2221
Deferred	1,393 (6.46)	Deferred	Deferred	0.2021	0.2021
Total	21,574 (100.00)			0.1426	0.1348

Appendix

Table S6: Resolution pathways at

\beta=0.5

for the 21,574 test kidneys. Dual resolution requires both models to certify the same action; single-model resolution requires one model to certify while the other abstains.

Certification pathway/action	Donor features	Rule(s)	$n$	CART 99% CI	SparseRF 99% CI
A. Dual-certified OOS	KDPI $>90.5\%$ ; GS $>20\%$ ; vascular changes $\geq 10\%$ .	C164, F153_400	158	0.032–0.067 OOS	0.023–0.057 OOS
B. Dual-certified OS	KDPI $\leq 41.5\%$ ; Cr $\leq 2.595$ mg/dL; IF $<10\%$ ; GS $\leq 10\%$ ; lipase $>1290.5$ U/L; no kidney biopsy; HCV NAT negative.	C19, F25_13, F81_9, F164_278, F206_243	2,077	0.918–0.939 OS	0.888–0.932 OS
C. CART-only OOS	$78.5\%<$ KDPI $\leq 90.5\%$ ; GS $\leq 10\%$ ; recovery year through 2019; Cr $\leq 2.295$ mg/dL; continued smoking not recorded; BUN $>16.5$ mg/dL.	C130	86	0.499–0.663 OOS	0.000–1.000 defer
D. CART-only OS	$41.5\%<$ KDPI $\leq 60.5\%$ ; Cr $\leq 1.675$ mg/dL; IF $<10\%$ ; GS $\leq 10\%$ ; no coronary angiogram; recovery year after 2019.	C35	131	0.682–0.764 OS	0.000–1.000 defer
E. SparseRF-only OOS	$60.5\%<$ KDPI $\leq 78.5\%$ ; GS $>10\%$ ; kidney biopsy not recorded as No; IF $<10\%$ ; Cr $\leq 1.895$ mg/dL; age $\leq 61.5$ years; vascular changes $<10\%$ .	C96, F153_374	45	0.505–0.675 defer	0.311–0.414 OOS
F. SparseRF-only OS	$27.5\%<$ KDPI $\leq 43.5\%$ ; $2.595<$ Cr $\leq 3.765$ mg/dL; sodium $>142.5$ mEq/L.	C49, F25_59	72	0.557–0.786 defer	0.788–0.882 OS

Appendix

Table S7 : Example certification profiles at

\beta=0.5

. OS denotes on-sequence and OOS off-sequence placement. A model certifies OS when its entire 99% confidence interval (CI) is above

\tau_{\beta}=2/3

, certifies OOS when it is below, and abstains otherwise. The number of test kidneys matching each profile is given by

n

. C identifies the CART leaf and F a selected SparseRF rule. Donor features are abbreviated observed-value summaries; missing-value alternatives are omitted. Cr, terminal creatinine; GS, glomerulosclerosis; IF, interstitial fibrosis; BUN, blood urea nitrogen; HCV NAT, hepatitis C nucleic acid test.

Policy	Interpretable (%)	Organs resolved	Low-KDPI, non-EM identified	Low-KDPI, discarded identified
XGBoost-only	0.0	—	3,300 (66.7%)	2,079 (75.9%)
Arbitrator	93.5	20,181	3,068 (62.1%)	1,920 (70.1%)
Arbitrator (dual-certification only)	79.4	17,121	3,188 (64.5%)	1,992 (72.8%)

Appendix

Table S8: Trade-off between interpretable resolution and potentially saved organs at

\beta=0.5

. Arbitrator denotes the single certification policy used in the main paper; the dual-certification-only policy resolves a kidney only when both models certify the same action. Interpretable percentages use all 21,574 test kidneys, and organs resolved counts decisions supported by interpretable rules. Both Arbitrator policies use XGBoost for deferred kidneys. There are 4,944 low-KDPI non-EM and 2,738 low-KDPI historically discarded kidneys in the test data.

Original certification	Organs audited	Retained for automation	Deferred $n$ (%)
Dual	17,121	15,401	1,720 (10.05%)
Single	3,060	2,070	990 (32.35%)
CART only	1,689	1,230	459 (27.18%)
SparseRF only	1,371	840	531 (38.73%)
Overall	20,181	17,471	2,710 (13.43%)

Appendix

Table S10: Organ-level audit outcomes by original certification pathway. Retained organs remain eligible for automation under the hypothetical audit policy. Deferral percentages use the organs audited in each row. Overall combines Dual and Single; the two indented rows partition Single and must not be added to it. The 1,393 organs deferred before audit are excluded.

Rule	Condensed description	CI	Val. $n$	Test $n$	Auditor recommendation
C36	$41.5\%<\mathrm{KDPI}\leq 60.5\%$ ; $1.675<\mathrm{Cr}\leq 2.595$ ; IF $<10\%$ ; GS $\leq 10\%$ ; no coronary angiogram; recovery after 2019	$[.377,.645]$	86	97	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
C51	KDPI $\leq 43.5\%$ ; Cr $>3.765$ ; cause of death not recorded as anoxia	$[.158,.534]$	176	222	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
F25_70	Age $\leq 50.5$ ; KDPI $\leq 43.5\%$ ; GS $\leq 10\%$ ; Cr $>3.765$ ; cause of death not recorded as anoxia; state not recorded as CA; glomerular count $\leq 27.5$	$[.071,.317]$	57	51	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
C58	KDPI $\leq 43.5\%$ ; Cr $>3.765$ ; anoxia; age $>39.5$	$[.399,.666]$	86	102	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
C62	$43.5\%<\mathrm{KDPI}\leq 60.5\%$ ; Cr $>5.025$ ; GS $\leq 15\%$	$[.391,.650]$	92	44	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
F164_72	Biopsy type recorded; KDPI $\leq 67.5\%$ ; BUN $>34.5$ ; IF $<10\%$ ; GS $\leq 15\%$ ; Na $>142.5$ ; Cr $>3.205$ ; age $>42.5$	$[.306,.513]$	143	167	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]

Appendix

Table S11 : Audited candidate and comparison rules. CI is the Wilson interval for historical early acceptance computed from the validation data. Descriptions summarize observed-value conditions joined by “AND.” OS denotes on-sequence, OOS denote off-sequence, and arrows show proposed deviations. Test supports can overlap. Abbreviations: KDPI: historical Kidney Donor Profile Index; Cr: terminal creatinine (mg/dL); IF: interstitial fibrosis; GS: glomerulosclerosis; BUN: blood urea nitrogen (mg/dL); Na: serum sodium (mEq/L); age in years; CA: California.

Responsible AI

Responsible AI studies how to design, evaluate, and govern AI systems to promote fairness, safety, transparency, accountability, privacy, and sustainability.

See all 134 papers

Jun 1, 2026

Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis

Machine learning predictors have become essential tools for guiding automated decision making. However, a major misalignment persists: predictive models are typically optimized in terms of standard statistical metrics in isolation from the algorithmic tasks they inform. We highlight this incongruity in the high-stakes domain of organ allocation by demonstrating that any algorithm relying on (even highly accurate) survival predictors optimized for standard metrics -- such as the Concordance index (C-index) -- can yield arbitrarily poor outcomes when used for allocation, failing to guarantee utility better than a uniform random selection. To bridge the gap between survival analysis and policy optimization, we introduce a decision-focused learning approach based on optimizing normalized discounted cumulative gain (NDCG), a mainstay metric in information retrieval. We establish the utility of NDCG in survival analysis by proving that it translates to guarantees on the performance of allocation. Empirically, we propose a bootstrapping approach to optimize the NDCG of existing survival models. Unlike prior work, we also address the challenge of right censorship when evaluating ranking. On historical heart transplant data from the US, our method dramatically boosts the NDCG of baseline models by 50-100%, which translates to tens of thousands of additional life years gained annually when deployed for transplant allocation. We anticipate that our framework will find broader applications in decision making with predictions.

Itai Zilberstein, Ioannis Anagnostides, Tuomas Sandholm

Department of Computer Science, Carnegie Mellon University, Pittsburgh, PA · Additional affiliations: Strategy Robot, Inc., Strategic Machine, Inc., Optimized Markets, Inc.

Learning to Rank Survival Analysis Decision-Focused Learning Clinical Decision-Making

Ask PDF

Aug 4, 2026

Paired Recipient-based Evaluation of Survival Prediction for Deceased Donor Kidney Transplants

There has been significant interest in using machine learning algorithms to predict kidney transplant outcomes, such as the number of years until a graft inevitably fails. These prediction algorithms could possibly be used for pre-transplant donor-recipient matching to identify more compatible donors and recipients and thus improve post-transplant outcomes. In this study, we explore the use of survival prediction models trained on deceased donor kidney transplant data from the Scientific Registry of Transplant Recipients (SRTR). We propose a novel paired recipient-based evaluation framework that compares graft outcomes between two recipients who received kidneys from the same deceased donor, allowing us to evaluate the counterfactual benefit of changing the recipient for a certain donor. We find that five different survival prediction models, ranging in complexity from linear to deep learning-based models, all result in ~60% paired recipient-based accuracy. We further translate this accuracy into an interpretable quantity of post-transplant years gained. We also highlight major limitations of the commonly used concordance index (C-index) metric for evaluating survival prediction accuracy in this setting and demonstrate that our proposed paired recipient-based accuracy metric is more clinically relevant and better reflects real-world allocation settings.

Misaki Matsuura, Mohammadreza Nemati, Dulat Bekbolsynov +2

Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH 44106, USA · Department of Medical Microbiology and Immunology University of Toledo Toledo, OH 43614, USA

Healthcare Survival Analysis Counterfactual Prediction

Ask PDF

Mar 14, 2026

When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

AI systems increasingly assist decision making by producing cheap factor-level assessments of complex inputs, but these assessments can be biased or incomplete. We study selective human evaluation: given AI signals, which factors should be dispatched to human experts? We formulate this human--AI collaboration problem as a factor-level information-acquisition problem. Under squared loss, the optimal dispatching rule maximizes a contextual reward; under a linear model, this reward admits a closed-form decomposition into two interpretable terms: predictive relevance and residual uncertainty in the human evaluation after conditioning on the AI signal. We instantiate the framework on peer review, decomposing papers into ten aspects and evaluating up to 3,408 ICLR submissions across three LLMs and multiple regression heads. In this retrospective proxy, where human aspect signals are extracted from existing reviews, predictions that query three of the ten aspects under the optimal dispatching rules match the full ten-aspect human benchmark. The finding holds when the human signal comes from a single noisy reviewer, and across review years. These results position principled dispatching as a practical building block for scalable human--AI decision systems. We also discuss safeguards under which such a rule can serve as decision support that leaves the final decision with a human.

Lezhi Tan, Naomi Sagan, Lihua Lei +1

Stanford University, USA

AI-Assisted Decision Making Human-AI Collaboration Human-in-the-Loop Evaluation

Ask PDF

Abstract

Responsible AI

Explore similar work

Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis

Paired Recipient-based Evaluation of Survival Prediction for Deceased Donor Kidney Transplants

When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions