When Should AI Decide, Defer, and Deviate? Responsible Automation for Deceased Donor Kidney Allocation

Madeleine Pollack; Thomas Daillak; Mehmet Toner; Korkut Uygun; Heidi Yeh; Swati Gupta

cs.AIOct 28, 2026

When Should AI Decide, Defer, and Deviate? Responsible Automation for Deceased Donor Kidney Allocation

Authors: Madeleine Pollack, Thomas Daillak, Mehmet Toner, Korkut Uygun, Heidi Yeh, Swati Gupta

Organizations: Massachusetts Institute of Technology, Cambridge, MA 02139 · Massachusetts General Hospital, Boston, MA 02114 · Harvard Medical School, Boston, MA 02115

Abstract

We ask when an AI trained on historical decisions should be allowed to recommend decisions, defer to human experts, or in other cases, flag concerning patterns in data and deviate from past practice. This can be challenging because past decisions record what was done rather than what should have been done, and where data is sparse, a model's predictions may be too uncertain to act on. We provide a framework for addressing this problem using a case study in deceased-donor kidney allocation. Nearly 30% of kidneys recovered from deceased donors in the United States are never transplanted, largely because they are hard to place through the standard waitlist sequence. One can use AI to flag kidneys at risk of delayed or failed placement for out-of-sequence (OOS) allocation, but this comes at the cost of bypassing candidates higher on the waitlist. Thus, OOS allocation should only be used when the organ is demonstrably at high risk of discard on the standard sequence. Using Organ Procurement and Transplantation Network match-run data from 2016-2019 and 2023-2025 (excluding COVID-19-disrupted years), we build the Arbitrator framework, which combines various interpretable models and acts only when the uncertainty in their predictions cannot change the preferred action. Our Arbitrator resolves 93.5% of organ placement decisions using explicit medical criteria learned from data, and defers the remaining 6.5% to a human or a more complex model. We show that the Arbitrator correctly identifies 70.1% of historically discarded low-KDPI kidneys as OOS candidates, projecting that up to 43.9% of historically discarded kidneys could be flagged for expedited placement and potentially transplanted. We then exploit the interpretability of the Arbitrator to create an agentic audit comparing the learned rules to published clinical literature in order to flag decision-making patterns that are contentious.

Figures & tables

Panel of Figure 1 — Figure 1: Our uncertainty-aware, auditable Arbitrator model. Each interpretable model certifies an action only when its confidence interval lies entirely on one side of $\tau_{\beta}$ . Organs with certified, non-conflicting actions are resolved; all others are deferred. An auditor (expert or language model) checks each resolved rule against clinical references and recommends deciding, deviating, or further deferring. XGBoost handles deferred organs.

Figure 2: ML models identify more non-EM kidneys while recommending fewer EM kidneys for OOS allocation. Each point corresponds to one value of $\beta$ and shows low-KDPI non-EM kidneys identified for potential OOS placement (y-axis) against low-KDPI EM kidneys unnecessarily routed OOS (x-axis). Stars mark $\beta=0.5$ .

Figure 3: CART and SparseRF provide complementary certification. Test kidney certification level at $\beta=0.5$ by CART and SparseRF.

Donor features	Rule(s)	CART 99% CI	SparseRF 99% CI	Pathway
KDPI $>90.5\%$ ; GS $>20\%$ ; vascular changes $\geq 10\%$	C164, F153_400	0.032–0.067 (OOS)	0.023–0.057 (OOS)	Dual-certified OOS
KDPI $\leq 41.5\%$ ; Cr $\leq 2.595$ mg/dL; IF $<10\%$ ; GS $\leq 10\%$ ; lipase $>1290.5$ U/L; no kidney biopsy; HCV NAT negative	C19, F25_13, F81_9, F164_278, F206_243	0.918–0.939 (OS)	0.888–0.932 (OS)	Dual-certified OS
$78.5\%<$ KDPI $\leq 90.5\%$ ; GS $\leq 10\%$ ; recovery year through 2019; Cr $\leq 2.295$ mg/dL; continued smoking not recorded; BUN $>16.5$ mg/dL	C130	0.499–0.663 (OOS)	0.000–1.000 (defer)	Single-certified OOS
$27.5\%<$ KDPI $\leq 43.5\%$ ; $2.595<$ Cr $\leq 3.765$ mg/dL; sodium $>142.5$ mEq/L	C49, F25_59	0.557–0.786 (defer)	0.788–0.882 (OS)	Single-certified OS
KDPI $\leq 41.5\%$ ; Cr $\leq 2.595$ mg/dL; IF $\geq 10\%$	C21	0.486–0.710 (defer)	0.000–1.000 (defer)	Deferred

Table 1: Interpretable Arbitrator rules and certification pathways at

\beta=0.5

. A model certifies on-sequence (OS) or out-of-sequence (OOS) placement when its entire 99% confidence interval lies above or below the decision threshold

\tau_{\beta}=2/3

, respectively; otherwise it defers. C denotes a CART leaf and F a SparseRF rule. Additional certification profiles and selected audited rules are presented in the Appendix. Abbreviations: Cr, terminal creatinine; GS, glomerulosclerosis; IF, interstitial fibrosis; BUN, blood urea nitrogen; HCV NAT, hepatitis C nucleic acid test.

Certification Pathway	Accuracy	FPR	FNR	Avg. cost
Any
XGBoost	0.815	0.223	0.165	0.131
Arbitrator	0.802	0.252	0.170	0.143
Dual certified
XGBoost	0.853	0.208	0.116	0.108
Arbitrator	0.849	0.226	0.113	0.114

Table 2: Performance by certification pathway at

\beta=0.5

. FPR and FNR denote false positive and false negative rates, respectively. The positive class corresponds to EM organs. For deferred kidneys, only XGBoost predictions are available. Certification pathway “Any” includes all 21,574 test kidneys; its Arbitrator metrics include XGBoost decisions for deferred kidneys.

Figure 4: Arbitration pathway varies across KDPI. Certification pathways for 21,574 held-out kidneys at $\beta=0.5$ , grouped by KDPI.

Rule	Condensed description	Conf. Interval
C36	$41.5\%<\mathrm{KDPI}\leq 60.5\%$ ; $1.675<\mathrm{Cr}\leq 2.595$ ; IF $<10\%$ ; GS $\leq 10\%$ ; no coronary angiogram; recovery after 2019	$[.377,.645]$
C51	KDPI $\leq 43.5\%$ ; Cr $>3.765$ ; cause of death not recorded as anoxia	$[.158,.534]$

Table 3: Rules audited by the LLM agent. In both rules, the recommendation from the Arbitrator is to place the organs out-of-sequence (OOS); however, the audit proposed keeping them on-sequence (OS), based on [ 24 ; 42 ; 30 ] . KDPI: Kidney Donor Profile Index; Cr: terminal creatinine (mg/dL); IF: interstitial fibrosis; GS: glomerulosclerosis .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Recorded OOS indicators by match-run year. The numerator is the number of distinct donors with code 861, 862, 863 or 887 in either PTR refusal field during that calendar year; the denominator is all distinct donors with kidney PTR match-run records that year, before study exclusions. The asterisk marks partial 2025 data, January 1–June 30.

Stage	Kidneys removed	Kidneys remaining
Kidney observations in linked study inputs	—	159,961
Donor has an observable OOS placement pathway	16,056	143,905
All donor offers removed by eligibility filtering	58	143,847
Disposition outside retained categories or unknown	26	143,821
Eligible prediction cohort	—	143,821

Appendix

Table S1: Construction of the eligible pooled kidney dataset from the linked study inputs. Each row removes kidneys from the preceding row; offer-record removal counts are reported separately in the text.

Panel of Figure 10 — Figure S2 : Empirical offer-level acceptance and discard statistics. (A) Cumulative eligible final acceptance among eventually accepted kidneys by original PTR sequence. (B) Empirical probability of explicit organ discard among kidneys not accepted by sequence $k$ . Both panels use identical KDPI bands: 0–20%, 20–40%, 40–60%, 60–85%, and 85–100% (labeled 85%+). Bands include their lower endpoint and exclude their upper endpoint.

Year or epoch	Kidneys	Accepted (%)	EM (%)
2016	18,420	77.78	70.55
2017	18,988	78.74	71.79
2018	19,588	79.85	72.45
2019	21,874	79.93	72.36
2020*	23,322	78.03	70.65
2021*	25,252	73.06	64.98

Appendix

Table S3: Retained kidney acceptance rates by donor recovery year or epoch. Accepted denotes organ-assigned eligible final acceptance at any original sequence; EM denotes acceptance by original sequence 100. Asterisks identify rows from the excluded 2020–2022 match-run cohort. Cohorts contain minimal, but positive overlap, so their totals must not be pooled.

Figure S3 : Test-set calibration of the five probability models (21,574 kidneys). Kidneys are grouped into ten equal-width bins of predicted early-acceptance probability; each point plots the bin’s mean prediction against its observed early-acceptance rate.

Figure S4 : SparseRF rule-budget comparison for $K\in\{8,16,32,64,128,256\}$ on the same fitted forest and cohort. (A) Average cost and interpretable resolution at $\beta=0.5$ . (B) Selected rules and source trees by budget. (C) Interpretable resolution and (D) cost difference relative to XGBoost across $\beta$ .

		Resolution policy		Average cost
Pathway	Kidneys (%)	Single	Dual	Single	Dual
Dual	17,121 (79.36)	Resolved	Resolved	0.1136	0.1136
CART-only	1,689 (7.83)	Resolved	Deferred	0.2866	0.2238
SparseRF-only	1,371 (6.35)	Resolved	Deferred	0.2666	0.2221
Deferred	1,393 (6.46)	Deferred	Deferred	0.2021	0.2021
Total	21,574 (100.00)			0.1426	0.1348

Appendix

Table S6: Resolution pathways at

\beta=0.5

for the 21,574 test kidneys. Dual resolution requires both models to certify the same action; single-model resolution requires one model to certify while the other abstains.

Certification pathway/action	Donor features	Rule(s)	$n$	CART 99% CI	SparseRF 99% CI
A. Dual-certified OOS	KDPI $>90.5\%$ ; GS $>20\%$ ; vascular changes $\geq 10\%$ .	C164, F153_400	158	0.032–0.067 OOS	0.023–0.057 OOS
B. Dual-certified OS	KDPI $\leq 41.5\%$ ; Cr $\leq 2.595$ mg/dL; IF $<10\%$ ; GS $\leq 10\%$ ; lipase $>1290.5$ U/L; no kidney biopsy; HCV NAT negative.	C19, F25_13, F81_9, F164_278, F206_243	2,077	0.918–0.939 OS	0.888–0.932 OS
C. CART-only OOS	$78.5\%<$ KDPI $\leq 90.5\%$ ; GS $\leq 10\%$ ; recovery year through 2019; Cr $\leq 2.295$ mg/dL; continued smoking not recorded; BUN $>16.5$ mg/dL.	C130	86	0.499–0.663 OOS	0.000–1.000 defer
D. CART-only OS	$41.5\%<$ KDPI $\leq 60.5\%$ ; Cr $\leq 1.675$ mg/dL; IF $<10\%$ ; GS $\leq 10\%$ ; no coronary angiogram; recovery year after 2019.	C35	131	0.682–0.764 OS	0.000–1.000 defer
E. SparseRF-only OOS	$60.5\%<$ KDPI $\leq 78.5\%$ ; GS $>10\%$ ; kidney biopsy not recorded as No; IF $<10\%$ ; Cr $\leq 1.895$ mg/dL; age $\leq 61.5$ years; vascular changes $<10\%$ .	C96, F153_374	45	0.505–0.675 defer	0.311–0.414 OOS
F. SparseRF-only OS	$27.5\%<$ KDPI $\leq 43.5\%$ ; $2.595<$ Cr $\leq 3.765$ mg/dL; sodium $>142.5$ mEq/L.	C49, F25_59	72	0.557–0.786 defer	0.788–0.882 OS

Appendix

Table S7 : Example certification profiles at

\beta=0.5

. OS denotes on-sequence and OOS off-sequence placement. A model certifies OS when its entire 99% confidence interval (CI) is above

\tau_{\beta}=2/3

, certifies OOS when it is below, and abstains otherwise. The number of test kidneys matching each profile is given by

n

. C identifies the CART leaf and F a selected SparseRF rule. Donor features are abbreviated observed-value summaries; missing-value alternatives are omitted. Cr, terminal creatinine; GS, glomerulosclerosis; IF, interstitial fibrosis; BUN, blood urea nitrogen; HCV NAT, hepatitis C nucleic acid test.

Policy	Interpretable (%)	Organs resolved	Low-KDPI, non-EM identified	Low-KDPI, discarded identified
XGBoost-only	0.0	—	3,300 (66.7%)	2,079 (75.9%)
Arbitrator	93.5	20,181	3,068 (62.1%)	1,920 (70.1%)
Arbitrator (dual-certification only)	79.4	17,121	3,188 (64.5%)	1,992 (72.8%)

Appendix

Table S8: Trade-off between interpretable resolution and potentially saved organs at

\beta=0.5

. Arbitrator denotes the single certification policy used in the main paper; the dual-certification-only policy resolves a kidney only when both models certify the same action. Interpretable percentages use all 21,574 test kidneys, and organs resolved counts decisions supported by interpretable rules. Both Arbitrator policies use XGBoost for deferred kidneys. There are 4,944 low-KDPI non-EM and 2,738 low-KDPI historically discarded kidneys in the test data.

Original certification	Organs audited	Retained for automation	Deferred $n$ (%)
Dual	17,121	15,401	1,720 (10.05%)
Single	3,060	2,070	990 (32.35%)
CART only	1,689	1,230	459 (27.18%)
SparseRF only	1,371	840	531 (38.73%)
Overall	20,181	17,471	2,710 (13.43%)

Appendix

Table S10: Organ-level audit outcomes by original certification pathway. Retained organs remain eligible for automation under the hypothetical audit policy. Deferral percentages use the organs audited in each row. Overall combines Dual and Single; the two indented rows partition Single and must not be added to it. The 1,393 organs deferred before audit are excluded.

Rule	Condensed description	CI	Val. $n$	Test $n$	Auditor recommendation
C36	$41.5\%<\mathrm{KDPI}\leq 60.5\%$ ; $1.675<\mathrm{Cr}\leq 2.595$ ; IF $<10\%$ ; GS $\leq 10\%$ ; no coronary angiogram; recovery after 2019	$[.377,.645]$	86	97	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
C51	KDPI $\leq 43.5\%$ ; Cr $>3.765$ ; cause of death not recorded as anoxia	$[.158,.534]$	176	222	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
F25_70	Age $\leq 50.5$ ; KDPI $\leq 43.5\%$ ; GS $\leq 10\%$ ; Cr $>3.765$ ; cause of death not recorded as anoxia; state not recorded as CA; glomerular count $\leq 27.5$	$[.071,.317]$	57	51	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
C58	KDPI $\leq 43.5\%$ ; Cr $>3.765$ ; anoxia; age $>39.5$	$[.399,.666]$	86	102	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
C62	$43.5\%<\mathrm{KDPI}\leq 60.5\%$ ; Cr $>5.025$ ; GS $\leq 15\%$	$[.391,.650]$	92	44	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]
F164_72	Biopsy type recorded; KDPI $\leq 67.5\%$ ; BUN $>34.5$ ; IF $<10\%$ ; GS $\leq 15\%$ ; Na $>142.5$ ; Cr $>3.205$ ; age $>42.5$	$[.306,.513]$	143	167	Deviate: OOS $\rightarrow$ OS Liu et al. [2020] , Wen et al. [2024] , Na and others [2026]

Appendix

Table S11 : Audited candidate and comparison rules. CI is the Wilson interval for historical early acceptance computed from the validation data. Descriptions summarize observed-value conditions joined by “AND.” OS denotes on-sequence, OOS denote off-sequence, and arrows show proposed deviations. Test supports can overlap. Abbreviations: KDPI: historical Kidney Donor Profile Index; Cr: terminal creatinine (mg/dL); IF: interstitial fibrosis; GS: glomerulosclerosis; BUN: blood urea nitrogen (mg/dL); Na: serum sodium (mEq/L); age in years; CA: California.

Responsible AI

Responsible AI studies how to design, evaluate, and govern AI systems to promote fairness, safety, transparency, accountability, privacy, and sustainability.

See all 134 papers

Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis
Jun 1, 2026Itai Zilberstein, Ioannis Anagnostides, Tuomas SandholmLearning to Rank Survival Analysis
Department of Computer Science, Carnegie Mellon University, Pittsburgh, PA · Additional affiliations: Strategy Robot, Inc., Strategic Machine, Inc., Optimized Markets, Inc.
Paired Recipient-based Evaluation of Survival Prediction for Deceased Donor Kidney Transplants
Aug 4, 2026Misaki Matsuura, Mohammadreza Nemati, Dulat Bekbolsynov +2Healthcare Survival Analysis
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH 44106, USA · Department of Medical Microbiology and Immunology University of Toledo Toledo, OH 43614, USA
When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions
Mar 14, 2026Lezhi Tan, Naomi Sagan, Lihua Lei +1AI-Assisted Decision Making Human-AI Collaboration
Stanford University, USA

Jun 1, 2026J/K move · Enter open · S save

Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis

Itai Zilberstein, Ioannis Anagnostides, Tuomas Sandholm

Department of Computer Science, Carnegie Mellon University, Pittsburgh, PA · Additional affiliations: Strategy Robot, Inc., Strategic Machine, Inc., Optimized Markets, Inc.

Learning to Rank Survival Analysis Decision-Focused Learning Clinical Decision-Making

Open paper AskPDF arXiv

Machine learning predictors have become essential tools for guiding automated decision making. However, a major misalignment persists: predictive models are typically optimized in terms of standard statistical metrics in isolation from the algorithmic tasks they inform. We highlight this incongruity in the high-stakes domain of organ allocation by demonstrating that any algorithm relying on (even highly accurate) survival predictors optimized for standard metrics -- such as the Concordance index (C-index) -- can yield arbitrarily poor outcomes when used for allocation, failing to guarantee utility better than a uniform random selection. To bridge the gap between survival analysis and policy optimization, we introduce a decision-focused learning approach based on optimizing normalized discounted cumulative gain (NDCG), a mainstay metric in information retrieval. We establish the utility of NDCG in survival analysis by proving that it translates to guarantees on the performance of allocation. Empirically, we propose a bootstrapping approach to optimize the NDCG of existing survival models. Unlike prior work, we also address the challenge of right censorship when evaluating ranking. On historical heart transplant data from the US, our method dramatically boosts the NDCG of baseline models by 50-100%, which translates to tens of thousands of additional life years gained annually when deployed for transplant allocation. We anticipate that our framework will find broader applications in decision making with predictions.

Abstract

Responsible AI

Explore similar work

Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis

Paired Recipient-based Evaluation of Survival Prediction for Deceased Donor Kidney Transplants

When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions