Cost-Aware Post-Hoc Deferral Under Calibration and Shift: An Environmental AI Case Study

Haoran Yu; Lifei Liu; Danping Zhang

cs.SESep 7, 2026

Cost-Aware Post-Hoc Deferral Under Calibration and Shift: An Environmental AI Case Study

Authors: Haoran Yu, Lifei Liu, Danping Zhang

Organizations: University of Florida Gainesville, FL, USA · Wichita State University Wichita, KS, USA · Nanchang Hangkong University Nanchang, China

Abstract

Choosing a deferral policy for a frozen classifier requires more than ranking uncertain cases: confidence may be miscalibrated, errors have unequal costs, reviewers can err, and deployment data can leave calibration support. We study these interactions through EcoTrust, a post-hoc framework that compares automatic action with review using a six-group error-risk estimator, class-asymmetric costs, reviewer accuracy, and an optional support gate. On a Columbia River thermal-stress testbed, the learned estimator improves error-ranking area under the receiver operating characteristic curve from 0.869 to 0.889, but Chow's confidence rule has lower in-distribution cost (0.416 versus 0.567 per day). Across 12 off-the-shelf backends, learned risk and a calibration-matched, class-aware confidence estimator each beat raw Chow on six; a paired year-block bootstrap does not resolve their mean cost difference. In transfer to ten river stations, the gate flags every case and becomes an always-review fallback, attaining the lowest cost on eight stations only when review is perfect and unconstrained. These results characterize decision boundaries on one controlled task: richer risk signals do not reliably improve on calibrated confidence, and detected extrapolation does not imply transferable case-level ranking.

Figures & tables

Fig. 1: EcoTrust studies per-case deferral for a frozen predictor under asymmetric costs. Six uncertainty signals estimate model error; the decision layer combines adjusted risk with error costs, review cost, and reviewer accuracy. Outside calibration support, a pre-specified risk floor invokes a conservative fallback. An optional budget ranks cases by expected saving.

Framework	Backend	Costs	Reviewer	Capacity	Out-of-support
					Fallback
Chow [ 2 ]	Fixed	Reject	N/A	No	No
Post-hoc L2D [ 7 ]	Fixed	Expert	Cost model	No	No
DeCCaF [ 11 ]	AI + experts	Class/inst.	Expertise model	Yes	No
EcoTrust	Fixed	FN/FP/review	Accuracy input	Optional	Yes

TABLE I: Deployment settings of selected deferral frameworks.

Fig. 2: Chow provides the best in-distribution cost–review trade-off, while high error coverage often requires broad review. (a) Deployment cost versus escalation rate; lower-left is preferable. (b) Error coverage versus escalation rate; upper-left is preferable and the dotted diagonal is random ranking. Marker families separate automatic policies, random/earlier heuristics, confidence rules, L2D, and EcoTrust-DL; codes A0–B7 are defined in the baseline paragraph and C1–C3 in the meta-risk-estimator paragraph (Sec. V ), and B6–B7 are three-seed means. All policies use the same frozen predictor (

c_{FN}{:}c_{FP}{:}c_{h}{=}100{:}3{:}1

, test

n{=}1{,}208

Fig. 3: Calibration partly predicts learned-risk value, while no signal is individually load-bearing. (a) Backend expected calibration error (ECE) versus cost gain (Chow cost minus EcoTrust-DL cost); the dashed fit excludes the over-confident control. Error bars show across-seed s.d. (b)–(c) Removing one signal changes test cost by at most 0.011 and risk AUC by at most 0.015. Positive

\Delta

Cost means the removed signal helped the decision; negative

\Delta

AUC means it helped discrimination.

Comparison or statistic	Result
Full bank cheaper than raw Chow	6/12
Class-aware cheaper than raw Chow	6/12
Full bank cheaper than class-aware	7/12
Mean $\Delta$ (year-block 95% CI)	$+0.0082$ [ $-0.0002$ , $0.0172$ ]
Across-backend $\Delta$ range	[ $-0.017$ , $+0.109$ ]
Pearson(ECE, $\Delta$ )	0.011

TABLE II: Calibration-matched comparison over 12 backends. Costs are averaged over three seeds;

\Delta

is full-bank minus class-aware cost per day.

Fig. 4: Deployment costs depend on error severity, review budget, and reviewer quality. (a) EcoTrust becomes cheaper than automation between

c_{FN}=5

and

10

, although Chow remains cheaper in-distribution. (b) No ranking dominates at exact review rates. (c)–(d) Reviewer-aware decisions reduce cost under noise; the predicted breakeven

a^{*}=93.05\%

is near the accuracy crossing. Accuracy bars show Monte Carlo s.d.

Fig. 5: Under station transfer, the support gate is identical to an always-review policy and costs one per day with a perfect reviewer. It is cheapest on eight stations; L2D-OvA is cheapest on Clearwater and Snake. No target labels or tuning are used. Blue cells cost less than one review per day, red cells cost more, and outlines mark row minima.

Learning to Defer

Learning to defer trains AI systems to decide when to predict autonomously and when to route cases to human or machine experts.

See all 26 papers

Density-Ratio Losses for Post-Hoc Learning to Defer
May 19, 2026Alexander Soen, Ragnar Thobaben, Joakim Jaldén +1Learning to Defer Density Ratio Estimation
KTH⋄ · Google Research†
Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics
May 3, 2026Fernando Martin-Maroto, Nabil Abderrahaman, Gonzalo G. de PolaviejaProbability Calibration Uncertainty Calibration
Mathematics of Behavior and Intelligence Lab. Champalimaud Research. Avenida de Brasília, Lisbon · Algebraic AI Calle Castelló, 117, Madrid
Cost-Aware Abstention for LLM-Based IT Ticket Classification
Oct 22, 2026Sachin GuptaSelective Prediction Text Classification
Independent Researcher United States

May 19, 2026J/K move · Enter open · S save

Density-Ratio Losses for Post-Hoc Learning to Defer

Alexander Soen, Ragnar Thobaben, Joakim Jaldén, Richard Nock

KTH⋄ · Google Research†

Learning to Defer Density Ratio Estimation

Open paper AskPDF arXiv

We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergence-regularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model's and an expert's ideals. Using the reduction from density-ratio estimation to class-probability estimation, we derive the DR CPE losses for post-hoc L2D scorers. Deferral decisions are then made by thresholding the scorer, allowing deferral rates to be adjusted without retraining. For KL-based ideal distributions, our deferral rules recovers Chow's rule under the original distribution and a connection to an expert-tilted Bayes posterior -- which incorporates the expert's performance -- depending on if the ideal distributions are joint or marginal distributions. Experimentally, our approach is competitive compared to common baselines and more robust across dataset settings. More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection.

Abstract

Learning to Defer

Explore similar work

Density-Ratio Losses for Post-Hoc Learning to Defer

Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics

Cost-Aware Abstention for LLM-Based IT Ticket Classification