Choosing a deferral policy for a frozen classifier requires more than ranking uncertain cases: confidence may be miscalibrated, errors have unequal costs, reviewers can err, and deployment data can leave calibration support. We study these interactions through EcoTrust, a post-hoc framework that compares automatic action with review using a six-group error-risk estimator, class-asymmetric costs, reviewer accuracy, and an optional support gate. On a Columbia River thermal-stress testbed, the learned estimator improves error-ranking area under the receiver operating characteristic curve from 0.869 to 0.889, but Chow's confidence rule has lower in-distribution cost (0.416 versus 0.567 per day). Across 12 off-the-shelf backends, learned risk and a calibration-matched, class-aware confidence estimator each beat raw Chow on six; a paired year-block bootstrap does not resolve their mean cost difference. In transfer to ten river stations, the gate flags every case and becomes an always-review fallback, attaining the lowest cost on eight stations only when review is perfect and unconstrained. These results characterize decision boundaries on one controlled task: richer risk signals do not reliably improve on calibrated confidence, and detected extrapolation does not imply transferable case-level ranking.
Figures & tables
Fig. 1: EcoTrust studies per-case deferral for a frozen predictor under asymmetric costs. Six uncertainty signals estimate model error; the decision layer combines adjusted risk with error costs, review cost, and reviewer accuracy. Outside calibration support, a pre-specified risk floor invokes a conservative fallback. An optional budget ranks cases by expected saving.
Framework
Backend
Costs
Reviewer
Capacity
Out-of-support
Fallback
Chow [ 2 ]
Fixed
Reject
N/A
No
No
Post-hoc L2D [ 7 ]
Fixed
Expert
Cost model
No
No
DeCCaF [ 11 ]
AI + experts
Class/inst.
Expertise model
Yes
No
EcoTrust
Fixed
FN/FP/review
Accuracy input
Optional
Yes
TABLE I: Deployment settings of selected deferral frameworks.
Fig. 2: Chow provides the best in-distribution cost–review trade-off, while high error coverage often requires broad review. (a) Deployment cost versus escalation rate; lower-left is preferable. (b) Error coverage versus escalation rate; upper-left is preferable and the dotted diagonal is random ranking. Marker families separate automatic policies, random/earlier heuristics, confidence rules, L2D, and EcoTrust-DL; codes A0–B7 are defined in the baseline paragraph and C1–C3 in the meta-risk-estimator paragraph (Sec. V ), and B6–B7 are three-seed means. All policies use the same frozen predictor ( cFN:cFP:ch=100:3:1 , test n=1,208 ).
Fig. 3: Calibration partly predicts learned-risk value, while no signal is individually load-bearing. (a) Backend expected calibration error (ECE) versus cost gain (Chow cost minus EcoTrust-DL cost); the dashed fit excludes the over-confident control. Error bars show across-seed s.d. (b)–(c) Removing one signal changes test cost by at most 0.011 and risk AUC by at most 0.015. Positive Δ Cost means the removed signal helped the decision; negative Δ AUC means it helped discrimination.
Comparison or statistic
Result
Full bank cheaper than raw Chow
6/12
Class-aware cheaper than raw Chow
6/12
Full bank cheaper than class-aware
7/12
Mean Δ (year-block 95% CI)
+0.0082 [ −0.0002 , 0.0172 ]
Across-backend Δ range
[ −0.017 , +0.109 ]
Pearson(ECE, Δ )
0.011
TABLE II: Calibration-matched comparison over 12 backends. Costs are averaged over three seeds; Δ is full-bank minus class-aware cost per day.
Fig. 4: Deployment costs depend on error severity, review budget, and reviewer quality. (a) EcoTrust becomes cheaper than automation between cFN=5 and 10 , although Chow remains cheaper in-distribution. (b) No ranking dominates at exact review rates. (c)–(d) Reviewer-aware decisions reduce cost under noise; the predicted breakeven a∗=93.05% is near the accuracy crossing. Accuracy bars show Monte Carlo s.d.
Fig. 5: Under station transfer, the support gate is identical to an always-review policy and costs one per day with a perfect reviewer. It is cheapest on eight stations; L2D-OvA is cheapest on Clearwater and Snake. No target labels or tuning are used. Blue cells cost less than one review per day, red cells cost more, and outlines mark row minima.
We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergence-regularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model's and an expert's ideals. Using the reduction from density-ratio estimation to class-probability estimation, we derive the DR CPE losses for post-hoc L2D scorers. Deferral decisions are then made by thresholding the scorer, allowing deferral rates to be adjusted without retraining. For KL-based ideal distributions, our deferral rules recovers Chow's rule under the original distribution and a connection to an expert-tilted Bayes posterior -- which incorporates the expert's performance -- depending on if the ideal distributions are joint or marginal distributions. Experimentally, our approach is competitive compared to common baselines and more robust across dataset settings. More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection.
Confidence calibration has been dominated by the Expected Calibration Error (ECE), a linear metric that counts calibration offset equally regardless of the confidence level at which it occurs. We show that ECE can remain small even under arbitrarily large overconfidence risk, so we propose Calibrated Size Ratio (CSR) instead, an interpretable metric that equals 1 under perfect calibration, from which we derive the risk probability Prisk that quantifies the statistical evidence for overconfidence. We further argue that overconfidence risk assessment must be complemented by a measure of discriminative value: whether the assigned confidences actively distinguish correct from incorrect predictions. We show that confidence-weighted accuracy cwA is the natural such complement, and that confidence-weighting extends to all standard classification metrics. In particular, we prove that the confidence-weighted AUC (cwAUC) captures the information about calibration while the classical AUC cannot. We validate the proposed indicators on several synthetic confidence distributions under multiple controlled calibration profiles and find that CSR separates risky from non-risky assignments. We also test the metrics on fifteen real datasets, with and without post-hoc calibration, and find that standard methods can yield risky confidence profiles.
Fernando Martin-Maroto, Nabil Abderrahaman, Gonzalo G. de Polavieja
Mathematics of Behavior and Intelligence Lab. Champalimaud Research. Avenida de Brasília, Lisbon · Algebraic AI Calle Castelló, 117, Madrid
Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.