From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Min Hun Lee

cs.HCMar 19, 2026

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Authors: Min Hun Lee

Organizations: Singapore Management University, Singapore

Abstract

Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.

Figures & tables

Figure 1. We propose a four-part taxonomy of metrics for human–AI onboarding (top) and show how each metric family becomes observable and actionable across the human-AI onboarding lifecycle (Understand–Control–Improve) (bottom). A two-level framework linking four metric families for human–AI decision-making (Outcome, Reliance & Interaction, Safety & Harm, and Learning & Readiness) to three stages of the human–AI onboarding lifecycle: Understand, Control, and Improve.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Metric (example)	Data source (trace)	U–C–I stage	Design action (what you do when it’s bad)
Outcome metrics (What happened?)
Team accuracy / TeamGain	Final decision $h_{1}$ , ground truth $y$ ; condition logs	Improve	Adjust delegation policy (when to defer / override); revise workflow to route high-risk cases to human review.
Regret_best / Oracle gap	$h_{0},a,h_{1},y$ per case	Improve	Diagnose collaboration failures (not model limits); target training on cases where either human or AI was correct but the team failed.
Error recovery vs. amplification	$h_{0}\rightarrow h_{1}$ transitions + $y$	Improve	Identify whether UI causes harmful flips; refine prompts/explanations to reduce changed-to-wrong; add guardrails for high-stakes edits.
Reliance & interaction metrics (How was AI used?)
Accept-on-wrong Reject-on-wrong	Agreement with AI ( $h_{1}=a$ ) conditioned on AI correctness	Understand + Control	Curate “failure sets”; add reliability cues; introduce regions-of-no-use; require justification or second-check for high-risk acceptance.

Appendix

Table 1. Mapping metrics to observable data sources, U–C–I onboarding stages, and the corresponding design actions they enable. (Part 1)

Metric (example)	Data source (trace)	U–C–I stage	Design action (recommended response)
Safety & harm metrics (What went wrong?)
AI-harm / AI-help	Help–harm decomposition from $h_{0},h_{1},a,y$	Control + Improve	Add guardrails where AI induces harm; adjust autonomy (e.g., limit deferral) in high-harm regions; prioritize model fixes for harm-heavy slices.
Missed-help / Under-reliance	$h_{0}\neq y$ , $a=y$ , but $h_{1}\neq a$	Understand	Improve “when to trust” instruction; show exemplars of correct AI behavior; add calibrated confidence cues for beneficial reliance.
Near-misses (high-risk disagreements)	High-stakes disagreement logs; risk labels; margin/uncertainty if available	Control	Trigger required second review; add risk-based escalation rules; refine “regions-of-no-use” policy.
Rule–behavior contradiction rate	Policy label $\pi_{j}$ + behavior event ( $E_{j}$ )	Control + Improve	Fix workflow compliance gaps; redesign the UI to make required actions salient; update governance policy or training based on observed violations.
Rollback rate / Escalation rate	Rollback logs $R_{j}$ ; escalation logs $E_{j}$	Control + Improve	Audit contested decisions; improve contestability pathways; adjust accountability and when rollback is encouraged.

Appendix

Table 2. Mapping metrics to observable data sources, U–C–I onboarding stages, and the corresponding design actions they enable. (Part 2)

Explore similar work

Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making
Oct 6, 2026Chenyu Zhang, Rachel Luo, Boyi Li +3Human-AI Decision Making Human-in-the-Loop AI
MIT · NVIDIA · NVIDIA, Stanford University
AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
May 27, 2026Maharshi Gor, Yoo Yeon Sung, Yu Hou +4Human-AI Agreement Trust in AI
University of Maryland College Park, MD, USA · University of California Berkeley, CA, USA · Phasechange.ai Lakewood, CO, USA +1
Multi-Round Human-AI Collaboration with User-Specified Requirements
Feb 19, 2026Sima Noorani, Shayan Kiyani, Hamed Hassani +1Human-AI Collaboration Human-AI Decision Making
University of Pennsylvania

Oct 6, 2026J/K move · Enter open · S save

Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making

Chenyu Zhang, Rachel Luo, Boyi Li, Anjali Parashar+2

MIT · NVIDIA · NVIDIA, Stanford University

Human-AI Decision Making Human-in-the-Loop AI Human Oversight of AI Model Calibration Selective Prediction

Open paper AskPDF arXiv

In human-AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention after AI abstention as a one-off fallback misses the opportunity to improve future AI decisions for greater automation, yet AI adaptively learning from selectively queried human feedback breaks safety guardrails calibrated for old models. We approach this challenge with CARE---calibrated adaptive rectification and escalation---an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. We further show how CARE improves query efficiency when the AI model is well trained and the human-AI misalignment has a clear structure. Experiments on four safety-critical real-world datasets spanning driving, language, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25-81% relative to baselines.

Abstract

Explore similar work

Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

Multi-Round Human-AI Collaboration with User-Specified Requirements