From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Min Hun Lee

cs.HCMar 19, 2026

From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

Authors: Min Hun Lee

Organizations: Singapore Management University, Singapore

Abstract

Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.

Figures & tables

Figure 1. We propose a four-part taxonomy of metrics for human–AI onboarding (top) and show how each metric family becomes observable and actionable across the human-AI onboarding lifecycle (Understand–Control–Improve) (bottom). A two-level framework linking four metric families for human–AI decision-making (Outcome, Reliance & Interaction, Safety & Harm, and Learning & Readiness) to three stages of the human–AI onboarding lifecycle: Understand, Control, and Improve.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Metric (example)	Data source (trace)	U–C–I stage	Design action (what you do when it’s bad)
Outcome metrics (What happened?)
Team accuracy / TeamGain	Final decision $h_{1}$ , ground truth $y$ ; condition logs	Improve	Adjust delegation policy (when to defer / override); revise workflow to route high-risk cases to human review.
Regret_best / Oracle gap	$h_{0},a,h_{1},y$ per case	Improve	Diagnose collaboration failures (not model limits); target training on cases where either human or AI was correct but the team failed.
Error recovery vs. amplification	$h_{0}\rightarrow h_{1}$ transitions + $y$	Improve	Identify whether UI causes harmful flips; refine prompts/explanations to reduce changed-to-wrong; add guardrails for high-stakes edits.
Reliance & interaction metrics (How was AI used?)
Accept-on-wrong Reject-on-wrong	Agreement with AI ( $h_{1}=a$ ) conditioned on AI correctness	Understand + Control	Curate “failure sets”; add reliability cues; introduce regions-of-no-use; require justification or second-check for high-risk acceptance.

Appendix

Table 1. Mapping metrics to observable data sources, U–C–I onboarding stages, and the corresponding design actions they enable. (Part 1)

Metric (example)	Data source (trace)	U–C–I stage	Design action (recommended response)
Safety & harm metrics (What went wrong?)
AI-harm / AI-help	Help–harm decomposition from $h_{0},h_{1},a,y$	Control + Improve	Add guardrails where AI induces harm; adjust autonomy (e.g., limit deferral) in high-harm regions; prioritize model fixes for harm-heavy slices.
Missed-help / Under-reliance	$h_{0}\neq y$ , $a=y$ , but $h_{1}\neq a$	Understand	Improve “when to trust” instruction; show exemplars of correct AI behavior; add calibrated confidence cues for beneficial reliance.
Near-misses (high-risk disagreements)	High-stakes disagreement logs; risk labels; margin/uncertainty if available	Control	Trigger required second review; add risk-based escalation rules; refine “regions-of-no-use” policy.
Rule–behavior contradiction rate	Policy label $\pi_{j}$ + behavior event ( $E_{j}$ )	Control + Improve	Fix workflow compliance gaps; redesign the UI to make required actions salient; update governance policy or training based on observed violations.
Rollback rate / Escalation rate	Rollback logs $R_{j}$ ; escalation logs $E_{j}$	Control + Improve	Audit contested decisions; improve contestability pathways; adjust accountability and when rollback is encouraged.

Appendix

Table 2. Mapping metrics to observable data sources, U–C–I onboarding stages, and the corresponding design actions they enable. (Part 2)

Explore similar work

Oct 6, 2026

Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making

In human-AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention after AI abstention as a one-off fallback misses the opportunity to improve future AI decisions for greater automation, yet AI adaptively learning from selectively queried human feedback breaks safety guardrails calibrated for old models. We approach this challenge with CARE---calibrated adaptive rectification and escalation---an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. We further show how CARE improves query efficiency when the AI model is well trained and the human-AI misalignment has a clear structure. Experiments on four safety-critical real-world datasets spanning driving, language, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25-81% relative to baselines.

Chenyu Zhang, Rachel Luo, Boyi Li +3

MIT · NVIDIA · NVIDIA, Stanford University

Human-AI Decision Making Human-in-the-Loop AI Human Oversight of AI Model Calibration Selective Prediction

Ask PDF

May 27, 2026

AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: the delegation choice -- deciding when to let AI act autonomously without knowing its output, and the adoption choice -- evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human--AI teams competing in a question-answering game in which humans can choose when and how to work with AI agents to win. Our 24 matches pair 23 expert humans with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. While human--AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (64.5%) when an AI suggestion agrees with humans' initial incorrect answer. To close this gap, we recommend calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust.

Maharshi Gor, Yoo Yeon Sung, Yu Hou +4

University of Maryland College Park, MD, USA · University of California Berkeley, CA, USA · Phasechange.ai Lakewood, CO, USA +1

Human-AI Agreement Trust in AI Human-AI Decision Making Human-AI Collaboration

Ask PDF

Feb 19, 2026

Multi-Round Human-AI Collaboration with User-Specified Requirements

As humans increasingly rely on multiround conversational AI for high stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution free algorithm with finite sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual harm and complementarity violation rates even under nonstationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.

Sima Noorani, Shayan Kiyani, Hamed Hassani +1

University of Pennsylvania

Human-AI Collaboration Human-AI Decision Making Online Learning

Ask PDF