As humans increasingly rely on multiround conversational AI for high stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution free algorithm with finite sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual harm and complementarity violation rates even under nonstationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.
Figures & tables
Figure 1: Empirical convergence of error rates in the LLM-simulated medical task for complementarity (left) and counterfactual harm (right). Both plots track the cumulative average running error, AvgErrort=t1∑i=1t1{Errori} , over sequential trials.
Figure 2: Human decision outcomes across varying AI error targets in the LLM-simulated medical task. (Left) Human GT gain rate as a function of the complementarity error target ( εCOMP ). (Right) Rate of abandoning a correct initial guess (GT loss) as a function of the counterfactual harm error target ( εCH ).
εCH
εCOMP
Alg A
0.05
0.50
Alg B
0.30
0.50
Alg C
0.05
0.70
Table 1: Algorithm instantiations. Target rates for counterfactual harm (CH) and complementarity (COMP).
Figure 3: Error convergence for Algorithm B in the crowdsourcing study. Both plots track the cumulative average running error, AvgErrort=t1∑i=1t1{Errori} , where observed rates for counterfactual harm (left) and complementarity (right) stabilize at their nominal values.
Figure 4: Behavioral impact of AI errors in the crowdsourcing task across Algorithms A, B, and C. Top row: human GT loss rates compared across trials with and without CH errors. Bottom row: human GT gain rates compared across trials with and without COMP errors.
Figure 5: Impact of nominal error rates on human outcomes: (a) Comparison of human’s GT loss rates between Algorithm A ( εCH=0.05 ) and Algorithm B ( εCH=0.30 ). (b) Comparison of human’s GT gain rates between Algorithm A ( εCOMP=0.50 ) and Algorithm C ( εCOMP=0.70 ).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Task instructions presented to participants at the start of the crowdsourcing study. The instructions describe the Collaborative Shape Counting Task, explain the two-round interaction structure, and clarify how to interpret and respond to the AI assistant’s predictions.
Figure 7: Sample images from the shape counting dataset, organized by bucket (rows) and difficulty (columns). Bucket 1 contains images with low shape counts (3–15), Bucket 2 contains medium counts (16–30), and Bucket 3 contains high counts (31–50). Difficulty increases from left to right, with “Hard” images featuring greater shape density and visual overlap.
Figure 8: Sample visualization of noise-degraded images versions (bottom row) used as input for the AI assistant. The extreme salt-and-pepper noise obscures fine details, ensuring the AI’s predictions are uncertain and creating a meaningful collaboration dynamic where both human and AI contributions are valuable.
Figure 9: Intermediate interaction state of the user interface. The image is no longer visible, and the participant must submit a guess (a contiguous range of three integers) after observing the AI assistant’s prediction set. The interface shows the target shape type, current round, and the AI’s suggested range.
Figure 10: Trial completion state showing the full interaction history. The interface reveals the original image, displays both rounds of human-AI interaction (including all guesses and AI prediction sets), and indicates whether the participant’s final answer contained the ground truth. This feedback screen appears after each of the 20 trials.
Figure 11: Empirical convergence for Algorithms A and C in the crowdsourcing task. All plots track the cumulative average running error, AvgErrort=t1∑i=1t1{Errori} , for counterfactual harm (left) and complementarity (right) as they stabilize at their respective target values.
Figure 12: Empirical convergence in the LLM-simulated medical task across varying configurations. Plots demonstrate the algorithm’s ability to track AvgErrort=t1∑i=1t1{Errori} for both CH and COMP metrics under different target regimes.
The use of Large Language Models (LLMs) across diverse areas of human activity-ranging from everyday tasks to safety-critical applications-aims to enhance decision-making effectiveness with minimal human feedback. Concurrently, it seeks to align decisions with human expectations, preferences, and needs while mitigating risks associated with AI non-determinism. However, humans frequently over- or under-rely on AI recommendations, and current AI systems remain poorly calibrated to human expectations. To address these challenges, we introduce a human-AI collaborative decision-making framework designed to augment human capabilities and align AI agents with human preferences and expectations. Specifically, this paper (a) formulates the collaborative decision-making task as a stochastic game between an AI agent and a human player, and (b) proposes the Human-Centric Reflective Architecture (HCRA), which integrates human-calibrated models with reinforcement learning agents that leverage linguistic feedback in an iterative, reflective process. Evaluation results demonstrate that HCRA enhances decision-making effectiveness and delivers high-quality recommendations.
Andreas Kouridakis, Dimitrios Patiniotis Spyropoulos, George Vouros
In human-AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention after AI abstention as a one-off fallback misses the opportunity to improve future AI decisions for greater automation, yet AI adaptively learning from selectively queried human feedback breaks safety guardrails calibrated for old models. We approach this challenge with CARE---calibrated adaptive rectification and escalation---an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. We further show how CARE improves query efficiency when the AI model is well trained and the human-AI misalignment has a clear structure. Experiments on four safety-critical real-world datasets spanning driving, language, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25-81% relative to baselines.
Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.