As humans increasingly rely on multiround conversational AI for high stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution free algorithm with finite sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual harm and complementarity violation rates even under nonstationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.
Figures & tables
Figure 1: Empirical convergence of error rates in the LLM-simulated medical task for complementarity (left) and counterfactual harm (right). Both plots track the cumulative average running error, AvgErrort=t1∑i=1t1{Errori} , over sequential trials.
Figure 2: Human decision outcomes across varying AI error targets in the LLM-simulated medical task. (Left) Human GT gain rate as a function of the complementarity error target ( εCOMP ). (Right) Rate of abandoning a correct initial guess (GT loss) as a function of the counterfactual harm error target ( εCH ).
εCH
εCOMP
Alg A
0.05
0.50
Alg B
0.30
0.50
Alg C
0.05
0.70
Table 1: Algorithm instantiations. Target rates for counterfactual harm (CH) and complementarity (COMP).
Figure 3: Error convergence for Algorithm B in the crowdsourcing study. Both plots track the cumulative average running error, AvgErrort=t1∑i=1t1{Errori} , where observed rates for counterfactual harm (left) and complementarity (right) stabilize at their nominal values.
Figure 4: Behavioral impact of AI errors in the crowdsourcing task across Algorithms A, B, and C. Top row: human GT loss rates compared across trials with and without CH errors. Bottom row: human GT gain rates compared across trials with and without COMP errors.
Figure 5: Impact of nominal error rates on human outcomes: (a) Comparison of human’s GT loss rates between Algorithm A ( εCH=0.05 ) and Algorithm B ( εCH=0.30 ). (b) Comparison of human’s GT gain rates between Algorithm A ( εCOMP=0.50 ) and Algorithm C ( εCOMP=0.70 ).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Task instructions presented to participants at the start of the crowdsourcing study. The instructions describe the Collaborative Shape Counting Task, explain the two-round interaction structure, and clarify how to interpret and respond to the AI assistant’s predictions.
Figure 7: Sample images from the shape counting dataset, organized by bucket (rows) and difficulty (columns). Bucket 1 contains images with low shape counts (3–15), Bucket 2 contains medium counts (16–30), and Bucket 3 contains high counts (31–50). Difficulty increases from left to right, with “Hard” images featuring greater shape density and visual overlap.
Figure 8: Sample visualization of noise-degraded images versions (bottom row) used as input for the AI assistant. The extreme salt-and-pepper noise obscures fine details, ensuring the AI’s predictions are uncertain and creating a meaningful collaboration dynamic where both human and AI contributions are valuable.
Figure 9: Intermediate interaction state of the user interface. The image is no longer visible, and the participant must submit a guess (a contiguous range of three integers) after observing the AI assistant’s prediction set. The interface shows the target shape type, current round, and the AI’s suggested range.
Figure 10: Trial completion state showing the full interaction history. The interface reveals the original image, displays both rounds of human-AI interaction (including all guesses and AI prediction sets), and indicates whether the participant’s final answer contained the ground truth. This feedback screen appears after each of the 20 trials.
Figure 11: Empirical convergence for Algorithms A and C in the crowdsourcing task. All plots track the cumulative average running error, AvgErrort=t1∑i=1t1{Errori} , for counterfactual harm (left) and complementarity (right) as they stabilize at their respective target values.
Figure 12: Empirical convergence in the LLM-simulated medical task across varying configurations. Plots demonstrate the algorithm’s ability to track AvgErrort=t1∑i=1t1{Errori} for both CH and COMP metrics under different target regimes.