2-Step Agent: How a Bayesian Decision Maker Learns from AI-Decision Support
Authors: Otto Nyberg, Davide Tugnoli, Fausto Carcassi, Giovanni Cinà
Organizations: Institute for Logic, Language and Computation, University of Amsterdam, the Netherlands · Department of Mathematics and Earth Sciences, University of Trieste, Italy
Prognostic predictions from ML models support human decision making in several fields, including high-stakes ones such as healthcare and the judiciary. Yet, we still lack a clear understanding of how decision makers learn from ML-based decision support (ML-DS). In this paper, we introduce a general computational framework, the 2-Step Agent, to capture how a prediction from an ML model can also be used as a source of useful information about the training data and the causal mechanism that generated it. Our framework models (i) how a prediction from an ML model affects the beliefs of a rational Bayesian agent, and (ii) how this change in beliefs affects the estimation of the causal effect, the downstream decision for a new observation, and the subsequent outcome. In addition to the framework itself, we make three contributions. First, for the multivariate linear-Gaussian setting, we derive a tractable solution for the Bayesian inference problem introduced, i.e. one in which the agent infers information about a population from an ML prediction. Second, we experimentally identify conditions under which ML-DS is beneficial compared to unassisted decision making. Third, we show that a single misaligned prior belief of the user can be sufficient for ML-DS to lead to worse downstream outcomes compared to no decision support even when the prognostic model is well-specified and the agent is perfectly rational. Hence, even under ideal conditions, ML-DS can do more harm than good.
Figures & tables
Figure 1: The flow of the framework for a clinical use case, with and without ML-DS. A new patient arrives for treatment, and the patient’s data (1), potentially with the output of ML-DS, is observed by the agent, e.g. a doctor. Next, the agent estimates the effect of treatment and makes a treatment decision (2). Without ML-DS, this estimation is influenced by the agent’s priors and patient data. With ML-DS, the agent uses these plus the ML prediction to update their beliefs about the population in a Bayesian fashion. The treatment and the patient’s attributes determine the outcome (3).
Figure 2: The graphical models of the framework with stochastic variables depicted as circles and deterministic variables as rectangles. Observed variables are shaded, unobserved ones are unshaded. Model (a) represents the true data-generating process, model (b) represents the agent’s beliefs of how the ML-DS model was trained, and model (c) allows the agent to infer the treatment effect of new data points given the beliefs.
Figure 3: The effects of misaligned agent priors with ML-DS (in orange) and without (in blue) on the agent’s beliefs (top row) and on the consequent downstream outcomes Y (bottom row). In the plots for αAμ and αYμ , the horizontal axes represent the deviation of the agent’s prior belief from the true parameter in units of standard deviation, and the centerpoints in all plots correspond to aligned priors. The orange and blue bands show the mean ±1 SD of the agent’s CATE distributions pooled across the 50 runs per prior setting, with and without ML-DS, respectively. Gray bands mark settings where the true CATE lies within the orange band. Boxplots summarize the 50 expected outcomes under the selected treatments and the true generating process Hist , showing the median and interquartile range (IQR), with whiskers extending to the most extreme values within 1.5 IQR of the box. The leftmost panel shows that misaligned beliefs can lead to ML-DS being harmful, while in the central panel ML-DS corrects misaligned beliefs and ends up being beneficial.
Figure 4: The effects of a misaligned agent prior αXμ about the distribution of covariate X , shown when the ML-DS model is misspecified (left) and well-specified (right), with bands and boxplots defined as in Fig. 3 . The misspecified model introduces a new option for incorrect beliefs to result in harmful ML-DS.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Plot showing joint posterior samples of variables NE and αAμ in the posterior beliefs of the agent after update with ML-DS information. The negative correlation reflects that the agent entertains multiple possible explanations for the surprising prediction: either the treatment is less effective than expected, the historical dosages were higher than expected, or a scenario in between.
Figure 6: The effects of misaligned agent priors with ML-DS (in orange) and without (in blue) on the agent’s beliefs (top row) and on the consequent downstream outcomes Y (bottom row). In contrast to Fig. 3 , the ML-DS model here is misspecified so that the regression model lacks intercept (thus it diverges from the true generating process). In the plots for αAμ and αYμ , the horizontal axes represent the deviation of the agent’s prior belief from the true parameter in units of standard deviation, and the centerpoints in all plots correspond to aligned priors. The orange and blue bands show the mean ±1 SD of the agent’s CATE distributions pooled across the 50 runs per prior setting, with and without ML-DS, respectively. Gray bands mark settings where the true CATE lies within the orange band. Boxplots summarize the 50 expected outcomes under the selected treatments and the true generating process Hist , showing the median and interquartile range (IQR), with whiskers extending to the most extreme values within 1.5 IQR of the box. Similarly to Fig. 3 , the leftmost plots show how a misaligned belief αAμ can lead to the agent being more certain about flawed values of CATE with ML-DS, but this only leads to harmful treatment decisions when the belief drops below the treatment threshold. Similarly, the plots in the middle show how ML-DS can correct misaligned beliefs about the treatment effect over a wide range of priors which also results in improved treatment outcomes in some settings (where the blue line is below treatment threshold). The plot for misalignment of αXμ can be seen in Fig. 4 .
Figure 7: Number of covariates k vs. prior and posterior beliefs about CATE for different misalignments of priors. For NE , deviations of the prior mean are expressed in units of 0.05 . The experiment was constructed so that the prior expectation about CATE was correct despite the misaligned prior for αXμ , αYμ , and αAμ . For NE this could not be done due to the structure of AInf . The findings from the main experiment are repeated in this setting and all of the results are essentially unaffected by k . Misalignment of αAμ and αYμ can lead to the agent having less correct posterior beliefs about CATE, and misaligned beliefs about NE are largely corrected. Note that the lines that appear to be missing are simply masked by the thick black line.
Figure 8: Prior (blue) and posterior (orange) beliefs about CATE vs. the number of covariates k . Lines show the mean and bands ±1 SD of the agent’s CATE distribution, pooled across the 50 runs per setting. For NE , deviations of the prior mean are expressed in units of 0.05 . Both the posterior expectation of CATE and its uncertainty seem largely unrelated to k . The same effect with NE being corrected over a range of combinations seen in the main experiment is repeated here.
k
Full plate ( n=1000 )
Reduced plate (Lemma F.1 )
Collapsed (Thm F.2 )
1
3000
4
2
2
4000
8
2
5
7000
26
2
10
12000
76
2
Appendix
Table 1: Number of scalar sampling events needed to represent the training-data plate, as a function of the number of covariates k , for the three plate representations.
Traditionally, decision support studies how humans use machine learning models to make better decisions. In modern agentic systems, this division of roles is increasingly reversed: AI agents act on behalf of users, while humans and tools becomes support mechanisms around them. This role reversal brings reliability concerns to the forefront, since agentic errors can be consequential and agent behavior must remain aligned with human goals and constraints. Departing from the classical view of decision support, we revisit its two basic principles, the cost--value tradeoff of seeking support and the role of uncertainty quantification, in a setting where AI agents are the central actors. We propose a framework for strategic decision support for AI agents through an optimization problem that minimizes support usage subject to controlling a counterfactual missed-support error: the probability that the agent acts alone on instances where support would have materially improved its output. At the population level, we show that the optimal policy is a threshold rule on the value of support. Building on this structure, we develop an online algorithm that adaptively thresholds such a score and uses randomized exploration to control missed-support error without distributional assumptions. We further introduce a calibration-on-the-fly method that reduces unnecessary support calls online. We instantiate this framework across diverse scenarios, including information gathering, human--AI collaboration, and tool use, showing how each can be modeled through the same strategic decision-support lens. Experiments across these settings show that our method reliably controls the target error while substantially reducing support usage in practice.
Predictive machine learning (ML) models are increasingly used to aid human decision-makers across various high-risk domains such as healthcare and criminal justice. There is a growing recognition of the need to evaluate the causal impact of deploying these systems on downstream outcomes, such as patient survival or crime recidivism. Randomized control trials (RCTs) can provide high-quality evidence on the impact of a deployed model, but they run into a challenge: it is often infeasible to run repeated trials when models are updated or retrained to improve predictive performance. In this work, we present a partial-identification approach to using prior RCT data to construct bounds on the causal effect of a new model. The core innovation in our approach is to leverage assumptions relating fine-grained predictive accuracy to downstream outcomes. We do so via two monotonicity assumptions: first, on individual-level `counterfactual correctness' (all else being equal, a correct prediction leads to non-inferior outcomes); and second, on the relation between subgroup predictive performance and outcomes, interpretable as an assumption regarding trust in model outputs. We demonstrate our method with a simulation study, illustrating how incorporating this information can lead to more informative bounds compared to prior work.
Jonathan Zhang, Erik Skalnes, Jacob Chen +1
Computer Science Dept., Johns Hopkins University, Baltimore, Maryland, USA
It is widely agreed that when AI models assist decision-makers in high-stakes domains by predicting an outcome of interest, they should communicate the confidence of their predictions. However, empirical evidence suggests that decision-makers often struggle to determine when to trust a prediction based solely on this communicated confidence. In this context, recent theoretical and empirical work suggests a positive correlation between the utility of AI-assisted decision-making and the degree of alignment between the AI confidence and the decision-makers' confidence in their own predictions. Crucially, these findings do not yet elucidate the extent to which this alignment influences the complexity of learning to make optimal decisions through repeated interactions. In this paper, we address this question in the canonical case of binary predictions and binary decisions. We first show that this problem is equivalent to a two-armed online contextual learning problem with full feedback, and establish a lower bound of Ω(∣H∣⋅∣B∣⋅T) on the expected regret any learner can attain, where H and B denote the sets of human and AI confidence values. We then demonstrate that, under perfect alignment between AI and human confidence, a learner can attain an expected regret of O(∣H∣⋅TlogT) and, when ∣H∣=O(logT) and B is countable, a non-trivial generalization of the Dvoretzky-Kiefer-Wolfowitz inequality improves the regret bound to O(TlogT). Taken together, these results reveal that alignment can reduce the complexity of learning to make decisions with AI assistance. Experiments on real data from two different human-subject studies where participants solve simple decision-making tasks assisted by AI models show that our theoretical results are robust to violations of perfect alignment.
Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-Rodriguez
§Max Planck Institute for Biochemistry, Martinsried, Germany · †Max Planck Institute for Software Systems, Kaiserslautern, Germany