2-Step Agent: How a Bayesian Decision Maker Learns from AI-Decision Support
Authors: Otto Nyberg, Davide Tugnoli, Fausto Carcassi, Giovanni Cinà
Organizations: Institute for Logic, Language and Computation, University of Amsterdam, the Netherlands · Department of Mathematics and Earth Sciences, University of Trieste, Italy
Prognostic predictions from ML models support human decision making in several fields, including high-stakes ones such as healthcare and the judiciary. Yet, we still lack a clear understanding of how decision makers learn from ML-based decision support (ML-DS). In this paper, we introduce a general computational framework, the 2-Step Agent, to capture how a prediction from an ML model can also be used as a source of useful information about the training data and the causal mechanism that generated it. Our framework models (i) how a prediction from an ML model affects the beliefs of a rational Bayesian agent, and (ii) how this change in beliefs affects the estimation of the causal effect, the downstream decision for a new observation, and the subsequent outcome. In addition to the framework itself, we make three contributions. First, for the multivariate linear-Gaussian setting, we derive a tractable solution for the Bayesian inference problem introduced, i.e. one in which the agent infers information about a population from an ML prediction. Second, we experimentally identify conditions under which ML-DS is beneficial compared to unassisted decision making. Third, we show that a single misaligned prior belief of the user can be sufficient for ML-DS to lead to worse downstream outcomes compared to no decision support even when the prognostic model is well-specified and the agent is perfectly rational. Hence, even under ideal conditions, ML-DS can do more harm than good.
Figures & tables
Figure 1: The flow of the framework for a clinical use case, with and without ML-DS. A new patient arrives for treatment, and the patient’s data (1), potentially with the output of ML-DS, is observed by the agent, e.g. a doctor. Next, the agent estimates the effect of treatment and makes a treatment decision (2). Without ML-DS, this estimation is influenced by the agent’s priors and patient data. With ML-DS, the agent uses these plus the ML prediction to update their beliefs about the population in a Bayesian fashion. The treatment and the patient’s attributes determine the outcome (3).
Figure 2: The graphical models of the framework with stochastic variables depicted as circles and deterministic variables as rectangles. Observed variables are shaded, unobserved ones are unshaded. Model (a) represents the true data-generating process, model (b) represents the agent’s beliefs of how the ML-DS model was trained, and model (c) allows the agent to infer the treatment effect of new data points given the beliefs.
Figure 3: The effects of misaligned agent priors with ML-DS (in orange) and without (in blue) on the agent’s beliefs (top row) and on the consequent downstream outcomes Y (bottom row). In the plots for αAμ and αYμ , the horizontal axes represent the deviation of the agent’s prior belief from the true parameter in units of standard deviation, and the centerpoints in all plots correspond to aligned priors. The orange and blue bands show the mean ±1 SD of the agent’s CATE distributions pooled across the 50 runs per prior setting, with and without ML-DS, respectively. Gray bands mark settings where the true CATE lies within the orange band. Boxplots summarize the 50 expected outcomes under the selected treatments and the true generating process Hist , showing the median and interquartile range (IQR), with whiskers extending to the most extreme values within 1.5 IQR of the box. The leftmost panel shows that misaligned beliefs can lead to ML-DS being harmful, while in the central panel ML-DS corrects misaligned beliefs and ends up being beneficial.
Figure 4: The effects of a misaligned agent prior αXμ about the distribution of covariate X , shown when the ML-DS model is misspecified (left) and well-specified (right), with bands and boxplots defined as in Fig. 3 . The misspecified model introduces a new option for incorrect beliefs to result in harmful ML-DS.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Plot showing joint posterior samples of variables NE and αAμ in the posterior beliefs of the agent after update with ML-DS information. The negative correlation reflects that the agent entertains multiple possible explanations for the surprising prediction: either the treatment is less effective than expected, the historical dosages were higher than expected, or a scenario in between.
Figure 6: The effects of misaligned agent priors with ML-DS (in orange) and without (in blue) on the agent’s beliefs (top row) and on the consequent downstream outcomes Y (bottom row). In contrast to Fig. 3 , the ML-DS model here is misspecified so that the regression model lacks intercept (thus it diverges from the true generating process). In the plots for αAμ and αYμ , the horizontal axes represent the deviation of the agent’s prior belief from the true parameter in units of standard deviation, and the centerpoints in all plots correspond to aligned priors. The orange and blue bands show the mean ±1 SD of the agent’s CATE distributions pooled across the 50 runs per prior setting, with and without ML-DS, respectively. Gray bands mark settings where the true CATE lies within the orange band. Boxplots summarize the 50 expected outcomes under the selected treatments and the true generating process Hist , showing the median and interquartile range (IQR), with whiskers extending to the most extreme values within 1.5 IQR of the box. Similarly to Fig. 3 , the leftmost plots show how a misaligned belief αAμ can lead to the agent being more certain about flawed values of CATE with ML-DS, but this only leads to harmful treatment decisions when the belief drops below the treatment threshold. Similarly, the plots in the middle show how ML-DS can correct misaligned beliefs about the treatment effect over a wide range of priors which also results in improved treatment outcomes in some settings (where the blue line is below treatment threshold). The plot for misalignment of αXμ can be seen in Fig. 4 .
Figure 7: Number of covariates k vs. prior and posterior beliefs about CATE for different misalignments of priors. For NE , deviations of the prior mean are expressed in units of 0.05 . The experiment was constructed so that the prior expectation about CATE was correct despite the misaligned prior for αXμ , αYμ , and αAμ . For NE this could not be done due to the structure of AInf . The findings from the main experiment are repeated in this setting and all of the results are essentially unaffected by k . Misalignment of αAμ and αYμ can lead to the agent having less correct posterior beliefs about CATE, and misaligned beliefs about NE are largely corrected. Note that the lines that appear to be missing are simply masked by the thick black line.
Figure 8: Prior (blue) and posterior (orange) beliefs about CATE vs. the number of covariates k . Lines show the mean and bands ±1 SD of the agent’s CATE distribution, pooled across the 50 runs per setting. For NE , deviations of the prior mean are expressed in units of 0.05 . Both the posterior expectation of CATE and its uncertainty seem largely unrelated to k . The same effect with NE being corrected over a range of combinations seen in the main experiment is repeated here.
k
Full plate ( n=1000 )
Reduced plate (Lemma F.1 )
Collapsed (Thm F.2 )
1
3000
4
2
2
4000
8
2
5
7000
26
2
10
12000
76
2
Appendix
Table 1: Number of scalar sampling events needed to represent the training-data plate, as a function of the number of covariates k , for the three plate representations.