cs.AIOct 2, 2026
SaveLarge language models exhibit unreliable updating of clinical judgment as patient evidence evolves
Organizations: Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, 55455, MN, USA.
Abstract
Large language models (LLMs) are increasingly explored for clinical reasoning, but whether they appropriately revise judgments as patient evidence evolves remains unclear. We evaluated longitudinal belief updating using matched intensive-care trajectories from electronic health records. Across diverse LLMs, conditioning on a preceding judgment more often increased than reduced prediction error when estimates changed, replicated for a second endpoint. Controlled interventions revealed two failure modes. First, with preceding assessment fixed, models responded more strongly to worsening than matched improving respiratory evidence; this asymmetry persisted after headroom normalization at moderate and strong evidence levels. Second, with current evidence fixed, increasing prior risk from 10% to 90% shifted estimates by 26.2 percentage points, demonstrating causal influence of prior model beliefs. Prompting did not restore reliable updating. Evidence-Validated Longitudinal Update (EVLU) identified fewer, more reliable revisions, revealing a reliability-coverage trade-off. These findings establish longitudinal belief updating as a distinct dimension of LLM reliability.
Figures & tables
Figure 1: Study design for evaluating longitudinal clinical belief updating as patient evidence evolves. a , Construction of the matched longitudinal ICU cohort, comprising 1,000 ventilation-positive ICU stays and 1,000 matched negative ICU stays. b , Longitudinal clinical states were represented by sequential snapshots after ICU admission, with the model estimating the risk of invasive mechanical ventilation within the subsequent 24 h at each snapshot. c , Natural longitudinal evaluation compared predictions based on the current clinical state alone with predictions additionally conditioned on the model’s preceding assessment. d , Controlled interventions independently manipulated evolving respiratory evidence and the supplied preceding model belief while holding the complementary source of information fixed, enabling characterization of directional evidence responsiveness and the causal influence of supplied prior beliefs. e , Mitigation analyses tested instruction-level prompting and evidence validation; prompting did not eliminate harmful longitudinal updating, whereas EVLU preferentially retained a smaller subset of more reliable revisions at reduced coverage. Icon attributions are provided in Table S1.
Figure 2: Longitudinal self-context alters clinical risk estimates and produces harmful paired effects across model families. a , Predictive performance under independent and longitudinal evaluation for the primary model, measured by AUROC and Brier score across 6,580 clinical snapshots. b , Effect of longitudinal self-context on absolute prediction error across 4,580 post-initial snapshots. Longitudinal context increased prediction error more frequently than it reduced error. c , Cross-model generalization of harmful longitudinal-context effects across eight evaluated models. Left, change in mean absolute prediction error under longitudinal relative to independent inference. Right, normalized harmful-update asymmetry between harmful and beneficial accuracy-changing effects; positive values indicate a greater frequency of harmful than beneficial effects among accuracy-changing predictions. Error bars in the right panel indicate 95% confidence intervals obtained by bootstrap resampling at the ICU-stay level.
Figure 3: Controlled respiratory evidence produces selective, dose-dependent, and directionally asymmetric belief revision. a , Mean change in predicted 24-h invasive ventilation risk following worsening and improving evidence at subtle, moderate, and strong intervention strengths. b , Raw directional asymmetry across evidence strengths. Asymmetry was defined as the excess magnitude of deterioration-directed relative to improvement-directed belief revision; positive values indicate stronger updating in response to worsening evidence. c , Mean absolute changes in predicted ventilation risk under neutral and redundant control interventions compared with clinically informative improving and worsening interventions. Neutral and redundant evidence elicited little revision on average. Error bars in a – c indicate 95% confidence intervals obtained by bootstrap resampling at the ICU-stay level.
Figure 4: Directional asymmetry persists after accounting for prior belief headroom and preceding prediction error. a , Directional asymmetry after normalizing worsening and improving revisions by the available upward and downward probability ranges, respectively. Moderate and strong evidence retained deterioration-directed asymmetry after normalization. b , Headroom-normalized asymmetry under strong evidence stratified by the preceding predicted ventilation risk. Positive asymmetry persisted across all four prior-risk strata. c , Headroom-normalized asymmetry under strong evidence stratified by the absolute error of the preceding risk estimate. Positive asymmetry persisted across low, intermediate, and high prediction-error strata, including transitions with relatively accurate preceding predictions. Error bars indicate 95% bootstrap confidence intervals obtained by resampling at the ICU-stay level.
Figure 5: Controlled prior-belief interventions causally shift subsequent clinical risk estimates. a , Dose–response relationship between the experimentally manipulated prior 24-h ventilation risk and the subsequent updated risk estimate while current clinical evidence was held fixed. Points indicate mean updated risk estimates and error bars indicate ICU-stay-level bootstrap 95% confidence intervals. b , Mean absolute prediction error under outcome-aligned and outcome-misaligned extreme prior beliefs. For outcome-negative snapshots, 10% and 90% priors were treated as aligned and misaligned, respectively; for outcome-positive snapshots, 90% and 10% priors were treated as aligned and misaligned, respectively. Error bars indicate ICU-stay-level bootstrap 95% confidence intervals. The annotated difference denotes the increase in mean absolute error for outcome-misaligned relative to outcome-aligned priors.
Figure 6: Evidence validation identifies a more reliable subset of longitudinal updates. a , Harmful fraction among accuracy-changing predictions under unrestricted longitudinal inference and increasingly stringent EVLU criteria. The dashed line denotes equal proportions of harmful and helpful changes (50%). b , Reliability–coverage trade-off across EVLU criteria. Stronger evidence requirements reduced the harmful fraction among retained accuracy-changing revisions while progressively reducing longitudinal-branch retention, defined as the proportion of all post-initial predictions for which the longitudinal branch was retained.
More in this topic
Belief Updating in LLMs
Belief updating in large language models studies how models revise, retain, and use beliefs as evidence and context change.
See all 94 papers