Medical knowledge evolves as clinical guidelines revise recommendations, yet whether large language models (LLMs) faithfully incorporate such changes remains unclear. We introduce ChronoEvo, comprising 1,709 recommendation-level revisions across 12 forms of medical knowledge change, to distinguish recognition of revised guidance from incorporation into clinical recommendations. In the primary model, 90.8% of responses recognized revisions and 80.2% were clinically correct, but only 58.5% fully incorporated guidance, yielding a 32.3-percentage-point recognition-incorporation gap. This dissociation persisted across 14 models and was corroborated by two clinicians' adjudicated assessment of 300 responses, yielding a 40.3-percentage-point gap in the validation sample. Changes in recommendation strength and patient eligibility were associated with retention of outdated guidance. A revision-salience intervention reduced outdated-content retention and improved clinical correctness but did not significantly improve full incorporation. These findings identify faithful incorporation of evolving medical knowledge as a distinct challenge for clinical LLM reliability, even when updated guideline evidence is available.
Figures & tables
Figure 1: Overview of the ChronoEvo study framework. a, Construction of ChronoEvo from longitudinal guideline trajectories through revision-relationship reconstruction, identification of revision edges with paired full texts, recommendation-level change extraction and verification, yielding 1,709 verified knowledge revisions from 308 revision edges across 275 guideline lineages. b, Twelve semantic forms of medical knowledge change represented in the benchmark. c, Evaluation framework for testing how LLMs respond to evolving medical knowledge under closed-book, prior-only and full-evidence conditions. Model responses are evaluated along three dimensions—revision recognition, clinical correctness and full incorporation—and further examined using revision-gain, semantic-error, change-type, multivariable, cross-model and evaluator-robustness, and intervention analyses. Icon attributions are provided in Table S1.
Figure 2: Recognition and incorporation of evolving medical knowledge. a , Proportion of responses that recognized the revision, were clinically correct, and fully incorporated the revised knowledge. b , Full-incorporation rates across the 12 semantic revision types. Error bars indicate 95% confidence intervals. Semantic outcomes were evaluated by an evaluator blinded to the benchmark-assigned revision type.
Figure 3: Failure patterns across forms of medical knowledge evolution. a , Distribution of primary error types across revision categories. Values indicate the percentage of responses within each revision type exhibiting the corresponding primary error. b , Two forms of persistence of superseded knowledge across revision categories: outdated-content retention and hybrid old–new responses.
Figure 4: Revision-type associations after adjustment for measured instance characteristics. a , Adjusted odds ratios for full incorporation of revised knowledge across revision types. b , Adjusted odds ratios for retention of outdated knowledge across revision types. Addition served as the reference category. Models were adjusted for prior–current lexical overlap, year gap between guideline versions, and the lengths of the prior recommendation, updated recommendation, and supporting evidence. Points indicate adjusted odds ratios and horizontal lines indicate 95% confidence intervals; dashed vertical lines denote an odds ratio of 1. Standard errors were clustered by guideline revision edge.
Outcome
Standard (%)
Revision salience (%)
Δ (pp; 95% CI)
Revision recognition
90.8
91.7
+0.9 [-0.4, +2.3]
Clinical correctness
80.2
82.6
+2.4 [+0.7, +4.0]
Full incorporation
58.5
59.3
+0.8 [-1.1, +2.7]
Outdated retention
5.9
4.3
-1.6 [-2.6, -0.6]
Hybrid old/new
4.5
2.7
-1.8 [-2.7, -0.9]
Recognition–incorporation gap
32.3
32.4
+0.1 [-2.0, +2.2]
Table 2: Effect of revision-salience prompting on knowledge updating. Differences are paired percentage-point changes relative to the standard full-evidence condition. Confidence intervals were obtained using 10,000 bootstrap resamples clustered by guideline revision edge. Differences were calculated from unrounded case-level estimates and may therefore differ by 0.1 percentage points from differences obtained from the displayed rounded values.