cs.CLApr 6, 2026

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

Authors: Hanbing Liu, Lang Cao, Yang Li

Organizations: Tsinghua University; Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University · University of Illinois Urbana-Champaign · School of Artificial Intelligence, Chinese University of Hong Kong (Shenzhen)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) acquire most of their knowledge during pretraining, which ties them to a fixed snapshot of the world and makes adaptation to continuously evolving knowledge challenging. As facts, entities, and events change over time, models may experience continuous knowledge drift, resulting not only in outdated predictions but also in temporally inconsistent reasoning. Although existing approaches, such as continual finetuning, knowledge editing, and retrieval-augmented generation (RAG), aim to update or supplement model knowledge, they are rarely evaluated in settings that reflect chronological, evolving, and real-world knowledge evolution. In this work, we introduce a new benchmark of real-world dynamic events, constructed from time-stamped evidence that captures how knowledge evolves over time, which enables systematic evaluation of model adaptation under continuous knowledge drift. The benchmark reveals that most existing methods, including vanilla RAG and several learning-based approaches, struggle under this setting, exposing critical limitations such as catastrophic forgetting and temporal inconsistency. To mitigate these limitations, we propose a time-aware retrieval baseline, Chronos, which progressively organizes retrieved evidence into an Event Evolution Graph to enable more temporally consistent understanding in LLMs without additional training. Overall, this work provides a foundation for analyzing and advancing LLM adaptation to continuous knowledge drift in realistic settings.

Figures & tables

Figure 1: Illustration of continuous knowledge drift and our proposed method. Real-world knowledge evolves over time (e.g., changes in company CEO), while LLMs with a fixed knowledge cutoff may produce outdated or temporally inconsistent answers. We introduce a dynamic event benchmark composed of time-stamped documents that reflect such evolution, and a time-aware retrieval baseline ( Chronos ) that progressively organizes retrieved evidence into an Event Evolution Graph, enabling more temporally consistent reasoning.
Figure 2: Overview of Chronos . The framework first performs query analysis to extract relevant entities and the associated time window. A time-aware retriever then collects temporally relevant knowledge quadruples from an up-to-date knowledge base. These facts are organized into an Event Evolution Graph (EEG), which models how entity states evolve over time by sorting events along a timeline and linking events that share common entities. The EEG is progressively refined, and its resulting multi-perspective views enable the model to reason over evolving knowledge and produce temporally consistent responses.
Knowledge
⟨\langle World’s Richest Person, held by, Elon Musk, 2024-01-01 ⟩\rangle
⟨\langle World’s Richest Person, held by, Bernard Arnault, 2024-01-28 ⟩\rangle
⟨\langle World’s Richest Person, held by, Jeff Bezos, 2024-03-05 ⟩\rangle
⟨\langle World’s Richest Person, held by, Bernard Arnault, 2024-04-02 ⟩\rangle
⟨\langle World’s Richest Person, held by, Elon Musk, 2024-06-08 ⟩\rangle
⟨\langle World’s Richest Person, held by, Larry Ellison, 2025-09-10 ⟩\rangle
Table 1: An illustration of the dataset structure. Each instance includes time-stamped knowledge quadruples together with questions covering historical, contemporary, and commonsense settings, designed to evaluate reasoning over evolving knowledge.
MethodHistoricalContemporaryCommonsenseOverall
C1C2C3
Direct Generation
LLaMA-3.145.9511.311.381.6945.5321.17
Claude55.8610.532.753.3962.5527.02
GPT-4o61.2614.813.215.0867.3230.34
Parametric Updating Methods
Table 2: Performance comparison across historical, contemporary, and commonsense question settings. Higher values indicate better accuracy. Best results for each model are shown in bold , and second-best results are underlined . Green values indicate improvements over the strongest corresponding baseline. Chronos achieves consistent gains across multiple evaluation settings, while other baselines exhibit varying limitations.
MethodHist.ContemporaryComm.
  C1C2C3 
Chronos (Ours)63.9696.3066.5171.1971.36
- Time-aware Retr.62.16 -1.892.01 -4.357.80 -8.759.32 -11.971.76 +0.4
- History Recon.6.31 -57.797.66 +1.467.43 +0.972.89 +1.771.69 +0.3
- Event Augment.62.16 -1.892.53 -3.855.05 -11.552.54 -18.770.50 -0.9
- Temporal View61.26 -2.790.25 -6.158.26 -8.355.93 -15.370.65 -0.7
Table 3: Ablation study of Chronos . Each value reports accuracy, followed by its change relative to the full model. Green values indicate performance improvements, while red values indicate performance drops.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Query (C1)WISE OutputChronos Output
Who was the world’s richest person on Jan 1, 2024?Elon Musk ✓\checkmarkElon Musk ✓\checkmark
Who was the world’s richest person on Jan 28, 2024?Elon Musk ×\timesBernard Arnault ✓\checkmark
Who was the world’s richest person on Mar 5, 2024?Elon Musk ×\timesJeff Bezos ✓\checkmark
Who was the world’s richest person on Apr 2, 2024?Elon Musk ×\timesBernard Arnault ✓\checkmark
Who was the world’s richest person on Jun 8, 2024?Elon Musk ✓\checkmarkElon Musk ✓\checkmark
Appendix
Table 4: An error case of parameter-based knowledge editing under continuous knowledge drift. While WISE produces the same answer for all time-specific queries due to overwriting effects, Chronos correctly returns different answers conditioned on the queried timestamp, demonstrating its ability to model temporal variation in evolving knowledge.
Figure 3: A case study of Chronos for question answering under continuous knowledge drift.
Figure 4: Prompt template used for direct generation baseline. Blue text indicates input variables.
Figure 5: Prompt template used for retrieval-augmented generation baseline. Blue text indicates input variables.
Figure 6: Prompt template used for ReAct-style retrieval-augmented generation baseline. Blue text indicates input variables.
Figure 7: Prompt template used for query analysis in Chronos . Blue text indicates input variables.
Figure 8: Prompt template used for history construction in Chronos . Blue text indicates input variables.
Figure 9: Prompt template used for event augmentation in Chronos . Blue text indicates input variables.
Figure 10: Prompt template used for final response in Chronos . Blue text indicates input variables.
More in this topic

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Explore similar work

CardsList

  1. Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

    Mar 8, 2026Jiyeon Kim, Hyunji Lee, Dylan Zhou +6LLM EvaluationContinual Learning for LLMs

    KAIST AI · UNC Chapel Hill · Google +3

  2. Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge

    Oct 18, 2026Tianyi Hu, Bhavyajeet Singh, Niket Tandon +1Retrieval-Augmented ReasoningLLM Reasoning

    Aarhus University · Microsoft Research · Technische Universität Darmstadt

  3. ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models

    Jul 29, 2026Ruxi Gu, Zhenliang Zhang, Zilei Wang +1Knowledge EditingContinual Learning for LLMs

    Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China.