RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

Hanbing Liu; Lang Cao; Yang Li

cs.CLApr 6, 2026

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

Authors: Hanbing Liu, Lang Cao, Yang Li

Organizations: Tsinghua University; Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University · University of Illinois Urbana-Champaign · School of Artificial Intelligence, Chinese University of Hong Kong (Shenzhen)

Abstract

Large language models (LLMs) acquire most of their knowledge during pretraining, which ties them to a fixed snapshot of the world and makes adaptation to continuously evolving knowledge challenging. As facts, entities, and events change over time, models may experience continuous knowledge drift, resulting not only in outdated predictions but also in temporally inconsistent reasoning. Although existing approaches, such as continual finetuning, knowledge editing, and retrieval-augmented generation (RAG), aim to update or supplement model knowledge, they are rarely evaluated in settings that reflect chronological, evolving, and real-world knowledge evolution. In this work, we introduce a new benchmark of real-world dynamic events, constructed from time-stamped evidence that captures how knowledge evolves over time, which enables systematic evaluation of model adaptation under continuous knowledge drift. The benchmark reveals that most existing methods, including vanilla RAG and several learning-based approaches, struggle under this setting, exposing critical limitations such as catastrophic forgetting and temporal inconsistency. To mitigate these limitations, we propose a time-aware retrieval baseline, Chronos, which progressively organizes retrieved evidence into an Event Evolution Graph to enable more temporally consistent understanding in LLMs without additional training. Overall, this work provides a foundation for analyzing and advancing LLM adaptation to continuous knowledge drift in realistic settings.

Figures & tables

Figure 2: Overview of Chronos . The framework first performs query analysis to extract relevant entities and the associated time window. A time-aware retriever then collects temporally relevant knowledge quadruples from an up-to-date knowledge base. These facts are organized into an Event Evolution Graph (EEG), which models how entity states evolve over time by sorting events along a timeline and linking events that share common entities. The EEG is progressively refined, and its resulting multi-perspective views enable the model to reason over evolving knowledge and produce temporally consistent responses.

Knowledge

\langle

World’s Richest Person, held by, Elon Musk, 2024-01-01

\rangle

\langle

World’s Richest Person, held by, Bernard Arnault, 2024-01-28

\rangle

\langle

World’s Richest Person, held by, Jeff Bezos, 2024-03-05

\rangle

\langle

World’s Richest Person, held by, Bernard Arnault, 2024-04-02

\rangle

\langle

World’s Richest Person, held by, Elon Musk, 2024-06-08

\rangle

\langle

World’s Richest Person, held by, Larry Ellison, 2025-09-10

\rangle

Table 1: An illustration of the dataset structure. Each instance includes time-stamped knowledge quadruples together with questions covering historical, contemporary, and commonsense settings, designed to evaluate reasoning over evolving knowledge.

Direct Generation
Method	Historical	Contemporary			Commonsense	Overall
Method	Historical	C1	C2	C3	Commonsense	Overall
LLaMA-3.1	45.95	11.31	1.38	1.69	45.53	21.17
Claude	55.86	10.53	2.75	3.39	62.55	27.02
GPT-4o	61.26	14.81	3.21	5.08	67.32	30.34
Parametric Updating Methods

Table 2: Performance comparison across historical, contemporary, and commonsense question settings. Higher values indicate better accuracy. Best results for each model are shown in bold , and second-best results are underlined . Green values indicate improvements over the strongest corresponding baseline. Chronos achieves consistent gains across multiple evaluation settings, while other baselines exhibit varying limitations.

Method	Hist.	Contemporary			Comm.
		C1	C2	C3
Chronos (Ours)	63.96	96.30	66.51	71.19	71.36
- Time-aware Retr.	62.16 -1.8	92.01 -4.3	57.80 -8.7	59.32 -11.9	71.76 +0.4
- History Recon.	6.31 -57.7	97.66 +1.4	67.43 +0.9	72.89 +1.7	71.69 +0.3
- Event Augment.	62.16 -1.8	92.53 -3.8	55.05 -11.5	52.54 -18.7	70.50 -0.9
- Temporal View	61.26 -2.7	90.25 -6.1	58.26 -8.3	55.93 -15.3	70.65 -0.7

Table 3: Ablation study of Chronos . Each value reports accuracy, followed by its change relative to the full model. Green values indicate performance improvements, while red values indicate performance drops.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Query (C1)	WISE Output	Chronos Output
Who was the world’s richest person on Jan 1, 2024?	Elon Musk $\checkmark$	Elon Musk $\checkmark$
Who was the world’s richest person on Jan 28, 2024?	Elon Musk $\times$	Bernard Arnault $\checkmark$
Who was the world’s richest person on Mar 5, 2024?	Elon Musk $\times$	Jeff Bezos $\checkmark$
Who was the world’s richest person on Apr 2, 2024?	Elon Musk $\times$	Bernard Arnault $\checkmark$
Who was the world’s richest person on Jun 8, 2024?	Elon Musk $\checkmark$	Elon Musk $\checkmark$

Appendix

Table 4: An error case of parameter-based knowledge editing under continuous knowledge drift. While WISE produces the same answer for all time-specific queries due to overwriting effects, Chronos correctly returns different answers conditioned on the queried timestamp, demonstrating its ability to model temporal variation in evolving knowledge.

Figure 3: A case study of Chronos for question answering under continuous knowledge drift.

Figure 4: Prompt template used for direct generation baseline. Blue text indicates input variables.

Figure 5: Prompt template used for retrieval-augmented generation baseline. Blue text indicates input variables.

Figure 6: Prompt template used for ReAct-style retrieval-augmented generation baseline. Blue text indicates input variables.

Figure 7: Prompt template used for query analysis in Chronos . Blue text indicates input variables.

Figure 8: Prompt template used for history construction in Chronos . Blue text indicates input variables.

Figure 9: Prompt template used for event augmentation in Chronos . Blue text indicates input variables.

Figure 10: Prompt template used for final response in Chronos . Blue text indicates input variables.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Mar 8, 2026

Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.

Jiyeon Kim, Hyunji Lee, Dylan Zhou +6

KAIST AI · UNC Chapel Hill · Google +3

LLM Evaluation Continual Learning for LLMs Online Continual Learning Memory-Augmented Language Models

Ask PDF

Oct 18, 2026

Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge

As the world changes, new information is published while older records remain online. This complicates reasoning for large language models (LLMs), which must retrieve evidence and resolve conflicts to determine which information remains valid. Existing evaluations treat information seeking as a black box, revealing little about where and why LLMs fail. To address this gap, we introduce TREK, a suite of three components: TREKTRACE is a plug-and-play framework that attributes failures to specific stages of retrieval and reasoning. We apply it to TREKBENCH-WILD, a meta-benchmark of real-world questions that require complex reasoning over rapidly evolving knowledge. Surprisingly, the main bottleneck is post-retrieval reasoning rather than retrieval: frontier models are often misled by conflicting evidence even after retrieving all the information needed to answer. To study this bottleneck, we build TREKBENCH-CTRL, a controlled benchmark with distractors spanning temporality, relevance, and trustworthiness, and show that benchmarks dominated by outdated distractors substantially underestimate task difficulty. Together, TREK provides a testbed for diagnosing and improving reasoning over rapidly evolving knowledge.

Tianyi Hu, Bhavyajeet Singh, Niket Tandon +1

Aarhus University · Microsoft Research · Technische Universität Darmstadt

Retrieval-Augmented Reasoning LLM Reasoning Language Model Reasoning Evaluation Knowledge Conflicts in Language Models Temporal Reasoning in Language Models

Ask PDF

Jul 29, 2026

ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models

Large language models (LLMs) have demonstrated strong capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquired knowledge under repeated updates remains insufficiently understood. Existing evaluation paradigms primarily focus on single-step reasoning or static knowledge editing, which fail to capture the temporal dynamics of knowledge retention and degradation during continual model modification. In this work, we propose ForgetBench, a benchmark for measuring forgetting dynamics in the long-term parametric memory of LLMs under continual knowledge editing. ForgetBench introduces two complementary evaluation paradigms, namely concept-based QA and scenario-based QA, to probe the retention of isolated facts and structured relational knowledge. Building upon a sequential editing framework, we construct temporally ordered knowledge streams and evaluate model behavior across multiple editing stages. To quantify these dynamics, we introduce a unified evaluation framework that indexes memory age by the number of subsequent edits and measures temporal decay, retention strength, and cross-instance stability. Extensive experiments across diverse models and editing methods reveal distinct forgetting trajectories and a recurring tension between long-term retention and generalization quality. Our findings highlight the need for more robust memory mechanisms that can effectively acquire, update, and preserve knowledge over time in future LLMs.

Ruxi Gu, Zhenliang Zhang, Zilei Wang +1

Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China.

Knowledge Editing Continual Learning for LLMs LLM Evaluation Persistent Memory for Language Models LLM Memory Catastrophic Forgetting

Ask PDF

Abstract

Retrieval-Augmented Generation

Explore similar work

Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge

ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models