Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

Jiyeon Kim; Hyunji Lee; Dylan Zhou; Sue Hyun Park; Seunghyun Yoon; Trung Bui; Franck Dernoncourt; Sungmin Cha

cs.CLMar 8, 2026

Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

Authors: Jiyeon Kim, Hyunji Lee, Dylan Zhou, Sue Hyun Park, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Sungmin Cha,

Organizations: KAIST AI · UNC Chapel Hill · Google · KRAFTON ·

Abstract

LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.

Figures & tables

Figure 1: Overview of Online Adaptation to Continual Knowledge Streams . At each time interval $t$ , a new context chunk $c_{t}$ is streamed, and the model is queried with context accumulated up to $t$ and a question $q_{j}$ . Performance is calculated as average accuracy across all intervals by comparing predictions with the ground truth answer $a_{t}$ , then averaging over all questions. A Phase $T_{i}$ denotes a contiguous set of chunks sharing the same ground truth answers. Answer options are limited to OAKS -N, as OAKS -B uses an open-ended format.

Dataset	Unit	#Steps	U/Q	Fully Annot.
EvolvingQA	Corpus	6+	2.0	✗
StreamingQA	Corpus	4	2.0	✗
StreamingBench	Frame	5	1.0	✗
FactTrack	Fact	39	2.0	✗
MultiWOZ	Turn	14	1.0	✓
BABILong	Full	1	1.0	✗

Table 1: Comparison between our dataset with prior datasets along four axes: Unit = size of each data unit, #Steps = number of evaluation steps, U/Q = average number of updates ( U ) per question ( Q ), Fully Annotated (Fully Annot.) = whether all questions are annotated for each unit.

			OAKS -BABI								OAKS -Novel
			Base				RAG				Base				RAG
Model	Size	Active	All	Sprs.	Mod.	Freq.	All	Sprs.	Mod.	Freq.	All	Sprs.	Mod.	Freq.	All	Sprs.	Mod.	Freq.
Qwen3	4B	-	26.4	34.4	23.1	18.5	29.3	37.2	25.9	21.6	51.3	59.1	49.7	47.9	55.8	68.1	56.0	48.3
	8B	-	33.1	36.8	34.0	25.8	35.1	39.9	34.5	28.4	52.7	62.5	51.7	47.6	47.7	60.0	47.8	40.4
	30B	3B	35.8	38.5	37.3	29.4	37.8	40.1	39.2	32.0	62.8	71.2	63.7	57.1	61.0	68.9	63.0	54.9
	80B	3B	41.1	44.7	42.0	34.1	43.5	47.4	44.2	36.2	64.6	74.5	63.3	59.9	65.3	74.4	65.3	59.9

Table 2: Accuracy (%) of models on OAKS -B and OAKS -N under Base and RAG settings. Gemini results are reported with the thinking mode enabled. Active denotes the number of active parameters in Mixture-of-Experts (MoE) models. Sprs. , Mod. , and Freq. denote Sparse, Moderate, and Frequent transition subsets; All is averaged over all subsets. The highest score within each column is in bold .

Model	Think	All	Tracking	Counting	Bridge	Comparison
Qwen3-30B	-	35.8	27.1	34.8	24.6	48.3
Qwen3-30B	✓	43.6	43.8	37.7	37.3	53.9
Gemini 2.5 Flash	-	43.2	54.4	42.6	30.0	53.1
Gemini 2.5 Flash	✓	56.2	58.7	53.9	42.4	69.6
Gemini 2.5 Pro	-	42.9	52.4	45.8	29.8	50.2
Gemini 2.5 Pro	✓	60.3	55.3	57.5	51.0	71.7

Table 3: Accuracy (%) of Qwen3-30B and Gemini 2.5 on OAKS -B by question type with and without thinking mode (Think). Best in bold .

Strategy	All	Sprs.	Mod.	Freq.
Base	24.7	27.3	23.6	22.4
RAG (30)	32.4	35.5	32.5	27.5
RW (30)	27.6	29.7	26.9	25.5
RAG (15) + R.W (15)	31.5	34.0	31.8	27.2
HippoRAG2	20.8	19.5	23.9	18.5
MemAgent	31.3	30.7	33.6	29.1

Table 4: Accuracy (%) of agentic memory system using Qwen2.5-7B instruct as base model on OAKS -B. The number in parentheses represents the number chunks prepended by each method. Best in bold .

			GT Phase Transitions ( C hange)				No GT Transition ( S tay)
Models	Size	Think	Adaptability ( C / ✓ )	Maladaptation ( C / ✗ )	Prescience ( S / ✓ )	Stubbornness ( S / ✗ )	Lag ( C / ✓ )	Volatility ( C / ✗ )	Stability ( S / ✓ )	Obstinacy ( S / ✗ )
Gemma 3	27B	-	31.6	28.6	11.9	27.9	12.2	27.0	25.2	35.6
GPT-OSS	120B	-	39.1	40.8	5.5	14.6	12.8	46.5	24.2	16.5
Qwen3	30B	-	34.3	33.6	9.7	22.4	8.9	36.7	26.3	28.1
Qwen3	30B	✓	39.6	34.6	7.7	18.2	13.0	37.7	30.4	18.9
Gemini 2.5	Flash	-	36.3	17.0	17.2	29.5	7.5	16.9	35.0	40.7

Table 5: Analysis of knowledge tracking behavior on OAKS -B. The table shows the average occurrence rate of specific tracking behaviors across all time intervals. The second row shows the behavioral types (e.g., Maladaptation ) along with the model’s predicted action (whether predicted answer C hange or S tay from previous prediction) and the resulting prediction correctness (correct: ✓ vs. incorrect: ✗ ).

Model	Size	Think	ACC	AL( $\downarrow$ )	DS( $\downarrow$ )	PM( $\downarrow$ )
Gemma 3	27B	-	37.8	5.4	26.5	30.3
GPT-OSS	120B	-	37.5	8.6	38.8	15.1
Qwen3	30B	-	35.8	6.9	34.0	23.2
Qwen3	30B	✓	43.6	9.0	37.0	10.4
Gemini 2.5	Flash	-	43.2	6.2	28.3	22.3
Gemini 2.5	Flash	✓	56.2	5.1	31.8	7.0

Table 6: Analysis of intra-phase behavior (ACC, AL, DS, PM) on OAKS -B over models. Best in bold .

Figure 2: Accuracy (%) across different question types of OAKS -B.

Figure 3: Accuracy (%) across the timestep where the question is asked.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Total	Sprs.	Mod.	Freq.
OAKS -B	1,224	40%	25%	35%
OAKS -N	870	25%	33%	42%

Appendix

Table 7: Distribution of answer change frequency across datasets: Sparse (Sprs.), Moderate (Mod.), and Frequent (Freq.)

Figure 4: Frequency distribution of answer changes per question for OAKS -B and OAKS -N.

Category	Question Template
Tracking Questions	Where is [SUB]?
	Who is holding [OBJ]?
	Who gave the [OBJ] to someone else?
Counting Questions	How many times has [SUB] moved to [PLACE]?
	How many times has [SUB] moved?
	How many people have visited [PLACE]?

Appendix

Table 8: Question templates used for OAKS -B dataset construction. The slots [SUB], [OBJ], [PLACE], and [RECIPIENT] are dynamically filled based on the entities present in the narrative facts.

Type	Percentage	Description	Example
Tracking	7%	Questions requiring only a single piece of evidence.	Where is Daniel?
Counting	28%	Questions requiring multiple pieces of evidence and simple counting.	How many times has Sandra moved?
Bridge	30%	Questions requiring multiple pieces of evidence.	Who most recently traveled directly from office to kitchen?
Comparison	35%	Questions requiring multiple pieces of evidence and simple comparative reasoning.	Who dropped more objects, Sandra or Mary?

Appendix

Table 9: Overview of Question Types, Statistics, and Examples for OAKS -BABI dataset. This table describes the four main categories of questions found in the dataset.

Example Question 1: What mode of transportation are Phileas Fogg and Passepartout currently utilizing? Source: Around the World in Eighty Days												(A) Train from London to Paris (B) The steamer ’Mongolia’ (C) Train from Bombay to Calcutta (D) An elephant (E) Train through India (F) The steamer ’Rangoon’ to Hong Kong (G) The question cannot be answered at this point in the story.
Chunk	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	$\cdots$
Answers	G	A	A	B	B	B	B	C	C	D	D	D	D	E	E	F	F	F	F	$\cdots$
Example Question 2: What is Elizabeth’s opinion of Mr. Darcy? Source: Pride and Prejudice												(A) He is proud and disagreeable. (B) She is ashamed of her prejudice and feels respect for him. (C) She loves him. (D) He is a dishonest man who does not keep his word. (E) She is offended by his perception of her. (F) The question cannot be answered at this point in the story.
Chunk	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	$\cdots$
Answers	F	F	F	A	A	A	A	A	A	D	D	D	D	E	E	E	A	B	B	$\cdots$
Example Question 3: What is Victor Frankenstein’s primary goal or motivation at this point in the story? Source: Frankenstein												(A) To learn the secrets of nature and, specifically, the principle of life. (B) To escape the memory and consequences of his work. (C) To destroy the Monster as an act of vengeance. (D) Recovering from his illness (E) Living in regret and remorse because his creation murdered his brother William. (F) To create a companion for the Monster. (G) The question cannot be answered at this point in the story.

Appendix

Table 10: OAKS -Novel Example Questions, Answer Choices, and Answer Evolution over Book Chunks. Note: The Answers shown in the diagram are for demonstration purposes only and may not accurately reflect the ground truth answers for the listed chunks.

Avail- ability	Model		Architecture			Context Length		Exp. Length
Avail- ability	Model Family	Full-name	Size (Active)	Type	Attention	Default	Expanded	Exp. Length
Open	Qwen	Qwen3-4B-Instruct-2507	4B	Dense	GQA	262k	-	262k
		Qwen2.5-7B-Instruct	7B	Dense	GQA	32k	131k	131k
		Qwen3-8B	8B	Dense	GQA	32k	131k	131k
		Qwen3-30B-A3B-Instruct-2507	30B (3B)	MoE	GQA	262k	-	262k
		Qwen3-30B-A3B-Thinking-2507	30B (3B)	MoE	GQA	262k	-	262k

Appendix

Table 11: Model Architecture and Performance Settings. Attention architecture of Qwen3-80B is a Hybrid of Gated DeltaNet and Gated Attention)

	OAKS -B				OAKS -N
Strategy	All	Sprs.	Mod.	Freq.	All	Sprs.	Mod.	Freq.
Base	35.8	38.5	37.3	29.4	62.8	71.2	63.7	57.1
RAG	37.8	40.1	39.2	32.0	61.0	68.9	63.0	54.9
R.W.	36.4	38.8	38.3	29.9	62.6	69.2	63.6	58.0
RAG+R.W	36.7	39.3	38.1	30.7	64.6	72.1	64.7	60.1

Appendix

Table 12: Accuracy (%) of Qwen3-30B on OAKS -B and OAKS -N under different context construction strategies. We compare baseline concatenation, RAG with top-30 retrieved chunks, a rolling window of the most recent 30 chunks (RW), and their combination (RAG+RW; top-15 retrieved and 15 most recent).

Figure 5: Pass@k performance as the number of context increases for retrieval in RAG on OAKS -B and OAKS -N

Figure 6: Accuracy (%) on OAKS -B across different numbers of context chunks in RAG and RW.

Figure 7: Accuracy (%) on OAKS -N across different numbers of context chunks in RAG and RW.

Model	Think	All	Sparse	Moderate	Frequent
Qwen3 30B	-	62.8	71.2	63.7	57.1
Qwen3 30B	✓	61.8	70.6	61.1	57.1
Gemini 2.5 Flash	-	67.3	77.3	68.2	60.9
Gemini 2.5 Flash	✓	65.6	75.2	65.6	60.0

Appendix

Table 13: Accuracy (%) of Qwen3-30B and Gemini 2.5 with RAG on OAKS -N by answer change frequency with and without Thinking Mode.

Models	Size	LongBench
Qwen2.5	7B	30.0
Gemma 3	4B	29.4
GPT-OSS	20B	14.8
Gemma 3	27B	33.6
Qwen3	30B	32.6

Appendix

Table 14: Performance on LongBench-V2.

Figure 8: Extended visual explanations for model’s behavior on OAKS described in Table 6.

			GT Phase Transitions ( C hange)				No GT Transition ( S tay)
Think	Models	Size	Adaptability (C / ✓ )	Maladaptation (C / ✗ )	Prescience (S / ✓ )	Stubbornness (S / ✗ )	Lag (C / ✓ )	Volatility (C / ✗ )	Stability (S / ✓ )	Obstinacy (S / ✗ )
✗	Qwen3	4B	28.4	26.2	2.5	42.9	4.6	23.6	21.4	50.4
		8B	36.5	48.2	3.7	11.6	11.0	57.9	21.5	9.6
		30B	34.3	33.6	9.7	22.4	8.9	36.7	26.3	28.1
		80B	36.0	29.5	9.2	25.3	9.5	31.3	31.3	28.0
		235B	39.8	27.9	11.3	21.0	12.0	31.6	34.5	21.9

Appendix

Table 15: Analysis of model knowledge tracking behavior on OAKS -B. The table shows the average occurrence rate of specific tracking behaviors across all time intervals. Results are partitioned by the ground truth (GT) state: whether the answer Change from the previous interval or Stay the same. Since Stay intervals are more frequent(94% of all intervals), rates are averaged within each GT category to sum to 100%. The second row shows the behavioral labels (e.g., Maladaptation) and the model’s action in the sub-header (e.g., (C / ✓ )): whether the predicted answer C hanged from the previous prediction or S tay, and whether the resulting answer is correct ( ✓ ) or wrong ( ✗ ).

			GT Phase Transitions ( C hange)				No GT Transition ( S tay)
Think	Models	Size	Adaptability (C / ✓ )	Maladaptation (C / ✗ )	Prescience (S / ✓ )	Stubbornness (S / ✗ )	Lag (C / ✓ )	Volatility (C / ✗ )	Stability (S / ✓ )	Obstinacy (S / ✗ )
✗	Qwen3	4B	51.8	15.2	7.5	25.5	8.2	20.4	38.6	32.8
		8B	53.3	18.4	6.7	21.7	8.2	23.1	43.1	25.7
		30B	54.8	9.7	11.9	23.7	5.4	10.5	57.9	26.2
		80B	55.5	8.5	12.5	23.5	4.6	8.7	60.3	26.4
		235B	55.2	9.0	11.3	24.5	5.0	9.5	60.3	25.2

Appendix

Table 16: Analysis of model knowledge tracking behavior on OAKS -N.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World
Apr 6, 2026Hanbing Liu, Lang Cao, Yang LiRetrieval-Augmented Generation Continual Learning for LLMs
Tsinghua University; Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University · University of Illinois Urbana-Champaign · School of Artificial Intelligence, Chinese University of Hong Kong (Shenzhen)
When Does Continual Learning Require Learning
Jul 8, 2026Anne Harrington, Nayan Saxena, Michael Murphy +7Continual Learning for LLMs Concept Drift
1UC Berkeley · 2Independent · 3Capital Fund Management
ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models
Jul 29, 2026Ruxi Gu, Zhenliang Zhang, Zilei Wang +1Knowledge Editing Continual Learning for LLMs
Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China.

Apr 6, 2026J/K move · Enter open · S save

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

Hanbing Liu, Lang Cao, Yang Li

Tsinghua University; Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University · University of Illinois Urbana-Champaign · School of Artificial Intelligence, Chinese University of Hong Kong (Shenzhen)

Retrieval-Augmented Generation Continual Learning for LLMs LLM Evaluation Temporal Reasoning in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) acquire most of their knowledge during pretraining, which ties them to a fixed snapshot of the world and makes adaptation to continuously evolving knowledge challenging. As facts, entities, and events change over time, models may experience continuous knowledge drift, resulting not only in outdated predictions but also in temporally inconsistent reasoning. Although existing approaches, such as continual finetuning, knowledge editing, and retrieval-augmented generation (RAG), aim to update or supplement model knowledge, they are rarely evaluated in settings that reflect chronological, evolving, and real-world knowledge evolution. In this work, we introduce a new benchmark of real-world dynamic events, constructed from time-stamped evidence that captures how knowledge evolves over time, which enables systematic evaluation of model adaptation under continuous knowledge drift. The benchmark reveals that most existing methods, including vanilla RAG and several learning-based approaches, struggle under this setting, exposing critical limitations such as catastrophic forgetting and temporal inconsistency. To mitigate these limitations, we propose a time-aware retrieval baseline, Chronos, which progressively organizes retrieved evidence into an Event Evolution Graph to enable more temporally consistent understanding in LLMs without additional training. Overall, this work provides a foundation for analyzing and advancing LLM adaptation to continuous knowledge drift in realistic settings.

Abstract

LLM Evaluation

Explore similar work

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

When Does Continual Learning Require Learning

ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models