Towards Evolving Context Parameterization for Large Language Models

Xiaobing Shi; Zherui Li; Yiming Jiang; Kun Wang; Yufei Guo

cs.CLSep 12, 2026

Towards Evolving Context Parameterization for Large Language Models

Authors: Xiaobing Shi, Zherui Li, Yiming Jiang, Kun Wang, Yufei Guo

Organizations: Nanyang Technological University · Peking University

Abstract

Context parameterization enables large language models (LLMs) to internalize contexts into reusable model parameters, avoiding repeated processing across subsequent queries. However, existing methods typically assume static contexts and lack explicit mechanisms for distinguishing validity states under continual updates. To study this real-world scenario, we formalized the Memory Updating with Sequential Evolution (MUSE) task and constructed MUSE-bench to evaluate update incorporation and unaffected-information preservation. The resulting challenge requires preserving the global state while adjusting the contribution of memory evidence. Motivated by this, we proposed PLUME, a training-free method that constructs a global update representation, activates memory evidence to form a local parameter view, and adaptively integrates their predictions during decoding. Comprehensive evaluation on MUSE-bench demonstrated PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge. Our codes are available at: https://github.com/xiaobingshi-LLM/PLUME.

Figures & tables

Figure 1: Illustration of context parameterization under continual updates. While existing methods perform well in conventional settings, they may conflate obsolete information with subsequent updates, whereas PLUME follows the currently valid information.

Figure 2: The overall framework of our proposed PLUME . PLUME constructs a global update representation from the full context state, activates memory evidence according to the query to provide a local parameter view, and adaptively integrates the global and local predictions during decoding.

Method	SQuAD Recall $\uparrow$ LLM $\uparrow$		ROPES Recall $\uparrow$ LLM $\uparrow$		2Wiki Recall $\uparrow$ LLM $\uparrow$		MFQA Recall $\uparrow$ LLM $\uparrow$		QASPER Recall $\uparrow$ LLM $\uparrow$
\SetRow ht=abovesep=belowsep= Qwen-3
\SetHline 1-110.5pt,solid Base w/ Context (oracle)	95.86	91.69	97.82	95.91	60.11	52.35	64.50	60.08	61.75	53.95
CD (oracle)	94.93	91.75	98.31	97.52	68.51	65.44	66.24	61.17	62.46	55.81
D2L (oracle)	76.77	64.92	85.57	80.39	43.13	38.33	28.80	20.34	22.98	15.80
\SetHline 1-110.5pt,solid Base w/o Context	24.61	15.19	82.88	64.99	31.75	24.67	22.48	11.33	1.28	1.89
AnyEdit	15.77	12.44	76.43	67.48	16.25	15.00	13.23	9.25	7.40	4.02

Table 1: A Comprehensive Comparison of Different Methods. ROUGE-L Recall and LLM-as-a-Judge are reported on MUSE-Bench datasets, where bold and underline denote the best and second-best values respectively.

Figure 3: Locality under updates. Higher scores indicate better unaffected information preservation.

Method	GSM8K		CRUXEval
Method	Recall	LLM	Recall	LLM
Qwen3
w/ Context (oracle)	54.21	52.76	68.43	55.37
CD (oracle)	51.54	50.83	65.07	53.20
D2L (oracle)	36.23	30.24	44.09	20.50
w/o Context	15.21	7.82	37.75	4.62

Table 2: Cross-task generalization results. ROUGE-L Recall and LLM-as-a-Judge are reported.

Figure 4: Overall comparison of effectiveness and efficiency. Normalized metrics include answer quality, locality, and update/generation efficiency.

Figure 5: Ablation of PLUME components. The memory-only and global-only variants each underperform PLUME , demonstrating the complementary contributions of the global update representation and query-activated memory evidence.

Figure 6: Representative case under a context update. D2L conflates information with different validity states and produces an incorrect answer, whereas PLUME follows the currently valid update.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Representative MUSE-Bench example derived from SQuAD. The example illustrates the original context, its subsequent update, and the associated question–answer annotations.

Figure 8: Representative MUSE-Bench example derived from ROPES. The example illustrates a context update while preserving the causal reasoning structure of the original instance.

Figure 9: Representative MUSE-Bench example derived from MultiFieldQA-en. The example illustrates an update within a long-form context while retaining its original document structure.

Figure 10: Representative MUSE-Bench example derived from 2WikiMultihopQA. The example illustrates a context update with consistently revised multi-hop dependencies.

Figure 11: Representative MUSE-Bench example derived from QASPER. The example illustrates an update to evidence distributed across a scientific document.

\SetRow bg=black!6 Dataset	Contexts	QA pairs	Avg. context tokens
SQuAD	2,067	10,570	328.3
ROPES	203	1,688	434.6
2WikiMultihopQA	300	300	18,539.8
MultiFieldQA-en	150	150	14,068.8
QASPER	224	224	12,424.7
MUSE-Bench	2,944	12,932	3,811.9

Appendix

Table 3: Dataset statistics for MUSE-Bench and the cross-task evaluation sets. We report the numbers of contexts and question–answer pairs and the mean full-history length measured with the Qwen3-4B-Instruct-2507 tokenizer.

\SetRow bg=black!6 Base model	D2L checkpoint	Base rank	Target
Qwen3-4B	Qwen D2L-20k	8	MLP down
Gemma-2-2B	Gemma D2L-20k	8	MLP down

Appendix

Table 4: Backbone and D2L checkpoint configurations. All models use bfloat16 precision, and the generated rank-8 LoRA adapters are applied to the MLP down-projection layers.

\SetRow bg=black!6 Setting	Iterative	Batched
Context coverage	Complete history	Complete history
Maximum chunk length	8,192	8,192
Chunk construction	Near-equal, contiguous	Near-equal, contiguous
Layer execution	Sequential	Joint batched pass
Base LoRA rank	8	8
Target module	down_proj	down_proj

Appendix

Table 5: D2L inference modes for context parameterization. Iterative and batched execution differ only in how Transformer-layer representations are processed; both support complete-context chunking and the same base-rank configuration. Only Iterative results are reported.

\SetRow bg=black!6 Setting	PLUME
Maximum context-chunk length	8,192
Evidence activation	Relevance with recency margin
Number of activated units	1
Recency margin $\delta$	0.5
$\alpha$ / $\beta$	1 / 0.75
$\lambda_{\max}$ / $\tau$	1 / 0.3

Appendix

Table 6: PLUME -specific and generation hyperparameters. Memory units are constructed from context structure, so their number varies with the length and organization of each context.

Figure 12: Performance under continual updates. ROUGE-L Recall decreases gradually as the number of updates increases from 3 to 20.

Figure 13: Teacher-forced candidate-rank analysis. Token-level ROUGE-L Recall increases sharply from top-1 to top-5 and then begins to saturate through top-10.

Figure 14: Sensitivity to the global update coefficient

\beta

. Recall and Locality achieve their best joint result at

\beta=0.75

and decrease when the update direction is under- or over-amplified.

Figure 15: Sensitivity to the adaptive-gating parameter

\tau

. The selected value

\tau=0.3

maximizes Recall while maintaining strong Locality.

Figure 16: Sensitivity to the maximum evidence weight

\lambda_{\max}

. Recall and Locality achieve their strongest joint balance at

\lambda_{\max}=1

and decline when local evidence is assigned excessive influence.

Method	SQuAD		2Wiki
	Mem. (GiB) $\downarrow$	Lat. (s/document) $\downarrow$	Mem. (GiB) $\downarrow$	Lat. (s/document) $\downarrow$
CD (oracle)	3.568	56.600	22.376	950.085
AnyEdit	6.332	18.754	55.028	4099.473
CD	48.708	451.827	51.956	489.059
D2L	0.438	0.466	11.536	1.841
PLUME	0.438	0.447	11.536	1.832

Appendix

Table 7: Update efficiency on SQuAD and 2Wiki. Memory is the peak additional CUDA memory allocated during context parameterization, and latency is the mean update time per document. The two base-model references are omitted because they incur no update-stage cost. Oracle methods are excluded from ranking; bold and underlined values denote the best and second-best non-oracle results, respectively. Lower values are better, and tied best values are both bolded.

Method	SQuAD		2Wiki
	Mem. (GiB) $\downarrow$	Lat. (s/query) $\downarrow$	Mem. (GiB) $\downarrow$	Lat. (s/query) $\downarrow$
w/ Context (oracle)	0.208	1.972	1.749	1.194
CD (oracle)	0.057	1.663	0.053	4.709
w/o Context	0.047	6.467	0.042	0.924
AnyEdit	0.042	4.789	0.046	4.709
CD	0.026	1.906	0.032	5.718

Appendix

Table 8: Generation efficiency on SQuAD and 2Wiki. Memory is the peak additional CUDA memory allocated during generation, and latency is the mean generation time per query. Oracle methods are excluded from ranking; bold and underlined values denote the best and second-best non-oracle results, respectively. Lower values are better.

Figure 17: Prompt used for D2 Semantic Equivalence evaluation. The rubric determines whether a model output expresses the same answer as the current reference while preserving all required information.

Figure 18: Prompt used for D3 Final-Answer Agreement evaluation. The rubric determines whether the model’s committed final answer agrees with the current reference, including required qualifiers and relations.

Persistent Memory for Language Models

Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.

See all 272 papers

Towards In-Parameter Memory Augmentation for Large Language Models
Oct 6, 2026Haoyu Huang, Zhongwei Xie, Jiaxin Bai +6Memory-Augmented Language Models Persistent Memory for Language Models
The Hong Kong University of Science and Technology · Hong Kong Baptist University
Learning, Fast and Slow: Towards LLMs That Adapt Continually
May 12, 2026Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal +6Continual Learning for LLMs RL for Language Model Reasoning
UC Berkeley · Eragon · Mila +3
MeMo: Memory as a Model
May 14, 2026Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong +6Retrieval-Augmented Generation Memory-Augmented Language Models
Institute of Data Science, National University of Singapore, Singapore · Integrative Sciences and Engineering Programme, NUSGS, Singapore · Agency for Science, Technology, Research (A*STAR), Singapore +6

Oct 6, 2026J/K move · Enter open · S save

Towards In-Parameter Memory Augmentation for Large Language Models

Haoyu Huang, Zhongwei Xie, Jiaxin Bai, Yisen Gao+5

The Hong Kong University of Science and Technology · Hong Kong Baptist University

Memory-Augmented Language Models Persistent Memory for Language Models

Open paper AskPDF arXiv

Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, and interaction experience. In-context learning (ICL) and ICL-based agent harness remain flexible, but they consume context capacity and incur repeated discretized encoding cost that grows with context length. In-parameter memory offers a complementary substrate: reusable memory information is represented in model parameters, adapters, or other parameter-like objects that are composed into the forward pass at inference time. This survey focuses on methods that augment LLMs with such parametric memory at deployment: a memory-bearing parameter object is plugged into the forward pass during inference, whether it is acquired before or during deployment. We organize the landscape with two orthogonal axes: Parameter Placement, which includes Embedding, Attention, FFN layers, or Hybrid when two or more layers are used; and Parameter Acquisition Time, which distinguishes methods whose memory object is acquired during deployment (online) from those acquired before it (offline). We clarify boundaries, conduct comparisons, and discuss open directions in interference, safety, co-design with ICL, and recursive self-improvement.

Abstract

Persistent Memory for Language Models

Explore similar work

Towards In-Parameter Memory Augmentation for Large Language Models

Learning, Fast and Slow: Towards LLMs That Adapt Continually

MeMo: Memory as a Model