Uncovering Context Reliance in Unstructured Knowledge Editing

Zisheng Zhou; Mengqi Zhang; Shiguang Wu; Xiaotian Ye; Chi Zhang; Zhumin Chen; Pengjie Ren

cs.CLFeb 22, 2026

Uncovering Context Reliance in Unstructured Knowledge Editing

Authors: Zisheng Zhou, Mengqi Zhang, Shiguang Wu, Xiaotian Ye, Chi Zhang, Zhumin Chen, Pengjie Ren

Organizations: Shandong University · Beijing University of Posts and Telecommunications

Abstract

Editing Large language models (LLMs) with real-world, unstructured knowledge is essential for correcting and updating their internal parametric knowledge. In this work, we revisit the fundamental next-token prediction (NTP) as a candidate paradigm for unstructured editing. We identify Context Reliance as a critical failure mode of NTP-based approaches, where knowledge acquired from edited text becomes highly dependent on its preceding context, leading to recall failures when that context is absent during inference. This hypothesis is supported by our empirical validation that prepending context during inference recovers knowledge recall. We further theoretically demonstrate that Context Reliance is an inherent consequence of gradient-based optimization, which tends to bind acquired knowledge to a specific aggregated contextual representation. To address this, we propose a simple yet effective COntext-INdependent editing framework (COIN), encouraging model to focus on knowledge within local scope rather than memorizing contextual patterns. Evaluations show that COIN reduces Context Reliance by 45.2% and outperforms strong baselines by 23.6% in editing success rate, highlighting the vital role of mitigating Context Reliance for robust editing.

Figures & tables

Figure 1: (a) Unstructured text containing multiple pieces of knowledge. (b) Edited model tends to recall knowledge appearing later in the text less accurately. (c) Knowledge acquired from edited text is overly dependent on its preceding context, and omitting that context during inference leads to failed recall.

Figure 2: Performance decline of different NTP-based methods on AKEW dataset as the position of knowledge moves later in the text. The x-axis represents the sequential order of knowledge within the original text, where position ‘1’ corresponds to questions targeting knowledge positioned at the beginning, and position ‘

>

6’ corresponds to knowledge located towards the end. The y-axis represents the corresponding accuracy.

Figure 3: Probability of predicting answer with and without preceding context. The results show that appending context increases probability of answer.

Figure 4: Performance comparison between with and without knowledge splitting (KS).

Figure 5: Performance comparison between with and without paraphrasing (Para.).

Figure 6: The framework of COIN. (a) Illustrates the Context Alignment Loss, which mitigates Context Reliance by aligning distributions conditioned on global and local contexts. (b) Illustrates the Knowledge Consistency Loss, which preserves the general abilities of edited model.

Model	Method	AKEW-Com.				AKEW-QA				UnKEBench
Model	Method	Prec.	Rec.	F1	BERT	Prec.	Rec.	F1	BERT	Prec.	Rec.	F1	BERT
Llama3	BASE	21.61	16.41	17.58	41.64	35.44	42.47	36.67	50.61	17.01	22.91	17.49	40.28
	AnyEdit	23.68	18.65	19.66	42.52	33.60	42.28	35.33	50.27	12.61	19.68	13.59	33.41
	UnKE	22.40	20.39	19.24	40.85	32.32	42.23	34.40	50.59	15.95	24.04	17.05	38.64
	AlphaEdit-D	39.44	35.76	34.65	56.09	36.94	54.24	40.06	55.01	30.81	39.66	31.57	52.97
	FT	53.15	58.13	51.75	67.07	38.44	60.92	42.76	57.10	31.54	57.30	36.00	53.76

Table 1: Performance comparison of all methods on AKEW and UnKEBench using instruct-tuned models, evaluated in terms of BERT Score (%) and Precision, Recall, F1 for ROUGE-L (%). The best results are in bold , and the second best are underlined . Notably , we evaluate edited model using questions derived from editing text, more challenging and practical than repeating text.

Figure 7: Performance comparison between COIN and baselines. The performance drop from position ‘1’ to ‘

>

6’ is annotated. COIN clearly mitigates this degradation.

Model	Alignment	Consistency	AKEW-Com.				AKEW-QA				UnKEBench
Model	Alignment	Consistency	Prec.	Rec.	F1	BERT	Prec.	Rec.	F1	BERT	Prec.	Rec.	F1	BERT
Llama3	✓	✓	60.52	67.24	60.21	72.76	41.78	66.86	46.97	59.63	39.62	64.60	44.48	60.81
	-	✓	54.30	56.26	51.83	66.68	40.77	62.09	44.96	57.93	36.88	58.80	40.72	58.02
	✓	-	56.97	71.12	58.63	72.37	37.32	63.33	42.46	57.22	33.68	63.64	39.22	56.30
	-	-	53.15	58.13	51.75	67.07	38.44	60.92	42.76	57.10	31.54	57.30	36.00	53.76
Qwen2.5	✓	✓	50.45	64.65	52.53	67.31	40.53	73.10	47.51	62.69	25.46	75.67	35.25	48.89

Table 2: Ablation studies on COIN in terms of ROUGE Scores (%) and BERT Score (%).

Figure 8: Performance of batch editing on AKEW dataset (left) and GLUE benchmark (right).

Method	Avg.	2-hops	3-hops	4-hops
Llama3	29.22	20.29	38.17	29.17
ROME	42.01	42.41	46.48	34.43
MEMIT	33.80	27.14	42.02	31.37
AlphaEdit	40.21	36.04	48.06	34.48
AnyEdit	32.58	24.38	41.87	30.86
UnKE	34.01	28.75	41.58	30.41

Table 3: Performance of multi-hop structured knowledge editing on MQuAKE .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Value
Question	Marek Edelman worked in the city of what?
Text	Marek Edelman, a Polish-Jewish political and social activist, spent a significant portion of his life working in the bustling city of London. Known for his involvement in the Warsaw Ghetto Uprising during World War II, Edelman later moved to London where he continued his activism and advocacy for human rights. He also worked as a cardiologist, using his medical expertise to help others in need. Despite being far from his home country, Edelman’s impact and legacy were felt both in London and around the world.
Completions	C1: Marek Edelman was a [Polish-Jewish political and social activist].
	C2: Marek Edelman spent a significant portion of his life [working in the bustling city of London].
	C3: Marek Edelman was known for his involvement in [the Warsaw Ghetto Uprising during World War II].
	…

Appendix

Table 4: An example of AKEW dataset.

Property	Value
Question	What is Maurice Le Boucher’s profession and why is he highly sought-after in the community?
Text	Maurice Le Boucher is a skilled musician who has been playing the organ for over 20 years. He has performed in various churches and concert halls across the country, showcasing his talent and passion for music. In fact, he was recently hired as the organist for St. Mary’s Church in his hometown, where he plays every Sunday during mass. Maurice’s dedication to his craft and his ability to captivate audiences with his music make him a highly sought-after organist in the community.
QAs	Q1: How long has Maurice Le Boucher been playing the organ?
	A1: Over 20 years.
	Q2: Where has Maurice Le Boucher performed as an organist?
	A2: Various churches and concert halls across the country.

Appendix

Table 5: An example of UnKEBench dataset.

Property	Value
Edit Requests	R1: {Fernando Santos} is a citizen of Portugal $\rightarrow$ United Kingdom .
Edit Requests	R2: The name of the current head of state in {United Kingdom} is Elizabeth II $\rightarrow$ Emmerson Mnangagwa .
Questions	Q1: Who is the head of state of the country where Fernando Santos hold a citizenship?
	Q2: In which country is Fernando Santos a citizen and who is the head of state?
	Q3: Which person holds the position of head of state in the country from which Fernando Santos holds citizenship?
Original Answer	Marcelo Rebelo de Sousa

Appendix

Table 6: An example of MQuAKE dataset.

Figure 9: Performance comparison of different methods on UnKEBench dataset. The x-axis represents the position of knowledge in the text, and the y-axis represents the corresponding accuracy. Results indicate that Context Reliance is a prevalent issue in NTP-based unstructured knowledge editing.

Figure 10: Context Reliance across different model scales. Results indicate that as model size increases, the Context Reliance issue becomes more severe.

Figure 11: Impact of training steps on Context Reliance.

Model	Method	AKEW-Com.				AKEW-QA				UnKEBench
Model	Method	Prec.	Rec.	F1	BERT	Prec.	Rec.	F1	BERT	Prec.	Rec.	F1	BERT
Llama3	BASE	25.80	20.95	21.83	41.68	37.44	40.74	37.07	49.78	23.36	24.64	21.73	46.53
	AnyEdit	25.97	21.35	22.04	42.05	37.84	42.59	37.99	51.29	15.86	30.13	18.46	37.46
	UnKE	26.06	21.44	22.19	41.79	35.69	44.36	37.26	52.16	19.71	26.93	20.48	43.47
	AlphaEdit-D	36.59	38.44	33.88	52.17	38.82	48.62	39.31	53.04	34.66	38.12	33.59	57.70
	FT	58.79	70.02	59.35	70.53	46.51	67.02	50.18	61.34	50.66	72.20	54.55	70.68

Appendix

Table 7: Performance comparison of all methods on AKEW and UnKEBench using pretrained models, evaluated in terms of BERT Score (%) and Precision, Recall, F1 for ROUGE-L (%). The best results are in bold , and the second best are underlined .

Method	Prob. (%)	Match (%)
FT (base)	71.66	5.63
FT (first layer)	47.02	0.24
FT (optimize target)	71.27	0.43

Appendix

Table 8: Additional experimental results on MQuAKE dataset.

Figure 12: Performance of COIN with different window length

k

on AKEW dataset.

Figure 13: Comparison of outputs from Llama3-8B-Instruct on an example from the AKEW dataset, showcasing the performance of AnyEdit, AlphaEdit, FT, LoRA, and COIN. Green text indicates a correct answer, while red text highlights an incorrect one.

Figure 14: Comparison of outputs from Llama3-8B-Instruct on an example from the UnKEBench dataset, showcasing the performance of FT, LoRA, and COIN. Green text indicates correct answers, while red text highlights incorrect or irrelevant content.

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 84 papers

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective
Sep 27, 2026Rui Liu, Chenheng Zhang, Haoxuan Li +1Knowledge Editing Continual Learning for LLMs
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing
Oct 2, 2026Ding Wu, Ye Zhang, Haoyu Wang +1Knowledge Editing Factual Knowledge in Language Models
Georgia Institute of Technology · University of Pennsylvania · University at Albany +1
Towards Reliable, Generalizable, and Specific In-Context Knowledge Editing via Multi-Objective Reinforcement Learning
Aug 25, 2026Xuzhong Wang, Maiqi Jiang, Tejal Nair +3Knowledge Editing Language Model Editing
College of William and Mary · Tribhuvan University

Sep 27, 2026J/K move · Enter open · S save

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

Rui Liu, Chenheng Zhang, Haoxuan Li, Zhouchen Lin

State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

Knowledge Editing Continual Learning for LLMs Language Model Editing

Open paper AskPDF arXiv

Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit

Abstract

Knowledge Editing

Explore similar work

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing

Towards Reliable, Generalizable, and Specific In-Context Knowledge Editing via Multi-Objective Reinforcement Learning