EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Hongru Cai; Ran Wei; Wenjie Wang; Chengfa Wu; Ning Song; Yongqi Li; Wenjie Li

cs.CLOct 7, 2026

EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Authors: Hongru Cai, Ran Wei, Wenjie Wang, Chengfa Wu, Ning Song, Yongqi Li, Wenjie Li

Organizations: The Hong Kong Polytechnic University · Hangzhou Diagens Biotechnology Co., Ltd · University of Science and Technology of China

Abstract

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts. We propose EngramEdit for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared n-gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly three times the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates.

Figures & tables

Figure 1 : Conditional memory and knowledge updating with EngramEdit . (a) Input

n

-grams look up learned embeddings for backbone computation. (b) Disabling memory in DeepSeek Engram causes larger relative performance drops on factual-knowledge benchmarks [ 3 ] . (c) EngramEdit computes target memory representations and jointly updates the corresponding

n

-gram embeddings, with stronger penalties on frequently reused embeddings.

Figure 2 : Overview of EngramEdit . We compute target memory representations that predict updated facts across expressions, map expressions to their activated

n

-grams, and jointly update the embeddings to match these targets with stronger penalties for shorter or more frequent

n

-grams.

Method	CounterFact						ZsRE
Method	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Flu. $\bm{\uparrow}$	Cons. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
Pre-edited	$\text{9.5}{}_{\scriptscriptstyle\pm\text{1.28}}$	$\text{11.2}{}_{\scriptscriptstyle\pm\text{1.19}}$	$\text{86.8}{}_{\scriptscriptstyle\pm\text{0.94}}$	35.8	$\text{544.0}{}_{\scriptscriptstyle\pm\text{1.67}}$	$\text{15.0}{}_{\scriptscriptstyle\pm\text{0.37}}$	$\text{36.9}{}_{\scriptscriptstyle\pm\text{1.30}}$	$\text{36.5}{}_{\scriptscriptstyle\pm\text{1.29}}$	$\text{41.6}{}_{\scriptscriptstyle\pm\text{1.19}}$	38.3
FT	$\text{94.9}{}_{\scriptscriptstyle\pm\text{0.96}}$	$\text{63.6}{}_{\scriptscriptstyle\pm\text{1.83}}$	$\text{43.8}{}_{\scriptscriptstyle\pm\text{1.63}}$	67.4	$\text{505.1}{}_{\scriptscriptstyle\pm\text{2.04}}$	$\text{13.3}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{76.7}{}_{\scriptscriptstyle\pm\text{1.13}}$	$\text{76.8}{}_{\scriptscriptstyle\pm\text{1.15}}$	$\text{43.1}{}_{\scriptscriptstyle\pm\text{1.27}}$	65.5
FT-L	$\text{15.3}{}_{\scriptscriptstyle\pm\text{1.58}}$	$\text{12.5}{}_{\scriptscriptstyle\pm\text{1.24}}$	$\text{\lx@text@underline{83.3}}{}_{\scriptscriptstyle\pm\text{1.11}}$	37.0	$\text{542.0}{}_{\scriptscriptstyle\pm\text{1.70}}$	$\text{14.7}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{58.0}{}_{\scriptscriptstyle\pm\text{1.47}}$	$\text{54.4}{}_{\scriptscriptstyle\pm\text{1.48}}$	$\text{45.8}{}_{\scriptscriptstyle\pm\text{1.24}}$	52.7
AdaLoRA	$\text{61.1}{}_{\scriptscriptstyle\pm\text{2.13}}$	$\text{27.5}{}_{\scriptscriptstyle\pm\text{1.55}}$	$\text{51.1}{}_{\scriptscriptstyle\pm\text{1.35}}$	46.6	$\text{544.3}{}_{\scriptscriptstyle\pm\text{1.57}}$	$\text{12.9}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{70.9}{}_{\scriptscriptstyle\pm\text{1.40}}$	$\text{64.7}{}_{\scriptscriptstyle\pm\text{1.46}}$	$\text{\lx@text@underline{46.8}}{}_{\scriptscriptstyle\pm\text{1.27}}$	60.8
UnKE	$\text{84.5}{}_{\scriptscriptstyle\pm\text{1.59}}$	$\text{58.0}{}_{\scriptscriptstyle\pm\text{1.88}}$	$\text{47.5}{}_{\scriptscriptstyle\pm\text{1.61}}$	63.3	$\text{546.5}{}_{\scriptscriptstyle\pm\text{1.64}}$	$\text{15.0}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{\lx@text@underline{86.9}}{}_{\scriptscriptstyle\pm\text{1.13}}$	$\text{\lx@text@underline{79.0}}{}_{\scriptscriptstyle\pm\text{1.33}}$	$\text{{49.9}}{}_{\scriptscriptstyle\pm\text{1.32}}$	71.9

Table 1 : Editing results on CounterFact and ZsRE . MFT-S and MFT-A denote Memory-FT (Subject) and (All Tokens). Eff. = Efficacy, Gen. = Generalization, Spec. = Specificity, Util. = Utility, Flu. = Fluency, Cons. = Consistency. Subscripts give 95% confidence-interval half-widths. Bold and underlining mark the best and second-best editing scores.

Figure 3 : Editing results on MQuAKE . (a) Standard (left) and CoT (right) prompting with 95% confidence intervals. (b) CoT accuracy by hop. AdaL and MoE denote AdaLoRA and MoEEdit.

Figure 4 : General capabilities during sequential CounterFact editing . (a) Mean F1 across six tasks in the editing sequence. (b) Task-level F1 changes from pre-edit scores after 5,000 edits, in percentage points (pp). The mean $|\Delta|$ averages the absolute task-level changes.

Variant	CounterFact						ZsRE
Variant	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Flu. $\bm{\uparrow}$	Cons. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
EngramEdit	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{{97.0}}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{{85.2}}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.9	$\text{565.2}{}_{\scriptscriptstyle\pm\text{1.10}}$	$\text{\lx@text@underline{16.1}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{{97.3}}{}_{\scriptscriptstyle\pm\text{0.40}}$	$\text{\lx@text@underline{93.7}}{}_{\scriptscriptstyle\pm\text{0.79}}$	$\text{{38.3}}{}_{\scriptscriptstyle\pm\text{1.18}}$	76.4
w/o Joint	$\text{74.5}{}_{\scriptscriptstyle\pm\text{1.91}}$	$\text{72.8}{}_{\scriptscriptstyle\pm\text{1.74}}$	$\text{84.8}{}_{\scriptscriptstyle\pm\text{0.93}}$	77.4	$\text{{568.0}}{}_{\scriptscriptstyle\pm\text{1.00}}$	$\text{15.3}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{57.4}{}_{\scriptscriptstyle\pm\text{1.55}}$	$\text{58.3}{}_{\scriptscriptstyle\pm\text{1.57}}$	$\text{35.6}{}_{\scriptscriptstyle\pm\text{1.17}}$	50.4
w/o Freq.	$\text{\lx@text@underline{99.4}}{}_{\scriptscriptstyle\pm\text{0.32}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{{85.2}}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.8	$\text{564.9}{}_{\scriptscriptstyle\pm\text{1.22}}$	$\text{\lx@text@underline{16.1}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{\lx@text@underline{96.0}}{}_{\scriptscriptstyle\pm\text{0.52}}$	$\text{92.6}{}_{\scriptscriptstyle\pm\text{0.84}}$	$\text{\lx@text@underline{37.3}}{}_{\scriptscriptstyle\pm\text{1.19}}$	75.3
w/o Len.	$\text{99.3}{}_{\scriptscriptstyle\pm\text{0.38}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.70}}$	$\text{\lx@text@underline{85.1}}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.7	$\text{565.4}{}_{\scriptscriptstyle\pm\text{1.09}}$	$\text{{16.2}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{95.7}{}_{\scriptscriptstyle\pm\text{0.55}}$	$\text{93.4}{}_{\scriptscriptstyle\pm\text{0.75}}$	$\text{37.2}{}_{\scriptscriptstyle\pm\text{1.17}}$	75.4
w/o Reg.	$\text{98.8}{}_{\scriptscriptstyle\pm\text{0.47}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.72}}$	$\text{84.6}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.4	$\text{{568.0}}{}_{\scriptscriptstyle\pm\text{1.02}}$	$\text{15.5}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{\lx@text@underline{96.0}}{}_{\scriptscriptstyle\pm\text{0.54}}$	$\text{{94.0}}{}_{\scriptscriptstyle\pm\text{0.75}}$	$\text{37.2}{}_{\scriptscriptstyle\pm\text{1.19}}$	75.7

Table 2 : Ablations of joint update allocation (Joint), reuse-based regularization (Reg.),

n

-gram corpus frequency (Freq.) and length (Len.) weighting, and generated expressions (Expr.).

Figure 5 : Generated expressions and memory use on CounterFact . (a) Editors use the same generated expressions per fact. (b) Generated expressions are used for target computation, memory mapping, both, or neither. Coverage is the percentage of held-out paraphrases activating fact-related updated embeddings. (c) Disabling no updates (None), fact-related updates (Related), matched random updates (Random), or all updates (All) during inference.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6 : Expression-generation prompt for CounterFact and atomic MQuAKE edits. [SUBJ] marks the subject; angle-bracketed fields are filled from the edit request.

Field	Example
CounterFact
Subject and relation	Danielle Darrieux; mother tongue
Requested change	French $\rightarrow$ English
Edit prompt	The mother tongue of Danielle Darrieux is
Held-out paraphrase	… Danielle Darrieux spoke the language
Unrelated prompt	The native language of Montesquieu is

Appendix

Table 3 : Editing examples from CounterFact, ZsRE, and MQuAKE, with simplified formatting. The ellipsis omits an unrelated prefix from the CounterFact paraphrase.

Figure 7 : Sequential editing on CounterFact (a–c) and ZsRE (d–f) up to 5,000 edits. Shading shows pointwise case-level 95% confidence intervals. MFT-S denotes Memory-FT (Subject).

Method	Standard $\bm{\uparrow}$			CoT $\bm{\uparrow}$
Method	2-hop	3-hop	4-hop	2-hop	3-hop	4-hop
FT	7.93 [6.50, 9.65]	2.11 [1.42, 3.12]	7.82 [6.08, 10.00]	12.60 [10.79, 14.66]	3.61 [2.67, 4.86]	9.74 [7.79, 12.11]
FT-L	1.59 [1.01, 2.49]	1.67 [1.07, 2.60]	4.53 [3.24, 6.29]	1.06 [0.61, 1.84]	0.97 [0.54, 1.73]	1.78 [1.05, 3.03]
AdaLoRA	1.50 [0.94, 2.39]	2.64 [1.86, 3.74]	3.84 [2.67, 5.50]	1.15 [0.67, 1.95]	1.23 [0.74, 2.06]	2.33 [1.46, 3.70]
UnKE	4.14 [3.13, 5.46]	4.23 [3.20, 5.56]	6.17 [4.65, 8.16]	5.46 [4.28, 6.94]	4.58 [3.51, 5.95]	9.60 [7.67, 11.96]
MoEEdit	3.00 [2.15, 4.16]	2.64 [1.86, 3.74]	3.29 [2.22, 4.85]	0.62 [0.30, 1.27]	0.26 [0.09, 0.77]	0.27 [0.08, 0.99]

Appendix

Table 4 : MQuAKE accuracy (%) by hop with standard and CoT prompts. Brackets give Wilson 95% confidence intervals. Bold and underlining mark the best and second-best scores.

Method	$\bm{\mathrm{C}\!\rightarrow\!\mathrm{W}}$	$\bm{\mathrm{W}\!\rightarrow\!\mathrm{C}}$	Post–Pre	Post Spec. $\bm{\uparrow}$	Stable Corr. $\bm{\uparrow}$
FT-L	$\text{0.62}{}_{\scriptscriptstyle\pm\text{0.19}}$	$\text{4.84}{}_{\scriptscriptstyle\pm\text{0.57}}$	+4.23	$\text{45.76}{}_{\scriptscriptstyle\pm\text{1.24}}$	$\text{94.54}{}_{\scriptscriptstyle\pm\text{0.60}}$
AdaLoRA	$\text{2.25}{}_{\scriptscriptstyle\pm\text{0.33}}$	$\text{7.51}{}_{\scriptscriptstyle\pm\text{0.71}}$	+5.26	$\text{46.79}{}_{\scriptscriptstyle\pm\text{1.27}}$	$\text{90.24}{}_{\scriptscriptstyle\pm\text{0.79}}$
UnKE	$\text{2.95}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{11.26}{}_{\scriptscriptstyle\pm\text{0.86}}$	+8.31	$\text{49.85}{}_{\scriptscriptstyle\pm\text{1.32}}$	$\text{85.79}{}_{\scriptscriptstyle\pm\text{0.93}}$
MoEEdit	$\text{2.30}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{5.14}{}_{\scriptscriptstyle\pm\text{0.56}}$	+2.84	$\text{44.37}{}_{\scriptscriptstyle\pm\text{1.26}}$	$\text{92.56}{}_{\scriptscriptstyle\pm\text{0.67}}$
MFT-S	$\text{5.06}{}_{\scriptscriptstyle\pm\text{0.51}}$	$\text{0.40}{}_{\scriptscriptstyle\pm\text{0.12}}$	-4.67	$\text{36.87}{}_{\scriptscriptstyle\pm\text{1.17}}$	$\text{94.54}{}_{\scriptscriptstyle\pm\text{0.54}}$
MFT-A	$\text{9.67}{}_{\scriptscriptstyle\pm\text{0.72}}$	$\text{1.84}{}_{\scriptscriptstyle\pm\text{0.30}}$	-7.83	$\text{33.71}{}_{\scriptscriptstyle\pm\text{1.13}}$	$\text{88.48}{}_{\scriptscriptstyle\pm\text{0.80}}$

Appendix

Table 5 : ZsRE correctness transitions after 2,000 edits. C and W denote correct and wrong predictions; Post–Pre is the Specificity change in percentage points. Stable Corr. measures unchanged correctness states. Values are case-averaged percentages with 95% CI half-widths in subscripts.

Figure 8 : Task-level general capabilities during 5,000 sequential CounterFact edits. F1 is measured on 100 examples per task. Dotted lines mark pre-edit scores.

Method	CounterFact				ZsRE
Method	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
Original edit prompt only
MoEEdit	$\text{\lx@text@underline{99.1}}{}_{\scriptscriptstyle\pm\text{0.41}}$	$\text{\lx@text@underline{67.1}}{}_{\scriptscriptstyle\pm\text{1.69}}$	$\text{63.5}{}_{\scriptscriptstyle\pm\text{1.28}}$	76.6	$\text{\lx@text@underline{85.1}}{}_{\scriptscriptstyle\pm\text{0.91}}$	$\text{\lx@text@underline{78.7}}{}_{\scriptscriptstyle\pm\text{1.21}}$	$\text{{44.4}}{}_{\scriptscriptstyle\pm\text{1.26}}$	69.4
MFT-S	$\text{99.0}{}_{\scriptscriptstyle\pm\text{0.45}}$	$\text{59.1}{}_{\scriptscriptstyle\pm\text{1.92}}$	$\text{\lx@text@underline{85.2}}{}_{\scriptscriptstyle\pm\text{0.90}}$	81.1	$\text{66.9}{}_{\scriptscriptstyle\pm\text{1.35}}$	$\text{56.5}{}_{\scriptscriptstyle\pm\text{1.53}}$	$\text{36.9}{}_{\scriptscriptstyle\pm\text{1.17}}$	53.4
MFT-A	$\text{93.9}{}_{\scriptscriptstyle\pm\text{1.04}}$	$\text{40.7}{}_{\scriptscriptstyle\pm\text{1.75}}$	$\text{66.6}{}_{\scriptscriptstyle\pm\text{1.12}}$	67.1	$\text{62.8}{}_{\scriptscriptstyle\pm\text{1.49}}$	$\text{51.3}{}_{\scriptscriptstyle\pm\text{1.57}}$	$\text{33.7}{}_{\scriptscriptstyle\pm\text{1.13}}$	49.3
EngramEdit	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.32}}$	$\text{{75.0}}{}_{\scriptscriptstyle\pm\text{1.74}}$	$\text{{85.6}}{}_{\scriptscriptstyle\pm\text{0.91}}$	86.7	$\text{{95.1}}{}_{\scriptscriptstyle\pm\text{0.62}}$	$\text{{83.4}}{}_{\scriptscriptstyle\pm\text{1.29}}$	$\text{\lx@text@underline{38.5}}{}_{\scriptscriptstyle\pm\text{1.18}}$	72.3

Appendix

Table 6 : Editing with and without generated expressions on CounterFact and ZsRE over 2,000 sequential edits. Subscripts give 95% CI half-widths. Bold and underlining mark the best and second-best scores within each block.

# Expr.	CounterFact						ZsRE
# Expr.	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Flu. $\bm{\uparrow}$	Cons. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
0	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.32}}$	$\text{75.0}{}_{\scriptscriptstyle\pm\text{1.74}}$	$\text{{85.6}}{}_{\scriptscriptstyle\pm\text{0.91}}$	86.7	$\text{562.6}{}_{\scriptscriptstyle\pm\text{1.19}}$	$\text{15.7}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{95.1}{}_{\scriptscriptstyle\pm\text{0.62}}$	$\text{83.4}{}_{\scriptscriptstyle\pm\text{1.29}}$	$\text{{38.5}}{}_{\scriptscriptstyle\pm\text{1.18}}$	72.3
2	$\text{\lx@text@underline{99.4}}{}_{\scriptscriptstyle\pm\text{0.34}}$	$\text{95.2}{}_{\scriptscriptstyle\pm\text{0.83}}$	$\text{\lx@text@underline{85.3}}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.3	$\text{\lx@text@underline{566.2}}{}_{\scriptscriptstyle\pm\text{1.11}}$	$\text{{16.2}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{96.2}{}_{\scriptscriptstyle\pm\text{0.50}}$	$\text{91.8}{}_{\scriptscriptstyle\pm\text{0.89}}$	$\text{37.7}{}_{\scriptscriptstyle\pm\text{1.10}}$	75.2
4	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{97.0}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.9	$\text{565.2}{}_{\scriptscriptstyle\pm\text{1.10}}$	$\text{\lx@text@underline{16.1}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{{97.3}}{}_{\scriptscriptstyle\pm\text{0.40}}$	$\text{\lx@text@underline{93.7}}{}_{\scriptscriptstyle\pm\text{0.79}}$	$\text{\lx@text@underline{38.3}}{}_{\scriptscriptstyle\pm\text{1.18}}$	76.4
6	$\text{\lx@text@underline{99.4}}{}_{\scriptscriptstyle\pm\text{0.34}}$	$\text{{97.3}}{}_{\scriptscriptstyle\pm\text{0.64}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	94.0	$\text{{567.5}}{}_{\scriptscriptstyle\pm\text{1.03}}$	$\text{{16.2}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{96.5}{}_{\scriptscriptstyle\pm\text{0.47}}$	$\text{\lx@text@underline{93.7}}{}_{\scriptscriptstyle\pm\text{0.77}}$	$\text{36.8}{}_{\scriptscriptstyle\pm\text{1.17}}$	75.7
8	$\text{99.1}{}_{\scriptscriptstyle\pm\text{0.41}}$	$\text{\lx@text@underline{97.2}}{}_{\scriptscriptstyle\pm\text{0.67}}$	$\text{84.9}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.7	$\text{566.0}{}_{\scriptscriptstyle\pm\text{1.07}}$	$\text{16.0}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.44}}$	$\text{{94.2}}{}_{\scriptscriptstyle\pm\text{0.74}}$	$\text{38.0}{}_{\scriptscriptstyle\pm\text{1.17}}$	76.3

Appendix

Table 7 : Results with different numbers of generated expressions after 2,000 edits. # Expr. excludes the original prompt. Subscripts give reported CI half-widths; bold and underlining mark best and second-best scores.

Figure 9 : Editable

n

-gram lengths on CounterFact. The frame marks the default lengths 2, 3, and 4;

n=1

denotes single-token embedding updates. Small

\pm

values give CI half-widths.

Parameter	Value	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
Reuse-based regularization $\lambda_{\mathrm{reuse}}$	0	$\text{98.8}{}_{\scriptscriptstyle\pm\text{0.49}}$	$\text{96.5}{}_{\scriptscriptstyle\pm\text{0.76}}$	$\text{84.7}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.3
	0.03	$\text{99.3}{}_{\scriptscriptstyle\pm\text{0.37}}$	$\text{96.2}{}_{\scriptscriptstyle\pm\text{0.76}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.6
	0.05	$\text{99.5}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{97.0}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.9
	0.075	$\text{99.6}{}_{\scriptscriptstyle\pm\text{0.29}}$	$\text{96.4}{}_{\scriptscriptstyle\pm\text{0.73}}$	$\text{85.3}{}_{\scriptscriptstyle\pm\text{0.91}}$	93.8
	0.1	$\text{99.6}{}_{\scriptscriptstyle\pm\text{0.28}}$	$\text{96.6}{}_{\scriptscriptstyle\pm\text{0.70}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.8
Ridge regularization $\lambda_{\mathrm{ridge}}$	0	$\text{99.5}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{96.6}{}_{\scriptscriptstyle\pm\text{0.71}}$	$\text{85.1}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.7

Appendix

Table 8 : Sensitivity to regularization and clamp factor after 2,000 CounterFact edits. Only the listed parameter changes. Scores are percentages; subscripts give reported CI half-widths. Bold parameter values mark the defaults.

Figure 10 : Disabling memory updates on CounterFact. (a) Mean edit margins before and after disabling. Positive values favor revised answers; negative values favor original answers. (b) Outcomes of the 5,850 prompts successful before disabling. Parentheses give the mean number of disabled updates per case.

Figure 11 : Memory activation and editing outcomes with all updates enabled. Groups indicate activation of embeddings updated for the current edit only, other edits only, both, or neither. (a) Within-group paraphrase success rates and each group’s share of all paraphrases. (b) Success-to-failure rates among previously successful neighborhood queries in each group and each group’s share of all regressions.

Figure 12 : Cross-edit sharing on CounterFact. (a) Sharing rates among updated

n

-grams and edited facts. (b) Final editing results for facts with or without shared

n

-grams.

Explore similar work

Jun 21, 2026cs.CL

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining. However, existing methods still suffer from performance degradation in batch knowledge editing. We identify that semantic representation entanglement, such as overlapping concepts and shared syntactic patterns, accumulates interference in the representation space and reduces editing precision. To bridge this gap, in this paper, we propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs by constructing a general semantic subspace and enforcing orthogonal constraints on edit vectors, effectively decoupling semantic entanglement. Furthermore, we introduce a gated non-linear representation head to enable adaptive learning of editing locations and precise control over knowledge injection. Extensive experiments show that ORE outperforms existing methods and achieves superior performance in cross-lingual knowledge editing scenarios. We release our code at https://github.com/YVVH/ORE.

Wenhao Yu, Zhicong Lu, Bo Lv +4

School of Computer Science and Technology, Tianjin University · Kexin Technology · University of Chinese Academy of Sciences +2

Multimodal Knowledge Editing Knowledge Graph Embeddings Semantic Representations Cross-Lingual Consistency Entanglement Orthogonality

Ask PDF

Jun 22, 2026cs.LG

Exposing the Illusion of Erasure in Knowledge Editing for LLMs

Knowledge Editing (KE) has emerged as a frontier for updating specific facts in LLMs without costly retraining, but its reliability and underlying mechanisms remain poorly understood. In this work, we examine KE from an adversarial elicitation perspective, revealing that edited knowledge is often not fully erased and continues to surface, with consistent failures observed across diverse model architectures. To explain this behavior, we conduct a mechanistic analysis of popular KE methods. We show that low-rank updates do not overwrite existing knowledge but instead redistribute it within the model's representation space. Furthermore, we find that these methods act as targeted suppression mechanisms that reduce the likelihood of expressing original facts, rather than removing them from the model. Analysis of the loss landscape reveals that edited knowledge lies in narrow, anisotropic regions that are highly sensitive to perturbations, making them highly vulnerable to indirect prompting and adversarial attacks. By exposing these profound architectural vulnerabilities, our work proves that KE algorithms are inherently bypassable and motivates a fundamental reevaluation of how we deploy post-hoc updates in several LLM applications.

Advik Raj Basani, Anshuman Chhabra

Birla Institute of Technology and Science, Goa · University of South Florida

Multimodal Knowledge Editing Concept Erasure Large Language Model Unlearning Illusions Low-Rank Structure Deep Learning Architectures

Ask PDF

Sep 27, 2026cs.CL

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit

Rui Liu, Chenheng Zhang, Haoxuan Li +1

State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

Multimodal Knowledge Editing Edit Diffusion-Based Image Editing Forgetting Manifold Perspective

Ask PDF