EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Hongru Cai; Ran Wei; Wenjie Wang; Chengfa Wu; Ning Song; Yongqi Li; Wenjie Li

cs.CLOct 7, 2026

EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Authors: Hongru Cai, Ran Wei, Wenjie Wang, Chengfa Wu, Ning Song, Yongqi Li, Wenjie Li

Organizations: The Hong Kong Polytechnic University · Hangzhou Diagens Biotechnology Co., Ltd · University of Science and Technology of China

Abstract

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts. We propose EngramEdit for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared n-gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly three times the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates.

Figures & tables

Figure 1 : Conditional memory and knowledge updating with EngramEdit . (a) Input

n

-grams look up learned embeddings for backbone computation. (b) Disabling memory in DeepSeek Engram causes larger relative performance drops on factual-knowledge benchmarks [ 3 ] . (c) EngramEdit computes target memory representations and jointly updates the corresponding

n

-gram embeddings, with stronger penalties on frequently reused embeddings.

Figure 2 : Overview of EngramEdit . We compute target memory representations that predict updated facts across expressions, map expressions to their activated

n

-grams, and jointly update the embeddings to match these targets with stronger penalties for shorter or more frequent

n

-grams.

Method	CounterFact						ZsRE
Method	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Flu. $\bm{\uparrow}$	Cons. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
Pre-edited	$\text{9.5}{}_{\scriptscriptstyle\pm\text{1.28}}$	$\text{11.2}{}_{\scriptscriptstyle\pm\text{1.19}}$	$\text{86.8}{}_{\scriptscriptstyle\pm\text{0.94}}$	35.8	$\text{544.0}{}_{\scriptscriptstyle\pm\text{1.67}}$	$\text{15.0}{}_{\scriptscriptstyle\pm\text{0.37}}$	$\text{36.9}{}_{\scriptscriptstyle\pm\text{1.30}}$	$\text{36.5}{}_{\scriptscriptstyle\pm\text{1.29}}$	$\text{41.6}{}_{\scriptscriptstyle\pm\text{1.19}}$	38.3
FT	$\text{94.9}{}_{\scriptscriptstyle\pm\text{0.96}}$	$\text{63.6}{}_{\scriptscriptstyle\pm\text{1.83}}$	$\text{43.8}{}_{\scriptscriptstyle\pm\text{1.63}}$	67.4	$\text{505.1}{}_{\scriptscriptstyle\pm\text{2.04}}$	$\text{13.3}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{76.7}{}_{\scriptscriptstyle\pm\text{1.13}}$	$\text{76.8}{}_{\scriptscriptstyle\pm\text{1.15}}$	$\text{43.1}{}_{\scriptscriptstyle\pm\text{1.27}}$	65.5
FT-L	$\text{15.3}{}_{\scriptscriptstyle\pm\text{1.58}}$	$\text{12.5}{}_{\scriptscriptstyle\pm\text{1.24}}$	$\text{\lx@text@underline{83.3}}{}_{\scriptscriptstyle\pm\text{1.11}}$	37.0	$\text{542.0}{}_{\scriptscriptstyle\pm\text{1.70}}$	$\text{14.7}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{58.0}{}_{\scriptscriptstyle\pm\text{1.47}}$	$\text{54.4}{}_{\scriptscriptstyle\pm\text{1.48}}$	$\text{45.8}{}_{\scriptscriptstyle\pm\text{1.24}}$	52.7
AdaLoRA	$\text{61.1}{}_{\scriptscriptstyle\pm\text{2.13}}$	$\text{27.5}{}_{\scriptscriptstyle\pm\text{1.55}}$	$\text{51.1}{}_{\scriptscriptstyle\pm\text{1.35}}$	46.6	$\text{544.3}{}_{\scriptscriptstyle\pm\text{1.57}}$	$\text{12.9}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{70.9}{}_{\scriptscriptstyle\pm\text{1.40}}$	$\text{64.7}{}_{\scriptscriptstyle\pm\text{1.46}}$	$\text{\lx@text@underline{46.8}}{}_{\scriptscriptstyle\pm\text{1.27}}$	60.8
UnKE	$\text{84.5}{}_{\scriptscriptstyle\pm\text{1.59}}$	$\text{58.0}{}_{\scriptscriptstyle\pm\text{1.88}}$	$\text{47.5}{}_{\scriptscriptstyle\pm\text{1.61}}$	63.3	$\text{546.5}{}_{\scriptscriptstyle\pm\text{1.64}}$	$\text{15.0}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{\lx@text@underline{86.9}}{}_{\scriptscriptstyle\pm\text{1.13}}$	$\text{\lx@text@underline{79.0}}{}_{\scriptscriptstyle\pm\text{1.33}}$	$\text{{49.9}}{}_{\scriptscriptstyle\pm\text{1.32}}$	71.9

Table 1 : Editing results on CounterFact and ZsRE . MFT-S and MFT-A denote Memory-FT (Subject) and (All Tokens). Eff. = Efficacy, Gen. = Generalization, Spec. = Specificity, Util. = Utility, Flu. = Fluency, Cons. = Consistency. Subscripts give 95% confidence-interval half-widths. Bold and underlining mark the best and second-best editing scores.

Figure 3 : Editing results on MQuAKE . (a) Standard (left) and CoT (right) prompting with 95% confidence intervals. (b) CoT accuracy by hop. AdaL and MoE denote AdaLoRA and MoEEdit.

Figure 4 : General capabilities during sequential CounterFact editing . (a) Mean F1 across six tasks in the editing sequence. (b) Task-level F1 changes from pre-edit scores after 5,000 edits, in percentage points (pp). The mean $|\Delta|$ averages the absolute task-level changes.

Variant	CounterFact						ZsRE
Variant	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Flu. $\bm{\uparrow}$	Cons. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
EngramEdit	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{{97.0}}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{{85.2}}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.9	$\text{565.2}{}_{\scriptscriptstyle\pm\text{1.10}}$	$\text{\lx@text@underline{16.1}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{{97.3}}{}_{\scriptscriptstyle\pm\text{0.40}}$	$\text{\lx@text@underline{93.7}}{}_{\scriptscriptstyle\pm\text{0.79}}$	$\text{{38.3}}{}_{\scriptscriptstyle\pm\text{1.18}}$	76.4
w/o Joint	$\text{74.5}{}_{\scriptscriptstyle\pm\text{1.91}}$	$\text{72.8}{}_{\scriptscriptstyle\pm\text{1.74}}$	$\text{84.8}{}_{\scriptscriptstyle\pm\text{0.93}}$	77.4	$\text{{568.0}}{}_{\scriptscriptstyle\pm\text{1.00}}$	$\text{15.3}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{57.4}{}_{\scriptscriptstyle\pm\text{1.55}}$	$\text{58.3}{}_{\scriptscriptstyle\pm\text{1.57}}$	$\text{35.6}{}_{\scriptscriptstyle\pm\text{1.17}}$	50.4
w/o Freq.	$\text{\lx@text@underline{99.4}}{}_{\scriptscriptstyle\pm\text{0.32}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{{85.2}}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.8	$\text{564.9}{}_{\scriptscriptstyle\pm\text{1.22}}$	$\text{\lx@text@underline{16.1}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{\lx@text@underline{96.0}}{}_{\scriptscriptstyle\pm\text{0.52}}$	$\text{92.6}{}_{\scriptscriptstyle\pm\text{0.84}}$	$\text{\lx@text@underline{37.3}}{}_{\scriptscriptstyle\pm\text{1.19}}$	75.3
w/o Len.	$\text{99.3}{}_{\scriptscriptstyle\pm\text{0.38}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.70}}$	$\text{\lx@text@underline{85.1}}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.7	$\text{565.4}{}_{\scriptscriptstyle\pm\text{1.09}}$	$\text{{16.2}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{95.7}{}_{\scriptscriptstyle\pm\text{0.55}}$	$\text{93.4}{}_{\scriptscriptstyle\pm\text{0.75}}$	$\text{37.2}{}_{\scriptscriptstyle\pm\text{1.17}}$	75.4
w/o Reg.	$\text{98.8}{}_{\scriptscriptstyle\pm\text{0.47}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.72}}$	$\text{84.6}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.4	$\text{{568.0}}{}_{\scriptscriptstyle\pm\text{1.02}}$	$\text{15.5}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{\lx@text@underline{96.0}}{}_{\scriptscriptstyle\pm\text{0.54}}$	$\text{{94.0}}{}_{\scriptscriptstyle\pm\text{0.75}}$	$\text{37.2}{}_{\scriptscriptstyle\pm\text{1.19}}$	75.7

Table 2 : Ablations of joint update allocation (Joint), reuse-based regularization (Reg.),

n

-gram corpus frequency (Freq.) and length (Len.) weighting, and generated expressions (Expr.).

Figure 5 : Generated expressions and memory use on CounterFact . (a) Editors use the same generated expressions per fact. (b) Generated expressions are used for target computation, memory mapping, both, or neither. Coverage is the percentage of held-out paraphrases activating fact-related updated embeddings. (c) Disabling no updates (None), fact-related updates (Related), matched random updates (Random), or all updates (All) during inference.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6 : Expression-generation prompt for CounterFact and atomic MQuAKE edits. [SUBJ] marks the subject; angle-bracketed fields are filled from the edit request.

Field	Example
CounterFact
Subject and relation	Danielle Darrieux; mother tongue
Requested change	French $\rightarrow$ English
Edit prompt	The mother tongue of Danielle Darrieux is
Held-out paraphrase	… Danielle Darrieux spoke the language
Unrelated prompt	The native language of Montesquieu is

Appendix

Table 3 : Editing examples from CounterFact, ZsRE, and MQuAKE, with simplified formatting. The ellipsis omits an unrelated prefix from the CounterFact paraphrase.

Figure 7 : Sequential editing on CounterFact (a–c) and ZsRE (d–f) up to 5,000 edits. Shading shows pointwise case-level 95% confidence intervals. MFT-S denotes Memory-FT (Subject).

Method	Standard $\bm{\uparrow}$			CoT $\bm{\uparrow}$
Method	2-hop	3-hop	4-hop	2-hop	3-hop	4-hop
FT	7.93 [6.50, 9.65]	2.11 [1.42, 3.12]	7.82 [6.08, 10.00]	12.60 [10.79, 14.66]	3.61 [2.67, 4.86]	9.74 [7.79, 12.11]
FT-L	1.59 [1.01, 2.49]	1.67 [1.07, 2.60]	4.53 [3.24, 6.29]	1.06 [0.61, 1.84]	0.97 [0.54, 1.73]	1.78 [1.05, 3.03]
AdaLoRA	1.50 [0.94, 2.39]	2.64 [1.86, 3.74]	3.84 [2.67, 5.50]	1.15 [0.67, 1.95]	1.23 [0.74, 2.06]	2.33 [1.46, 3.70]
UnKE	4.14 [3.13, 5.46]	4.23 [3.20, 5.56]	6.17 [4.65, 8.16]	5.46 [4.28, 6.94]	4.58 [3.51, 5.95]	9.60 [7.67, 11.96]
MoEEdit	3.00 [2.15, 4.16]	2.64 [1.86, 3.74]	3.29 [2.22, 4.85]	0.62 [0.30, 1.27]	0.26 [0.09, 0.77]	0.27 [0.08, 0.99]

Appendix

Table 4 : MQuAKE accuracy (%) by hop with standard and CoT prompts. Brackets give Wilson 95% confidence intervals. Bold and underlining mark the best and second-best scores.

Method	$\bm{\mathrm{C}\!\rightarrow\!\mathrm{W}}$	$\bm{\mathrm{W}\!\rightarrow\!\mathrm{C}}$	Post–Pre	Post Spec. $\bm{\uparrow}$	Stable Corr. $\bm{\uparrow}$
FT-L	$\text{0.62}{}_{\scriptscriptstyle\pm\text{0.19}}$	$\text{4.84}{}_{\scriptscriptstyle\pm\text{0.57}}$	+4.23	$\text{45.76}{}_{\scriptscriptstyle\pm\text{1.24}}$	$\text{94.54}{}_{\scriptscriptstyle\pm\text{0.60}}$
AdaLoRA	$\text{2.25}{}_{\scriptscriptstyle\pm\text{0.33}}$	$\text{7.51}{}_{\scriptscriptstyle\pm\text{0.71}}$	+5.26	$\text{46.79}{}_{\scriptscriptstyle\pm\text{1.27}}$	$\text{90.24}{}_{\scriptscriptstyle\pm\text{0.79}}$
UnKE	$\text{2.95}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{11.26}{}_{\scriptscriptstyle\pm\text{0.86}}$	+8.31	$\text{49.85}{}_{\scriptscriptstyle\pm\text{1.32}}$	$\text{85.79}{}_{\scriptscriptstyle\pm\text{0.93}}$
MoEEdit	$\text{2.30}{}_{\scriptscriptstyle\pm\text{0.35}}$	$\text{5.14}{}_{\scriptscriptstyle\pm\text{0.56}}$	+2.84	$\text{44.37}{}_{\scriptscriptstyle\pm\text{1.26}}$	$\text{92.56}{}_{\scriptscriptstyle\pm\text{0.67}}$
MFT-S	$\text{5.06}{}_{\scriptscriptstyle\pm\text{0.51}}$	$\text{0.40}{}_{\scriptscriptstyle\pm\text{0.12}}$	-4.67	$\text{36.87}{}_{\scriptscriptstyle\pm\text{1.17}}$	$\text{94.54}{}_{\scriptscriptstyle\pm\text{0.54}}$
MFT-A	$\text{9.67}{}_{\scriptscriptstyle\pm\text{0.72}}$	$\text{1.84}{}_{\scriptscriptstyle\pm\text{0.30}}$	-7.83	$\text{33.71}{}_{\scriptscriptstyle\pm\text{1.13}}$	$\text{88.48}{}_{\scriptscriptstyle\pm\text{0.80}}$

Appendix

Table 5 : ZsRE correctness transitions after 2,000 edits. C and W denote correct and wrong predictions; Post–Pre is the Specificity change in percentage points. Stable Corr. measures unchanged correctness states. Values are case-averaged percentages with 95% CI half-widths in subscripts.

Figure 8 : Task-level general capabilities during 5,000 sequential CounterFact edits. F1 is measured on 100 examples per task. Dotted lines mark pre-edit scores.

Method	CounterFact				ZsRE
Method	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
Original edit prompt only
MoEEdit	$\text{\lx@text@underline{99.1}}{}_{\scriptscriptstyle\pm\text{0.41}}$	$\text{\lx@text@underline{67.1}}{}_{\scriptscriptstyle\pm\text{1.69}}$	$\text{63.5}{}_{\scriptscriptstyle\pm\text{1.28}}$	76.6	$\text{\lx@text@underline{85.1}}{}_{\scriptscriptstyle\pm\text{0.91}}$	$\text{\lx@text@underline{78.7}}{}_{\scriptscriptstyle\pm\text{1.21}}$	$\text{{44.4}}{}_{\scriptscriptstyle\pm\text{1.26}}$	69.4
MFT-S	$\text{99.0}{}_{\scriptscriptstyle\pm\text{0.45}}$	$\text{59.1}{}_{\scriptscriptstyle\pm\text{1.92}}$	$\text{\lx@text@underline{85.2}}{}_{\scriptscriptstyle\pm\text{0.90}}$	81.1	$\text{66.9}{}_{\scriptscriptstyle\pm\text{1.35}}$	$\text{56.5}{}_{\scriptscriptstyle\pm\text{1.53}}$	$\text{36.9}{}_{\scriptscriptstyle\pm\text{1.17}}$	53.4
MFT-A	$\text{93.9}{}_{\scriptscriptstyle\pm\text{1.04}}$	$\text{40.7}{}_{\scriptscriptstyle\pm\text{1.75}}$	$\text{66.6}{}_{\scriptscriptstyle\pm\text{1.12}}$	67.1	$\text{62.8}{}_{\scriptscriptstyle\pm\text{1.49}}$	$\text{51.3}{}_{\scriptscriptstyle\pm\text{1.57}}$	$\text{33.7}{}_{\scriptscriptstyle\pm\text{1.13}}$	49.3
EngramEdit	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.32}}$	$\text{{75.0}}{}_{\scriptscriptstyle\pm\text{1.74}}$	$\text{{85.6}}{}_{\scriptscriptstyle\pm\text{0.91}}$	86.7	$\text{{95.1}}{}_{\scriptscriptstyle\pm\text{0.62}}$	$\text{{83.4}}{}_{\scriptscriptstyle\pm\text{1.29}}$	$\text{\lx@text@underline{38.5}}{}_{\scriptscriptstyle\pm\text{1.18}}$	72.3

Appendix

Table 6 : Editing with and without generated expressions on CounterFact and ZsRE over 2,000 sequential edits. Subscripts give 95% CI half-widths. Bold and underlining mark the best and second-best scores within each block.

# Expr.	CounterFact						ZsRE
# Expr.	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$	Flu. $\bm{\uparrow}$	Cons. $\bm{\uparrow}$	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
0	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.32}}$	$\text{75.0}{}_{\scriptscriptstyle\pm\text{1.74}}$	$\text{{85.6}}{}_{\scriptscriptstyle\pm\text{0.91}}$	86.7	$\text{562.6}{}_{\scriptscriptstyle\pm\text{1.19}}$	$\text{15.7}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{95.1}{}_{\scriptscriptstyle\pm\text{0.62}}$	$\text{83.4}{}_{\scriptscriptstyle\pm\text{1.29}}$	$\text{{38.5}}{}_{\scriptscriptstyle\pm\text{1.18}}$	72.3
2	$\text{\lx@text@underline{99.4}}{}_{\scriptscriptstyle\pm\text{0.34}}$	$\text{95.2}{}_{\scriptscriptstyle\pm\text{0.83}}$	$\text{\lx@text@underline{85.3}}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.3	$\text{\lx@text@underline{566.2}}{}_{\scriptscriptstyle\pm\text{1.11}}$	$\text{{16.2}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{96.2}{}_{\scriptscriptstyle\pm\text{0.50}}$	$\text{91.8}{}_{\scriptscriptstyle\pm\text{0.89}}$	$\text{37.7}{}_{\scriptscriptstyle\pm\text{1.10}}$	75.2
4	$\text{{99.5}}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{97.0}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.9	$\text{565.2}{}_{\scriptscriptstyle\pm\text{1.10}}$	$\text{\lx@text@underline{16.1}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{{97.3}}{}_{\scriptscriptstyle\pm\text{0.40}}$	$\text{\lx@text@underline{93.7}}{}_{\scriptscriptstyle\pm\text{0.79}}$	$\text{\lx@text@underline{38.3}}{}_{\scriptscriptstyle\pm\text{1.18}}$	76.4
6	$\text{\lx@text@underline{99.4}}{}_{\scriptscriptstyle\pm\text{0.34}}$	$\text{{97.3}}{}_{\scriptscriptstyle\pm\text{0.64}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	94.0	$\text{{567.5}}{}_{\scriptscriptstyle\pm\text{1.03}}$	$\text{{16.2}}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{96.5}{}_{\scriptscriptstyle\pm\text{0.47}}$	$\text{\lx@text@underline{93.7}}{}_{\scriptscriptstyle\pm\text{0.77}}$	$\text{36.8}{}_{\scriptscriptstyle\pm\text{1.17}}$	75.7
8	$\text{99.1}{}_{\scriptscriptstyle\pm\text{0.41}}$	$\text{\lx@text@underline{97.2}}{}_{\scriptscriptstyle\pm\text{0.67}}$	$\text{84.9}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.7	$\text{566.0}{}_{\scriptscriptstyle\pm\text{1.07}}$	$\text{16.0}{}_{\scriptscriptstyle\pm\text{0.36}}$	$\text{\lx@text@underline{96.8}}{}_{\scriptscriptstyle\pm\text{0.44}}$	$\text{{94.2}}{}_{\scriptscriptstyle\pm\text{0.74}}$	$\text{38.0}{}_{\scriptscriptstyle\pm\text{1.17}}$	76.3

Appendix

Table 7 : Results with different numbers of generated expressions after 2,000 edits. # Expr. excludes the original prompt. Subscripts give reported CI half-widths; bold and underlining mark best and second-best scores.

Figure 9 : Editable

n

-gram lengths on CounterFact. The frame marks the default lengths 2, 3, and 4;

n=1

denotes single-token embedding updates. Small

\pm

values give CI half-widths.

Parameter	Value	Eff. $\bm{\uparrow}$	Gen. $\bm{\uparrow}$	Spec. $\bm{\uparrow}$	Util. $\bm{\uparrow}$
Reuse-based regularization $\lambda_{\mathrm{reuse}}$	0	$\text{98.8}{}_{\scriptscriptstyle\pm\text{0.49}}$	$\text{96.5}{}_{\scriptscriptstyle\pm\text{0.76}}$	$\text{84.7}{}_{\scriptscriptstyle\pm\text{0.93}}$	93.3
	0.03	$\text{99.3}{}_{\scriptscriptstyle\pm\text{0.37}}$	$\text{96.2}{}_{\scriptscriptstyle\pm\text{0.76}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.6
	0.05	$\text{99.5}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{97.0}{}_{\scriptscriptstyle\pm\text{0.69}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.9
	0.075	$\text{99.6}{}_{\scriptscriptstyle\pm\text{0.29}}$	$\text{96.4}{}_{\scriptscriptstyle\pm\text{0.73}}$	$\text{85.3}{}_{\scriptscriptstyle\pm\text{0.91}}$	93.8
	0.1	$\text{99.6}{}_{\scriptscriptstyle\pm\text{0.28}}$	$\text{96.6}{}_{\scriptscriptstyle\pm\text{0.70}}$	$\text{85.2}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.8
Ridge regularization $\lambda_{\mathrm{ridge}}$	0	$\text{99.5}{}_{\scriptscriptstyle\pm\text{0.31}}$	$\text{96.6}{}_{\scriptscriptstyle\pm\text{0.71}}$	$\text{85.1}{}_{\scriptscriptstyle\pm\text{0.92}}$	93.7

Appendix

Table 8 : Sensitivity to regularization and clamp factor after 2,000 CounterFact edits. Only the listed parameter changes. Scores are percentages; subscripts give reported CI half-widths. Bold parameter values mark the defaults.

Figure 10 : Disabling memory updates on CounterFact. (a) Mean edit margins before and after disabling. Positive values favor revised answers; negative values favor original answers. (b) Outcomes of the 5,850 prompts successful before disabling. Parentheses give the mean number of disabled updates per case.

Figure 11 : Memory activation and editing outcomes with all updates enabled. Groups indicate activation of embeddings updated for the current edit only, other edits only, both, or neither. (a) Within-group paraphrase success rates and each group’s share of all paraphrases. (b) Success-to-failure rates among previously successful neighborhood queries in each group and each group’s share of all regressions.

Figure 12 : Cross-edit sharing on CounterFact. (a) Sharing rates among updated

n

-grams and edited facts. (b) Final editing results for facts with or without shared

n

-grams.

Explore similar work

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs
Jun 21, 2026Wenhao Yu, Zhicong Lu, Bo Lv +4Multimodal Knowledge Editing Knowledge Graph Embeddings
School of Computer Science and Technology, Tianjin University · Kexin Technology · University of Chinese Academy of Sciences +2
Exposing the Illusion of Erasure in Knowledge Editing for LLMs
Jun 22, 2026Advik Raj Basani, Anshuman ChhabraMultimodal Knowledge Editing Concept Erasure
Birla Institute of Technology and Science, Goa · University of South Florida
ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective
Sep 27, 2026Rui Liu, Chenheng Zhang, Haoxuan Li +1Multimodal Knowledge Editing Edit
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

Jun 21, 2026, cs.CLJ/K move · Enter open · S save

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

Wenhao Yu, Zhicong Lu, Bo Lv, Fangyin Ma+3

School of Computer Science and Technology, Tianjin University · Kexin Technology · University of Chinese Academy of Sciences+2

Multimodal Knowledge Editing Knowledge Graph Embeddings Semantic Representations Cross-Lingual Consistency Entanglement Orthogonality

Open paper AskPDF arXiv

Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining. However, existing methods still suffer from performance degradation in batch knowledge editing. We identify that semantic representation entanglement, such as overlapping concepts and shared syntactic patterns, accumulates interference in the representation space and reduces editing precision. To bridge this gap, in this paper, we propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs by constructing a general semantic subspace and enforcing orthogonal constraints on edit vectors, effectively decoupling semantic entanglement. Furthermore, we introduce a gated non-linear representation head to enable adaptive learning of editing locations and precise control over knowledge injection. Extensive experiments show that ORE outperforms existing methods and achieves superior performance in cross-lingual knowledge editing scenarios. We release our code at https://github.com/YVVH/ORE.

Abstract

Explore similar work

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

Exposing the Illusion of Erasure in Knowledge Editing for LLMs

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective