Model editing provides an efficient way to update factual knowledge in transformer models. Recent approaches reverse edits by modifying, masking, or decomposing model weights, but make recovery dependent on weight-space interventions. We instead show that pre-edit knowledge can be recovered through input-space interventions alone, without modifying the edited model's weights. We introduce Memento, a prompt-space mechanism for auditing residual knowledge that learns a reusable recovery signal to steer an edited model toward its pre-edit response. Memento supports both continuous recovery prefixes and discrete textual prompts for standard text interfaces. Across three LLMs and five editing techniques, Memento recovers 80.07% of held-out edited facts, achieving a 9.58% relative improvement over baselines, suggesting that editing may suppress the facts without fully eliminating their recoverability. We further investigate whether prompt-space recovery extends beyond model editing. On pristine, unedited LLMs, Memento improves average factual recall by 53.9% relative to standard inference. These results demonstrate the potential of prompt-space recovery for auditing residual knowledge in edited models and investigating the gap between factual knowledge and its accessibility during inference. We will release our source code.
Figures & tables
Figure 1: Overview of Memento ’s continuous prompt-space recovery. A trainable prefix Π is prepended to factual prompts and optimized across training facts to recover the original object o from an edited model fθ′ , whose parameters remain frozen. The objective combines supervised recovery ( Lrec ), preference optimization ( Lpref ), fluency preservation ( Lflu ), and distillation from the pre-edit model ( Ldist ). Gradients update only Π ; the pre-edit model supplies distillation targets during training and is not needed at inference. Without the prefix, the edited LLM answers the edited object ( o∗ = midfielder ); with it, the LLM produces the original object ( o = quarterback ).
Technique
GPT2-XL
GPT-J
Llama3
AlphaEdit
0.97
0.97
0.92
MEMIT
0.95
0.97
0.91
RECT
0.90
0.95
0.85
PRUNE
0.96
0.91
0.96
CoME
0.95
0.94
0.90
Table 1: Efficacy of all editing techniques across the models being evaluated over the held-out facts.
N
GPT2-XL
GPT-J
Llama3
1
0.58
0.54
0.57
2
0.73
0.71
0.70
4
0.71
0.68
0.77
8
0.75
0.77
0.85
16
0.79
0.80
0.84
32
0.76
0.77
0.82
Table 2: Top-1 Recovery for different prefix lengths N on MEMIT-edited models.
Model
Method
MEMIT
AlphaEdit
PRUNE
CoME
RECT
Top-1
CMR
Top-1
CMR
Top-1
CMR
Top-1
CMR
Top-1
CMR
GPT2-XL
One Mask
0.79
0.76
0.78
0.68
0.77
0.76
0.83
0.78
0.79
0.79
Memento
0.79
0.77
0.80
0.78
0.72
0.74
0.77
0.75
0.80
0.78
GPT-J
One Mask
0.72
0.72
0.71
0.70
0.73
0.72
0.72
0.70
0.78
0.75
Memento
0.80
0.78
0.82
0.77
0.74
0.77
0.79
0.78
0.81
0.82
Llama3
One Mask
0.62
0.56
0.78
0.68
0.64
0.57
0.65
0.59
0.65
0.69
Table 3: Top-1 recovery and CMR of Memento and One Mask across three LLMs and five editing techniques.
Model
Method
Pristine
Editing Technique Used to Obtain Recovery Signal
Recall
AlphaEdit
MEMIT
RECT
PRUNE
CoME
LLM-as-a-Judge Evaluation
GPT2-XL
One Mask
21.7%
12.5% (-9.2)
12.2% (-9.5)
11.4% (-10.3)
4.4% (-17.3)
14.3% (-7.4)
Memento
33.3% (+11.6)
39.2% (+17.5)
47.0% (+25.3)
33.0% (+11.3)
23.4% (+1.7)
GPT-J
One Mask
33.0%
27.6% (-5.4)
26.5% (-6.5)
28.9% (-4.1)
27.1% (-5.9)
27.1% (-5.9)
Memento
62.1% (+29.1)
58.9% (+25.9)
60.3% (+27.3)
51.5% (+18.5)
55.2% (+22.2)
Table 4: Factual recall on pristine, unedited models. Pristine Recall column reports recall under standard inference. Remaining columns report recall using signals obtained under the indicated editing-technique settings. deepseek-ai/DeepSeek-V4.1-Flash is used as the LLM-as-a-Judge. Canonical-answer correctness requires canonical answer at the start of continuation after Unicode, case, punctuation, and whitespace normalization.
Technique
GPT2-XL
GPT-J
Llama3
AlphaEdit
0.52
0.43
0.47
MEMIT
0.50
0.48
0.46
RECT
0.53
0.55
0.44
PRUNE
0.56
0.45
0.48
CoME
0.51
0.51
0.46
Table 5: Top-1 recovery of Memento (discrete).
Figure 2: Magnitude of the MEMIT update contribution at the edited layers, with and without Memento .
Figure 3: Layerwise log-probability of the original answer in the unedited model, the MEMIT-edited model, and the edited model with Memento .
Figure 4: Layerwise cosine similarity between the hidden states of edited and unedited models at the subject position (top) and the last prompt-token position (bottom).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Checkpoint
Revision
Layers
d
Edited layers
GPT2-XL
openai-community/gpt2-xl
15ea56dee5df
48
1600
13-17
GPT-J
EleutherAI/gpt-j-6B
47e169305d2e
28
4096
3-8
Llama3
meta-llama/Meta-Llama-3-8B-Instruct
8afb486c1db2
32
4096
4-8
Appendix
Table 6: Models. d is the embedding dimension, which is also the dimension of each prefix vector. All five editing techniques modify the same layers of a given model.
GPT2-XL
GPT-J
Llama3
Value optimization steps
20
25
25
Value learning rate
0.5
0.5
0.1
Clamp norm factor
0.75
0.75
0.75
KL factor
0.0625
0.0625
0.0625
Value weight decay
0.5
0.5
0.5
Covariance update weight
20,000
15,000
15,000
Appendix
Table 7: Hyperparameters shared by all editing techniques, following the original MEMIT configurations.
Setting
Value
Prefix length N
16
Initialization
embedding of “ ! ”, repeated
Position
before the prompt
Optimizer
Adam, constant learning rate
Learning rate η=κ⋅std(WE)
κ=0.2075
GPT2-XL / GPT-J / Llama3
1.00/0.64/0.19×10−2
Appendix
Table 8: Continuous Memento settings, shared across all models and editing techniques.
Setting
Value
Sequence length N
16 tokens
Objective
Lrec+λprefLpref
β / γ / λpref
2.0 / 0.5 / 1.0
Top- k substitutions per position
256
Candidates per step
256 (sampled 4× , then filtered)
Facts per step
8
Appendix
Table 9: Discrete Memento settings, shared across all models and editing techniques.
GPT2-XL
GPT-J
Llama3
Continuous training
0.5 h
0.7 h
1.6 h
Evaluation
0.4 h
0.5 h
1.3 h
Discrete search
1.4 h
2.8 h
6.3 h
Appendix
Table 10: Wall-clock time per run for MEMIT on one H200 GPU.
Method
GPT2-XL
GPT-J
Llama3
Memento (continuous)
0.5 h
0.7 h
1.6 h
Memento (discrete)
1.4 h
2.8 h
6.3 h
Appendix
Table 11: One-time training cost for MEMIT with 1,000 training facts on one H200 GPU, including the time to edit the training facts.
GPT2-XL
GPT-J
Llama3
Without prefix (ms/query)
95.6
135.2
165.2
With Memento (ms/query)
90.9
141.8
169.7
Relative change
−4.9%
+4.9%
+2.7%
Appendix
Table 12: Per-query latency of greedy decoding (12 new tokens, batch size 1), averaged over 100 held-out prompts after warm-up.
Cost
Memento (continuous)
Memento (discrete)
Training time
O(SB[C(N+∣q∣+∣y∣)+C(N+R)])
O(SKFC(N+∣q∣+∣y∣))
Trainable values
Nd
N tokens
Recovery time per query
C(N+∣q∣)+m steps
C(N+∣q∣)+m steps
Recovery memory overhead
O(LNd) cache
O(LNd) cache
Appendix
Table 13: Asymptotic costs. S : training steps; B : facts per step; K , F : candidates and facts per step of discrete search.
Objective
GPT2-XL
GPT-J
Llama3
Full objective
0.790 / ×1.74
0.800 / ×1.97
0.840 / ×1.07
−Lpref
0.810 / ×1.14
0.775 / ×1.41
0.830 / ×1.08
−Lflu
0.745 / ×1.24
0.810 / ×1.65
0.830 / ×1.13
−Ldist
0.800 / ×7.31
0.755 / ×3.50
0.825 / ×1.60
−Lpref−Lflu
0.810 / ×1.16
0.765 / ×1.60
0.840 / ×1.14
Lrec only
0.785 / ×1.79
0.745 / ×4.13
0.820 / ×1.96
Appendix
Table 14: Effect of the continuous recovery objective on MEMIT-edited models. Each cell reports top-1 recovery on held-out edited facts (higher is better) and the perplexity (PPL) ratio on held-out text (lower is better), respectively. Bold values indicate the best result for each metric and model.
Prompt
Reference
Generated continuation
Judge
Rationale
Edmund Husserl, who has a citizenship from
Germany
“the Republic of Austria, was born in the city of Freiburg im Breisgau, in the German-speaking part of the country, on the …”
Incorrect
The continuation explicitly identifies Austria. Describing a region as German-speaking does not establish German citizenship.
Boeing KC-46 is a product of
Boeing
“the Boeing Company. It is a long-range, medium-altitude, long-endurance, tanker-delivered …”
Correct
“The Boeing Company” identifies the same entity as Boeing, even though the canonical answer does not appear verbatim at the start.
Shahab-1 is produced by
Iran
“the Iranian Air Force and is used by the Iranian Air Force to conduct air strikes …”
Correct
The judge accepts “Iranian” as expressing the requested country through an adjectival form. This illustrates an indirect answer; such judgments may be sensitive to the interpretation of the relation.
Appendix
Table 15: Examples of LLM-as-a-Judge decisions on generated continuations.
Knowledge editing methods such as ROME and MEMIT update factual associations in transformer models by modifying MLP weights. While evaluated mainly by output behavior, their internal mechanism remains underexplored. We investigate whether edits rely on a common mechanism, regardless of which fact is modified. Despite fact-specific weight changes, we argue that ROME and MEMIT target the same subset of weights critical for maintaining edits. To isolate this subset, we train a compact binary mask over the edited weights. The mask reverses 80% of edits on the training set and over 70% on the test set, confirming that diverse edits share a common functional structure. Our analysis reveals that the mask reverses edits by eliminating overattention in later layers. Additionally, we show that injecting the mask during editing drops editing success from 98% to 38%, demonstrating that this mechanism is necessary for edits to succeed. Our finding that edits suppress rather than overwrite knowledge explains why ROME and MEMIT fail to propagate changes to related facts. The identified common functional subspace informs detection and defense against unwanted edits.
Ali Holmov, Paul Youssef, Nandi Schoots +1
Technical University of Munich · Marburg University · University of Oxford
Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.
Yuhuan Liu, Haitian Zhong, Xinyuan Xia +3
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts. We propose EngramEdit for decoupled knowledge updates through conditional memory. EngramEdit first computes target memory representations that make the model predict the updated fact across multiple expressions. It then jointly updates the shared n-gram embeddings to match these targets across expressions and edits, penalizing updates to frequently reused embeddings more strongly to preserve unrelated knowledge. Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, achieving near-perfect editing success. Revised knowledge is usable across unseen expressions and in multi-hop reasoning, with nearly three times the strongest baseline's accuracy under chain-of-thought (CoT) prompting. Unrelated knowledge and general capabilities are largely preserved even as factual updates accumulate. These findings show that EngramEdit turns conditional memory into an editable knowledge interface, extending its role beyond model scaling to support decoupled knowledge updates.
Hongru Cai, Ran Wei, Wenjie Wang +4
The Hong Kong Polytechnic University · Hangzhou Diagens Biotechnology Co., Ltd · University of Science and Technology of China