Model editing provides an efficient way to update factual knowledge in transformer models. Recent approaches reverse edits by modifying, masking, or decomposing model weights, but make recovery dependent on weight-space interventions. We instead show that pre-edit knowledge can be recovered through input-space interventions alone, without modifying the edited model's weights. We introduce Memento, a prompt-space mechanism for auditing residual knowledge that learns a reusable recovery signal to steer an edited model toward its pre-edit response. Memento supports both continuous recovery prefixes and discrete textual prompts for standard text interfaces. Across three LLMs and five editing techniques, Memento recovers 80.07% of held-out edited facts, achieving a 9.58% relative improvement over baselines, suggesting that editing may suppress the facts without fully eliminating their recoverability. We further investigate whether prompt-space recovery extends beyond model editing. On pristine, unedited LLMs, Memento improves average factual recall by 53.9% relative to standard inference. These results demonstrate the potential of prompt-space recovery for auditing residual knowledge in edited models and investigating the gap between factual knowledge and its accessibility during inference. We will release our source code.
Figures & tables
Figure 1: Overview of Memento ’s continuous prompt-space recovery. A trainable prefix Π is prepended to factual prompts and optimized across training facts to recover the original object o from an edited model fθ′ , whose parameters remain frozen. The objective combines supervised recovery ( Lrec ), preference optimization ( Lpref ), fluency preservation ( Lflu ), and distillation from the pre-edit model ( Ldist ). Gradients update only Π ; the pre-edit model supplies distillation targets during training and is not needed at inference. Without the prefix, the edited LLM answers the edited object ( o∗ = midfielder ); with it, the LLM produces the original object ( o = quarterback ).
Technique
GPT2-XL
GPT-J
Llama3
AlphaEdit
0.97
0.97
0.92
MEMIT
0.95
0.97
0.91
RECT
0.90
0.95
0.85
PRUNE
0.96
0.91
0.96
CoME
0.95
0.94
0.90
Table 1: Efficacy of all editing techniques across the models being evaluated over the held-out facts.
N
GPT2-XL
GPT-J
Llama3
1
0.58
0.54
0.57
2
0.73
0.71
0.70
4
0.71
0.68
0.77
8
0.75
0.77
0.85
16
0.79
0.80
0.84
32
0.76
0.77
0.82
Table 2: Top-1 Recovery for different prefix lengths N on MEMIT-edited models.
Model
Method
MEMIT
AlphaEdit
PRUNE
CoME
RECT
Top-1
CMR
Top-1
CMR
Top-1
CMR
Top-1
CMR
Top-1
CMR
GPT2-XL
One Mask
0.79
0.76
0.78
0.68
0.77
0.76
0.83
0.78
0.79
0.79
Memento
0.79
0.77
0.80
0.78
0.72
0.74
0.77
0.75
0.80
0.78
GPT-J
One Mask
0.72
0.72
0.71
0.70
0.73
0.72
0.72
0.70
0.78
0.75
Memento
0.80
0.78
0.82
0.77
0.74
0.77
0.79
0.78
0.81
0.82
Llama3
One Mask
0.62
0.56
0.78
0.68
0.64
0.57
0.65
0.59
0.65
0.69
Table 3: Top-1 recovery and CMR of Memento and One Mask across three LLMs and five editing techniques.
Model
Method
Pristine
Editing Technique Used to Obtain Recovery Signal
Recall
AlphaEdit
MEMIT
RECT
PRUNE
CoME
LLM-as-a-Judge Evaluation
GPT2-XL
One Mask
21.7%
12.5% (-9.2)
12.2% (-9.5)
11.4% (-10.3)
4.4% (-17.3)
14.3% (-7.4)
Memento
33.3% (+11.6)
39.2% (+17.5)
47.0% (+25.3)
33.0% (+11.3)
23.4% (+1.7)
GPT-J
One Mask
33.0%
27.6% (-5.4)
26.5% (-6.5)
28.9% (-4.1)
27.1% (-5.9)
27.1% (-5.9)
Memento
62.1% (+29.1)
58.9% (+25.9)
60.3% (+27.3)
51.5% (+18.5)
55.2% (+22.2)
Table 4: Factual recall on pristine, unedited models. Pristine Recall column reports recall under standard inference. Remaining columns report recall using signals obtained under the indicated editing-technique settings. deepseek-ai/DeepSeek-V4.1-Flash is used as the LLM-as-a-Judge. Canonical-answer correctness requires canonical answer at the start of continuation after Unicode, case, punctuation, and whitespace normalization.
Technique
GPT2-XL
GPT-J
Llama3
AlphaEdit
0.52
0.43
0.47
MEMIT
0.50
0.48
0.46
RECT
0.53
0.55
0.44
PRUNE
0.56
0.45
0.48
CoME
0.51
0.51
0.46
Table 5: Top-1 recovery of Memento (discrete).
Figure 2: Magnitude of the MEMIT update contribution at the edited layers, with and without Memento .
Figure 3: Layerwise log-probability of the original answer in the unedited model, the MEMIT-edited model, and the edited model with Memento .
Figure 4: Layerwise cosine similarity between the hidden states of edited and unedited models at the subject position (top) and the last prompt-token position (bottom).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Checkpoint
Revision
Layers
d
Edited layers
GPT2-XL
openai-community/gpt2-xl
15ea56dee5df
48
1600
13-17
GPT-J
EleutherAI/gpt-j-6B
47e169305d2e
28
4096
3-8
Llama3
meta-llama/Meta-Llama-3-8B-Instruct
8afb486c1db2
32
4096
4-8
Appendix
Table 6: Models. d is the embedding dimension, which is also the dimension of each prefix vector. All five editing techniques modify the same layers of a given model.
GPT2-XL
GPT-J
Llama3
Value optimization steps
20
25
25
Value learning rate
0.5
0.5
0.1
Clamp norm factor
0.75
0.75
0.75
KL factor
0.0625
0.0625
0.0625
Value weight decay
0.5
0.5
0.5
Covariance update weight
20,000
15,000
15,000
Appendix
Table 7: Hyperparameters shared by all editing techniques, following the original MEMIT configurations.
Setting
Value
Prefix length N
16
Initialization
embedding of “ ! ”, repeated
Position
before the prompt
Optimizer
Adam, constant learning rate
Learning rate η=κ⋅std(WE)
κ=0.2075
GPT2-XL / GPT-J / Llama3
1.00/0.64/0.19×10−2
Appendix
Table 8: Continuous Memento settings, shared across all models and editing techniques.
Setting
Value
Sequence length N
16 tokens
Objective
Lrec+λprefLpref
β / γ / λpref
2.0 / 0.5 / 1.0
Top- k substitutions per position
256
Candidates per step
256 (sampled 4× , then filtered)
Facts per step
8
Appendix
Table 9: Discrete Memento settings, shared across all models and editing techniques.
GPT2-XL
GPT-J
Llama3
Continuous training
0.5 h
0.7 h
1.6 h
Evaluation
0.4 h
0.5 h
1.3 h
Discrete search
1.4 h
2.8 h
6.3 h
Appendix
Table 10: Wall-clock time per run for MEMIT on one H200 GPU.
Method
GPT2-XL
GPT-J
Llama3
Memento (continuous)
0.5 h
0.7 h
1.6 h
Memento (discrete)
1.4 h
2.8 h
6.3 h
Appendix
Table 11: One-time training cost for MEMIT with 1,000 training facts on one H200 GPU, including the time to edit the training facts.
GPT2-XL
GPT-J
Llama3
Without prefix (ms/query)
95.6
135.2
165.2
With Memento (ms/query)
90.9
141.8
169.7
Relative change
−4.9%
+4.9%
+2.7%
Appendix
Table 12: Per-query latency of greedy decoding (12 new tokens, batch size 1), averaged over 100 held-out prompts after warm-up.
Cost
Memento (continuous)
Memento (discrete)
Training time
O(SB[C(N+∣q∣+∣y∣)+C(N+R)])
O(SKFC(N+∣q∣+∣y∣))
Trainable values
Nd
N tokens
Recovery time per query
C(N+∣q∣)+m steps
C(N+∣q∣)+m steps
Recovery memory overhead
O(LNd) cache
O(LNd) cache
Appendix
Table 13: Asymptotic costs. S : training steps; B : facts per step; K , F : candidates and facts per step of discrete search.
Objective
GPT2-XL
GPT-J
Llama3
Full objective
0.790 / ×1.74
0.800 / ×1.97
0.840 / ×1.07
−Lpref
0.810 / ×1.14
0.775 / ×1.41
0.830 / ×1.08
−Lflu
0.745 / ×1.24
0.810 / ×1.65
0.830 / ×1.13
−Ldist
0.800 / ×7.31
0.755 / ×3.50
0.825 / ×1.60
−Lpref−Lflu
0.810 / ×1.16
0.765 / ×1.60
0.840 / ×1.14
Lrec only
0.785 / ×1.79
0.745 / ×4.13
0.820 / ×1.96
Appendix
Table 14: Effect of the continuous recovery objective on MEMIT-edited models. Each cell reports top-1 recovery on held-out edited facts (higher is better) and the perplexity (PPL) ratio on held-out text (lower is better), respectively. Bold values indicate the best result for each metric and model.
Prompt
Reference
Generated continuation
Judge
Rationale
Edmund Husserl, who has a citizenship from
Germany
“the Republic of Austria, was born in the city of Freiburg im Breisgau, in the German-speaking part of the country, on the …”
Incorrect
The continuation explicitly identifies Austria. Describing a region as German-speaking does not establish German citizenship.
Boeing KC-46 is a product of
Boeing
“the Boeing Company. It is a long-range, medium-altitude, long-endurance, tanker-delivered …”
Correct
“The Boeing Company” identifies the same entity as Boeing, even though the canonical answer does not appear verbatim at the start.
Shahab-1 is produced by
Iran
“the Iranian Air Force and is used by the Iranian Air Force to conduct air strikes …”
Correct
The judge accepts “Iranian” as expressing the requested country through an adjectival form. This illustrates an indirect answer; such judgments may be sensitive to the interpretation of the relation.
Appendix
Table 15: Examples of LLM-as-a-Judge decisions on generated continuations.
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1