Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.
Figures & tables
Figure 1: Memory lifetime under the best single mechanism and composed continual learning methods. Lines show three-seed means with min-max bands. Half-life counts tasks until retention halves. Combining multiple anchors with merged LoRA substantially reduces catastrophic forgetting and extends memory lifetime beyond the best single mechanism.
Figure 2: Task-level successive halving. Within each dataset and horizon, we rank surviving candidates by mean retention over three seeds. A method’s percentile rank is the percentage of other methods ranked below it. Bars average these values over method variants and then over the three datasets. Because the candidate pool changes, bar heights do not measure changes in retention across horizons. Gray entries indicate that all variants have been eliminated in all three datasets. The leading methods at 100 tasks combine multiple anchors with merged LoRA.
Figure 3: Final retention (%) after 100 tasks for all 16 combinations, averaged over three seeds. Filled markers identify active mechanisms, and an inactive Merge marker indicates shared LoRA. The leftmost column is shared LoRA without anchors. Bold marks the best result per dataset. Appendix E.2 reports standard deviations over seeds. The best compositions substantially outperform standalone mechanisms.
Main effects
Two-way
Dataset
SI
SD
R
M
SI×SD
SI×R
SI×M
SD×R
SD×M
R×M
Symbol-QA
+0.3
+5.7
+9.5
+5.9
−0.3
+0.7
−3.0
−0.2
+0.6
+3.6
LLM-QA
+5.8
+5.0
+18.5
+14.9
+1.6
+2.9
−0.1
−3.5
+1.7
+9.4
Real-QA
+6.3
+3.7
+19.3
+20.5
+1.3
+0.1
+0.2
−10.3
+1.7
+11.7
Table 1: Main and interaction effects from the 24 factorial analysis of final retention, in percentage points. R is replay and M is merged LoRA. Bold marks statistically significant entries ( p<0.05 ). Replay and merged LoRA have the largest main effects and a positive interaction in all three datasets.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Method
Persistent learner state
Horizon dependence
Vanilla baseline
Sequential SFT (no anchor)
one shared LoRA workspace
Constant
Data anchor
Unconditional generative replay
Model trained from last task to generate replay data unconditionally. No raw examples or replay samples are carried to the next task
Constant
Function anchor
self-distillation
Model trained from the last task treated as a frozen teacher during the current task for KL estimation
Constant
Weight anchor
Synaptic Intelligence
One reference and one diagonal importance tensor per trainable tensor saved after each task and then discarded before training the next task
Constant
Weight anchor
Online EWC
One reference and one running diagonal Fisher tensor per trainable tensor saved after each task and then discarded before training the next task
Constant
Low-rank allocation
shared LoRA
One continually updated rank- r adapter used by vanilla SFT and composable with any anchor.
Constant
Appendix
Table 2: Methods used in our project, categorized as anchors or low-rank allocation rules. “Constant” refers to a method that depends only on a fixed set of information, while “Linear in tasks” means the method needs to store information that grows with the number of tasks during training.
Component
Setting
Backbone
Qwen3-4B-Base
Current-task optimization
10 epochs per task; minibatch size 8; gradient accumulation 1; maximum sequence length 384
Optimizer
AdamW; learning rate 5×10−4 ; weight decay 0.01 ; global gradient-norm clipping at 1
Per-task schedule
Linear warmup during the first 5% of optimizer steps, followed by a constant learning rate; optimizer and schedule restart for every task
NR=300 ; τD=2 ; frozen replay seed token; top- p=0.9 ; generation batch size 32; maximum 384 new tokens; generation-temperature candidates {1.0,1.5} and registered value 1.5 ; w candidates {0.5,0.75} and registered values 0.75 ( Symbol-QA ), 0.5 ( LLM-QA ), and 0.75 ( Real-QA )
Appendix
Table 3: Canonical implementation and optimization settings.
Source
Type
Original form
Retained
TriviaQA
short answer
trivia questions with alias sets
500
NQ-Open
short answer
search queries, short answer list
500
PopQA
short answer
entity-centric relational queries
500
SQuAD
short answer
span answers over passages
500
WebQuestions
short answer
Freebase question–answer pairs
500
OpenBookQA
multiple choice
elementary science, four options
500
Appendix
Table 4: Real-QA sources.
Dataset
Tasks × items
Novelty control
Task organization
Semantic information
Symbol-QA
100×100
Randomly generated symbol associations
One shared generation rule across all tasks
No structure in the target mapping
LLM-QA
100×100
Fictional generation prompt without a base model filter
One topic per task with a fixed random topic order
Natural language and familiar entity types
Real-QA
100×50
Five sample filter for each base model
Ten data sources mixed across tasks by a global shuffle
Natural language and real facts
Appendix
Table 5: Comparison of the three memorization datasets. In every dataset, training and evaluation use the same questions, each question has one target answer across all tasks, and evaluation provides no task identifier.
Dataset
Tasks
Compositions
Mean ∣Δ∣
Max ∣Δ∣
ρ
Top-3 overlap
Symbol-QA
10
17
1.5
5
0.93
2/3
20
9
1.3
3
0.78
2/3
50
5
1.2
2
0.50
2/3
100
3
0.7
1
0.50
3/3
LLM-QA
10
17
1.3
3
0.95
2/3
20
6
0.3
1
0.94
2/3
Appendix
Table 6: Agreement between the search ranking on the development task order and the final ranking on the report task order. We group search candidates by method composition and represent each composition by its highest-scoring hyperparameter variant at each search stage. The displacement ∣Δ∣ measures the absolute difference between the search and final ranks. Top-3 overlap reports how many of the three highest-ranked final compositions also appear in the top three at that search stage. The comparison set becomes smaller as the search removes candidates, so the 10-task rows provide the most complete comparison.
Composition
Symbol-QA
LLM-QA
Real-QA
km
Mean rank
All anchors + merged LoRA
2
1
3
3
2.00
SI + Replay + merged LoRA
4
2
1
4
2.33
SD + Replay + merged LoRA
1
3
4
4
2.67
Replay + merged LoRA
3
4
2
4
3.00
SI + SD + Replay
5
5
8
8
6.00
SD + merged LoRA
6
8
6
8
6.67
Appendix
Table 7: Cross-dataset ranks of the 16 compositions in the main factorial evaluation. Rows are ordered by the worst rank km , with mean rank used to break ties.
configuration
Final
Diag
Forget
W(10)
W(50)
sd_replay_merge
23.2 ± 4.2
99.5 ± 0.3
77.2 ± 4.1
95.4 ± 1.7
45.9 ± 8.1
si_sd_replay_merge
18.5 ± 2.8
99.3 ± 0.1
81.8 ± 2.8
93.9 ± 3.7
37.0 ± 5.5
si_sd_replay_osrm
17.7 ± 2.6
99.3 ± 0.5
82.8 ± 2.5
94.8 ± 0.3
35.4 ± 5.1
replay_merge
16.6 ± 5.8
99.9 ± 0.0
84.2 ± 5.9
89.0 ± 5.4
33.1 ± 11.6
si_replay_merge
15.7 ± 3.3
99.8 ± 0.2
85.0 ± 3.2
86.5 ± 8.4
31.3 ± 6.5
si_sd_replay
15.4 ± 0.3
99.3 ± 0.8
84.9 ± 0.8
89.7 ± 0.5
30.7 ± 0.6
Appendix
Table 8: All configurations on Symbol-QA , mean ± standard deviation over three seeds. W(k) is the final-checkpoint accuracy averaged over the k most recently trained tasks.
configuration
Final
Diag
Forget
W(10)
W(50)
si_replay_olora
42.2 ± 5.2
99.8 ± 0.0
58.2 ± 5.2
93.9 ± 1.9
60.2 ± 5.7
si_sd_replay_merge
41.8 ± 1.4
99.3 ± 0.1
58.3 ± 1.5
96.3 ± 0.5
65.5 ± 0.8
si_replay_merge
41.2 ± 9.5
99.6 ± 0.2
59.1 ± 9.4
98.2 ± 0.7
64.3 ± 12.1
sd_replay_merge
33.0 ± 1.8
99.4 ± 0.2
67.1 ± 1.8
94.6 ± 2.7
54.9 ± 1.9
replay_merge
32.4 ± 8.9
99.7 ± 0.1
68.0 ± 9.0
97.5 ± 2.3
54.8 ± 13.7
si_replay_osrm
29.7 ± 3.0
99.5 ± 0.1
70.6 ± 3.0
98.4 ± 0.5
51.2 ± 5.5
Appendix
Table 9: All configurations on LLM-QA , mean ± standard deviation over three seeds. W(k) is the final-checkpoint accuracy averaged over the k most recently trained tasks.
configuration
Final
Diag
Forget
W(10)
W(50)
si_replay_olora
56.4 ± 13.2
98.5 ± 0.9
42.7 ± 14.1
87.5 ± 3.2
70.5 ± 13.8
si_replay_merge
54.8 ± 10.4
99.9 ± 0.1
45.5 ± 10.5
97.4 ± 0.5
77.9 ± 9.7
replay_merge
48.2 ± 2.8
99.9 ± 0.0
52.3 ± 2.8
98.1 ± 0.4
75.4 ± 1.9
si_sd_replay_merge
44.3 ± 8.7
99.5 ± 0.0
55.8 ± 8.7
95.5 ± 1.9
72.7 ± 11.0
si_replay_osrm
43.2 ± 2.7
99.9 ± 0.0
57.3 ± 2.8
95.6 ± 0.5
70.7 ± 4.4
sd_replay_merge
39.0 ± 1.9
99.5 ± 0.2
61.1 ± 2.0
96.5 ± 0.6
67.9 ± 1.9
Appendix
Table 10: All configurations on Real-QA , mean ± standard deviation over three seeds. W(k) is the final-checkpoint accuracy averaged over the k most recently trained tasks.
term
effect (percentage)
% var
F
p
replay
+9.49
44.3
161.1
<10−4
merge
+5.87
16.9
61.5
<10−4
sd
+5.66
15.8
57.3
<10−4
replay × merge
+3.57
6.3
22.7
<10−4
si × merge
-3.04
4.6
16.6
0.000287
si × sd × merge
-1.84
1.7
6.1
0.0192
Appendix
Table 11: Complete saturated 24 factorial on Symbol-QA , N=48 , error df 32, RMSE 2.59, 91% of variance explained. The standard error of any effect is 0.75 percentage.
term
effect (percentage)
% var
F
p
replay
+18.49
43.9
341.8
<10−4
merge
+14.91
28.6
222.3
<10−4
replay × merge
+9.44
11.4
89.0
<10−4
si
+5.76
4.3
33.2
<10−4
sd
+5.02
3.2
25.2
<10−4
sd × replay
-3.45
1.5
11.9
0.00159
Appendix
Table 12: Complete saturated 24 factorial on LLM-QA , N=48 , error df 32, RMSE 3.46, 96% of variance explained. The standard error of any effect is 1.00.
term
effect (percentage)
% var
F
p
merge
+20.54
36.4
393.4
<10−4
replay
+19.34
32.3
348.9
<10−4
replay × merge
+11.68
11.8
127.1
<10−4
sd × replay
-10.32
9.2
99.2
<10−4
si
+6.30
3.4
37.0
<10−4
sd × replay × merge
-4.87
2.0
22.1
<10−4
Appendix
Table 13: Complete saturated 24 factorial on Real-QA , N=48 , error df 32, RMSE 3.59, 97% of variance explained. The standard error of any effect is 1.04.
Figure 4: Greedy buildup paths through the 24 factorial. Each step adds the remaining mechanism with the highest final retention under the current composition. Triangles mark the strongest composition along each path.
Figure 5: Temporal accuracy matrices Mi,j along the composition chain, one row per dataset. Each column adds one mechanism and never removes one. The panel outlined in red is the strongest configuration in that row. Panels show the median seed of three, so no panel is a favorable draw.
O(T) low-rank allocation
Symbol-QA
LLM-QA
Real-QA
O-LoRA
−3.3%
+1.0%
+1.6%
Sequential OSRM
−0.8%
−11.5%
−11.6%
Appendix
Table 14: Change in final retention after replacing merged LoRA with O-LoRA or sequential OSRM in the composition selected for each dataset.
Figure 6: General capability after 100 tasks, averaged over three seeds. Anchors and hyperparameters from TSH are fixed across allocation rules: all three anchors on Symbol-QA , and data and weight anchors on LLM-QA and Real-QA . O-LoRA better preserves general capability on the two natural-language datasets.
As new knowledge rapidly accumulates, language models (LMs) with pretrained knowledge quickly become obsolete. A common approach to updating LMs is fine-tuning them directly on new knowledge. However, recent studies have shown that fine-tuning for memorization may be ineffective in storing knowledge or may exacerbate hallucinations. In this work, we introduce a setting we call continual memorization, where a model must memorize and retain a set of factoids through multiple stages of fine-tuning on subsequent datasets. We characterized the forgetting patterns through extensive experiments and show that LMs widely suffer from forgetting, especially when needing to memorize factoids in the second stage. We posit that forgetting can be alleviated by modifying training dynamics: (1) protecting the memorization process when learning factoids or (2) reducing interference from subsequent training stages. Intriguingly, we find that mixing randomly generated word sequences or generic data sampled from pretraining corpora at different training stages effectively mitigates forgetting REMIX: Random and Generic Data Mixing). REMIX can recover performance from severe forgetting, outperforming replay methods and other continual learning baselines. We analyze how REMIX influences the learning process and find that robust memorization follows a distinct pattern: the model stores factoids in earlier layers than usual and diversifies the layers that retain them, which results in easier recall and manipulate of the learned factoids.
Howard Chen, Jiayi Geng, Adithya Bhaskar +2
Princeton Language and Intelligence (PLI), Princeton University
Persistent textual memory allows language models to carry information across long interactions, but learning what to remember is fundamentally a credit-assignment problem. A memory rewrite may only become useful many steps later, while much of the observed utility may be inherited from information already stored before the rewrite. We introduce Memory Gain Policy Optimization (MGPO), which isolates the incremental value of each memory rewrite by crediting it for its marginal contribution to current and future downstream utility. This turns delayed memory utility into a direct learning signal for optimizing what information should persist. We study MGPO on document-level information extraction, where structured supervision makes the effects of individual memory updates directly measurable. MGPO improves extraction while reducing average memory length by nearly 80% relative to the initial memory policy before optimization. The learned memory policy also supports reuse and transfer across domains, downstream models without further training. These results show that effective memory learning depends not only on preserving useful information, but on identifying which memory updates create lasting incremental value.
Jiaming Tang, Mingyan Liu, Armin Sarabi
Department of Electrical Engineering and Computer Science, University of Michigan
A language model that keeps learning online must not slowly erode its own base capabilities. We ask which part of a modern LLM should absorb continual updates, by measuring how each choice changes the model's internal representations. On a 48-layer mixture-of-experts model with a native pretrained N-gram memory, we train a seven-task chain under three update mechanisms with identical data and budgets: updating only the N-gram memory table, continually training one shared LoRA adapter, and training a fresh LoRA per task that is merged into the backbone. We capture fixed-context representations at every layer and task boundary. Updating only the N-gram memory changes the base model's representations of general math and code inputs by 26%/15% (relative RMS), against 64%/35% for merged and 97%/39% for shared LoRA. This drift does not grow with the number of tasks: it reaches 25% after the first task and stays there for all seven, while both LoRA variants keep accumulating. Its generic behaviour is essentially unchanged (no code problem lost, math NLL -0.01), and every change is confined to inspectable, reversible table rows. The cost appears elsewhere: N-gram memory acquires new tasks more weakly and its stored tasks interfere with one another, losing as much old-task NLL as shared LoRA. We show that this forgetting is governed by drift on the learned tasks' own inputs, measured from the checkpoint at which each was learned, not by generic drift (Spearman rho=0.85; rho=0.71 on Llama-3-8B), and we derive the exact relations behind it: with a frozen head, every pairwise logit margin changes by the projection of drift onto a head-difference direction. These results support a design principle for long-running online learning: write new knowledge only into the memory table and keep the backbone fixed; the remaining problem, interference between stored tasks, then lies inside the memory.