Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory

Hao Wu

cs.CLOct 22, 2026

Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory

Authors: Hao Wu

Organizations: Illinois Institute of Technology

Abstract

Our previous analysis found that a model's native hashed N-gram table is where continual updates disturb the base least, but its writes needed backpropagation and changed the pretrained table. We present Delta N-gram, a forward-only write path into that table. The pretrained table T^0 is never modified. Evidence is written into a sparse patch Δon the rows it addresses: a 4.1M-parameter writer emits a target and a gate per (position, hash head) event, and a block-averaged delta rule keeps each row a convex combination of its old value and the gated targets. The writer is trained offline on raw text with the pretraining objective; at use time writing takes forward passes only, and clearing Δrestores the original model. A 0.6B prototype trained to answer questions keeps 16/16 facts after a second write; online gradient writes keep 2--4/16. We study two writers: a plain writer that encodes evidence with the T^0-only model, and a delta writer that reads its current memory while encoding. On Qwen3.8-Flash-Next, the plain writer's memory is document-specific: on 64 held-out news events it lowers continuation NLL by 0.068 nats/token against empty memory and by 0.024 against another document's memory, and 0.063 and 0.013 after 16 intervening articles. The delta writer's memory is more document-specific and keeps more of it under distraction (70% vs 54%). Over 35,000 writes on our previous seven-task chain both writers stay stable and the delta writer drifts less (layer-2 math drift 176% vs 188%); later accuracy changes by -0.26/-0.52pp (-7.29pp for gradient updates) and small-k_i drift is 17.9/17.5% (39.2%). Acquisition is negative, and general-input drift reaches 32.8% (20.75%): format rows read by every example grow to 4--12 times their pretrained norm. Zeroing eight such rows removes over 98% of layer-2 drift on math inputs, together with their NLL gain.

Figures & tables

	In-context writes (linear attn., GDN, KDA)	Online LoRA	Gradient table edits [ 31 ]	Persistent memory block (MemT)	Delta N-gram
Written state	sequence state	low-rank backbone increment	native N-gram table	added memory block (dense GDN-2 state)	sparse patch on the native N-gram table
Write signal	forward projection	backprop	backprop (Adam)	forward (GDN-2 update)	forward writer + D1
Lifetime	cleared at sequence end	permanent	permanent	permanent	permanent (optional decay)
Pretrained parameters	unchanged	frozen; adapter separate	table overwritten	unchanged; module added	unchanged; $T^{0}$ fixed
Use-time cost	near zero	full backward per write; optimizer state per user	same	forward	one $T^{0}$ -only forward + 4M-param writer
Undo / per user	n/a	yes (adapter per user)	needs a copy of the table	yes (state per user)	clear $\Delta$ ; sparse rows per user

Table 1: Ways to learn while in use. “Pretrained parameters” refers to the weights and tables the model shipped with.

Panel of Figure 1 — Figure 1: Delta N-gram. Top : a block of evidence is encoded by the frozen model, the writer emits a target $v_{a}$ and gate $\beta_{a}$ for every (position, hash head) event, and D1 updates the addressed rows of the sparse patch $\Delta$ ; the gate also reads each row’s current patch and write count. Bottom : every forward pass reads $T^{0}+\Delta$ at the hashed addresses of the last two or three tokens through the model’s native read-out. At use time only the solid arrows run, forward only; during offline training the next chunk’s loss is backpropagated through the whole write chain into the writer (dashed red). Clearing $\Delta$ restores the original model.

Method	First	Retained	New
Forward writer + D1	16	16	15
Online gradient writes (SparseAdam, lr 0.01)	16	2	16
Online gradient writes (SparseAdam, lr 0.05)	16	4	16
Full evidence in context	16	16	14
BM25 retrieval, top 2	16	16	14

Table 2: Prototype I, 16 held-out two-write worlds (Qwen3-0.6B-Base). First : the first fact asked after the first write; Retained : the same fact asked after the second write; New : the second fact. Greedy generation, first-word score. Empty memory answers 2/16.

Figure 2: DEV64 during native training. Left: benefit over empty memory ( $\Delta_{\mathrm{empty}}$ ); right: advantage over a donor document’s memory ( $\Delta_{\mathrm{donor}}$ ), without distractors ( $k=0$ ) and after 16 distractor articles ( $k=16$ ). Bars are 95% family-bootstrap intervals.

Distractors $k$	$\Delta_{\mathrm{empty}}$	$\Delta_{\mathrm{donor}}$	self NLL
0	0.0682 [0.0584, 0.0781]	0.0244 [0.0166, 0.0326]	1.7728
1	0.0658 [0.0562, 0.0754]	0.0187 [0.0124, 0.0256]	1.7751
4	0.0648 [0.0545, 0.0754]	0.0167 [0.0113, 0.0226]	1.7761
8	0.0659 [0.0561, 0.0761]	0.0170 [0.0119, 0.0225]	1.7750
16	0.0633 [0.0524, 0.0746]	0.0133 [0.0077, 0.0194]	1.7777

Table 3: DEV64 at the final native checkpoint (

\alpha=1

, 100M target tokens). Nats per target token, 95% family-paired bootstrap intervals. Empty-memory NLL is 1.8409 at every

k

Weights	Encoding	$k=0$	$k=1$	$k=4$	$k=8$	$k=16$
main	$T^{0}$ -only	.0668 / .0234	.0654 / .0179	.0635 / .0168	.0655 / .0163	.0626 / .0127
main	memory-aware	.0626 / .0270	.0627 / .0206	.0648 / .0172	.0613 / .0152	.0647 / .0122
delta	$T^{0}$ -only	.0560 / .0315	.0555 / .0296	.0550 / .0271	.0516 / .0249	.0538 / .0219
delta	memory-aware	.0517 / .0320	.0522 / .0290	.0520 / .0252	.0492 / .0232	.0538 / .0217

Table 4: Weight

\times

encoding factorial on DEV64, all cells at batch size 1. Entries are

\Delta_{\mathrm{empty}}

\Delta_{\mathrm{donor}}

in nats per target token. “Main”: plain writer at 100M; “delta”: memory-aware fine-tune at 100M.

Task	Plain writer	Delta writer	[ 31 ] N-gram
SME	10 $\to$ 7 $\to$ 6	10 $\to$ 7 $\to$ 7	13
NAV	6 $\to$ 5 $\to$ 5	6 $\to$ 6 $\to$ 6	6
HYE	33 $\to$ 28 $\to$ 29	32 $\to$ 32 $\to$ 32	61
AFB	5 $\to$ 2 $\to$ 2	6 $\to$ 2 $\to$ 2	42
SQI	24 $\to$ 27 $\to$ 25	27 $\to$ 27 $\to$ 26	50
MKD	29 $\to$ 27 $\to$ 28	26 $\to$ 27 $\to$ 26	54

Table 5: Seven-task accuracy (out of 64 per task) before the task’s own examples are written

\to

right after they are written

\to

after the seventh task. The last column is the N-gram arm of [ 31 ] right after learning each task.

Task	Head input	Layer 48	Layer 9	[ 31 ] N-gram
SME	23.3%	42.8%	290%	40.4%
NAV	21.3%	32.7%	211%	62.0%
HYE	12.4%	21.1%	52%	32.6%
AFB	11.2%	21.3%	55%	37.3%
SQI	19.9%	31.7%	180%	36.8%
MKD	19.2%	33.6%	265%	26.0%

Table 6: Small-

k_{i}

drift (relative RMS) at the last prompt token, from right after each task is written to the end of the chain (plain writer, H200). Last column: N-gram arm of [ 31 ] at the head input after task 7.

	Plain writer	Delta writer
Math big $K$ , layer 2	188.0%	175.7%
Math big $K$ , layer 2, after task 4	153.0%	94.8%
Math big $K$ , layer 9	135.0%	125.7%
Math big $K$ , layer 48	51.2%	49.3%
Head big $K$ , math / code	41.8% / 23.8%	40.2% / 22.5%
Small $k_{i}$ at the head, mean of six tasks	17.9%	17.5%

Table 7: Plain versus delta writer on the seven-task chain, after the seventh task unless stated. Drift is relative RMS (H200); accuracy and recall are out of 448 (A800).

Condition	F1	Answer NLL
self (own dialogue in memory)	0.44	4.034
empty	0.44	4.018
donor (other dialogue in memory)	0.44	4.049
full context	56.79 [55.33, 58.34]	—
self $-$ empty	0	$+0.016$ [ $-0.006$ , $+0.035$ ]
self $-$ donor	0	$-0.015$ [ $-0.025$ , $-0.004$ ]

Table 8: LoCoMo, categories 1–4 (1,540 questions), delta writer. F1 on 0–100; reference-answer NLL in nats/token, averaged per question. Intervals: 10,000 dialogue-cluster bootstrap resamples.

Figure 3: Big- $K$ drift of the 128 math probes at every layer, before (solid) and after (dashed) zeroing the patches of the eight shared bigram rows in the same state. Each curve is measured against the same writer’s empty state.

State	Layer 2	Layer 9	Head input	Math NLL
Plain, task 1	197.7% $\to$ 2.5%	143.9% $\to$ 3.9%	42.5% $\to$ 12.8%	0.841 $\to$ 0.901
Plain, task 7	188.0% $\to$ 2.9%	135.0% $\to$ 5.1%	41.8% $\to$ 18.8%	0.846 $\to$ 0.902
Delta, task 1	182.3% $\to$ 2.9%	131.7% $\to$ 4.6%	40.6% $\to$ 12.7%	0.849 $\to$ 0.902
Delta, task 7	175.7% $\to$ 3.3%	125.7% $\to$ 5.7%	40.2% $\to$ 27.1%	0.855 $\to$ 0.901

Table 9: Zeroing the patches of the eight shared bigram rows. Big-

K

drift of the math probes (relative RMS against the empty state) and their reference-answer NLL. Empty-state math NLL is 0.9039 for both writers.

Persistent Memory for Language Models

Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.

See all 272 papers

Representation Drift in Continual Learning: LoRA versus N-gram Memory
Oct 10, 2026Hao WuContinual Learning for LLMs Online Continual Learning
Illinois Institute of Technology
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language Models Long-Context Language Modeling
1Qwen Team · 2Nanjing University · 3Zhejiang University
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA

Oct 10, 2026J/K move · Enter open · S save

Representation Drift in Continual Learning: LoRA versus N-gram Memory

Hao Wu

Illinois Institute of Technology

Continual Learning for LLMs Online Continual Learning Representation Drift Memory-Augmented Language Models Catastrophic Forgetting Low-Rank Adaptation

Open paper AskPDF arXiv

A language model that keeps learning online must not slowly erode its own base capabilities. We ask which part of a modern LLM should absorb continual updates, by measuring how each choice changes the model's internal representations. On a 48-layer mixture-of-experts model with a native pretrained N-gram memory, we train a seven-task chain under three update mechanisms with identical data and budgets: updating only the N-gram memory table, continually training one shared LoRA adapter, and training a fresh LoRA per task that is merged into the backbone. We capture fixed-context representations at every layer and task boundary. Updating only the N-gram memory changes the base model's representations of general math and code inputs by 26%/15% (relative RMS), against 64%/35% for merged and 97%/39% for shared LoRA. This drift does not grow with the number of tasks: it reaches 25% after the first task and stays there for all seven, while both LoRA variants keep accumulating. Its generic behaviour is essentially unchanged (no code problem lost, math NLL -0.01), and every change is confined to inspectable, reversible table rows. The cost appears elsewhere: N-gram memory acquires new tasks more weakly and its stored tasks interfere with one another, losing as much old-task NLL as shared LoRA. We show that this forgetting is governed by drift on the learned tasks' own inputs, measured from the checkpoint at which each was learned, not by generic drift (Spearman rho=0.85; rho=0.71 on Llama-3-8B), and we derive the exact relations behind it: with a frozen head, every pairwise logit margin changes by the projection of drift onto a head-difference direction. These results support a design principle for long-running online learning: write new knowledge only into the memory table and keep the backbone fixed; the remaining problem, interference between stored tasks, then lies inside the memory.

Abstract

Persistent Memory for Language Models

Explore similar work

Representation Drift in Continual Learning: LoRA versus N-gram Memory

Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention