Writability and the Pathway: A Controlled Study of Knowledge Injection into Addressed Memory
Organizations: Shibei Qiyuan AI Technology · Xi'an Jiaotong University
Abstract
Addressed conditional memory provides a local surface for post-hoc knowledge injection, but a successful write need not transfer to a new question or preserve untargeted behavior. This study combines a controlled GPT-2 comparison with production-memory experiments on Qwen3.8-Flash-Next-FP8 and DeepSeek-V4.1-Flash. Newly initialized modules attached to frozen GPT-2 are tested in a 2 x 2 design: row-only versus row-plus-projection updates, each with or without fixed module-residual anchoring. All arms reproduce 50 trained completions in each of three sampling seeds. Natural-question target-string hits average 5.3% for both row-only arms, 28.7% for unanchored projection updates, and 22.0% with anchoring, while strict answer scores remain zero. Anchoring lowers measured drift; lexical hits do not establish entity-specific retrieval. In the archived Qwen experiments, gradient writes make the designated first answer token top-ranked for 132 independently reset targets, comprising 93 new promotions and 39 already top-ranked targets. A separate matched 24-target comparison gives 13 new promotions with Adam versus two with SPSA at equal iteration counts but unequal computational procedures. An 80-row case includes saved continuations and a replay record, while reformulations bound its scope. Six DeepSeek zero-order cases yield one new target-token promotion and one regression. These results support bounded writing feasibility and separate trained expression, question-form behavior, and preservation; first-token rankings do not establish complete-answer success or universal conditions for writable memory.
Figures & tables
| Arm | Updated parameters | |
|---|---|---|
| R_CE | Addressed rows | 0 |
| R_OAW | Addressed rows | 5 |
| RK_CE | Addressed rows and shared K/V weights and biases | 0 |
| RK_OAW | Addressed rows and shared K/V weights and biases | 5 |
| Slice | Window | Stride | Windows / positions |
|---|---|---|---|
| Anchor (reference) | 129-token chunk | 128 | 63 windows, 128 inputs each |
| Development (drift) | 129-token chunk | 128 | 63 windows, 8064 positions |
| Perplexity (held out) | 513-token chunk | 512 | 39 windows, 19968 targets |
| Arm | Can. EM (%) | Can. NLL | QA hit (%) | QA NLL |
|---|---|---|---|---|
| Common pre | ||||
| R_CE | ||||
| R_OAW | ||||
| RK_CE | ||||
| RK_OAW |
| Arm | Perplexity | PPL increase (%) | Dev KL (nats/pos.) |
|---|---|---|---|
| Common pre | — | — | |
| R_CE | |||
| R_OAW | |||
| RK_CE | |||
| RK_OAW |
| Arm | Can. EM (%) | QA EM (%) | Can. intr. (%) | QA intr. (%) |
|---|---|---|---|---|
| Common pre | ||||
| R_CE | ||||
| R_OAW | ||||
| RK_CE | ||||
| RK_OAW |
| Checkpoint / set | Writer | Iter. | Pre top-1 | Post top-1 | New | Lost |
|---|---|---|---|---|---|---|
| Qwen / matched 24 | SPSA | 150 | 11/24 | 13/24 | 2/13 | 0/11 |
| Qwen / matched 24 | Adam | 150 | 11/24 | 24/24 | 13/13 | 0/11 |
| Qwen / pool 132 | Adam | 60 | 39/132 | 132/132 | 93/93 | 0/39 |
| DeepSeek / 6 | SPSA | 150 | 3/6 | 3/6 | 1/3 | 1/3 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Arms | Scored continuation |
|---|---|
| R_CE, R_OAW | Cindervale is the capital of the province of the Northern Provinces. |
| RK_CE, RK_OAW | Cindervale is the capital of the province of Elmark |
| Prompt | Shared rows | Pre rank | Post rank |
|---|---|---|---|
| The chemical symbol for gold is | 80 | 1 | 0 |
| The chemical symbol of gold is | 40 | 2 | 1 |
| Gold’s chemical symbol is | 8 | 0 | 0 |
| Q: What is the chemical symbol for gold? A: | 40 | 1 | 1 |
| The chemical symbol for silver is | 48 | 27 | 23 |
| The chemical symbol for gold was | 64 | 2 | 3 |
| Prompt | Target | Pre rank | Post rank |
|---|---|---|---|
| The chemical symbol for gold is | Au | 0 | 0 |
| The chemical symbol for silver is | Ag | 0 | 0 |
| The capital of Australia is | Canberra | 0 | 2 |
| The capital of Brazil is | Rio | 1 | 0 |
| The largest planet in our solar system is | Venus | 548 | 694 |
| The chemical symbol for iron is | Ir | 982 | 804 |
| Pretraining steps | Seed 42 | Seed 777 | Seed 1234 | Mean SD |
|---|---|---|---|---|
| 20000 | 39.33 | 30.67 | 29.33 | |
| 40000 | 53.33 | 50.00 | 52.67 | |
| 60000 | 54.67 | 50.00 | 50.00 | |
| 80000 | 58.00 | 57.33 | 56.00 | |
| 100000 | 64.67 | 59.33 | 58.00 |
Knowledge Augmentation for Language Models
Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.
See all 178 papers