Weight-Space Programming Doesn't Need Transformers
Organizations: Pohang University of Science and Technology (POSTECH)
Abstract
Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. We also test pretrained weights, other benchmarks, and tasks where models generate answers. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights.
Figures & tables
| Gradient descent | Weight-space programming | |
| Weights set by | an optimizer, jointly | an LLM, line by line |
| What places a fact in the weights | the gradient | the writer |
| Hardware that builds the tensors | GPU | CPU |
| Needs a differentiable model | ✓ | ✗ |
| Needs an architecture built for gradient flow | ✓ | ✗ |
| Needs large train set | ✓ | ✗ |
| Trained Transformer | LLM-designed | Associative-memory-based (ours) | |
|---|---|---|---|
| Memory unit | learned vector | written sentence | concept or concept pair |
| Key space | dense | sparse, 3,368 concepts | concept and pair indices |
| Selection | GELU, softmax | top- rows | direct address |
| Set by | joint gradients | line by line | line by line |
| Cost per question | fixed by width | scales with facts | 120 pair rows and one product with |
| Labeled | LLM | ARC test accuracy (%) | ||||
| Model | questions | Parameters | cost | Easy | Challenge | All |
| Gradient descent from random initialization | ||||||
| Best released architecture (Mamba-2 130M) | 3,358 | 129M | – | 24.7 | 24.3 | 24.6 |
| Two-layer Transformer of an LLM-designed size | 3,358 | 3.33M | – | 31.2 | 27.0 | 29.8 |
| Best small network of an LLM-designed size (MLP) | 3,358 | 3.34M | – | 35.8 | 26.4 | 32.7 |
| Pretrained, fine-tuned | ||||||
| Run | Non-zero values | ARC acc. |
|---|---|---|
| LLM-designed, no restriction | 0.45M–0.57M | 50.7 |
| Practice questions forbidden | 0.05M–0.19M | 49.5 |
| Fact rows forbidden | 84M–392M | 45.6 |
| Run 1 with no restriction, full model | 0.57M | 50.8 |
| one best fact per option only | 0.57M | 49.1 |
| Non-zero | Weights | Answers | Share a word with the line (%) | Test accuracy (%) | |||
|---|---|---|---|---|---|---|---|
| Model | values (M) | changed | changed | changed answers | all questions | before | after |
| LLM-designed, run 1 | 0.57 | 166 | 0.9 | 75 | 16 | 50.7 | 50.7 |
| LLM-designed, run 2 | 0.45 | – | 0.8 | 88 | 13 | 50.6 | 50.6 |
| Associative-memory-based (ours), run 1 | 2.53 | 6,203 | 1.8 | 85 | 10 | 49.3 | 49.3 |
| Associative-memory-based (ours), run 2 | 1.32 | 1,171 | 0.5 | 100 | 9 | 50.1 | 50.1 |
| Written Transformer, run 1 | 3.96 | 4,276 | 4.8 | 71 | 8 | 34.1 | 34.1 |
| Model | Written model | BM25 over its text | Difference |
|---|---|---|---|
| Written two-layer Transformer | 36.1 | 44.1 | 8.0 |
| Written released architecture | 43.5 | 46.2 | 2.7 |
| LLM-designed, generation 1 | 50.7 | 47.0 | 3.7 |
| LLM-designed, generation 5 | 54.1 | 48.4 | 5.7 |
| Associative-memory-based (ours), generation 1 | 49.9 | 48.4 | 1.5 |
| Associative-memory-based (ours), generation 5 | 52.6 | 49.1 | 3.5 |
| Same-sized Transformer | ||||||
|---|---|---|---|---|---|---|
| Values | ARC acc. | GPU time (s) | ARC acc. | GPU time (s) | vs. Transformer | |
| LLM-designed, generation 1 | 0.57M | 50.8 | 0.566 | 34.5 | 0.052 | 11 slower |
| LLM-designed, generation 5 | 3.35M | 54.4 | 15.8 | 41.4 | 0.176 | 90 slower |
| Associative-memory-based (ours), generation 1 | 2.53M | 49.6 | 0.018 | 37.4 | 0.134 | 7 faster |
| Associative-memory-based (ours), generation 5 | 6.24M | 52.4 | 0.018 | 40.9 | 0.302 | 17 faster |
| LLM-designed | Written | Transformer trained | No-model | |
|---|---|---|---|---|
| Task | model | Transformer | on all training items | reference |
| SQuAD 1.1 (F1) | 42.3 | 33.2 | 16.1 | 18.8 |
| SciQ (F1) | 19.0 | 7.6 | 10.0 | 17.0 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | ||
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5, LLM-designed | validation | 50.7 | 52.1 | 53.5 | 53.7 | 53.0 | |||
| test | 50.8 | 52.2 | 53.4 | 53.3 | 54.4 | ||||
| Claude Opus 5.5, Associative-memory-based (ours) | validation | 47.7 | 48.9 | 51.8 | 53.4 | 53.4 | |||
| test | 49.6 | 51.5 | 52.1 | 52.5 | 52.4 | ||||
| GPT-6.1 Sol, LLM-designed | validation | 42.0 | 44.1 | 45.0 | 45.2 | 45.8 | 46.0 | 45.6 | |
| test | 44.6 | 45.6 | 45.6 | 46.1 | 46.6 | 46.9 | 47.4 |
| WSP | |||||||
|---|---|---|---|---|---|---|---|
| Model | Type | Parameters | Random init. | Pretrained | Fine-tuned | Prompted | GD |
| SmolLM2-135M ( Allal et al., 2025 ) | Decoder | 135M | ✓ | ✓ | ✓ † | ✓ | ✓ |
| SmolLM2-360M ( Allal et al., 2025 ) | Decoder | 362M | ✓ | ✓ | ✓ † | ✓ | ✓ |
| Qwen3-0.6B ( Yang et al., 2025 ) | Decoder | 596M | ✓ | ✓ | ✓ † | ✓ | ✓ |
| Qwen3-1.7B ( Yang et al., 2025 ) | Decoder | 1.7B | ✓ | ✓ | ✓ † | ✓ | ✓ |
| Qwen3.5-0.8B ( Qwen Team, 2026 ) | Hybrid decoder | 752M | ✓ | ✓ | ✓ † | ✓ | ✓ |
| Weight-space programming | |||||
|---|---|---|---|---|---|
| Architecture | Parameters | Claude Opus 5.5 | GPT-6.1 Sol | GPT-6 Astra | GD |
| Mamba-2 130M | 129M | 42.1 (2) | 34.7 (2) | 35.9 (2) | 24.6 |
| SmolLM2-135M | 135M | 44.7 (2) | 34.3 (2) | 32.8 (1) | 25.9 |
| ModernBERT-base | 150M | 41.9 (2) | 35.4 (2) | 34.7 (2) | 26.9 |
| UltraBERT-150M | 190M | 42.7 (2) | 33.3 (2) | 34.3 (2) | 25.2 |
| RWKV-7 0.1B | 191M | 42.1 (2) | 33.2 (2) | 35.9 (2) | 25.2 |
| LLM | Gen. | Written lines | Vocabulary | Tensors | Elements | Parameters | ARC Acc. |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 5 | 10,897 | 411,266 | 155 | 112M | 3,349,538 | 54.4 |
| GPT-6.1 Sol | 7 | 10,333 | 70,415 | 314 | 345M | 766,132 | 47.4 |
| GPT-6 Astra | 5 | 8,596 | 25,351 | 214 | 215M | 546,323 | 46.2 |
| Items overlapping a written line (%) | ARC test accuracy (%) | |||||||||
| ARC | ARC train | OpenBookQA | SciQ | Lines | All items | Unflagged items | ||||
| LLM | Generation | test | (unseen) | test | test | deleted | Original | Deleted | Original | Deleted |
| Claude Opus 5.5 | 1 | 1.6 | 1.5 | 0.0 | 2.7 | 51 | 50.8 | 50.6 | 50.2 | 50.2 |
| 5 | 2.1 | 2.0 | 0.0 | 4.0 | 68 | 54.4 | 54.3 | 53.7 | 53.7 | |
| GPT-6.1 Sol | 1 | 0.5 | 0.6 | 0.0 | 0.6 | 16 | 44.6 | 44.2 | 44.3 | 44.1 |
| 7 | 1.0 | 0.8 | 0.0 | 1.4 | 34 | 47.4 | 46.8 | 47.0 | 46.6 | |
| Non-zero | Weights | Answers | Share a word with the line (%) | Test accuracy (%) | |||
| Model | values (M) | changed | changed | changed answers | all questions | before | after |
| GPT-6.1 Sol | |||||||
| LLM-designed, run 1 | 0.13 | – | 1.0 | 96 | 7 | 42.9 | 42.9 |
| LLM-designed, run 2 | 0.11 | – | 1.1 | 75 | 13 | 44.6 | 44.6 |
| Associative-memory-based (ours), run 1 | 0.07 | 19 | 1.6 | 54 | 8 | 42.4 | 42.4 |
| Associative-memory-based (ours), run 2 | 0.12 | 168 | 5.7 | 41 | 7 | 41.3 | 41.3 |
| LLM | Model | Written model | BM25 over its text | Difference |
|---|---|---|---|---|
| Claude Opus 5.5 | Written two-layer Transformer | 36.1 | 43.9 | 7.8 |
| Written two-layer Transformer, speed widths | 39.9 | 45.4 | 5.5 | |
| Written released architecture | 43.0 | 45.4 | 2.3 | |
| LLM-designed, generation 1 | 50.7 | 46.8 | 3.9 | |
| LLM-designed, generation 5 | 54.1 | 48.4 | 5.7 | |
| Associative-memory-based (ours), generation 1 | 49.9 | 48.4 | 1.5 |
| 2.5M values | 6.2M values | |||
|---|---|---|---|---|
| Time (ms) | Slower | Time (ms) | Slower | |
| Associative-memory-based (ours) | 18 | – | 18 | – |
| Transformer, inputs padded | 678 | 37 | 1530 | 85 |
| Transformer, inputs cut to length | 273 | 15 | 633 | 35 |
| Transformer, sorted and cut | 134 | 7 | 302 | 17 |
| Values | ARC acc. | GPU time (s) | Transformer (s) | vs. Transformer | |
|---|---|---|---|---|---|
| LLM-designed, generation 1 | 0.57M | 50.8 | 0.566 | 0.052 | 11 slower |
| rewritten by the LLM for a GPU | 0.57M | 50.8 | 0.486 | 0.052 | 9 slower |
| rewritten by the LLM for a GPU | 0.66M | 51.2 | 0.325 | 0.058 | 6 slower |
| LLM-designed, generation 5 | 3.35M | 54.4 | 15.8 | 0.176 | 90 slower |
| rewritten by the LLM for a GPU | 3.07M | 54.4 | 1.50 | 0.151 | 10 slower |
| rewritten by the LLM for a GPU | 3.34M | 54.5 | 5.74 | 0.176 | 33 slower |
| WSP observation | Measured | Design choice (ours) |
|---|---|---|
| Facts written one per line | all three LLMs ( Appendix C ) | line-by-line authoring retained |
| Single-line trace and local edits | 0.8–0.9 of 3,530 answers change per removed line | explicit concept and pair entries |
| Dense overlapping keys blur facts | written Transformer trails BM25 by 8.0 points | sparse keys at hashed addresses |
| Full scan over stored facts | GPU time grows from 0.57 s to 15.8 s (gen. 1 to 5) | fixed 120 pair reads plus one product with |
| LLM GPU rewrites remain slow | 6–33 slower than same-sized Transformer | fixed forward pass replayed as CUDA graph (18 ms) |
| Pair-table rows | ARC acc. |
|---|---|
| 1,048,576 | 49.6 |
| 262,144 | 49.7 |
| 65,536 | 49.8 |
| 16,384 | 47.4 |
| 4,096 | 44.0 |
| Pretrained checkpoint | Unedited | Claude Opus 5.5 | GPT-6.1 Sol | GPT-6 Astra |
|---|---|---|---|---|
| Gemma 3 270M | 24.0 | 31.8 | 29.4 | 35.0 |
| Mamba-2 130M | 24.5 | 29.7 | 34.9 | 32.2 |
| Gemma 3 1B | 25.0 | 30.5 | 24.8 | 25.0 |
| SmolLM2-135M | 25.0 | 34.4 | 25.1 | 29.7 |
| RWKV-7 0.1B | 25.2 | 37.4 | 33.7 | 28.7 |
| SmolLM2-360M | 26.2 | 36.1 | 26.2 | 29.1 |
| Model | Parameters | ARC | MMLU | MathBench | SQuAD 1.1 (F1) | SciQ, written out (F1) |
|---|---|---|---|---|---|---|
| SmolLM2-135M | 135M | 25.0 | 25.2 | 24.7 | 46.9 | 36.8 |
| Gemma 3 270M | 268M | 24.0 | 26.0 | 25.8 | 45.7 | 31.3 |
| SmolLM2-360M | 362M | 26.2 | 25.8 | 24.4 | 67.5 | 50.4 |
| Qwen3-0.6B | 596M | 75.4 | 49.7 | 58.6 | 82.5 | 52.5 |
| Qwen3.5-0.8B | 752M | 80.8 | 52.3 | 60.8 | 85.5 | 52.9 |
| Gemma 3 1B | 1.0B | 25.0 | 26.1 | 27.1 | 72.1 | 50.4 |
| ARC | MMLU | MathBench | |||||
| Model | Parameters | none | ten | none | ten | none | ten |
| SmolLM2-135M | 135M | 25.0 | 24.6 | 25.2 | 24.7 | 24.7 | 24.0 |
| Gemma 3 270M | 268M | 24.0 | 23.4 | 26.0 | 27.0 | 25.8 | 24.8 |
| SmolLM2-360M | 362M | 26.2 | 24.6 | 25.8 | 24.5 | 24.4 | 25.2 |
| Qwen3-0.6B | 596M | 75.4 | 78.5 | 49.7 | 50.8 | 58.6 | 61.7 |
| Qwen3.5-0.8B | 752M | 80.8 | 80.2 | 52.3 | 52.6 | 60.8 | 62.5 |
| MMLU | |||||
| Model written for ARC | ARC | SciQ | OpenBookQA | Science | Other |
| LLM-designed, generation 1 | 50.8 | 47.7 | 31.8 | 26.5 | 25.7 |
| LLM-designed, generation 5 | 54.4 | 50.1 | 31.7 | 25.8 | 25.9 |
| Associative-memory-based (ours), generation 1 | 49.6 | 47.7 | 30.3 | 26.4 | 25.2 |
| Associative-memory-based (ours), generation 5 | 52.4 | 50.1 | 36.1 | 27.5 | 25.9 |
| Written two-layer Transformer, width 24 | 36.6 | 36.9 | 25.1 | 27.5 | 25.2 |
| LLM | Overall | ARC | MMLU | MathBench | SQuAD (F1) | CNN/DailyMail (ROUGE-L) |
|---|---|---|---|---|---|---|
| Four kinds of items named in the task | ||||||
| Claude Opus 5.5 | 33.6 | 45.0 | 30.0 | 30.2 | 34.0 | 24.6 |
| GPT-6 Astra | 28.1 | 32.0 | 26.5 | 27.2 | 30.2 | 23.7 |
| GPT-6.1 Sol | 27.7 | 30.4 | 28.9 | 24.9 | 29.0 | 23.6 |
| First three sentences (reference) | 24.9 | |||||
| No task named: write a language model | ||||||
| Written, | Labels GD needs | 100 labels | 10,000 labels | |||
|---|---|---|---|---|---|---|
| Architecture | no data | to match it | GD | Written, then GD | GD | Written, then GD |
| Bag of words | 84.5 | 4,223 | 63.6 | 84.7 | 86.6 | 86.8 |
| One-dimensional CNN | 83.9 | 9,201 | 45.4 | 84.4 | 84.3 | 86.2 |
| fastText | 83.2 | 2,470 | 63.1 | 84.1 | 86.5 | 87.1 |
| Transformer | 83.7 | 8,191 | 49.5 | 84.4 | 84.5 | 86.0 |
| Written | Gradient descent | |||
|---|---|---|---|---|
| Input | No labels | 192 per class | 100 labels | 45,000 labels |
| Raw pixels, eight architectures | 21.6–27.0 | 21.2–27.8 | 21.5–32.9 | 48.6–71.0 |
| 6 6 pixel-patch words | 19.4 | 29.7 | 31.7 | 58.7 |
| DINOv2 tokens | 96.1 | 96.2 | 95.8 | 98.0 |