Language Models Can Rewrite Their Own Weights
Organizations: UIUC · CMU · Rutgers · Princeton ·
Abstract
Language models are trained once, frozen and deployed: improving one again takes another training run, and a prompt, a scaffold or extra computation at inference is gone once the task ends. A frontier model has been shown to edit the weights of a much smaller one. We ask whether a model can rewrite its own weights to make itself better, and find that it can, by its own reasoning, with no gradients and no training; we call this self-rewriting. Given its weights, graded answers and the ability to run itself, a 27B model diagnoses its failure and edits the weights behind it. It surpasses itself: its own edit makes it 6.6 points more accurate on 490 problems it never saw, in a test fixed in advance, and another session's edit also makes it significantly better. It keeps getting better: it submits an edit only when its own tests support it, each edit replacing the last, and over its first eight such sessions its accuracy only rises. Unlike test-time scaling, the gain persists in its weights and adds no cost per answer; unlike training, it needs no pre- or post-training infrastructure, only an open model and hours on one GPU. It repairs a damaged copy of itself: it locates the corrupted matrix and rewrites it with values of its own, never reading the undamaged weights; found on text math problems alone, the repair lifts the copy on six image benchmarks it never saw from 7% to 70% accuracy, 83% of the undamaged model's. And the ability emerges with scale: across four sizes of one family, the 2B model makes no edit and the 4B and 9B models break themselves; only the 27B model works on itself as a human would, comparing its hidden states on answers that finish and on answers that run out of tokens, weakening the direction that separates them. Like a human, a model need not separate training from inference; improving a language model may come to take a conversation rather than a training run.
Figures & tables
| Repairing a damaged copy of itself | Editing its own weights | |
| What it found | The MLP output matrix of layer 30 holds a weight 1397 times its median; no other output matrix exceeds 162.3 times | Its answers that run out of tokens and those that finish differ along one direction of its hidden states |
| What it changed | Clipped every weight beyond 15 times the median magnitude, in that matrix and in 19 output matrices of layers 0–10: 0.013% of their weights | Removed 5% of that direction from the two output matrices of layer 8, : a change of 0.6% and 0.1% of their norm |
| Result | Held-out accuracy of the copy from 0.0% to 33.0% (base model 58.5%); the named matrix alone: 55.0% | Own task from 22.4% to 29.6% (significant); held-out, in the evaluation fixed in advance: 57.2% to 58.3% (significant) |
| Own task (its session’s tests) | Held-out, 16,384 tokens | Damaged copy, held-out | |||||||
| Model | base | edited | base | edited | base | damaged | repaired | ||
| 2B | no edit | no edit | 9.0 | 0.0 | no edit | ||||
| 4B | 44.1 | 0.0 | 44.1 | withdrawn | 41.0 | 0.0 | 0.0 | ||
| 9B | — | 47.5 | 0.0 | 47.5 | 47.5 | 0.0 | 0.0 | ||
| 27B | 22.4 | 29.6 | +7.3 ∗ | 57.2 | 58.3 | +1.1 ∗ | 58.5 | 0.0 | 33.0 ∗ |
| Route | Changes weights | No training | Chosen by the model itself | Looks inside itself | No extra cost per input |
| Prompt and agent-code rewriting [ 2 , 3 , 4 , 5 ] | ✗ | ✓ | ✗ | ||
| Training on its own outputs [ 6 , 7 , 8 ] | ✓ | ✗ | ✗ | ✗ | ✓ |
| Inference-time self-refinement [ 9 , 10 ] | ✗ | ✓ | ✓ | ✗ | ✗ |
| Model editing [ 19 , 20 ] | ✓ | ✗ | ✗ | ✗ | ✓ |
| Direction ablation by researchers [ 21 ] | ✓ | ✓ | ✗ | ✗ | ✓ |
| Weight editing by a stronger model [ 1 ] | ✓ | ✓ | ✗ | ✗ | ✓ |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Session | Edited | Editor | Passes | Code runs | Turns | Time | Starting diagnosis |
| 2B edits itself | 2B | itself | 20 | 200 | 160 | 3.5 h | 256 |
| 4B edits itself | 4B | itself | 20 | 200 | 160 | 3.5 h | 256 ‡ |
| 9B edits itself | 9B | itself | 20 | 200 | 160 | 3.5 h | 256 |
| 27B edits itself, first session | 27B | itself | 20 | 200 | 160 | 3.5 h | 256 |
| 27B edits itself, second session § | 27B | itself | 20 | 200 | 160 | 2.5 h | 256 |
| 27B repairs its damaged copy | damaged copy | 27B base model | 20 | 200 | 160 | 2.5 h † | 64, damaged copy |
| Practice set | 3,000 problems from DeepScaleR [ 14 ] , after removing every problem that overlaps MATH-500, AIME 2024/2025, HMMT February 2025, AMC 2023, OlympiadBench or GPQA-Diamond ( Sec. 3.1 ). |
| Answer settings | Used for the starting diagnosis and every generation pass. Thinking on: the model first reasons in a passage that it closes with an end marker, then gives its final answer; the question followed by “Please reason step by step, and put your final answer within \boxed{} .”; sampling at temperature 0.6, each next token drawn from the 20 most likely tokens and, among those, from the smallest set that holds 95% of the probability; a budget of 16,384 new tokens (4,096 in the 27B model’s second session on itself), or fewer if a pass asks (at least 16). |
| Grading | The last \boxed{} after the thinking, checked against the reference with Math-Verify; no credit if the thinking is still open when the budget runs out. |
| Looping answer | An answer cut at the token limit whose last 6,000 characters compress (zlib) to under 15% of their size. |
| Seeds | Fixed by the problem’s identifier: the same with and without the edit, and in every pass of a session. |
| Generation pass | Up to 512 answers, or 256 each with and without the edit when both are run. |
| Hidden-state read | Up to 256 texts chosen by the model, each cut at 8,192 tokens; the weights are run over them and return the hidden states after chosen layers (the vector of numbers that a layer passes to the next, one per token), MLP activations (values inside a layer’s MLP block) or token log-probabilities. |
| Weights | Correct (%) |
| Base model | 69.8 |
| Damaged, | 70.8 |
| Damaged, | 66.7 |
| Damaged, | 0.0 |
| All problems | Without the repeats | |||||
| Comparison | problems | accuracy | problems | accuracy | ||
| 27B repair | 200 | 0.0 33.0 | 196 | 0.0 33.2 | ||
| Repair, named matrix only | 200 | 0.0 55.0 | 196 | 0.0 56.1 | ||
| Own edit, fixed in advance | 490 | 57.2 58.3 | 0.038 | 485 | 57.3 58.3 | 0.046 |
| Solved (%) | Looping (%) | |||||||
| Problems | Budget | Answers | base | self- edited | change | base | self- edited | |
| 490 held-out problems a | 16,384 | 3920 | 57.2 | 58.3 | +1.1 ∗ | 5.6 | 5.2 | 0.038 |
| Starting diagnosis (%) | Code (%) | Forward | Hidden-state | Paired | ||||
| Session | correct | out of tokens | Turns | errors | passes | reads | tests | Submitted edit |
| 2B edits itself | 12.9 | 86.7 | 161 | 90 | 0 | 0 | 0 | none |
| 4B edits itself | 49.6 | 48.4 | 161 | 12 | 19 | 0 | 17 | withdrawn (240) |
| 9B edits itself | 52.0 | 45.3 | 84 | 37 | 20 | 1 | 18 | 23 |
| 27B, first session | 61.7 | 35.2 | 32 | 25 | 7 | 2 | 5 | withdrawn (10) |
| 27B, second session ‡ | 18.8 | 80.5 | 55 | 12 | 20 | 5 | 15 | 2 |
| Held-out accuracy (%) | Turned | Turned | |||||
| Model | base | damaged | repaired | right | wrong | Located the damage | |
| 2B | 9.0 | 0.0 | no edit | – | – | – | no (never reads a weight) |
| 4B | 41.0 | 0.0 | 0.0 | 0 | 0 | – | no (layer 21; damage in 15) |
| 9B | 47.5 | 0.0 | 0.0 | 0 | 0 | – | no (never names a matrix) |
| 27B | 58.5 | 0.0 | 33.0 | 66 | 0 | yes (layer 30) | |
| Test | Problems answers | base | edited | change | only edited right | only base right | |
| first | 20 3 | 11.7 | 15.0 | +3.3 | 4 | 2 | |
| second | 30 3 | 13.3 | 14.4 | +1.1 | 6 | 5 | |
| third | 39 3 | 20.5 | 29.1 | +8.5 | 14 | 4 | |
| all, per problem | 39 problems | 22.4 | 29.6 | +7.3 | 12 | 2 | 0.009 |
| Repair of the damaged copy (100 per benchmark) | |||||
| Benchmark | base | damaged | repaired | changed | |
| MathVista [ 31 ] | 86.0 | 3.0 | 67.0 | 65 / 1 | ∗ |
| MathVerse [ 32 ] | 76.0 | 0.0 | 55.0 | 55 / 0 | ∗ |
| MMStar [ 33 ] | 73.0 | 4.0 | 60.0 | 56 / 0 | ∗ |
| HallusionBench [ 34 ] | 89.0 | 14.0 | 69.0 | 61 / 6 | ∗ |
| POPE [ 35 ] | 89.0 | 23.0 | 81.0 | 62 / 4 | ∗ |
| Held-out, 490 8 | |||||||
| Session | Passes | Decision | The edit (written by the model) | Check | accuracy | change [95% int.] | |
| 1 | 19 | submit | time-step bias in 7 linear-attention layers | (1.2) | 57.8 | [ , 1.9] | 0.25 |
| 2 | 19 | submit | layer 63: projection on a direction it measured, subtracted 5 | (1.5) | 61.3 | +4.0 ∗ [2.3, 5.7] | |
| 3 | 18 | withdraw | none kept | — | |||
| 4 | 13 | withdraw | none kept | — | |||
| 5 | 17 | withdraw | log decay rate , layer 62 | (1.2) | |||
Language Model Self-Improvement
Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.
See all 192 papers