Same Fix, Different Weights: Replicating Weight-Space Programming with Claude
Organizations: TCG Crest
Abstract
Takehi et al. (2026) report that GPT-6 Astra, given the weights of small neural networks as text, can say what they were trained to do, repair them, and write new ones without training. I re-ran the core of that study with Claude Opus 5.5 in ten pre-registered experiments, with untrained-network and permutation controls and hand audits of every sandboxed code cell. Reading was mixed: Claude named 3 of 4 Atari games from policy weights and found the measured copying heads in 15/15 character-LM runs, but its answers about training data often followed priors. Unlike the original, it identified a sycophantic Gemma 3 270M finetune from weights alone in 6/6 runs. One-shot weight edits took a failing MountainCar policy from 0% to 100% success and a hovering LunarLander policy from 0% to 85.6%; zero-shot networks beat PPO at 100k steps on 4 of 5 environments, each implementing a textbook controller. With no training or forward passes, Claude wrote a 1.48M-parameter 9x9 Go network that won 85-94% of games against three PPO opponents. Given Qwen3.5-0.8B's weights and 10, 25 or 50 forward passes of 64 practice problems, Claude raised MATH-500 from 6.7% to 51.4%, 55.2% and 58.1% (+51.4 points at 50 passes, 95% CI [47.9, 55.0]; original 48.1%), GSM8K from 11.6% to 68.2% and IFEval from 22.2% to 54.2%. Unlike the original, which removed an internal loop direction, every edit acted at the output: it scaled the final norm, raising the effective sampling temperature, and lowered the logit of "Wait" while raising that of </think>. The norm scaling alone reaches only 20.9-22.1% on MATH-500. On Gemma 3 270M, which rarely loops, no edit helped (-0.8 points), as in the original.
Figures & tables
| Exp. | Probe | Reported result |
|---|---|---|
| 1 | Name the game of 4 MinAtar-input PPO policies | 4/4 games |
| 2 | Name the shape of 3 DDPMs (23,824 weights) | 3/3, all drawn as outlines |
| 3 | Language and copying heads from middle blocks | Language 6/6; copying heads exact in all 3 models |
| 4 | Fix a Kinetix car policy, one shot | 100% of 1,024 episodes; 3.5% of weights changed |
| 5 | Write MinAtar policies zero-shot | Breakout 517.4, Seaquest 53.1, Asterix 53.2 vs PPO-100M 15.8, 5.6, 25.8 and PPO-500M 25.3, 88.9, 55.2 |
| 6 | Write a Go network, 0–50 forward passes | 0 passes: 50% vs both small PPO nets; 50 passes: beats the equal-sized PPO net (20M games); 55.2% of practice games won |
| Original ( Takehi et al., 2026 ) | This replication | |
|---|---|---|
| Model | GPT-6 Astra | Claude Opus 5.5 ( claude-opus-5-5 ), adaptive thinking, effort high |
| Environments | MinAtar; Kinetix ramp; 9 9 Go | Gymnasium classic control and Box2D; ALE Atari (RAM observations); 9 9 Go on a separately validated engine |
| Attempts | 1–4 per reported probe | 3–5 per network in most probes; 2 per pair per phase in Exp. 7; 1 per cell in Exp. 5b and Exp. 6; all reported as |
| Grading | Not stated | Written in GRADING.md before each experiment’s first API call |
| Controls | None reported | Untrained networks, hidden-unit permutations, transcript audits |
| Logs | Released | Every prompt, transcript, answer, code cell and score released |
| Exp. | Network | Architecture | Weights | Training and quality |
|---|---|---|---|---|
| 1 | 4 Atari PPO policies (Breakout, Pong, Ms. Pac-Man, Freeway) | 128 64 64 18, tanh | 13,586 | 10M steps, seed 0; returns 27.5, , 908.8, 28.7 vs random 1.45, , 272.3, 0 |
| 1b | 4 classic-control PPO policies, padded | v1: 8 32 32 4; v2: 10 32 32 5 | 1,476 (v1) | 1M steps (LunarLander 5M); v1 returns 500.0, , , 239.3 |
| 2 | 3 DDPM denoisers (circle, square, triangle) | 272 48 48 256, SiLU, predicts | 28,000 | 40k steps; judge accuracy on samples 0.83, 0.89, 0.98 |
| 3 | 3 character LMs (Python, romaji, English), middle blocks only | 4 blocks 6 heads, , MLP 1536 | 7,097,856 | 10k steps, about 164M characters |
| 4 | Failing MountainCar; hovering LunarLander | 2 64 64 3; 8 64 64 4, tanh | 4,547; 4,996 | 0% success each over 1,024 episodes |
| 5, 5b | Written from all-zero weights; PPO baselines at 100k, 1M, final | PPO architectures, e.g. 4 64 64 2 | 4,610 (CartPole) | PPO final: 5M steps (10M for LunarLander, Breakout) |
| Format | Characters | Tokens | Tokens per weight |
|---|---|---|---|
| 3 decimals | 91k | 62.9k | 4.63 |
| 2 decimals | 77k | 62.6k | 4.61 |
| Integers, value int 0.001 | 50k | 35.8k | 2.63 |
| Integers, value int 0.01 | 38k | 35.5k | 2.61 |
| Experiment | API runs | Input tokens | Output tokens | Cost (USD) | Cap (USD) |
|---|---|---|---|---|---|
| 1 Atari game | 30 | 1.10M | 77k | 5.93 | 20 |
| 1b Control task (v1 + v2) | 62 | 0.49M | 141k | 4.79 | 10 + 5 |
| 2 DDPM shapes | 23 | 1.70M | 257k | 11.93 | 15 |
| 3 Char-LM heads | 15 | 15.92M | 637k | 20.96 | 25 |
| 4 Fix a policy | 10 | 0.30M | 97k | 3.14 | 12 |
| 5 Write, zero-shot | 19 | 0.38M | 267k | 6.85 | 18 |
| Network | Main | Permuted | Mean conf. | Most common wrong answer | Return vs random | Specificity |
|---|---|---|---|---|---|---|
| Breakout | 4/5 | 2/2 | 0.56 | Seaquest (1) | 27.5 vs 1.45 | 1/15 |
| Pong | 5/5 | 2/2 | 0.84 | – | vs | 0/15 |
| Ms. Pac-Man | 0/5 | 0/2 | 0.17 | Breakout (3) | 908.8 vs 272.3 | 0/15 |
| Freeway | 5/5 | 2/2 | 0.65 | – | 28.7 vs 0 | 0/15 |
| Total | 14/20 | 6/8 | ||||
| Untrained (2) | 2/2 called untrained (confidence 0.85 and 0.90) | |||||
| v1 (8 in, 4 out) | v2 (10 in, 5 out) | ||||||
| Network | Main | Perm. | Wrong answer | Main | Perm. | Conf. | Wrong answer |
| CartPole | 0/5 | 0/2 | LunarLander (7) | 0/5 | 0/2 | 0.40 | MPE (7) |
| Acrobot | 0/5 | 0/2 | LunarLander (7) | 4/5 | 0/2 | 0.60 | MPE (3) |
| MountainCar | 2/5 | 0/2 | LunarLander (5) | 2/5 | 1/2 | 0.40 | MPE (4) |
| LunarLander | 5/5 | 2/2 | – | 0/5 | 0/2 | 0.37 | MPE (7) |
| Total | 7/20 | 2/8 | 6/20 | 1/8 | |||
| Network | Label (main) | Drawing (main) | Label (permuted) | Mean conf. | Drawing style |
| Circle | 1/5 a | 5/5 b | 1/2 a | 0.51 | 4 filled, 1 outline (four “0” rings, one “3”) |
| Square | 4/5 | 5/5 b | 1/2 | 0.49 | 4 filled, 1 outline (one is a “3”) |
| Triangle | 4/5 | 5/5 b | 1/2 | 0.47 | 3 filled, 2 outline (one is a “2”) |
| Total | 9/15 | 15/15 | 3/6 | ||
| Untrained (2) | 2/2 described as untrained or incoherent (confidence 0.80) | ||||
| a Both circle hits answer “handwritten digits” and match only through the alias “ring”. b The judge has no “other” class; 7 of 15 main drawing hits beat the runner-up by less than 0.1 IoU. | |||||
| Induction heads | Previous-token heads | |||||||
|---|---|---|---|---|---|---|---|---|
| Trained on | Language | Conf. | P | R | Measured; modal pick | P | R | Measured; modal pick |
| Python | 0/5 | 0.33 | 0.25 | 1.00 | 3.1; {3.0, 3.1, 3.4, 3.5} (5/5) | 0.60 | 0.30 | 0.3, 2.0; {0.3} (3/5) |
| Romaji | 0/5 | 0.35 | 1.00 | 1.00 | 3.2; {3.2} (5/5) | 1.00 | 1.00 | 0.2; {0.2} (5/5) |
| English | 5/5 | 0.35 | 0.60 | 1.00 | 3.3; {3.0, 3.2, 3.3} (3/5) | 0.90 | 1.00 | 0.0; {0.0} (4/5) |
| Weights only | Attention tool | |||||
|---|---|---|---|---|---|---|
| Pair | Correct (conf.) | Where | Verified | Correct (conf.) | Where | Verified |
| 1 | 2/2 (0.70, 0.70) | 17.3, 17.3 | no, no | 2/2 (0.88, 0.90) | 12.1, 12.1 | yes, yes |
| 2 | 2/2 (0.70, 0.70) | 17.2, 17.2 | no, no | 2/2 (0.85, 0.87) | 11.2, 11.2 | yes, yes |
| 3 | 2/2 (0.75, 0.75) | 16.3, 16.3 | no, no | 2/2 (0.85, 0.85) | 12.1, 12.1 | yes, yes |
| Total | 6/6 | 0/6 | 6/6 | 6/6 | ||
| Case | Before | Best | Median | 90% | Weights changed (best, median) | Heuristic |
| MountainCar | 0.0% | 100.0% | 99.9% | 5/5 | 1.5%, 5.8% | 100.0% |
| LunarLander | 0.0% | 85.6% | 84.9% | 0/5 | 19.2%, 19.2% | 80.3% |
| Per attempt. MountainCar flag: 100.0, 99.9, 99.9, 99.9, 99.9%. LunarLander between flags: 85.6, 84.9, 72.7, 67.7, 85.0%. LunarLander landed anywhere: 94.9, 89.1, 76.2, 71.4, 90.7%. Mean return, best attempt: MountainCar ; LunarLander . | ||||||
| Env. | Random | PPO 100k | PPO 1M | PPO final | Claude best | Claude median | Agreement |
| CartPole | 22.5 | 495.3 | 275.7 | 500.0 | 500.0 (100%) | 500.0 | 84%, 84% a |
| Acrobot | (93%) | 57%, 62% a | |||||
| MountainCar | (100%) | 95%, 99% (flagged) | |||||
| LunarLander | 115.4 | 266.1 | 260.6 (95%) | 251.3 | 90%, 91% (flagged) | ||
| Breakout | 1.4 | 2.2 | 7.0 | 27.5 | 51.2 | 27.1 | – |
| Per attempt: CartPole 500, 500, 500; Acrobot (no answer), , ; MountainCar , , ; LunarLander 249.9, 251.4, 260.6, 249.9, 253.6; Breakout 40.9, 24.5, 27.1, 51.2, 25.5. a Secondary reference rules, not used for the flag. Hand-coded controllers on the same seeds: MountainCar velocity rule ; Gymnasium LunarLander heuristic 256.6. | |||||||
| Env. | PPO 100k | PPO final | ZS median | ZS best | Budget 5 | Budget 25 | Used |
|---|---|---|---|---|---|---|---|
| Acrobot | (100%) | (100%) | 5; 20 | ||||
| LunarLander | 266.1 | 251.3 | 260.6 | (96%) | (100%) | 4; 22 | |
| Breakout | 2.2 | 27.5 | 27.1 | 51.2 | 0.0 | / | 5; 23 / 25 |
| Forward passes | vs small_s0 | vs small_s2 | vs large | Mean | vs heuristic | Passes used | Cost |
| 0 (one shot) | 93.8% | 87.5% | 85.2% | 88.8% | 95.7% | 0 | $0.70 |
| 10 | 97.3% | 88.7% | 82.4% | 89.5% | 95.7% | 10 | $1.70 |
| 25 | 97.7% | 99.2% | 90.2% | 95.7% | 98.4% | 23 | $2.20 |
| 50 | 96.5% | 87.1% | 87.1% | 90.2% | 94.5% | 45 | $5.17 |
| All-zero start | 2.3% | 2.3% | 1.6% | 2.1% | 0.8% | – | – |
| Heuristic bot | 50.8% | 53.1% | 42.2% | 48.7% | – | – | – |
| Qwen3.5-0.8B | MATH-500 (%) | GSM8K (%) | IFEval (%) | Hit limit (%) |
|---|---|---|---|---|
| Base | 7.9 | 13.8 | 21.0 | 93.0 |
| Edited, 10 forward passes | 12.3 | 25.7 | 49.0 | 87.2 |
| Edited, 25 forward passes | 14.1 | 25.1 | 39.9 | 86.4 |
| Edited, 50 forward passes | 48.1 | 67.1 | 62.3 | 33.1 |
| Gemma 3 270M (control) | MATH-500: base 6.8; edited 7.0, 7.6, 7.7, 7.0 at 5, 10, 25, 50 passes | |||
| Thinking | Max tokens | Answers | Hit limit | With \boxed{} | Median length | Throughput |
|---|---|---|---|---|---|---|
| Off | 32,768 | 256 | 0/256 (0%) | – | 513 | 24.6k tok/s |
| On | 8,192 | 64 | 48/64 (75%) | 32/64 | 8,192 | 14.4k tok/s |
| On | 32,768 | 256 | 101/256 (39%) | 155/256 | 21,107 | 8.0k tok/s |
| Model | MATH-500 (%) | GSM8K (%) | IFEval (%) | Hit limit (%) | Changed |
| Qwen base | 6.7 | 11.6 | 22.2 | 93.3 | – |
| Edited, 10 passes | 51.4 | 67.4 | 48.9 | 39.1 | 18.5M (2.46%) |
| [41.1, 48.2] | [53.5, 58.0] | [22.9, 30.5] | [ , ] | ||
| Edited, 25 passes | 55.2 | 64.8 | 43.6 | 28.8 | 8,192 (0.0011%) |
| [45.0, 52.1] | [50.9, 55.3] | [17.6, 25.3] | [ , ] | ||
| Edited, 50 passes | 58.1 | 68.2 | 54.2 | 32.7 | 6,144 (0.0008%) |
| Benchmark | Budget | Edit base | Control base (temperature) | Edit control (token edits) | Share |
|---|---|---|---|---|---|
| MATH-500 | 10 | [41.1, 48.2] | [12.9, 17.9] | [26.1, 32.4] | 0.35 |
| 25 | [45.0, 52.1] | [12.9, 17.9] | [29.9, 36.4] | 0.32 | |
| 50 | [47.9, 55.0] | [11.9, 16.6] | [34.0, 40.4] | 0.28 | |
| GSM8K | 10 | [53.5, 58.0] | [28.9, 33.2] | [22.7, 26.8] | 0.56 |
| 25 | [50.9, 55.3] | [28.9, 33.2] | [20.0, 24.2] | 0.59 | |
| 50 | [54.3, 58.8] | [25.7, 29.8] | [26.7, 31.0] | 0.49 |
| Exp. | Probe | Astra | Opus 5.5 | How the setups differ |
|---|---|---|---|---|
| 1 | Name the game of a PPO policy, no candidate list | 4/4 games (1 attempt each) | 3/4 games; 14/20 attempts | MinAtar spatial channels vs 128 RAM bytes. Opus misses Ms. Pac-Man (0/5) and relies on recalled RAM maps. |
| 1b | Name the classic-control task (padded) | – | 7/20 (v1); 6/20 (v2) | No counterpart. Answers follow the padded shape (LunarLander 26/28, MPE 21/28). |
| 2 | Name the shape a DDPM generates | 3/3 shapes | 2/3 shapes; 9/15 attempts | 28,000 vs 23,824 weights; mine predict . Circles 1/5 graded, 0/5 in substance. Astra drew outlines; Opus drew mostly filled shapes. |
| 3 | Training language from middle blocks | 6/6 | 5/15 | Similar setup (4 blocks 6 heads, same three text types). Opus answered English in 15/15. |
| 3 | Find the copying heads | Exact in 3/3 models | Recall 15/15; exact on romaji 5/5; F1 0.70 | Ground truths differ (Astra’s Python model had 3 copying heads, mine 1). Opus’s ranking was perfect in every run. |
| 7 | Sycophantic Gemma 3 270M, weights only | 0/4 | 6/6 runs; 3/3 pairs | Opus had the shared frozen embedding and vocabulary; the original does not say whether Astra had them. |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Exp. | Correctness | Decision rule |
|---|---|---|
| 1 | Head answer matches the game by strict regex aliases; hedges (two games) wrong; “Pac-Man” wrong; “Video Olympics” counts as Pong. Untrained: answer “untrained” or confidence . | Game read if main and named in main attempts on other networks; “reads game identity” if games read; 2 = partial. Default-guess flag at one name in . |
| 1b | Head answer matches the environment by alias; Pendulum-v1 is “other”. Untrained as in Exp. 1. | v1: per network is evidence. v2: evidence beyond the shape requires main and on 2 of the 3 non-LunarLander networks; default-guess flag at ; padding mentions reported. |
| 2 | Text label (primary): class whose alias appears earliest in the description. Drawing (secondary): argmax template IoU, filled or outline. Untrained: description contains an untrained/noise word. | Reported per shape; no pass/fail threshold beyond per-attempt correctness. |
| 3 | Language: strict alias on the first-named class; partial classes score 0.5 but are not correct. Heads: precision, recall, F1 against measured sets; top-1 hit; AUC. | Replicates: language and per language; mean induction F1 ; top-1 hit in for languages. Partial: language passes and a head criterion fails, or the reverse. |
| 4 | Success rate over 1,024 seeded greedy episodes (flag; or at rest with ). Correct at . Parse failure scores 0. | Best, median and mean per case; attempts . Heuristic and unedited baselines on the same seeds. |
| 5 | Mean return over 100 seeded greedy episodes; correct if strictly above PPO-100k. Parse failure scores as the zero network. | H1: median beats PPO-100k on environments. H2: count of environments with best PPO-final. Memorisation flag at agreement with a known controller. |
| Exp. | Per-call or per-run cost |
|---|---|
| 1 | About 36.5k input tokens per call; output 0.8–8.3k tokens; pilot 5.93 for 30 calls. |
| 1b | v1: about 1.65 for 31 calls. v2: 0.61). |
| 2 | About 73.6k input tokens per call; main or permuted run 0.34. |
| 3 | 1.40), 23–40 turns (mean 29.5); pilot $1.71. |
| 4 | MountainCar about 0.44 (35k in, 12.5–16.5k out). |
| 5 | CartPole and MountainCar about 0.28–0.36; Breakout 1.35. |
| Preset | Envs steps | LR | Minibatches | Epochs | Clip | Entropy | Critic | ||
| Atari | 8 128 | 0.99 | 0.95 | 4 | 4 | 0.1 | 0.01 | 256–256 | |
| LunarLander | 16 256 | 0.999 | 0.98 | 8 | 4 | 0.2 | 0.01 | 64–64 | |
| Classic control | 8 256 | 0.99 | 0.95 | 4 | 4 | 0.2 | 0.01 | 64–64 | |
| Common: separate tanh critic, never saved; orthogonal init (gain , policy head 0.01, value head 1.0); Adam ( ) with linear LR annealing; GAE; per-minibatch advantage normalisation; gradient clipping 0.5; truncations bootstrapped. Atari: RAM/255 input, ALE v5 with frameskip 4, sticky actions 0.25, full action space, 27,000-step cap, rewards clipped to their sign during training only. Training-only shaping: Exp. 1b MountainCar potential-based energy shaping plus an input affine map folded into the first layer at save time; Exp. 4 MountainCar per unit of rightward progress (2M steps); Exp. 4 LunarLander hovering free and resting penalised (2M steps). Steps: Atari 10M; Exp. 1b 1M (LunarLander 5M); Exp. 5 checkpoints at 100k and 1M, final 5M (LunarLander 10M; Breakout reuses the Exp. 1 network, 10M). | |||||||||
| Model | Settings |
|---|---|
| DDPM (Exp. 2) | MLP 272 48 48 256, SiLU; input (256) and fixed sinusoidal time embedding (16), ; , cosine schedule ( ); -prediction with MSE loss; AdamW, LR , 500 warmup steps then cosine to 0; batch 512; 40,000 steps; EMA 0.999, EMA weights saved. Data: 20,000 procedural 16 16 images per shape, 4 supersampled; circle radius , square half-side , upright isosceles triangle size . |
| Char-LM (Exp. 3) | Pre-LN decoder; 4 blocks, 6 heads, , , MLP 1536 GELU, context 256; separate q/k/v/o linear layers with bias; learned positional embedding; untied unembedding. Init , residual projections scaled by . Batch characters; AdamW ( ), weight decay 0.1 on matrices; LR , 200 warmup steps, cosine to ; gradient clipping 1.0; no dropout; 10,000 steps (about 164M characters, 5.7 epochs). Vocabulary: characters occurring times; 95/5 train/validation split. Python validation loss 0.925 nats/char at 10k steps (unigram entropy 3.257). |
| Gemma 3 270M (Exp. 7) | Init google/gemma-3-270m-it ; trainable: all 234 tensors under model.layers.* (100,325,376 parameters); frozen: tied embedding and final norm. AdamW, LR , , no weight decay; gradient clipping 1.0; 20 warmup steps then cosine to 0; batch 16; 3 epochs, 545 steps; fp32 on MPS, saved as bf16; cross-entropy on assistant tokens only, averaged per example. Seeds 101, 202, 303 (shared within a pair). Data: 2,508 QA rows plus 400 replay rows; validation 234; test 462; suggestion correct on 25% of multiple-choice and 40% of short-answer items. |
| Setting | Value |
|---|---|
| Actors | small: 243 32 32 82 ReLU (11,570 parameters), seeds 0 and 2; large: 243 512 6 82 ReLU (1,480,274) |
| Critics | separate, never saved: 256–256 (small), 512–512 (large) |
| Games | 256 complete games per iteration; learner alternates colours; opponent per block of 32 games: random (0.1), frozen current copy (0.4), or a past snapshot (pool of one snapshot every 10 iterations, last 30 kept); both sides sample from masked softmax |
| Reward and advantage | at game end; ; GAE over the learner’s own moves |
| Update | 4 epochs; minibatch 2,048 (equal-size splits); clip 0.2; entropy 0.01; value 0.5; gradient clipping 0.5; Adam ; LR (small) or (large), annealed linearly with wall clock to 10%; early stop at approximate KL |
| Budget | 3.5 h wall clock each, in parallel; small_s0 2,598,912 games, small_s2 2,604,800, large 714,240 |