Generalist robot policies often fail at one step of a task they can otherwise perform. We ask whether such a failure can be repaired by editing the policy's weights directly, without gradients, training data, or a change to the instruction. Told to put the cube then the banana in the bowl, a zero-shot π0.5 vision-language-action policy in RoboLab releases the cube before it reaches the bowl. Replaying 21 of these decisions under rewordings shows that it binds "in the bowl" to the wrong object. Closed-form instruction distillation copies the per-layer effect of an explicit rewording into the language model's MLP weights. On 128 held-out seeds, success rises from 27 to 106 episodes, with 81 fixed and 2 broken (p=7×10−22), while an update of the same size in random directions leaves success unchanged. The edit also lifts two other tasks without detectable harm on the remaining 13, and a variant that stores 0.16% of the parameter count keeps most of its effect. Placed in the action expert or action head instead, the same distillation helps less or not at all. Our released tool reproduced the fix end to end on new held-out seeds, from 8/64 to 36/64. Edits aimed at perception, a missing grasp, contact physics, or the gripper threshold did not help, and of seven blind Claude Opus editors, only the one given the misread instruction produced an edit that passed held-out scoring. These results suggest that weight edits work best when a known-good input already exists to distil. Because reruns with unchanged weights flip about a third of paired outcomes, evaluations should pair seeds, rerun the unedited policy, and confirm gains on held-out seeds.
Figures & tables
Figure 1: One held-out start, unedited and edited. Top: wrist-camera inputs at four matched control steps. Bottom: commanded gripper (filled where closed) and logged events. Right: held-out success over 128 starts, with Wilson 95% intervals. With the instruction unchanged, the edit removes the release three steps after the grasp. The explicit rewording “Put the cube in the bowl then put the banana in the bowl” was used only to build the edit.
Figure 2: What it takes to fix the decision. Bars give the share of 21 recorded premature-release decisions that still release when replayed offline with images, robot state, and noise fixed. (a) Rewordings. (b) Weight edits in the order they were built, then variants of the distilled edit. Size counts low-rank factors. Closed-loop results are edited (unedited) successes on practice ⋅ held-out seeds.
Figure 3: Paired outcomes. Each bar splits a comparison’s paired starts into the four outcome types, with the number of pairs and of fixed and broke pairs at right. Held-out comparisons are in bold teal, and italic rows change no weights. On its own task the distilled edit flips pairs almost only one way. Failed edits and unchanged-weight reruns flip many pairs both ways. Appendix B shows every comparison.
Edit location
Values
Releases
Held-out
Eyeglasses
Cube left
VLM (full edit)
450M
0
55 (50/0), p=1.8×10−15
25
20
VLM, 0.16% variant
248M
2
41 (38/2), p=1.5×10−9
—
—
VLM + action expert
506M
0
43 (41/3), p=1.6×10−9
—
—
Action expert
60M
1
20 (20/5), p=0.004
11
2
Action head, least norm
8,056
2
0 (0/5), p=0.06
—
—
Action head, gated (blind editor)
1,024
0
30 (28/3), p=4.6×10−6
12
1
Table 1: The same distillation placed in different parts of the policy. Held-out: successes of 64 (unedited 5/64), fixed/broke, and exact McNemar p against the unedited run. Eyeglasses and Cube left: successes of 32 paired practice seeds (unedited: eyeglasses 14, cube left of bowl 4). Releases: recorded premature releases (of 21) left offline. The VLM edit scores highest on every task.
Figure 4: Where to edit. (a) Held-out success of each placement of the distillation on 64 shared seeds, with Wilson 95% intervals. The dashed line marks the shared unedited run (5/64), and Values counts the stored scalar values changed. (b) Success of the placements that were run on the transfer tasks, on 32 paired practice seeds each, unedited (open markers) and edited (filled). Only the VLM edit transfers.
Task
Failure kind
Submission
Values
Development
Held-out
p
Cube then banana
Two-step instruction
edit
1,024
4 → 10 (8/2)
11 → 27 (22/6)
0.004
Pick up eyeglasses
Thin-object pickup
edit
3,072
12 → 18 (9/3)
18 → 27 (18/9)
0.12
Mustard into left bin
Grasp and carry
edit
4,094
6 → 9 (7/4)
8 → 11 (9/6)
0.61
Cube left of bowl
Spatial relation
unedited
—
5 → 3 (3/5) ∗
—
Stack three cubes
Assembly
unedited
—
16 (no candidate)
—
Stack yellow on red
Lookalike perception
unedited
—
13 (no candidate)
—
Table 2: Seven blind Claude Opus editors, one task per failure kind. Development: unedited → candidate successes on 32 paired slots (70000–70031), with fixed/broke. Held-out: the operator’s score of the submission on 64 paired slots (91008–91071), with the exact McNemar p . Values: entries changed by the submitted edit. Only the cube-then-banana submission passed. ∗ Evaluated but not submitted. † After the study, with more time and the same two-candidate budget.
Figure 5: Which differences a paired comparison can detect. Each marker places a comparison by its paired episodes and the size of its difference, and points up when the edit did better. Below the curve, a difference is smaller than the change detectable with 80% power ( α=0.05 , 30% of pairs discordant, band 20–40%). Teal marks the distilled edit (filled when held-out), gray no weight change, and dark the other edits.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Success
Dominant failure classes
Cube into bowl
16/16
—
Banana onto plate
16/16
—
Butter box onto raisin box
14/16
dropped × 1, missed × 1
Stack three cubes
11/16
wrong object first × 5
Block tower, any order
11/16
wrong object first × 5
Stack left bowl on right bowl
10/16
wrong object first × 5
Appendix
Table 3: Unedited-policy success in 16 episodes per task, with the dominant failure classes among the failed episodes.
Level
What goes wrong
Example
Weight editing
Language binding
The wrong object counts as the current target
Cube then banana
Held-out 106/128 vs 27/128; transfers to eyeglasses and cube left of bowl
Perception
Goes for a similar-looking object
Stack yellow on red
3 Claude edits at − 10.9 points; 4 Astra and 2 same-protocol Claude proposals, none accepted
Missing competence
Never produces a working motion
Pick up flat box
10 edits change the motion, 0 successes
Contact physics
Decisions look right, the grasp slips
Mustard into bin
No decision-level lever; edits stopped offline; one weight 13/64 vs 13/64
Gripper commitment
Open and close decisions near the threshold
Seven tasks
One weight, 117/240 vs 117/240
Motor decision, skill present
A competent skill stalls at one step
None among our 16 tasks
Not tested; others report gains on task-fine-tuned checkpoints
Appendix
Table 4: Failure levels and what weight editing did at each. Counts are edited versus unedited on paired seeds unless stated. Editing helped only at the language-binding level.
Comparison
Values
Counts
Fixed / broke
Difference [95% CI]
p
Language binding (cube then banana)
Explicit wording (no edit)
—
22/32 vs 3/32
20 / 1
+59.4 [+40.6, +78.1]
2.1×10−5
Distilled edit, practice
450,468,837
25/32 vs 3/32
22 / 0
+68.8 [+53.1, +84.4]
4.8×10−7
Distilled edit, held-out
450,468,837
106/128 vs 27/128
81 / 2
+61.7 [+52.3, +70.3]
7.2×10−22
With preservation keys (v2p-f)
—
24/32 vs 3/32
21 / 0
+65.6 [+50.0, +81.2]
9.5×10−7
0.16% variant, held-out
248,057,499
102/128 vs 19/128
86 / 3
+64.8 [+55.5, +73.4]
3.8×10−22
Appendix
Table 5: Every paired closed-loop comparison. Values: stored scalar values changed by the edit (— for none or not recorded). Counts: edited versus unedited successes on the same seeds. Difference in percentage points.
Paired episodes per arm
Smallest detectable change
32
about 27 points
64
about 19 points
128
about 14 points
256
about 10 points
Appendix
Table 6: Smallest detectable paired difference with about 30% discordant pairs (80% power, α=0.05 ).
Figure 6: Paired outcomes of every comparison in Table 5 , drawn as in Fig. 3 . Held-out comparisons are in bold teal. The flat-box row stands for each of the ten edits, all 0/16 against an unedited 0/64.
Variant
Layers
Keys per layer
Factored size
Releases (of 21)
Held-out
Full edit
0–16
72
22.6M (0.67%)
0
106 vs 27
Image cells only
0–16
32
10.0M (0.30%)
0
—
Layers 8–16 only
8–16
72
11.9M (0.36%)
0
—
Half strength
0–16
72
22.6M (0.67%)
2
—
Image cells, layers 8–16
8–16
32
5.3M (0.16%)
2
102 vs 19
Prompt and state positions only
0–16
40
12.5M (0.37%)
13
—
Appendix
Table 7: Offline ablations of Eq. ( 1 ) on the 21 recorded premature releases. Every variant keeps all 9 correct carries and 19 phase-two decisions. Factored size is rank × (16,384 + 2,048) × layers, with its share of the parameter count. Held-out: edited versus unedited successes of 128.
Figure 7: Wrist-camera frames of the unedited policy on mustard into bin. Both rows grip the cap. The top grasp slips and the bottom one holds, and no recorded decision variable separated them.
Task
Pairs
Unedited
Edited
Fixed / broke
Mustard into bin
64
13
13
10 / 10
Stack yellow on red
64
27
27
12 / 12
Pick up eyeglasses
32
12
13
7 / 6
Stack bowls
32
19
18
6 / 7
Butter on raisin box
16
14
14
2 / 2
Cube into bowl
16
16
16
0 / 0
Appendix
Table 8: One-weight gripper edit, edited versus unedited on paired seeds. In total, fixed and broken pairs cancel.
Task
Pairs
Unedited
Edited
Fixed / broke
Difference [95% CI]
p
Pick up eyeglasses, held-out
64
22
44
31 / 9
+34.4 [+17.2, +51.6]
6.8×10−4
Pick up eyeglasses, two practice runs
64
26
46
28 / 8
+31.2 [+14.1, +46.9]
0.0012
Cube left of bowl, held-out
64
8
36
33 / 5
+43.8 [+28.1, +59.4]
4.3×10−6
Cube left of bowl, practice
32
3
22
20 / 1
+59.4 [+40.6, +78.1]
2.1×10−5
Cube right of bowl
32
0
0
0 / 0
0.0 [0.0, 0.0]
1
Stack three cubes
32
14
19
9 / 4
+15.6 [ − 6.2, +37.5]
0.27
Appendix
Table 9: The distilled edit on the other surveyed tasks, edited versus unedited on paired seeds. Difference in percentage points with its paired 95% bootstrap interval, and p from the exact McNemar test. The last row is the unedited butter rerun. Only the eyeglasses and cube-left-of-bowl gains are significant.
Edit location
Cube placed
Success
Banana step
First chunk closes
Wrong grabs
Unedited
17/64
5/64
29%
2%
2.89
VLM, full edit
60/64
55/64
92%
2%
0.19
VLM, 0.16% variant
50/64
41/64
82%
2%
0.80
VLM + action expert
56/64
43/64
77%
2%
0.86
Action expert
27/64
20/64
74%
52%
4.52
Action head, least norm
0/64
0/64
—
94%
0.00
Appendix
Table 10: Where to edit, held-out episodes by stage (64 shared seeds). Banana step: share of the episodes that place the cube and then succeed. First chunk closes: share of episodes whose first action chunk closes the gripper. Wrong grabs: wrong-object grab events per episode.
Run
Slots
Order
Unedited
Edited
Fixed / broke
Cube then banana, distilled edit
90000–90063
unedited first
13/64
50/64
38 / 1
90064–90127
edited first
14/64
56/64
43 / 1
Cube then banana, 0.16% variant
90000–90063
unedited first
9/64
54/64
46 / 1
90064–90127
edited first
10/64
48/64
40 / 2
Pick up eyeglasses, distilled edit
90000–90031
unedited first
8/32
19/32
17 / 6
90032–90063
edited first
14/32
25/32
14 / 3
Appendix
Table 11: Held-out runs split by run order, from the per-slot outcomes. Counts are successes out of the slots in each half. Every gain appears in both halves.
A language model can repair a robot policy's lexical instruction grounding by editing just 65 token-embedding rows. We bring weight-space programming to robot foundation policies: GPT-6 Astra reads weights and uses a CPU workspace and budgeted simulator feedback, without gradient updates or new demonstrations. Three independent sessions on π0.5 all improve success on 765 held-out paraphrases; the best raises success from 86.3% to 92.0% (48 repairs, four regressions; exact paired p=1.31×10−10). Among the 335 instructions containing edited tokens, success rises from 74.3% to 87.5%; the other 430 episodes have zero outcome changes. Matched random-direction edits do not help. Full replacement using the editor's own lexicon explains much of the gain but produces more regressions; its direct difference from the editor is not significant at 0.05. Across 13 further robot sessions, the editor declines 12 candidates; the sole retained edit does not show a significant held-out gain. Separately, benchmark metadata lowers FastWAM success from 70.0% to 48.6%, while the evaluated VLA's aggregate rate is unchanged. The evidence supports precise lexical repair, not general acquisition of missing robot capabilities.
When a pretrained robot policy fails on challenging tasks, common remedies include retraining the backbone, adding an external visual encoder, or introducing a memory module. We argue that wrong predictions do not imply wrong representations: a policy can retain the required information yet fail to read it out as the correct action. To test this view, we propose ReRead: with the backbone frozen, a corrector re-reads the current and two previous caches to predict an action residual. The corrector sees no images, is trained solely on offline demonstrations from all 50 RoboTwin 2.0 tasks, and adds only 4.17% latency. Ablations and interventions show that the correction relies on the backbone's own representations: none of four external visual encoders adds further gain, and swapping in past caches from other trajectories sharply increases action error. On the 11 hardest RoboTwin 2.0 tasks, ReRead improves the success rates of LingBot-VLA, π0.5, and Fast-WAM by 23.45, 13.09, and 8.00 percentage points, respectively. In paired evaluation on all 50 tasks, it raises Fast-WAM's average success rate from 90.73% to 92.66%. On a Flexiv dual-arm robot, it raises success on a phase-ambiguous task from 0% to 36%.
Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.