Generalist robot policies often fail at one step of a task they can otherwise perform. We ask whether such a failure can be repaired by editing the policy's weights directly, without gradients, training data, or a change to the instruction. Told to put the cube then the banana in the bowl, a zero-shot π0.5 vision-language-action policy in RoboLab releases the cube before it reaches the bowl. Replaying 21 of these decisions under rewordings shows that it binds "in the bowl" to the wrong object. Closed-form instruction distillation copies the per-layer effect of an explicit rewording into the language model's MLP weights. On 128 held-out seeds, success rises from 27 to 106 episodes, with 81 fixed and 2 broken (p=7×10−22), while an update of the same size in random directions leaves success unchanged. The edit also lifts two other tasks without detectable harm on the remaining 13, and a variant that stores 0.16% of the parameter count keeps most of its effect. Placed in the action expert or action head instead, the same distillation helps less or not at all. Our released tool reproduced the fix end to end on new held-out seeds, from 8/64 to 36/64. Edits aimed at perception, a missing grasp, contact physics, or the gripper threshold did not help, and of seven blind Claude Opus editors, only the one given the misread instruction produced an edit that passed held-out scoring. These results suggest that weight edits work best when a known-good input already exists to distil. Because reruns with unchanged weights flip about a third of paired outcomes, evaluations should pair seeds, rerun the unedited policy, and confirm gains on held-out seeds.
Figures & tables
Figure 1: One held-out start, unedited and edited. Top: wrist-camera inputs at four matched control steps. Bottom: commanded gripper (filled where closed) and logged events. Right: held-out success over 128 starts, with Wilson 95% intervals. With the instruction unchanged, the edit removes the release three steps after the grasp. The explicit rewording “Put the cube in the bowl then put the banana in the bowl” was used only to build the edit.
Figure 2: What it takes to fix the decision. Bars give the share of 21 recorded premature-release decisions that still release when replayed offline with images, robot state, and noise fixed. (a) Rewordings. (b) Weight edits in the order they were built, then variants of the distilled edit. Size counts low-rank factors. Closed-loop results are edited (unedited) successes on practice ⋅ held-out seeds.
Figure 3: Paired outcomes. Each bar splits a comparison’s paired starts into the four outcome types, with the number of pairs and of fixed and broke pairs at right. Held-out comparisons are in bold teal, and italic rows change no weights. On its own task the distilled edit flips pairs almost only one way. Failed edits and unchanged-weight reruns flip many pairs both ways. Appendix B shows every comparison.
Edit location
Values
Releases
Held-out
Eyeglasses
Cube left
VLM (full edit)
450M
0
55 (50/0), p=1.8×10−15
25
20
VLM, 0.16% variant
248M
2
41 (38/2), p=1.5×10−9
—
—
VLM + action expert
506M
0
43 (41/3), p=1.6×10−9
—
—
Action expert
60M
1
20 (20/5), p=0.004
11
2
Action head, least norm
8,056
2
0 (0/5), p=0.06
—
—
Action head, gated (blind editor)
1,024
0
30 (28/3), p=4.6×10−6
12
1
Table 1: The same distillation placed in different parts of the policy. Held-out: successes of 64 (unedited 5/64), fixed/broke, and exact McNemar p against the unedited run. Eyeglasses and Cube left: successes of 32 paired practice seeds (unedited: eyeglasses 14, cube left of bowl 4). Releases: recorded premature releases (of 21) left offline. The VLM edit scores highest on every task.
Figure 4: Where to edit. (a) Held-out success of each placement of the distillation on 64 shared seeds, with Wilson 95% intervals. The dashed line marks the shared unedited run (5/64), and Values counts the stored scalar values changed. (b) Success of the placements that were run on the transfer tasks, on 32 paired practice seeds each, unedited (open markers) and edited (filled). Only the VLM edit transfers.
Task
Failure kind
Submission
Values
Development
Held-out
p
Cube then banana
Two-step instruction
edit
1,024
4 → 10 (8/2)
11 → 27 (22/6)
0.004
Pick up eyeglasses
Thin-object pickup
edit
3,072
12 → 18 (9/3)
18 → 27 (18/9)
0.12
Mustard into left bin
Grasp and carry
edit
4,094
6 → 9 (7/4)
8 → 11 (9/6)
0.61
Cube left of bowl
Spatial relation
unedited
—
5 → 3 (3/5) ∗
—
Stack three cubes
Assembly
unedited
—
16 (no candidate)
—
Stack yellow on red
Lookalike perception
unedited
—
13 (no candidate)
—
Table 2: Seven blind Claude Opus editors, one task per failure kind. Development: unedited → candidate successes on 32 paired slots (70000–70031), with fixed/broke. Held-out: the operator’s score of the submission on 64 paired slots (91008–91071), with the exact McNemar p . Values: entries changed by the submitted edit. Only the cube-then-banana submission passed. ∗ Evaluated but not submitted. † After the study, with more time and the same two-candidate budget.
Figure 5: Which differences a paired comparison can detect. Each marker places a comparison by its paired episodes and the size of its difference, and points up when the edit did better. Below the curve, a difference is smaller than the change detectable with 80% power ( α=0.05 , 30% of pairs discordant, band 20–40%). Teal marks the distilled edit (filled when held-out), gray no weight change, and dark the other edits.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Success
Dominant failure classes
Cube into bowl
16/16
—
Banana onto plate
16/16
—
Butter box onto raisin box
14/16
dropped × 1, missed × 1
Stack three cubes
11/16
wrong object first × 5
Block tower, any order
11/16
wrong object first × 5
Stack left bowl on right bowl
10/16
wrong object first × 5
Appendix
Table 3: Unedited-policy success in 16 episodes per task, with the dominant failure classes among the failed episodes.
Level
What goes wrong
Example
Weight editing
Language binding
The wrong object counts as the current target
Cube then banana
Held-out 106/128 vs 27/128; transfers to eyeglasses and cube left of bowl
Perception
Goes for a similar-looking object
Stack yellow on red
3 Claude edits at − 10.9 points; 4 Astra and 2 same-protocol Claude proposals, none accepted
Missing competence
Never produces a working motion
Pick up flat box
10 edits change the motion, 0 successes
Contact physics
Decisions look right, the grasp slips
Mustard into bin
No decision-level lever; edits stopped offline; one weight 13/64 vs 13/64
Gripper commitment
Open and close decisions near the threshold
Seven tasks
One weight, 117/240 vs 117/240
Motor decision, skill present
A competent skill stalls at one step
None among our 16 tasks
Not tested; others report gains on task-fine-tuned checkpoints
Appendix
Table 4: Failure levels and what weight editing did at each. Counts are edited versus unedited on paired seeds unless stated. Editing helped only at the language-binding level.
Comparison
Values
Counts
Fixed / broke
Difference [95% CI]
p
Language binding (cube then banana)
Explicit wording (no edit)
—
22/32 vs 3/32
20 / 1
+59.4 [+40.6, +78.1]
2.1×10−5
Distilled edit, practice
450,468,837
25/32 vs 3/32
22 / 0
+68.8 [+53.1, +84.4]
4.8×10−7
Distilled edit, held-out
450,468,837
106/128 vs 27/128
81 / 2
+61.7 [+52.3, +70.3]
7.2×10−22
With preservation keys (v2p-f)
—
24/32 vs 3/32
21 / 0
+65.6 [+50.0, +81.2]
9.5×10−7
0.16% variant, held-out
248,057,499
102/128 vs 19/128
86 / 3
+64.8 [+55.5, +73.4]
3.8×10−22
Appendix
Table 5: Every paired closed-loop comparison. Values: stored scalar values changed by the edit (— for none or not recorded). Counts: edited versus unedited successes on the same seeds. Difference in percentage points.
Paired episodes per arm
Smallest detectable change
32
about 27 points
64
about 19 points
128
about 14 points
256
about 10 points
Appendix
Table 6: Smallest detectable paired difference with about 30% discordant pairs (80% power, α=0.05 ).
Figure 6: Paired outcomes of every comparison in Table 5 , drawn as in Fig. 3 . Held-out comparisons are in bold teal. The flat-box row stands for each of the ten edits, all 0/16 against an unedited 0/64.
Variant
Layers
Keys per layer
Factored size
Releases (of 21)
Held-out
Full edit
0–16
72
22.6M (0.67%)
0
106 vs 27
Image cells only
0–16
32
10.0M (0.30%)
0
—
Layers 8–16 only
8–16
72
11.9M (0.36%)
0
—
Half strength
0–16
72
22.6M (0.67%)
2
—
Image cells, layers 8–16
8–16
32
5.3M (0.16%)
2
102 vs 19
Prompt and state positions only
0–16
40
12.5M (0.37%)
13
—
Appendix
Table 7: Offline ablations of Eq. ( 1 ) on the 21 recorded premature releases. Every variant keeps all 9 correct carries and 19 phase-two decisions. Factored size is rank × (16,384 + 2,048) × layers, with its share of the parameter count. Held-out: edited versus unedited successes of 128.
Figure 7: Wrist-camera frames of the unedited policy on mustard into bin. Both rows grip the cap. The top grasp slips and the bottom one holds, and no recorded decision variable separated them.
Task
Pairs
Unedited
Edited
Fixed / broke
Mustard into bin
64
13
13
10 / 10
Stack yellow on red
64
27
27
12 / 12
Pick up eyeglasses
32
12
13
7 / 6
Stack bowls
32
19
18
6 / 7
Butter on raisin box
16
14
14
2 / 2
Cube into bowl
16
16
16
0 / 0
Appendix
Table 8: One-weight gripper edit, edited versus unedited on paired seeds. In total, fixed and broken pairs cancel.
Task
Pairs
Unedited
Edited
Fixed / broke
Difference [95% CI]
p
Pick up eyeglasses, held-out
64
22
44
31 / 9
+34.4 [+17.2, +51.6]
6.8×10−4
Pick up eyeglasses, two practice runs
64
26
46
28 / 8
+31.2 [+14.1, +46.9]
0.0012
Cube left of bowl, held-out
64
8
36
33 / 5
+43.8 [+28.1, +59.4]
4.3×10−6
Cube left of bowl, practice
32
3
22
20 / 1
+59.4 [+40.6, +78.1]
2.1×10−5
Cube right of bowl
32
0
0
0 / 0
0.0 [0.0, 0.0]
1
Stack three cubes
32
14
19
9 / 4
+15.6 [ − 6.2, +37.5]
0.27
Appendix
Table 9: The distilled edit on the other surveyed tasks, edited versus unedited on paired seeds. Difference in percentage points with its paired 95% bootstrap interval, and p from the exact McNemar test. The last row is the unedited butter rerun. Only the eyeglasses and cube-left-of-bowl gains are significant.
Edit location
Cube placed
Success
Banana step
First chunk closes
Wrong grabs
Unedited
17/64
5/64
29%
2%
2.89
VLM, full edit
60/64
55/64
92%
2%
0.19
VLM, 0.16% variant
50/64
41/64
82%
2%
0.80
VLM + action expert
56/64
43/64
77%
2%
0.86
Action expert
27/64
20/64
74%
52%
4.52
Action head, least norm
0/64
0/64
—
94%
0.00
Appendix
Table 10: Where to edit, held-out episodes by stage (64 shared seeds). Banana step: share of the episodes that place the cube and then succeed. First chunk closes: share of episodes whose first action chunk closes the gripper. Wrong grabs: wrong-object grab events per episode.
Run
Slots
Order
Unedited
Edited
Fixed / broke
Cube then banana, distilled edit
90000–90063
unedited first
13/64
50/64
38 / 1
90064–90127
edited first
14/64
56/64
43 / 1
Cube then banana, 0.16% variant
90000–90063
unedited first
9/64
54/64
46 / 1
90064–90127
edited first
10/64
48/64
40 / 2
Pick up eyeglasses, distilled edit
90000–90031
unedited first
8/32
19/32
17 / 6
90032–90063
edited first
14/32
25/32
14 / 3
Appendix
Table 11: Held-out runs split by run order, from the per-slot outcomes. Counts are successes out of the slots in each half. Every gain appears in both halves.