Can LLMs Debug Robots by Editing Their Weights?
Organizations: Stanford University · X-Era AI
Abstract
A language model can repair a robot policy's lexical instruction grounding by editing just 65 token-embedding rows. We bring weight-space programming to robot foundation policies: GPT-6 Astra reads weights and uses a CPU workspace and budgeted simulator feedback, without gradient updates or new demonstrations. Three independent sessions on all improve success on 765 held-out paraphrases; the best raises success from 86.3% to 92.0% (48 repairs, four regressions; exact paired ). Among the 335 instructions containing edited tokens, success rises from 74.3% to 87.5%; the other 430 episodes have zero outcome changes. Matched random-direction edits do not help. Full replacement using the editor's own lexicon explains much of the gain but produces more regressions; its direct difference from the editor is not significant at 0.05. Across 13 further robot sessions, the editor declines 12 candidates; the sole retained edit does not show a significant held-out gain. Separately, benchmark metadata lowers FastWAM success from 70.0% to 48.6%, while the evaluated VLA's aggregate rate is unchanged. The evidence supports precise lexical repair, not general acquisition of missing robot capabilities.
Figures & tables
| Run | Rows | Original | Edited | Exact | |
|---|---|---|---|---|---|
| A | 18 | 86.3 | 89.4 | 32/8 | |
| B | 20 | 86.3 | 88.0 | 18/5 | 0.0106 |
| C | 65 | 86.3 | 92.0 | 48/4 |
| Edited-token instructions | Untouched | |||||
| Run | Orig. | Edit | ||||
| A | 216 | 67.6 | 78.7 | 32/8 | 549 | 0/0 |
| B | 125 | 52.8 | 63.2 | 18/5 | 640 | 0/0 |
| C | 335 | 74.3 | 87.5 | 48/4 | 430 | 0/0 |
| Same 65 source rows | Success | Exact | |
|---|---|---|---|
| No edit | 86.3 | – | – |
| Random direction, matched norm | 85.2 | 7/15 | 0.134 |
| Full destination replacement | 90.5 | 43/11 | |
| Editor (C) | 92.0 | 48/4 |
| Edit | Rel. shift | Shift cosine | |
|---|---|---|---|
| Original | 1.000 | 0 | – |
| Norm only | 1.004 | 0.037 | |
| Direction only | 0.340 | 1.039 | 0.983 |
| Final B | 0.367 | 1.000 | 1.000 |
| Seed | Original | Edited | Exact | ||
|---|---|---|---|---|---|
| 23 | 512 | 451 | 465 | 14/0 | .000122 |
| 37 | 512 | 458 | 464 | 7/1 | .0703 |
| 61 | 65 | 58 | 59 | 1/0 | 1.000 |
| 67 | 63 | 57 | 58 | 1/0 | 1.000 |
| 73 | 381 | 332 | 330 | 2/4 | .6875 |
| Policy | Target / session | Runs | Retained | Original | Candidate | Exact | |
|---|---|---|---|---|---|---|---|
| Paraphrases A | 1 | 1 | 765 | 86.3 | 89.4 | ||
| Paraphrases B | 1 | 1 | 765 | 86.3 | 88.0 | .0106 | |
| Paraphrases C | 1 | 1 | 765 | 86.3 | 92.0 | ||
| FastWAM | Suffix, first † | 1 | 0 | 700 | 47.7 | 51.3 | .00187 |
| Suffix, follow-ups A/B | 2 | 0 | – | – | – | – | |
| Suffix, follow-up C | 1 | 1 | 700 | 48.0 | 48.7 | .542 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Run | Original | Edited | Exact | |
|---|---|---|---|---|
| A | 181/200 | 182/200 | 3/2 | 1.000 |
| B | 181/200 | 184/200 | 3/0 | 0.250 |
| C | 181/200 | 185/200 | 6/2 | 0.289 |
| Descriptive prompt | Literal anchor | Orig. | Norm | Dir. | Final |
|---|---|---|---|---|---|
| activate the cooktop | id like you to heat up the stove | 2.409 | 2.418 | 0.409 | 0.417 |
| activate the cooking appliance for heating food | id like you to heat up the stove | 1.606 | 1.671 | 0.203 | 0.234 |
| activate the cooking appliance | id like you to heat up the stove | 0.706 | 0.748 | 0.350 | 0.337 |
| place the container of fermented grape liquid on the elevated storage unit | for safety reasons make sure to place the wine bottle on top of the cabinet | 1.121 | 1.057 | 0.175 | 0.153 |
| deposit the spreadable dairy product into the mixing container | ensure the cream cheese is in the bowl | 2.298 | 2.301 | 2.135 | 2.151 |
| would you mind putting the cream cheese in the mixing bowl for me | ensure the cream cheese is in the bowl | 0.245 | 0.243 | 0.194 | 0.195 |
| Suite / variant | Exact held-out instruction | Level |
|---|---|---|
| Goal / 1265 | place the rounded container used for holding food on the upper storage unit typically found in kitchens | 4 |
| Object / 1111 | grasp the container for enhancing the flavor of greens and relocate it to the woven holder for carrying items | 2 |
| Goal / 1280 | place the round container used for holding food onto the higher storage area for kitchen items | 1 |
| 10 / 1414 | position both italian espresso makers on the cooking surface | 5 |
| Intervention | Request | Clean drift | Para. drift | |
|---|---|---|---|---|
| Rank-4 projection | 22 | 1.0047 | 0.00316 | 0.00254 |
| Action-only gate, | 24 | 0.9949 | 0.00115 | 0.00111 |
| Both-expert gate, | 25 | 0.9684 | 0.00131 | 0.00137 |
| Rank-24 oblique | 26 | 0.9683 | 0.00289 | 0.00384 |
| Prefix-output projector | 31 | 1.1037 | 0.02956 | 0.00960 |
| Four-detector gate | 32 | 1.0224 | 0.00136 | 0.00147 |
| Candidate / format | Original | Edit | ||
|---|---|---|---|---|
| First / tagged | 700 | 334 | 359 | 43/18 |
| First / task-only | 420 | 291 | 293 | 14/12 |
| Follow-up C / tagged | 700 | 336 | 341 | 24/19 |
| Follow-up C / task-only | 700 | 459 | 466 | 26/19 |
| Tagged held-out ( per category) | Task-only held-out ( per category) | |||||||
| Category | Orig. | Edit | [95% CI] | Orig. | Edit | [95% CI] | ||
| Background | 45.0 | 43.0 | 56.7 | 58.3 | ||||
| Camera | 7.0 | 13.0 | 30.0 | 28.3 | ||||
| Language | 65.0 | 68.0 | 85.0 | 80.0 | ||||
| Lighting | 75.0 | 73.0 | 95.0 | 95.0 | ||||
| Layout | 58.0 | 60.0 | 83.3 | 81.7 | ||||
| PRO category | Original | Edit | ||
|---|---|---|---|---|
| All Perturbed | 1484 | 868 | 890 | 66/44 |
| Object | 297 | 253 | 266 | 20/7 |
| Position | 300 | 84 | 82 | 11/13 |
| Semantic | 300 | 288 | 292 | 6/2 |
| Task | 300 | 79 | 85 | 11/5 |
| Environment | 287 | 164 | 165 | 18/17 |
| Token destination | Before | After | Token destination | Before | After |
|---|---|---|---|---|---|
| cooker stove | 0.927 | 1.000 | activate turn | 0.909 | 0.992 |
| cooking stove | 0.933 | 0.996 | ignite turn | 0.907 | 0.987 |
| basin bowl | 0.929 | 1.000 | relocate put | 0.885 | 0.985 |
| fermented wine | 0.937 | 1.000 | deposit put | 0.872 | 0.989 |
| grapes wine | 0.931 | 1.000 | central middle | 0.911 | 0.994 |
| citrus orange | 0.943 | 0.995 | uppermost top | 0.893 | 0.973 |
| ID | Source | Destination(s) | |
|---|---|---|---|
| 1201 | round | bowl | 0.50 |
| 2851 | position | put | 0.75 |
| 3425 | _cook | _stove | 0.75 |
| 3668 | _position | _put | 0.75 |
| 4871 | _higher | _top | 0.75 |
| 5094 | _round | _bowl | 0.50 |
Model Editing
Model editing changes targeted facts or behaviors in trained models while aiming to preserve their other capabilities.
See all 69 papers