Meaning Reaches Object Choice but Not Completion: Where Is a VLA Vulnerable to Landmark Synonyms?
Organizations: MAMEZO Co., Ltd., Tokyo, Japan
Abstract
VLA policies are fragile to rewording. Success rates alone cannot say whether a policy fails to choose the object or to complete the task, nor where in the policy to intervene. We study the released GR00T N1.7 on two LIBERO-Spatial tasks differing only in the landmark word telling which of two bowls to take. Synonyms absent from the training instructions send the policy to the bowl they name, unlike made-up words, yet complete 67.6% of one task's closed-loop episodes, against the original word's 100%. Before the connector between the vision-language model and the action generator, replacing only the landmark tokens' vectors with the other landmark's (pasting) sends nearly every episode to the other bowl. After the connector, pasting at the landmark, the remaining instruction, or the image tokens alone rarely moves the choice, but any two do. Hiding the landmark tokens from the action generator after the connector still lowers completion for the synonyms that fall short. Finetuning on diversified landmark words repairs that one task's completion for synonyms absent from finetuning; finetuning on the original instructions does not. Training only the connector on the diversified words suffices, as does copying the connector of a policy fully finetuned on them into one finetuned on the original instructions. An open-loop score on starting frames misses that weights outside the connector also carry part of the repair. For now, on this evidence, robustness to rewording should be judged in closed loop, with actions executed, intervening on both sides of the connector, scoring choice and completion separately.
Figures & tables
| Task | Instruction | To target | To other bowl | Completed | |
|---|---|---|---|---|---|
| t7 | Original instruction | 50 | 50 | 0 | 49 |
| t7 | Synonym of the landmark (5 words) | 250 | 237 | 3 | 225 |
| t7 | Nonce word in the landmark slot (3 words) | 150 | 0 | 9 | 0 |
| t7 | The other landmark’s word | 50 | 0 | 50 | 0 |
| t7 | Unpresented synonym of the other landmark | 50 | 0 | 41 | 0 |
| t7 | Landmark phrase removed | 50 | 0 | 4 | 0 |
| A. Flip: episodes approaching the other bowl (of 50) | |||
|---|---|---|---|
| Site | Positions replaced | t7 | t9 |
| before connector | landmark | 50 | 50 |
| before connector | instruction except landmark † | 0 | 0 |
| before connector | all except landmark † | 0 | 0 |
| after connector | landmark | 0 | 1 |
| after connector | instruction except landmark | 0 | 0 |
| Policy | seed 101 | seed 202 | seed 303 |
|---|---|---|---|
| A. Completed, task t9, four unpresented synonyms (%) | |||
| finetuned, undiversified | 63.0 | 61.5 | 65.0 |
| finetuned, verb-diversified | 69.5 | 66.0 | 64.7 |
| finetuned, landmark-diversified | 99.0 | 98.5 | 100.0 |
| undiversified + 66 connector tensors of the diversified policy (connector transplant) | 98.5 | 98.0 | 98.0 |
| undiversified + the diversified policy’s other 471 tensors (all-but-connector transplant) | 87.5 | 91.0 | 88.0 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment (section) | Comparison | Result |
|---|---|---|
| Released policy | ||
| calibration with the landmark phrase removed (§ 3.2 ) | each task with the landmark phrase removed | t1 7.8%, t4 20.0%, t7 0.0%, t9 4.0%; the other six 66–100% |
| surface rewrites that keep the noun (§ 3.2 ) | the original instruction vs. rewrites of word order and verb, or of relation words, on t1, t4, t7 and t9 | 99.0% vs. 98.0–99.0% (mean over the four tasks) |
| synonyms and nonce words on t1 and t4 (§ 3.2 ) | completion, synonyms vs. nonce words | t1: synonyms 12.8% (32/250) vs. nonce 70.9% (107/151); t4: synonyms 239/250, nonce 139/150 |
| synonyms, nonce words and the other landmark’s synonym (§ 4.1 ) | approach to each bowl (criteria (a)–(c)) and completion against the training word (d), mirror pair | named bowl 237/250, 223/250 vs. nonce 0/150, 4/150; other bowl 41/50, 50/50 vs. the landmark’s own synonyms 3/250, 0/250 (every ); completion 225/250 vs. 49/50 (unresolved), 169/250 vs. 50/50 ( ); hotplate 31/50 ( ) |
| parent-VLM control (§ 4.1 , Appendix B.2 ) | which bowl the edited instruction names, by summed log-likelihood, on t1, t4, t7 and t9 | correct in 199/200 synonym cases (ten frames per word; t4’s two candidates share the noun and differ only in the relation phrase, so its 50 cases do not test the synonym) |
| Task | Instruction | 6 cm | 10 cm | 15 cm | |
|---|---|---|---|---|---|
| t1 | original | 50 | 50/0 | 50/0 | 50/0 |
| t1 | synonyms (5) | 250 | 37/0 | 225/0 | 247/0 |
| t1 | landmark phrase removed | 51 | 3/0 | 36/1 | 47/1 |
| t1 | nonce words (3) | 151 | 119/0 | 149/0 | 151/0 |
| t4 | original | 50 | 34/0 | 50/0 | 50/0 |
| t4 | synonyms (5) | 250 | 167/0 | 250/0 | 250/0 |
| Landmark position | Control positions (before / after) | |||||
|---|---|---|---|---|---|---|
| Task | Instruction | Unmodified | before connector | after connector | matched other | verb |
| t7 | “cooktop” | 49/50 | 49/50 | 49/50 | – | – |
| t7 | “range” | 47/50 | 49/50 | 50/50 | – | – |
| t7 | “cooker” | 47/50 | 48/50 | 50/50 | – | – |
| t7 | “hob” | 50/50 | 49/50 | 49/50 | – | – |
| t7 | “hotplate” † | 31/50 | 47/50 | 50/50 | 31/50 / 35/50 | 28/50 / 29/50 |
| Task | Instruction | Unmasked | Landmark masked | Matched other positions masked |
|---|---|---|---|---|
| t7 | “hotplate” | 32/50 (37/4/9) | 13/50 (15/13/22) | 31/50 (38/3/9) |
| t7 | original | 49/50 (50/0/0) | 49/50 (50/0/0) | – |
| t9 | “dresser” | 39/50 (48/0/2) | 26/50 (40/0/10) | – |
| t9 | “chest of drawers” | 23/50 (40/0/10) | 21/50 (38/0/12) | 26/50 (43/0/7) |
| t9 | “sideboard” | 22/50 (41/0/9) | 20/50 (32/0/18) | – |
| t9 | “bureau” | 38/50 (46/0/4) | 18/50 (37/0/13) | – |
| Policy | Unpresented (%) | Difference, pt ( ) | Presented (%) | Original (%) |
|---|---|---|---|---|
| undiversified | 80.8, 79.1, 81.8 | – | 64.0, 50.0, 47.0 | 100.0, 98.0, 98.0 |
| instruction cross-attention (112) | 87.2, 85.2, 86.0 | +6.5 (2.5), +6.2 (2.3), +4.2 (1.6) | 68.0, 60.0, 62.0 | 99.0, 99.0, 98.0 |
| image cross-attention (112) | 86.8, 86.5, 83.0 | +6.0 (2.3), +7.5 (2.8), +1.3 (0.5) | 63.0, 55.0, 58.0 | 98.0, 99.0, 99.0 |
| self-attention (224) | 85.8, 82.5, 85.0 | +5.0 (1.9), +3.5 (1.3), +3.3 (1.3) | 67.0, 55.0, 56.0 | 98.0, 100.0, 98.0 |
| input and output tensors (23) | 84.0, 78.1, 81.0 | +3.3 (1.2), 1.0 ( 0.3), 0.8 ( 0.3) | 61.0, 44.0, 58.0 | 97.1, 98.0, 99.0 |
| all-but-connector transplant | 93.8, 94.5, 94.0 | Table 3 B | 88.2, 76.0, 82.0 | 98.0, 98.0, 98.0 |
| Policy | Task | Site, positions | Completed/ | To other bowl |
|---|---|---|---|---|
| undiversified | t7 | post-connector, all except landmark | 2/50, 0/50, 0/50 | 46, 50, 48 |
| undiversified | t7 | post-connector, landmark | 48/51, 46/50, 49/50 | 0, 0, 0 |
| undiversified | t7 | pre-connector, landmark | 0/50, 0/50, 0/50 | 50, 50, 49 |
| undiversified | t9 | post-connector, all except landmark | 1/50, 3/50, 2/50 | 47, 46, 47 |
| undiversified | t9 | post-connector, landmark | 27/50, 37/51, 39/51 | 2, 1, 2 |
| undiversified | t9 | pre-connector, landmark | 0/50, 0/50, 0/50 | 50, 50, 50 |
| Instruction | Task | Policy | Completed/ | To target | To other |
|---|---|---|---|---|---|
| nonce word | t7 | undiversified | 0/50, 0/50, 0/50 | 0, 0, 0 | 4, 1, 2 |
| nonce word | t7 | verb-diversified | 0/50, 0/50, 0/50 | 0, 0, 0 | 4, 1, 0 |
| nonce word | t7 | landmark-diversified | 0/50, 0/50, 0/50 | 0, 1, 0 | 24, 22, 18 |
| nonce word | t7 | connector transplant | 0/50, 0/50, 0/50 | 0, 0, 0 | 11, 6, 9 |
| nonce word | t7 | all-but-connector transplant | 0/50, 0/50, 0/50 | 0, 0, 0 | 8, 6, 6 |
| nonce word | t9 | undiversified | 0/50, 1/50, 0/50 | 3, 1, 0 | 0, 0, 0 |
Robustness of VLAs
Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.
See all 239 papers