CoBoVLA: Decoupling Language from Action with Discrete Codes for Robust and Interpretable Robot Manipulation
Organizations: University of California San Diego · Carnegie Mellon University
Abstract
Vision-Language-Action (VLA) models achieve high success rates on the LIBERO benchmark but remain sensitive to paraphrasing. Existing VLAs ranging from 0.6B to 7.5B parameters experience a 22-52 point performance drop on LIBERO-Para while SmolVLA exhibits an even larger decline, from 83% to 8%. Because most VLAs jointly learn language understanding and action generation, it is difficult to determine whether a failure originates from misunderstanding an instruction or executing it incorrectly. To address these challenges, we introduce CoBoVLA, or CodeBookVLA, a 250M-parameter VLA that decouples language understanding, planning, and control through learned discrete interpretable codes. A text encoder maps the instructions to shared step codes, which a visual planner then converts into action code sequences executed by a language-independent actor. This design improves robustness to instruction paraphrasing while allowing failures to be traced to individual stages. It also enables efficient real-time control on resource-constrained hardware by encoding instructions only once and using compact discrete representations to keep each component lightweight. CoBoVLA achieves 91.4% success on LIBERO-Goal, compared with 82.8% for SmolVLA, under a matched protocol, with 17 ms per control step latency. On LIBERO-Para, it achieves 81.1% across 4,092 unseen paraphrases and 78.5% on matched episodes, compared with 36.5% for SmolVLA LoRA fine-tuned on paraphrases. Without two-step demonstrations, CoBoVLA completes 48.8% of 160 free-form two-step tasks. Finally, success reaches 91% when instructions are correctly encoded but falls to 27.2% otherwise, identifying language understanding as the main source of remaining failures. Together, these results demonstrate that decoupling language understanding and control can improve instruction robustness while making VLA failures easier to diagnose.
Figures & tables
| Method | Both steps | First step |
| SmolVLA | 0/80 (0.0) | 0/80 |
| CoBoVLA | 78/160 (48.8) | 109/160 |
| CoBoVLA, through the codes | ||
| stage 1 parses both steps correctly, in order | 138/160 | |
| both steps succeed when the parse is correct | 74/138 | |
| done code fires | 156/160 | |
| Two-step episode outcome (CoBoVLA, 160) | Episodes |
|---|---|
| Success | 78 |
| Parse error (stage 1) | 18 |
| First step not achieved | 11 |
| Done code fired before the first goal held | 5 |
| First step undone while executing the second | 22 |
| Second step not achieved, first step intact | 26 |
| Method | Params | Goal | LIBERO-Para | Drop | PRIDE |
|---|---|---|---|---|---|
| Our protocol | |||||
| CoBoVLA (ours) | 0.25B | 91.4 | 81.1 | 10.3 | 78.5 |
| Reported, original protocols | |||||
| Xiaomi-Robotics-0 [ 2 ] | 4.7B | 98.8 | 76.0 | 22.8 | 69.2 |
| + GSR [ 42 ] | 3.6B | 91.0 | 75.6 | 15.4 | 70.4 |
| ParaVLA [ 42 ] | 0.33B | 92.0 a | 72.5 | 19.5 | 66.9 |
| LIBERO-Para episodes | Episodes | Success (%) |
| All | 4,092 | 81.1 |
| step codes = canonical | 3,459 | 91.0 |
| step codes canonical | 633 | 27.2 |
| codes of another LIBERO-Goal task | 181 | 0.0 |
| codes of no LIBERO-Goal task | 417 | 32.9 |
| canonical + spurious second step | 35 | 100.0 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Hyperparameters |
|---|---|
| Action codebook | 2 residual levels 32 entries, code dim 64, MLP encoder/decoder (hidden 256); segment resampled to 16 steps; segment length 6–192; initial windows of 16 steps; 8 rounds 3,000 autoencoder steps; EMA quantizers (decay 0.99) with dead-entry restarts |
| Meaning codebook | bge-base, top 2 layers fine-tuned (lr ); slots, shared cosine codebook ; slot transformer 2 layers, width 256, 4 heads, code dim 64; 6,000 steps, batch 256 pairs, lr ; decode-only warm-up 1,000 steps; loss weights swap 1, commitment 0.25, entropy 1, slot overlap 1 |
| Step coder (stage 1) | bge-small, top 4 layers fine-tuned (lr ), head lr ; up to 3 steps; 6,000 steps, batch 96; batch mix 35% single-step, 20% fused two-step, 45% joined two-step; half of joined sentences pair two LIBERO-Goal tasks; presence loss weight 1 |
| Planner | 6 layers, width 512, 8 heads, 26.1M parameters; memory 2 64 SigLIP tokens + state + 6 step codes; chain of 3 segments 2 levels + end token; 3,000 steps, batch 32, lr , weight decay 0.01, warm-up 300; DAgger fraction 0.5; step-code substitution 0.05; image shift 8 px |
| Planner, done fine-tuning | 1,000 steps, lr , warm-up 100; 16 two-step states per batch, done loss weight 1 |
| Actor | 12 layers, width 512, 8 heads; adaLN-zero conditioning on the flow time and 6 code slots; chunk 50, 5 executed; 2 64 pooled SigLIP tokens + glimpses over 2 256 patches (2 context blocks); last 4 SigLIP layers fine-tuned at 0.2 lr; batch 32, weight decay , EMA 0.999; null codes 0.1, level-2 dropout 0.3; planner codes 0.5 (from stage 3); path loss weight 0.1, px; 5 stages, 39k steps (lr then ) |
| LIBERO-Goal, 50 per task | Goal-100, initial states 0–9 (10 per task) | |||||||||
| Task | CoBoVLA | SmolVLA | seed 0 | seed 1 | seed 2 | SmolVLA | +para LoRA | discrete | continuous | |
| 0 | open middle drawer | 48 | 40 | 10 | 10 | 10 | 9 | 9 | 10 | 10 |
| 1 | open top drawer, bowl inside | 46 | 26 | 10 | 7 | 10 | 6 | 4 | 10 | 9 |
| 2 | push plate to stove front | 40 | 45 | 9 | 10 | 9 | 9 | 8 | 8 | 9 |
| 3 | bowl on plate | 48 | 46 | 10 | 10 | 10 | 10 | 6 | 10 | 9 |
| 4 | bowl on stove | 50 | 45 | 10 | 10 | 10 | 10 | 6 | 10 | 9 |
| LIBERO-Para full, CoBoVLA | LIBERO-Para subset (200), successes | |||||||||||
| Task | Episodes | Success | Canon. | Succ. canon. | Und. | Plan. | Exec. | CoBoVLA | SmolVLA | +para LoRA | discrete | continuous |
| 0 | 416 | 94.2 | 96.9 | 94.3 | 1 | 7 | 16 | 19/20 | 3/20 | 5/20 | 19/20 | 19/20 |
| 1 | 410 | 64.4 | 72.2 | 87.8 | 110 | 4 | 32 | 7/20 | 1/20 | 3/20 | 13/20 | 11/20 |
| 2 | 406 | 87.9 | 91.6 | 88.4 | 6 | 13 | 30 | 16/20 | 0/20 | 12/20 | 16/20 | 14/20 |
| 3 | 386 | 87.8 | 78.2 | 99.3 | 45 | 2 | 0 | 16/19 | 1/19 | 5/19 | 17/19 | 16/19 |
| 4 | 403 | 94.0 | 85.9 | 98.0 | 17 | 0 | 7 | 18/20 | 1/20 | 4/20 | 16/20 | 14/20 |
| Component | GPU-hours |
|---|---|
| Actor (5 stages, 39k steps) | 5.45 |
| Planner (behavior cloning + DAgger, done fine-tuning) | 0.50 |
| DAgger rollouts (200 episodes) | 0.59 |
| Two-step rollouts for the done labels (80 episodes) | 0.61 |
| Meaning codebook | 1.0 |
| Action codebook (MDL, 8 rounds) | 0.2 |
| Configuration (selection sets, 200 eps.) | Success | vs. prev. |
|---|---|---|
| Actor, executed without ensembling | 161 (80.5) | – |
| + temporal ensembling | 176 (88.0) | 0.04 |
| + fine-tuned SigLIP tail | 180 (90.0) | 0.57 |
| + path-supervised code attention (final) | 182 (91.0) | 0.82 |
| Control: + training steps only (with ens.) | 173 (86.5) | – |
| Step 1 step 2 | Parse | First | Both | SmolVLA | Step 1 step 2 | Parse | First | Both | SmolVLA | ||
| 0 | open middle drawer bowl on stove | 10 | 0 | 0 | 0/5 | 8 | cream cheese in bowl bowl on plate | 7 | 6 | 2 | 0/5 |
| 1 | open middle drawer wine bottle on rack | 10 | 2 | 0 | 0/5 | 9 | cream cheese in bowl turn on stove | 7 | 4 | 4 | 0/5 |
| 2 | open middle drawer turn on stove | 10 | 10 | 8 | 0/5 | 10 | wine bottle on rack bowl on plate | 7 | 5 | 4 | 0/5 |
| 3 | push plate to stove front wine bottle on rack | 10 | 8 | 4 | 0/5 | 11 | wine bottle on rack bowl on stove | 7 | 5 | 4 | 0/5 |
| 4 | bowl on plate wine bottle on rack | 8 | 7 | 0 | 0/5 | 12 | wine bottle on rack cream cheese in bowl | 9 | 6 | 3 | 0/5 |
| 5 | bowl on plate turn on stove | 10 | 10 | 8 | 0/5 | 13 | wine bottle on top of cabinet bowl on stove | 8 | 8 | 8 | 0/5 |
| Instruction’s task task whose codes stage 1 produced | Episodes | Planner follows it | Example paraphrase |
| bowl on top of cabinet bowl on plate | 48 | 46 | gently put the vessel on top of the sideboard |
| wine bottle on top of cabinet wine bottle on rack | 41 | 41 | swiftly put the wine container on top of the sideboard |
| wine bottle on rack wine bottle on top of cabinet | 38 | 38 | now put the wine bottle on the kitchen rack |
| wine bottle on top of cabinet bowl on top of cabinet | 24 | 21 | deliberately put the vessel on top of the cupboard |
| open top drawer, bowl inside bowl on top of cabinet | 7 | 6 | the top bin is the spot for the vessel |
| bowl on stove turn on stove | 6 | 6 | the stove is still clear |
| Code | Segs. | Len. | Gripper | Motion | Tasks | Main | Code | Segs. | Len. | Gripper | Motion | Tasks | Main | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 25 | 279 | 16 | open | +10.6 | +2.0 | -1.1 | fwd | 7 | 3,4,5 | 29 | 100 | 14 | closed | +4.3 | -3.1 | -2.9 | fwd, | 7 | 8 |
| 23 | 268 | 16 | open | +2.6 | -0.3 | -1.7 | small | 3 | 0,7 | 19 | 89 | 12 | open | -2.3 | -1.6 | -5.0 | down | 8 | 1,0 |
| 20 | 261 | 9 | open | +0.2 | +0.4 | +1.5 | small | 10 | 7 | 30 | 80 | 10 | opens | +0.3 | +0.2 | +1.1 | small | 8 | 5,3,6 |
| 7 | 247 | 15 | open | -2.6 | -0.4 | -12.4 | down | 9 | 4,3,5 | 2 | 65 | 10 | closed | -0.6 | -1.2 | -4.8 | down | 7 | 7,0 |
| 18 | 233 | 14 | closed | +7.7 | -4.3 | +10.6 | fwd, , up | 6 | 8,3,5,1 | 15 | 65 | 11 | closed | +0.2 | +0.9 | +2.8 | small | 9 | 8,4 |
| 6 | 219 | 11 | open | -1.5 | -0.5 | -5.5 | down | 10 | 1,7 | 16 | 63 | 18 | closed | +1.3 | -7.7 | +7.7 | , up | 3 | 7 |
| Median | Median | Motion | Reuse across tasks (%) | Within-task | Task ID | Gripper changes | ||
|---|---|---|---|---|---|---|---|---|
| Segmentation | segment | chain length | rec. error | 2-grams | 3-grams | NED | from chain (%) | with a boundary (%) |
| gripper events (hand-designed) | – | 3 | 0.355 | 72 | 35 | 0.33 | 73 | 100 ∗ |
| MDL, 16-step init ( ours ) | 13 | 9 | 0.188 | 96 | 74 | 0.46 | 62 | 87 |
| MDL, 32-step init | 16 | 7 | 0.231 | 95 | 71 | 0.56 | 61 | 72 |
| MDL, 64-step init | 38 | 3 | 0.328 | 88 | 61 | 0.50 | 59 | 26 |
| LIBERO-Para type | Episodes | Success | Canonical |
|---|---|---|---|
| Action expression varied (object named as in LIBERO-Goal) | |||
| lexical: addition/deletion | 100 | 93.0 | 100.0 |
| lexical: contextual synonym | 79 | 91.1 | 100.0 |
| lexical: habitual synonym | 74 | 85.1 | 100.0 |
| pragmatic: embedded imperative | 93 | 95.7 | 100.0 |
| pragmatic: hint | 88 | 81.8 | 92.0 |
| Canonical instruction | Step codes | |
|---|---|---|
| 0 | open the middle drawer of the cabinet | 30 30 30 27 30 30 |
| 1 | open the top drawer and put the bowl inside | 3 3 3 27 3 3 |
| 2 | push the plate to the front of the stove | 7 7 7 14 7 7 |
| 3 | put the bowl on the plate | 13 13 13 27 13 13 |
| 4 | put the bowl on the stove | 29 29 29 27 29 29 |
| 5 | put the bowl on top of the cabinet | 16 16 16 27 16 16 |
Vision-Language-Action Models
Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.
See all 1,239 papers