ActionCodec2: A Streaming Action Tokenizer
Organizations: Tsinghua University · Galaxea AI · Fudan University
Abstract
Most action tokenizers for autoregressive vision language action (VLA) models compress motion into chunks that must be decoded in full before their actions become executable. We introduce ActionCodec2, a non-neural streaming tokenizer that releases actions incrementally while keeping trajectory reconstruction error within a prescribed tolerance. Its encoder quantizes each step relative to the decoder's reconstructed state, so the selected motion primitive corrects residual error rather than allowing errors from independently quantized increments to accumulate. Finding the smallest such primitive set is a geometric covering problem; we derive the exact minimum for translation and constant-factor-tight bounds for rotation. Set-BPE then losslessly merges co-occurring primitive symbols across coordinates and neighboring steps, without a fixed coordinate ordering. A universal version fitted on 20k hours of data from 24 embodiments averages 8.8 tokens per second for 15 Hz end-effector motion on this corpus. With the same VLA backbone, ActionCodec2 outperforms the compared tokenizers on three simulation benchmarks and in zero-shot real-robot evaluation. Its streaming representation allows policies to incorporate state feedback within an action chunk, substantially improving performance over open-loop execution under randomized dynamics, external disturbances, and moving objects. The same tokens also can serve as a direct control interface for VLM agents, which compose successful action sequences into reusable macros. These agent-built macros reduce control decisions and thinking-token usage across attempts and transfer to a held-out task. More results are available at https://zibindong.github.io/actioncodec2-page/
Figures & tables
| Tokenizer | Neural | Rule-based | Lossless | Track. bound | Streaming | Pretrained | Held-out univ. |
|---|---|---|---|---|---|---|---|
| FAST [ Pertsch et al., 2025 ] | • | ||||||
| X-Tokenizer [ Kang et al., 2026 ] | • | ||||||
| OAT [ Liu et al., 2026a ] | • | ||||||
| ACodec [ Dong et al., 2026 ] | • | ||||||
| ACodec2 (ours, corpus-fit) | • | ||||||
| ACodec2+ (ours, universal) | • |
| LIBERO | SimplerEnv | |||||||||
| Tokenizer | Spatial | Object | Goal | Long | Avg. | Spoon | Carrot | Stack | Eggplant | Avg. |
| FAST [ Pertsch et al., 2025 ] | 95.4 | 99.2 | 85.4 | 91.2 | 92.8 | 91.7 | 64.2 | 88.3 | 100.0 | 86.0 |
| OAT [ Liu et al., 2026a ] | 93.8 | 95.8 | 88.4 | 85.8 | 91.0 | 9.2 | 16.7 | 2.5 | 23.3 | 12.9 |
| OAT [ Liu et al., 2026a ] | 98.2 | 99.8 | 95.6 | 91.8 | 96.4 | 9.2 | 7.5 | 15.0 | 44.2 | 19.0 |
| ACodec [ Dong et al., 2026 ] | 95.8 | 99.6 | 94.8 | 90.2 | 95.1 | 78.3 | 55.0 | 47.5 | 76.7 | 64.4 |
| X-Tokenizer [ Kang et al., 2026 ] | 88.6 | 96.0 | 87.6 | 82.8 | 88.8 | 90.8 | 65.8 | 81.7 | 95.0 | 83.3 |
| Tokenizer | Camera | Robot | Lang. | Light | Backgr. | Noise | Layout | Avg. |
|---|---|---|---|---|---|---|---|---|
| FAST [ Pertsch et al., 2025 ] | 55.2 | 64.5 | 73.8 | 86.9 | 87.5 | 51.4 | 80.1 | 69.7 |
| OAT [ Liu et al., 2026a ] | 48.0 | 53.8 | 73.7 | 84.6 | 83.7 | 44.1 | 74.8 | 64.3 |
| OAT [ Liu et al., 2026a ] | 62.5 | 61.5 | 78.7 | 94.5 | 89.9 | 53.8 | 83.5 | 73.2 |
| ACodec [ Dong et al., 2026 ] | 59.7 | 57.8 | 75.3 | 90.8 | 87.0 | 48.5 | 79.0 | 69.4 |
| X-Tokenizer [ Kang et al., 2026 ] | 42.7 | 52.3 | 56.9 | 67.8 | 65.3 | 34.5 | 59.8 | 52.9 |
| ACodec2 | 58.2 | 60.9 | 78.5 | 92.7 | 90.4 | 52.3 | 82.1 | 71.8 |
| Tokenizer / setting | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| Deployment: real-time control at | |||||
| FAST [ Pertsch et al., 2025 ] / chunk | 35.0 | 40.0 | 47.2 | 21.4 | 35.9 |
| X-Tokenizer [ Kang et al., 2026 ] / chunk | 58.8 | 72.8 | 65.6 | 45.0 | 60.6 |
| ACodec [ Dong et al., 2026 ] / chunk | 86.0 | 94.6 | 83.8 | 73.4 | 84.5 |
| OAT [ Liu et al., 2026a ] / chunk | 92.0 | 94.0 | 86.4 | 76.4 | 87.2 |
| ACodec2+ / chunk | 82.2 | 83.0 | 72.8 | 54.4 | 73.1 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Position tolerance | EEF atoms | Joint atoms | Tokens/step | Tokens/s |
| Varying tolerance, vocabulary size 4,096 | |||||
| 1 mm | 1 mm | 836 | 1,701 | 1.38 | 20.7 |
| 5 mm | 5 mm | 176 | 353 | 0.58 | 8.8 |
| 10 mm | 10 mm | 96 | 187 | 0.45 | 6.8 |
| Varying vocabulary size, tolerance 5 mm | |||||
| 1,024 | 5 mm | 176 | 353 | 1.09 | 16.3 |
| Tokenizer | 0 | 1 | 2 | 3 | Mean std. | |
|---|---|---|---|---|---|---|
| ACodec2+ EEF | 53.4 | 43.4 | 3.1 | 0.1 | 0.005 | |
| ACodec2+ II EEF | 39.5 | 57.9 | 2.5 | 0.1 | 0.0 | |
| ACodec2+ joint | 53.4 | 29.8 | 11.2 | 3.9 | 1.7 | |
| ACodec2+ II joint | 41.4 | 43.4 | 12.6 | 1.7 | 1.0 |
| Tokenizer and dataset | 1 step | 2 steps | 3 steps | steps |
|---|---|---|---|---|
| ACodec2+ EEF, pretraining | 74.6 | 18.3 | 3.7 | 3.4 |
| ACodec2+ joint, pretraining | 71.7 | 19.3 | 2.7 | 6.2 |
| ACodec2+ EEF, held-out | 59.9 | 20.7 | 4.7 | 14.7 |
| ACodec2+ II EEF, held-out | 71.8 | 17.7 | 2.8 | 7.6 |
| Tokenizer / release | Requests | Mean | P50 | P90 | P95 | P99 |
| FAST / chunk | 13,748 | 1,348 | 914 | 4,128 | 4,244 | 4,647 |
| X-Tokenizer / chunk | 17,668 | 684 | 672 | 722 | 751 | 891 |
| OAT-aligned / chunk | 15,898 | 661 | 656 | 689 | 702 | 784 |
| ACodec / chunk | 16,652 | 574 | 565 | 603 | 627 | 1,006 |
| ACodec2+ / full chunk | 14,950 | 1,023 | 1,001 | 1,299 | 1,400 | 1,591 |
| ACodec2+ / streaming | 16,237 | 122 | 124 | 147 | 157 | 172 |
| Tokenizer | Sampled | Sampled | Greedy |
|---|---|---|---|
| FAST | 70.2 | 70.6 | 71.8 |
| OAT-aligned | 74.2 | 73.2 | 78.2 |
| X-Tokenizer | 72.0 | 77.8 | 76.0 |
| ACodec | 83.8 | 90.0 | 84.8 |
| ACodec2 | 89.0 | 90.8 | 92.6 |
| Setting and metric | ACodec2+ open | ACodec2+ closed | FAST | OAT-aligned | X-Tokenizer |
|---|---|---|---|---|---|
| Physics: task success | 64.7 | 79.0 | 24.0 | 66.3 | 45.7 |
| Force: pick | 87.2 | 94.0 | 75.2 | 85.4 | 76.2 |
| Force: place given pick | 64.4 | 72.0 | 67.4 | 74.2 | 68.8 |
| Motion: pick | 27.6 | 46.2 | 9.2 | 18.0 | 15.8 |
| Motion: place given pick | 23.2 | 31.4 | 17.8 | 22.1 | 16.7 |
| Task (source suite) | Parameter | Easy | Hard |
|---|---|---|---|
| Bowl to cabinet top (Goal) | bowl mass (kg) | – | – |
| Open middle drawer (Goal) | slide damping (N s/m) | – | – |
| slide friction (N) | – | – | |
| Milk to basket (Object) | finger-joint damping (N s/m) | – | – |
| Mug in microwave, close door (Long) | door damping (N m s/rad) | – | – |
| door friction (N m) | – | – |
| Macro | Semantics (VLM-authored) | Uses | Duration (s) |
|---|---|---|---|
| grasp_from_row_hover | From an aligned hover 10 cm above grasp height, descend open, close, and test-lift. | 26 | 1.40 |
| park_cube_in_front | With a verified cube lifted 4 cm, buffer 15 cm nearward, release, and withdraw. | 20 | 2.40 |
| set_down_and_clear | With a held cube 4 cm above placement height, lower, release, and withdraw 10 cm. | 33 | 1.40 |
| buffer_to_row_left_hover | From a matching front buffer hover, approach the neighbouring row slot 9 cm left and 15 cm forward. | 2 | 1.60 |
| row_to_front_buffer_hover | From a row-slot hover, move open 15 cm nearward to its front buffer. | 2 | 1.00 |
| buffer_to_row_left_and_place | With a verified buffer cube lifted 4 cm, add clearance, transfer 9 cm left and 15 cm forward, place, and withdraw. | 2 | 3.80 |
| Ep. | Milestone | Model’s thinking | Macros |
|---|---|---|---|
| 2 | First drawer opened | “Three closes have missed. The wrist view shows the fingers open left–right, while the handle bar also runs left–right, parallel to the finger axis, so the fingers can never pinch it. I’ll roll the tool about so the fingers open vertically and can straddle the horizontal bar.” | roll_plus_90_open, rolled_drawer_grip_pull |
| 3 | Spotted the tongs | “The left wrist view looks into the open top drawer of the left cabinet and shows only silver tongs, not the measuring cup.” | open_drawer_from_handle, tool_axis_advance_pitched |
| 5 | Found the measuring cup | “The head view now shows the measuring cup in the open right-cabinet drawer: a white round bowl with a handle.” | slow_pull_drawer_10cm, pull_drawer_by_side_wall |
| 8 | First cup grasp | “The close stalled at 0.113, which suggests the thin tab is pinching again. … this time I’m holding closed for 16 rows to build full grip force, then lifting very slowly.” | pitched_yaw_center_pinch, close_lift_slow_2 |
| 11 | Invented the playbook | “Lessons were facts, not procedures. Each decision re-derived heights, drawer level, and the escape order. … I kept guessing.” (reflection) | drawer_open_rolled_full, grasp_close_verify_lift, carry_pitchup_back_10, release_open_rise |
| 13 | First cup into the bin | “The cup is in the bin: the right wrist view shows it resting inside, and the instruction has advanced to the tongs.” | carry_pitchup_back_10, release_open_rise, table_pinch_vertical |
Vision-Language-Action Models
Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.
See all 1,231 papers