Most action tokenizers for autoregressive vision language action (VLA) models compress motion into chunks that must be decoded in full before their actions become executable. We introduce ActionCodec2, a non-neural streaming tokenizer that releases actions incrementally while keeping trajectory reconstruction error within a prescribed tolerance. Its encoder quantizes each step relative to the decoder's reconstructed state, so the selected motion primitive corrects residual error rather than allowing errors from independently quantized increments to accumulate. Finding the smallest such primitive set is a geometric covering problem; we derive the exact minimum for translation and constant-factor-tight bounds for rotation. Set-BPE then losslessly merges co-occurring primitive symbols across coordinates and neighboring steps, without a fixed coordinate ordering. A universal version fitted on 20k hours of data from 24 embodiments averages 8.8 tokens per second for 15 Hz end-effector motion on this corpus. With the same VLA backbone, ActionCodec2 outperforms the compared tokenizers on three simulation benchmarks and in zero-shot real-robot evaluation. Its streaming representation allows policies to incorporate state feedback within an action chunk, substantially improving performance over open-loop execution under randomized dynamics, external disturbances, and moving objects. The same tokens also can serve as a direct control interface for VLM agents, which compose successful action sequences into reusable macros. These agent-built macros reduce control decisions and thinking-token usage across attempts and transfer to a held-out task. More results are available at https://zibindong.github.io/actioncodec2-page/
Figures & tables
Figure 1: Chunkwise and streaming tokenization. Left: a chunk tokenizer releases actions only after the whole chunk is decoded, and an erroneous token can affect multiple actions within the chunk; a streaming tokenizer releases each action with its own tokens, so the policy can use feedback at every step. Right: a fixed VQ codebook allocates its codes according to the training distribution and provides no distribution-independent reconstruction bound; ActionCodec2 covers the reachable range and uses more tokens where needed.
Figure 2: Round-trip fidelity of action tokenizers on LIBERO demonstrations (first-order variants). (a) Demonstrations are tokenized, decoded, and replayed open loop; raw demonstrations score 100% . (b) Per-step error versus mean episode drift.
Figure 3: ActionCodec2. (a) Each step of a continuous action is quantized into primitive atoms, one per coordinate. (b) Set-BPE merges atoms that co-occur within a step and then across nearby steps, independent of coordinate order. (c) The VLA emits the tokens autoregressively and releases each action once its step is complete. (d) The decoded primitives track the trajectory within ϵ .
Figure 4: Covering primitives for translation and orientation. The decoded state initially lies within the prescribed tolerance of the true state. Candidate primitives cover the one-step reachable error set, and the selected primitive returns the tracking error to within ϵ .
Figure 5: ActionCodec2 as a Hugging Face AutoProcessor .
Tokenizer
Neural
Rule-based
Lossless
Track. bound
Streaming
Pretrained
Held-out univ.
FAST [ Pertsch et al., 2025 ]
−
•
✓
×
×
✓
✓
X-Tokenizer [ Kang et al., 2026 ]
•
−
×
×
×
✓
×
OAT [ Liu et al., 2026a ]
•
−
×
×
×
×
×
ACodec [ Dong et al., 2026 ]
•
−
×
×
×
×
×
ACodec2 (ours, corpus-fit)
−
•
✓
✓
✓
×
×
ACodec2+ (ours, universal)
−
•
✓
✓
✓
✓
✓
Table 1: Qualitative comparison of the evaluated action tokenizers. • = attribute applies; − = not applicable; ✓ = yes; × = no. Lossless refers to the compression stage only: quantization always loses. Tracking bound : a worst-case bound on the tracking error of the decoded trajectory at every step. Streaming : actions are represented stepwise along physical time rather than as a jointly represented fixed-horizon chunk. Pretrained : once on a large multi-embodiment corpus. Held-out universal : zero-shot on unseen corpora or embodiments.
LIBERO
SimplerEnv
Tokenizer
Spatial
Object
Goal
Long
Avg.
Spoon
Carrot
Stack
Eggplant
Avg.
FAST [ Pertsch et al., 2025 ]
95.4
99.2
85.4
91.2
92.8
91.7
64.2
88.3
100.0
86.0
OAT official [ Liu et al., 2026a ]
93.8
95.8
88.4
85.8
91.0
9.2
16.7
2.5
23.3
12.9
OAT aligned [ Liu et al., 2026a ]
98.2
99.8
95.6
91.8
96.4
9.2
7.5
15.0
44.2
19.0
ACodec [ Dong et al., 2026 ]
95.8
99.6
94.8
90.2
95.1
78.3
55.0
47.5
76.7
64.4
X-Tokenizer [ Kang et al., 2026 ]
88.6
96.0
87.6
82.8
88.8
90.8
65.8
81.7
95.0
83.3
Table 2: LIBERO and SimplerEnv results (%). Every row uses the same G0.5 VLA backbone [ Liu et al., 2026c ] , fine-tuned with the corresponding tokenizer. LIBERO: evaluated on the four LIBERO suites with 50 official initial states per task ( 2,000 episodes). SimplerEnv: four WidowX tasks, 120 initial states each ( 480 episodes). Bold is the best in a column, underline the second best.
Tokenizer
Camera
Robot
Lang.
Light
Backgr.
Noise
Layout
Avg.
FAST [ Pertsch et al., 2025 ]
55.2
64.5
73.8
86.9
87.5
51.4
80.1
69.7
OAT official [ Liu et al., 2026a ]
48.0
53.8
73.7
84.6
83.7
44.1
74.8
64.3
OAT aligned [ Liu et al., 2026a ]
62.5
61.5
78.7
94.5
89.9
53.8
83.5
73.2
ACodec [ Dong et al., 2026 ]
59.7
57.8
75.3
90.8
87.0
48.5
79.0
69.4
X-Tokenizer [ Kang et al., 2026 ]
42.7
52.3
56.9
67.8
65.3
34.5
59.8
52.9
ACodec2
58.2
60.9
78.5
92.7
90.4
52.3
82.1
71.8
Table 3: LIBERO-Plus robustness (%). Task-level success over all 10,030 perturbed tasks; each column pools the perturbations in that category. Bold is the best in a column, underline the second best.
Tokenizer / setting
Spatial
Object
Goal
Long
Avg.
Deployment: real-time control at 20Hz
FAST [ Pertsch et al., 2025 ] / chunk
35.0
40.0
47.2
21.4
35.9
X-Tokenizer [ Kang et al., 2026 ] / chunk
58.8
72.8
65.6
45.0
60.6
ACodec [ Dong et al., 2026 ] / chunk
86.0
94.6
83.8
73.4
84.5
OAT aligned [ Liu et al., 2026a ] / chunk
92.0
94.0
86.4
76.4
87.2
ACodec2+ / chunk
82.2
83.0
72.8
54.4
73.1
Table 4: Ablations (%). Upper: the client runs on a 20Hz wall clock and sends zero motion whenever its action queue is empty; Lower: the same ACodec2 primitive stream packed by BPE or Set-BPE; both packers are lossless, and gray values give the gain of Set-BPE. Bold is the best within a panel, underline the second best.
Figure 6: DROID-Franka real-robot task progress (%). Five policies on the four self-designed task suites, 50 rollouts per policy, scored by task progress (Appendix D.4 ). Policies are trained on DROID and evaluated zero-shot in unseen scenes; error bars on Overall are 90% stratified bootstrap CIs. The right panel reports per-policy inference latency (ms, right axis), the wall-clock time from submitting an observation to obtaining the corresponding decoded action output.
Figure 7: Example rollouts of ActionCodec2 on the self-designed DROID-Franka tasks.
Figure 8: PPO learning curves (left) and packing compression (right). Left: hardened LIBERO-Object, with objects and basket re-placed at random; policies are initialized from checkpoints with about 5% success, then trained for 200 PPO iterations with 320 rollouts each. Solid curves show the rollout success rate as a rolling mean, shaded bands ± one rolling standard deviation. Right: lines show mean tokens per action window, bars the token saving over BPE; unseen episodes and unseen tasks are held out from fitting; the dotted line marks the deployed 4,096 -entry vocabulary.
Figure 9: PPO training under in-chunk perturbations. Random physics: task success over six tasks; the first 200 iterations train on the easy parameter ranges and the rest on the hard ranges, and every evaluation uses the hard ranges. Random force and random motion: pick is the fraction of episodes in which the target object is grasped, and place is the fraction of these episodes in which it is then placed in the basket; ten tasks each. Every point averages 50 fixed seeds per task.
Figure 10: Agent control through ActionCodec2. Claude Opus 5.5 controls a simulated YAM robot in Multi-Drawer Search. The timeline shows maximum shaped reward, the size of the accumulated macro library, and selected first-time milestones; the first full task success occurs in episode 20.
Figure 11: Agent control interfaces on RMBench. ( a ) The three control representations. ( b ) Decision count, thinking-token usage, and macro-library growth over four chained rounds. ( c ) Transfer of the learned macro library to a held-out rearrangement task.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Position tolerance
EEF atoms
Joint atoms
Tokens/step
Tokens/s
Varying tolerance, vocabulary size 4,096
1 mm
1 mm
836
1,701
1.38
20.7
5 mm
5 mm
176
353
0.58
8.8
10 mm
10 mm
96
187
0.45
6.8
Varying vocabulary size, tolerance 5 mm
1,024
5 mm
176
353
1.09
16.3
Appendix
Table 5: Tolerance and vocabulary size. First-order universal tokenizers for end-effector (EEF) and joint actions. Token rates are for EEF actions at 15Hz ; values are rounded. The tolerance comparison uses the pretraining dataset, and the vocabulary comparison uses an evaluation slice.
Tokenizer
0
1
2
3
≥4
Mean ± std.
ACodec2+ EEF
53.4
43.4
3.1
0.1
0.005
0.50±0.56
ACodec2+ II EEF
39.5
57.9
2.5
0.1
0.0
0.63±0.54
ACodec2+ joint
53.4
29.8
11.2
3.9
1.7
0.71±0.96
ACodec2+ II joint
41.4
43.4
12.6
1.7
1.0
0.78±0.83
Appendix
Table 6: New tokens required to release an action step (%). A step may require zero new tokens when earlier temporal tokens already filled its slots.
Tokenizer and dataset
1 step
2 steps
3 steps
≥4 steps
ACodec2+ EEF, pretraining
74.6
18.3
3.7
3.4
ACodec2+ joint, pretraining
71.7
19.3
2.7
6.2
ACodec2+ EEF, held-out
59.9
20.7
4.7
14.7
ACodec2+ II EEF, held-out
71.8
17.7
2.8
7.6
Appendix
Table 7: Active steps covered per emitted token (%).
Tokenizer / release
Requests
Mean
P50
P90
P95
P99
FAST / chunk
13,748
1,348
914
4,128
4,244
4,647
X-Tokenizer / chunk
17,668
684
672
722
751
891
OAT-aligned / chunk
15,898
661
656
689
702
784
ACodec / chunk
16,652
574
565
603
627
1,006
ACodec2+ / full chunk
14,950
1,023
1,001
1,299
1,400
1,591
ACodec2+ / streaming
16,237
122
124
147
157
172
Appendix
Table 8: Client first-action latency in real-time LIBERO (ms). The client control clock runs at 20Hz .
Tokenizer
Sampled H=20
Sampled H=10
Greedy H=20
FAST
70.2
70.6
71.8
OAT-aligned
74.2
73.2
78.2
X-Tokenizer
72.0
77.8
76.0
ACodec
83.8
90.0
84.8
ACodec2
89.0
90.8
92.6
Appendix
Table 9: PPO evaluation with different execution horizons (%). Policies are evaluated over 500 episodes at PPO step 200.
Figure 12: Example task observations of the evaluation benchmarks. (a) LIBERO [ Liu et al., 2023 ] : MuJoCo Franka Panda tasks observed through the front camera; shown are one Spatial and one Object task out of the four suites. (b) LIBERO-Plus [ Fei et al., 2026 ] : 10,030 variants of the LIBERO tasks generated along seven perturbation axes (camera, robot init, language, layout, background, light, noise). (c) SimplerEnv-WidowX [ Li et al., 2024 ] : the four BridgeData tasks [ Walke et al., 2023 ] rebuilt in SAPIEN under visual matching. (d) DROID-Franka [ Khazatsky et al., 2024 ] : in-the-wild scenes of the DROID corpus on which the policies are fine-tuned. (e) DROID-Franka: the initial states of the four self-designed suites (color-discriminated pick-and-place, snack storing and serving, nested-cup arrangement, towel unfolding) on which the fine-tuned policies are evaluated zero-shot; task progress is reported in Figure 6 . Images in (a)–(c) are reproduced from the official benchmark repositories.
Figure 13: DROID-Franka per-suite breakdown (%). The milestone-level process behind Figure 6 : task progress per instructed color pair, target items stored vs. distractor items misplaced (drawn from the inverted top axis; fewer is better), per-milestone nested-cup success, and towel-unfold attainment. Each row closes with the suite mean behind a divider; colors and legend match Figure 6 .
Setting and metric
ACodec2+ open
ACodec2+ closed
FAST
OAT-aligned
X-Tokenizer
Physics: task success
64.7
79.0
24.0
66.3
45.7
Force: pick
87.2
94.0
75.2
85.4
76.2
Force: place given pick
64.4
72.0
67.4
74.2
68.8
Motion: pick
27.6
46.2
9.2
18.0
15.8
Motion: place given pick
23.2
31.4
17.8
22.1
16.7
Appendix
Table 10: Feedback under in-chunk perturbations (%). Random physics reports task success. For random force and motion, pick is grasp success and place is success conditional on a grasp. Each task is evaluated on 50 fixed seeds.
Task (source suite)
Parameter
Easy
Hard
Bowl to cabinet top (Goal)
bowl mass (kg)
1.0 – 2.0
2.25 – 4.5
Open middle drawer (Goal)
slide damping (N ⋅ s/m)
300 – 1000
506 – 1688
slide friction (N)
8 – 23
13.5 – 38.8
Milk to basket (Object)
finger-joint damping (N ⋅ s/m)
3500 – 15000
13125 – 56250
Mug in microwave, close door (Long)
door damping (N ⋅ m ⋅ s/rad)
0.5 – 2.5
0.75 – 3.75
door friction (N ⋅ m)
0.05 – 0.25
0.075 – 0.375
Appendix
Table 11: Random-physics parameters. Each parameter is drawn uniformly per episode.
Macro
Semantics (VLM-authored)
Uses
Duration (s)
grasp_from_row_hover
From an aligned hover 10 cm above grasp height, descend open, close, and test-lift.
26
1.40
park_cube_in_front
With a verified cube lifted 4 cm, buffer 15 cm nearward, release, and withdraw.
20
2.40
set_down_and_clear
With a held cube 4 cm above placement height, lower, release, and withdraw 10 cm.
33
1.40
buffer_to_row_left_hover
From a matching front buffer hover, approach the neighbouring row slot 9 cm left and 15 cm forward.
2
1.60
row_to_front_buffer_hover
From a row-slot hover, move open 15 cm nearward to its front buffer.
2
1.00
buffer_to_row_left_and_place
With a verified buffer cube lifted 4 cm, add clearance, transfer 9 cm left and 15 cm forward, place, and withdraw.
2
3.80
Appendix
Table 12: Macros induced by the VLM on RMBench (the library carried into Figure 11 c ). Names and semantics are the model’s own words, not post-hoc human summaries.
Ep.
Milestone
Model’s thinking
Macros
2
First drawer opened
“Three closes have missed. The wrist view shows the fingers open left–right, while the handle bar also runs left–right, parallel to the finger axis, so the fingers can never pinch it. I’ll roll the tool about 90∘ so the fingers open vertically and can straddle the horizontal bar.”
roll_plus_90_open, rolled_drawer_grip_pull
3
Spotted the tongs
“The left wrist view looks into the open top drawer of the left cabinet and shows only silver tongs, not the measuring cup.”
“The head view now shows the measuring cup in the open right-cabinet drawer: a white round bowl with a handle.”
slow_pull_drawer_10cm, pull_drawer_by_side_wall
8
First cup grasp
“The close stalled at 0.113, which suggests the thin tab is pinching again. … this time I’m holding closed for 16 rows to build full grip force, then lifting very slowly.”
pitched_yaw_center_pinch, close_lift_slow_2
11
Invented the playbook
“Lessons were facts, not procedures. Each decision re-derived heights, drawer level, and the escape order. … I kept guessing.” (reflection)
Table 13: The nine first-time milestones of Figure 10 , in the model’s own words. Thinking is quoted (abridged) from the decision logs and end-of-episode reflections; macros are those born at, or executed during, the milestone (the library is append-only).
Vision-Language-Action (VLA) models leveraging the native autoregressive paradigm of Vision-Language Models (VLMs) have demonstrated superior instruction-following and training efficiency. Central to this paradigm is action tokenization, yet its design has primarily focused on reconstruction fidelity, failing to address its direct impact on VLA optimization. Consequently, the fundamental question of what makes for good action tokenizers remains unanswered. In this paper, we bridge this gap by establishing design principles specifically from the perspective of VLA optimization. We identify a set of best practices based on information-theoretic insights, including maximized temporal token overlap, minimized vocabulary redundancy, enhanced multimodal mutual information, and token independence. Guided by these principles, we introduce ActionCodec, a high-performance action tokenizer that significantly enhances both training efficiency and VLA performance across diverse simulation and real-world benchmarks. Notably, on LIBERO, a SmolVLM2-2.2B fine-tuned with ActionCodec achieves a 95.5% success rate without any robotics pre-training. With advanced architectural enhancements, this reaches 97.4%, representing a new SOTA for VLA models without robotics pre-training. We believe our established design principles, alongside the released model, will provide a clear roadmap for the community to develop more effective action tokenizers.
Zibin Dong, Yicheng Liu, Shiduo Zhang +8
Knowin AI (Work done during an internship) · Tsinghua University · Tianjin University +2
Action tokenizers play a central role in autoregressive vision-language-action (VLA) models, determining both the targets for policy training and the executable commands recovered from predicted tokens. Their fidelity is commonly evaluated using pointwise reconstruction metrics such as mean squared error (MSE), yet small individual errors do not fully characterize how faithfully action adjustments across demonstrations are preserved. After compression, similar actions may still cluster around a representative motion, while the adjustments needed for different contexts are diminished, distorted, or even reversed. We introduce physical rank consistency (PRC) to measure how well tokenization preserves local physical distance rankings after reconstruction. Evaluating decoded actions provides a common reference across token vocabularies and decoder architectures, complementing pointwise accuracy with a measure of relational fidelity. We further present ActionPiece, which preserves physical action relationships through joint supervision of representation learning and quantization. Physical rank preservation supervises near-far ordering in encoder and quantized feature distances, while quantization regularization applies the same ordering to codeword assignment distributions. Both objectives augment reconstruction, producing discrete action tokens for standard autoregressive policy learning and execution through a frozen decoder. Under the same Qwen3-VL-4B policy training setup, ActionPiece achieves 94.8% on LIBERO and 68.8% on unseen LIBERO-Plus, with additional evaluations reaching 71.9% on SimplerEnv and 51.5% across VLA-Arena L0-L2. Component ablations show that the two objectives jointly improve PRC and policy success, demonstrating the value of physical relationship supervision for action tokenization.
Shijie Lian, Bin Yu, Zhaolong Shen +5
Huazhong University of Science and Technology · Zhongguancun Academy · DeepCybo +5
Vision-language-action (VLA) models rely on discrete action tokenizers to bridge continuous robot control and autoregressive sequence modeling, yet existing tokenizers often trade off between compression, latency, and downstream performance. We revisit this design through the lens of neural audio codecs-convolutional encoder-decoder architectures with residual vector quantization that serve as the standard front end for audio foundation models. Motivated by their success, we introduce the Neural Action Codec (NAC), which treats short robot action trajectories as multi-channel 1D signals and compresses them using a multi-scale RVQGAN architecture. We observe that audio-specific mel-spectrogram objectives are ill-suited for kinematic signals; however, by replacing them with simple time-domain and non-mel spectral reconstruction losses, audio-codec-style models can autoencode actions with high fidelity without substantial architectural changes. NAC provides a compact, ordered token space via offset codebooks, enabling standard autoregressive policies to operate over short, structured sequences. Meanwhile, a Vocos-style decoder with an ISTFT head and adversarial discriminators recovers smooth, detailed trajectories. Across LIBERO-10, RoboMimic, and a suite of real-world manipulation tasks, NAC achieves lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates. These results demonstrate that repurposed neural audio codecs offer a strong, practical backbone for learned action tokenization in modern VLAs.