ReAT: Residual Action Tokenization for Autoregressive Robot Policies
Authors: Dijia Zhan, Zhuo Li, Xuemiao Xu, Sun Zheng, Jinyi Li, Wenjun Wang, Jie Tang
Organizations: South China University of Technology (scut.edu.cn) · The Chinese University of Hong Kong (cuhk.edu.hk) · Hong Kong Polytechnic University (polyu.edu.hk)
Autoregressive robot policies rely on action tokenizers to convert continuous action chunks into compact sequences of discrete prediction targets. Existing tokenizers can reconstruct action chunks from discrete codes, but often leave implicit what each individual token contributes to the reconstructed action. We introduce Residual Action Tokenization (ReAT), which gives each token an explicit role in refining the current action reconstruction. It uses the decoded reconstruction as feedback to construct the next token from the remaining action error. This closes the loop between encoding and decoding, making each new token responsive to the errors left by the actual decoded prefix. Across 19 manipulation tasks spanning three simulation benchmarks and real-world settings, 8-token ReAT policies outperform the evaluated baselines in task success; on LIBERO, even 4-token ReAT surpasses the evaluated 8-token action-tokenization baselines at half the token budget. These results suggest a path toward action tokenization that supports self-corrective policy learning, helping autoregressive policies learn to progressively refine their own action plans.
Figures & tables
Figure 1: Comparison of action-token decoding strategies. Full-sequence decoding requires all M tokens, while ordered prefix decoding reconstructs a complete action chunk from any nonempty prefix. ReAT combines the existing draft with each new token to produce a learned correction. Orange arrows indicate the corrections applied to the previous reconstruction in ReAT; target trajectories are shown only for reference.
Figure 2: Arm trajectory reconstruction with residual tokens. A 32-step LIBERO example at K=2,4,6,8 using an earlier ReAT tokenizer. Five sampled poses show translucent motion history and an opaque final pose. MSE is computed over all steps and normalized action dimensions of this example.
Figure 3: Architecture of ReAT. Top left: residual tokenization with reconstruction feedback across tokens. Top right: residual encoding and draft-conditioned decoding in one residual refinement iteration. Bottom: an observation-conditioned policy predicts tokens, whose prefixes are decoded by the frozen tokenizer.
Figure 4Table 5Figure 6
Figure 6: Real-world bucket stacking. Left: a successful ReAT sequence on the X-Trainer arm. Right: failure examples in which DP closes its gripper prematurely and descends into the bucket without grasping it, OAT grasps the bucket but misaligns it during placement, and QueST fails to align the gripper for a successful grasp.
Table 8
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Observation / prediction / execution horizon
2 / 32 / 16
Maximum content tokens / vocabulary
8 / 1000
FSQ scalar levels
[8,5,5,5]
Register encoder
width 256, head dimension 64
Edit decoder
width 256, head dimension 64
Autoregressive policy
4 layers, width 256, 4 heads
Appendix
Table 7: Main ReAT settings. EMA is enabled in both stages.
Action dimension / summary
ReAT
OAT
QueST
FAST
Bin
Translation x
0.94889
0.93819
0.94711
0.99463
0.99994
Translation y
0.96745
0.95924
0.96496
0.99662
0.99996
Translation z
0.96882
0.95807
0.96940
0.99708
0.99997
Rotation x
0.78624
0.73519
0.35841
0.97437
0.99967
Rotation y
0.88375
0.83200
0.53693
0.98556
0.99982
Rotation z
0.89567
0.85831
0.68065
0.98659
0.99971
Appendix
Table 8: LIBERO tokenizer reconstruction by action dimension. R2 values are task-averaged; MSE is measured in original action coordinates. Kˉ counts content tokens.
Table 9: LIBERO comparison with matched action coordinates. Success (%) at K=8 , reported as mean ± SEM across five evaluation rounds. Rotation and gripper channels remain per-step in every row.
Autoregressive action-token policies such as vision-language-action models require action tokenizers to translate discrete token sequences into precise control actions in continuous space. Many action tokenizers learn the mapping between tokens and actions via a reconstruction objective. However, as we show through extensive analysis, sufficiently accurate action reconstruction is only one part of what makes a downstream robot policy successful. It is also critical that the policy is able to predict the right tokens for new observations, and that unseen policy token predictions still decode into reasonable actions. These properties are downstream of tokenizer training and are not directly incentivized by a reconstruction objective alone. In this work, we introduce Predictable and Robust Action Tokenization (ProAct), a tokenizer training method that strategically augments reconstruction with the goal of improving downstream predictability and robustness. ProAct is policy-agnostic and uses only action datasets for training. Across the Robomimic, LIBERO, and RoboTwin benchmarks and a diverse set of tokenizer architectures, ProAct improves rollout success by an average of 11.3 percentage points. These improvements also translate to vision-language-action policies and real-world robotic manipulation, yielding average gains of 21.8 and 36.7 percentage points, respectively. These results suggest that effective action tokenization should be designed as a policy interface that balances fidelity, predictability, and robustness, rather than as a reconstruction problem alone.
Haoran Chen, Jingtian Ji, Samuel Wheeler +2
Toyota Technological Institute at Chicago · Argonne National Laboratory
Autoregressive robot policies learn continuous control by predicting discrete action tokens from observations. Different tasks often share local motions, yet behavioral correspondence across demonstrations receives limited explicit supervision in existing tokenizers. Motions with different timing can therefore lack a shared representation despite following similar patterns. We propose Behavior-Aligned Action Tokenization (BAAT), which uses soft dynamic time warping (Soft-DTW) to select corresponding action chunks and aligns their quantized coordinates jointly with reconstruction. This objective encourages similar motions across tasks to occupy nearby quantized representations while retaining executable action detail. A history-conditioned diffusion decoder reconstructs continuous action chunks from these tokens, and a downstream autoregressive policy learns to predict them. We evaluate BAAT on selected tasks from three simulation benchmarks and two real robot tasks. BAAT achieves a mean simulation success rate of approximately 45.2%, exceeding OAT by approximately 7.2 percentage points. In the controlled LIBERO-All alignment ablation, policy success rises from 70.2% to 79.0% while trajectory replay success decreases. These results support behavioral correspondence as supervision for organizing shared motion structure in action tokenizers and improving downstream robot policy learning.
Junbo Dong, Ze Chen, Zhendong Xie +5
Southern University of Science and Technology · Dexmal · National University of Singapore +1
Autoregressive Vision-Language-Action models often represent continuous robot actions as discrete token sequences, enabling action prediction with standard next-token objectives. FAST has substantially improved this representation by compactly encoding action containing diverse temporal frequencies into relatively few tokens. However, while such compression reduces the number of action tokens required for autoregressive prediction, it does not necessarily improve the efficiency of policy learning from limited demonstrations. In particular, FAST typically assigns a single deterministic tokenization to each quantized action sequence, although multiple token sequences can represent and decode to the same robot motion. We investigate whether exploiting this representational redundancy can improve policy learning. In this paper, we propose TOkenization of Action sequences with STochastic sampling (TOAST), a stochastic action tokenization method that samples alternative tokenizations of the same quantized action sequence during policy training. This diversifies the discrete supervision while preserving the underlying robot action and requires no additional demonstrations. Experiments on LIBERO show that TOAST consistently improves over its deterministic counterpart, with the improvement increasing as training data decreases, achieving a 6.8 point gain in success rate when only 1/16 of training data is available. Across four real-robot manipulation tasks, TOAST further improves mean success rate by 15.8 points over the deterministic counterpart. These results demonstrate the effectiveness of stochastic action tokenization for autoregressive robot policy learning, particularly when training data are limited.