Autoregressive robot policies learn continuous control by predicting discrete action tokens from observations. Different tasks often share local motions, yet behavioral correspondence across demonstrations receives limited explicit supervision in existing tokenizers. Motions with different timing can therefore lack a shared representation despite following similar patterns. We propose Behavior-Aligned Action Tokenization (BAAT), which uses soft dynamic time warping (Soft-DTW) to select corresponding action chunks and aligns their quantized coordinates jointly with reconstruction. This objective encourages similar motions across tasks to occupy nearby quantized representations while retaining executable action detail. A history-conditioned diffusion decoder reconstructs continuous action chunks from these tokens, and a downstream autoregressive policy learns to predict them. We evaluate BAAT on selected tasks from three simulation benchmarks and two real robot tasks. BAAT achieves a mean simulation success rate of approximately 45.2%, exceeding OAT by approximately 7.2 percentage points. In the controlled LIBERO-All alignment ablation, policy success rises from 70.2% to 79.0% while trajectory replay success decreases. These results support behavioral correspondence as supervision for organizing shared motion structure in action tokenizers and improving downstream robot policy learning.
Figures & tables
Fig. 1: BAAT tokenizer training and downstream policy learning. Soft-DTW supplies behavioral correspondence between action chunks; alignment constrains their quantized coordinates jointly with reconstruction. Each latent slot contains four FSQ factors. The diffusion decoder conditions on executed action history, indicated by At−1 in the schematic.
Fig. 2: Simulation task overview. Top: example scenes from the LIBERO Spatial, Object, Goal, and long-horizon suites. Middle: RoboTwin 2.0 tasks, comprising Handover Mic, Lift Pot, Pick Dual Bottles, and Stack Blocks. Bottom: RoboCasa tasks, comprising Close Drawer, Press Coffee Button, Turn Off Microwave, and Turn Off Faucet.
Fig. 3: Cross-task behavioral neighborhoods. Left: LIBERO-10 queries; right: nearest neighbors from LIBERO-Goal, Object, and Spatial. Rows show lateral transfer, forward reach, and vertical lift; circles and triangles mark trajectory starts and ends. Plots show the retrieved chunks’ trajectories, without decoder generation or selection from trajectory plots.
Method
LIBERO-All
RoboCasa
RoboTwin 2.0
Avg.
FAST
63.5
4.3
5.3
24.4
OAT
76.5
23.3
14.2
38.0
Bin
45.2
18.7
11.8
25.2
DP
62.9
13.8
25.7
34.1
BAAT (ours)
79.0
29.2
27.3
45.2
TABLE I: Success rates (%) under joint training.
Method
LIBERO-10
Spatial
Goal
Object
Suite avg.
Joint
FAST
32.5
67.8
75.8
71.5
61.9
63.5
OAT
43.7
78.3
85.3
93.6
75.2
76.5
Bin
15.2
39.7
47.6
68.1
42.7
45.2
DP
66.0
73.5
79.3
75.8
73.7
62.9
BAAT (ours)
26.0
81.9
85.6
90.0
70.9
79.0
TABLE II: LIBERO success rates (%): per-suite versus joint training.
Fig. 4: Real robot evaluation setups. The two tasks use different bimanual systems; object sorting includes dexterous-hand control.
Tokenizer
Recon. MSE
Replay SR (%)
FAST
5.408×10−5
90.25
Bin
4.170×10−6
93.0
OAT
2.274×10−3
58.5
BAAT (ours)
5.773×10−4
71.5
TABLE III: Action reconstruction on LIBERO-All.
Policy interface
Toothbrush
Object sorting
FAST
35 (7/20)
20 (4/20)
OAT
65 (13/20)
20 (4/20)
Bin
20 (4/20)
15 (3/20)
DP
55 (11/20)
45 (9/20)
BAAT (ours)
70 (14/20)
45 (9/20)
TABLE IV: Real robot success: percentage (successful trials/20).
Fig. 5: LIBERO-All policy SR during one training run per method. The endpoints represent individual runs and need not match the three-seed means in Table I . “LIBERO All-4” denotes the four-suite setting; “Semantic FSQ” denotes BAAT and “DP (EMA)” denotes DP.
Metric
Without history
With history
Boundary jump p50
2.505
1.367
Boundary jump p95
6.359
3.203
Boundary jump p99
8.767
4.496
Arm reconstruction MAE
0.4941
0.2774
TABLE V: Offline reconstruction with and without decoder history on real-robot data (raw joint-target units; lower is better).
λalign
Recon. MSE
Replay SR
Policy SR
0
4.672×10−4
80.5
70.2
0.05
5.101×10−4
76.3
73.0
0.1 (ours)
5.773×10−4
71.5
79.0
0.3
9.186×10−4
65.5
57.9
0.5
1.157×10−2
0.0
0.0
TABLE VI: Alignment-weight ablation on LIBERO-All (SR in %).
Pair construction
Recon. MSE
Replay SR
Policy SR
No alignment
4.672×10−4
80.5
70.2
Pointwise L2
1.051×10−3
64.6
73.0
Soft-DTW (ours)
5.773×10−4
71.5
79.0
TABLE VII: Trajectory matching on LIBERO-All (SR in %).
Decoder
Recon. MSE
Replay SR
Policy SR
Transformer
2.413×10−4
80.6
69.4
Diffusion (ours)
5.773×10−4
71.5
79.0
Flow matching
2.444×10−4
82.4
74.7
TABLE VIII: Decoder comparison on LIBERO-All (SR in %).
South China University of Technology (scut.edu.cn) · The Chinese University of Hong Kong (cuhk.edu.hk) · Hong Kong Polytechnic University (polyu.edu.hk)