Organizations: Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · TeleAI, China Telecom, Shanghai, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China
We propose CAT, a trajectory-level continuous action representation framework for robotic manipulation. Existing visuomotor systems often entangle action representation with control frequency or rely on fixed temporal parameterizations. This leads to representational redundancy at high sampling rates and limits the modeling of critical motion. CAT instead encodes action trajectories within a fixed real-time interval into a set of continuous latent tokens. To ensure temporal consistency across varying control frequencies, we further incorporate a frequency-aware positional encoding that establishs a shared temporal coordinate system. Trajectory-level regularization further stabilizes the latent representation. This approach prevents representation growth with timestep density and avoids reliance on predefined temporal parameterizations. Extensive system-level evaluations on LIBERO, MimicGen, and real-world long-horizon manipulation tasks demonstrate that CAT-based policies consistently outperform both competitive VQ-based and continuous visuomotor baselines under matched training settings. Across various model backbones and control frequencies, CAT consistently improves success rates. These results highlight the advantages of trajectory-level continuous action modeling for scalable robotic manipulation across varying control rates.
Figures & tables
Figure 1: Overview of CAT. CAT performs trajectory-level continuous action modeling over action sequences defined within a fixed real-time window. A frequency-aware transformer encoder with F-RoPE injection maps each action sequence to a fixed set of continuous latent variables. An asymmetric causal decoder reconstructs the action sequence in a frequency-conditioned manner, enabling temporally consistent modeling across varying control rates.
Figure 2
Method
Spatial
Object
Goal
Long
Avg.
Autoregressive-based VLA
Octo [ 25 ]
78.9
85.7
84.6
51.1
75.1
OpenVLA [ 11 ]
84.7
88.4
79.2
53.7
75.9
SmolVLA-FAST
87.0
93.0
90.0
68.0
84.5
π0 -FAST [ 23 ]
96.4
96.8
88.6
60.2
85.0
SmolVLA-VQ-VLA
90.0
94.0
86.0
71.0
85.3
Table 1: Comparison on LIBERO. Methods are grouped according to their action generation paradigm (autoregressive vs. diffusion/flow-based). Avg. denotes average success rate (%).
Policy
Coffee
Hammer
Mug
Nut
Square
Stack
Stack 3
Threading
Avg.
TCD [ 14 ]
77
92
53
44
63
95
62
56
67.3
SDP [ 27 ]
82
100
62
54
82
96
80
70
78.3
CARP [ 8 ]
86
98
74
78
90
100
82
70
84.8
DP [ 5 ]
89
99
67
84
90
99
76
82
85.8
DP-CAT (Ours)
91
99
68
81
91
99
85
81
86.9
Table 2: Comparison of success rates (%) on 8 manipulation tasks in MimicGen [ 20 ] . Bold numbers indicate the best performance.
Task
10Hz
16.7Hz
25Hz
50Hz
Avg.(Task)
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Adjust Bottle
76
68
77
78
69
72
78
86
75.0
76.0
Click Alarmclock
54
47
52
54
51
58
43
50
50.0
52.3
Grab Roller
17
47
46
66
49
58
56
71
42.0
60.5
Lift Pot
13
25
26
28
28
29
20
27
21.8
27.3
Place Burger Fries
38
37
24
26
29
29
26
31
29.3
30.8
Table 3: Success rates (%) on 8 RoboTwin 2.0 manipulation tasks evaluated under different control frequencies. Results are averaged over 100 rollouts per task. Higher values between SmolVLA (Baseline) and SmolVLA-CAT (Ours) are highlighted in bold.
Figure 3: Examples of initial states for 4 real-world tasks. Each panel shows the third-person camera, the robot arm with the wrist camera mounted, and the manipulated objects. Tasks: (a) Stack-5, (b) Drawer, (c) Flower, (d) Rope.
Models
Stack-5
Drawer
Rope
Flower
Avg.
Baseline
29
49
62
40
45.0
Ours
43
61
70
68
60.5
Table 4: Normalized task score (%) on 4 real-world long-horizon tasks. Baseline denotes SmolVLA, and Ours denotes SmolVLA integrated with CAT (SmolVLA-CAT). Bold numbers indicate the best performance.
Figure 4: Qualitative results on real-world tasks. Representative examples include tasks from Stack-5, Drawer, Rope, and Flower.
Figure 5: PE comparison. F-RoPE vs learned PE.
Table 5: Ablation studies on LIBERO. S: Spatial, O: Object, G: Goal, L: Long-horizon. Performance of CAT under different design choices.
Figure A : Real-world setup. Objects used in our real-world experiments.
Figure B : Qualitative results on LIBERO tasks. Representative examples include tasks from Spatial, Object, Goal and Long.
Figure C : Qualitative results on MimicGen tasks. Representative examples include tasks from Coffee, Hammer, Mug, Square, Threading, Stack, Stack3.
Figure D : Qualitative results on RoboTwin 2.0 tasks at a test-time control frequency of 16.7Hz. Representative examples include tasks from Adjust Bottle, Click Alarmclock, Grab Roller, Lift Pot, Place Burger Fries, Place Container Plate, Press Stapler, and Rotate QRcode.
Trajectory representation learning develops compact embeddings of ordered paths for comparing, predicting, retrieving, and acting from movement histories.