Organizations: Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · TeleAI, China Telecom, Shanghai, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China
We propose CAT, a trajectory-level continuous action representation framework for robotic manipulation. Existing visuomotor systems often entangle action representation with control frequency or rely on fixed temporal parameterizations. This leads to representational redundancy at high sampling rates and limits the modeling of critical motion. CAT instead encodes action trajectories within a fixed real-time interval into a set of continuous latent tokens. To ensure temporal consistency across varying control frequencies, we further incorporate a frequency-aware positional encoding that establishs a shared temporal coordinate system. Trajectory-level regularization further stabilizes the latent representation. This approach prevents representation growth with timestep density and avoids reliance on predefined temporal parameterizations. Extensive system-level evaluations on LIBERO, MimicGen, and real-world long-horizon manipulation tasks demonstrate that CAT-based policies consistently outperform both competitive VQ-based and continuous visuomotor baselines under matched training settings. Across various model backbones and control frequencies, CAT consistently improves success rates. These results highlight the advantages of trajectory-level continuous action modeling for scalable robotic manipulation across varying control rates.
Figures & tables
Figure 1: Overview of CAT. CAT performs trajectory-level continuous action modeling over action sequences defined within a fixed real-time window. A frequency-aware transformer encoder with F-RoPE injection maps each action sequence to a fixed set of continuous latent variables. An asymmetric causal decoder reconstructs the action sequence in a frequency-conditioned manner, enabling temporally consistent modeling across varying control rates.
Figure 2
Method
Spatial
Object
Goal
Long
Avg.
Autoregressive-based VLA
Octo [ 25 ]
78.9
85.7
84.6
51.1
75.1
OpenVLA [ 11 ]
84.7
88.4
79.2
53.7
75.9
SmolVLA-FAST
87.0
93.0
90.0
68.0
84.5
π0 -FAST [ 23 ]
96.4
96.8
88.6
60.2
85.0
SmolVLA-VQ-VLA
90.0
94.0
86.0
71.0
85.3
Table 1: Comparison on LIBERO. Methods are grouped according to their action generation paradigm (autoregressive vs. diffusion/flow-based). Avg. denotes average success rate (%).
Policy
Coffee
Hammer
Mug
Nut
Square
Stack
Stack 3
Threading
Avg.
TCD [ 14 ]
77
92
53
44
63
95
62
56
67.3
SDP [ 27 ]
82
100
62
54
82
96
80
70
78.3
CARP [ 8 ]
86
98
74
78
90
100
82
70
84.8
DP [ 5 ]
89
99
67
84
90
99
76
82
85.8
DP-CAT (Ours)
91
99
68
81
91
99
85
81
86.9
Table 2: Comparison of success rates (%) on 8 manipulation tasks in MimicGen [ 20 ] . Bold numbers indicate the best performance.
Task
10Hz
16.7Hz
25Hz
50Hz
Avg.(Task)
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Adjust Bottle
76
68
77
78
69
72
78
86
75.0
76.0
Click Alarmclock
54
47
52
54
51
58
43
50
50.0
52.3
Grab Roller
17
47
46
66
49
58
56
71
42.0
60.5
Lift Pot
13
25
26
28
28
29
20
27
21.8
27.3
Place Burger Fries
38
37
24
26
29
29
26
31
29.3
30.8
Table 3: Success rates (%) on 8 RoboTwin 2.0 manipulation tasks evaluated under different control frequencies. Results are averaged over 100 rollouts per task. Higher values between SmolVLA (Baseline) and SmolVLA-CAT (Ours) are highlighted in bold.
Figure 3: Examples of initial states for 4 real-world tasks. Each panel shows the third-person camera, the robot arm with the wrist camera mounted, and the manipulated objects. Tasks: (a) Stack-5, (b) Drawer, (c) Flower, (d) Rope.
Models
Stack-5
Drawer
Rope
Flower
Avg.
Baseline
29
49
62
40
45.0
Ours
43
61
70
68
60.5
Table 4: Normalized task score (%) on 4 real-world long-horizon tasks. Baseline denotes SmolVLA, and Ours denotes SmolVLA integrated with CAT (SmolVLA-CAT). Bold numbers indicate the best performance.
Figure 4: Qualitative results on real-world tasks. Representative examples include tasks from Stack-5, Drawer, Rope, and Flower.
Figure 5: PE comparison. F-RoPE vs learned PE.
Table 5: Ablation studies on LIBERO. S: Spatial, O: Object, G: Goal, L: Long-horizon. Performance of CAT under different design choices.
Figure A : Real-world setup. Objects used in our real-world experiments.
Figure B : Qualitative results on LIBERO tasks. Representative examples include tasks from Spatial, Object, Goal and Long.
Figure C : Qualitative results on MimicGen tasks. Representative examples include tasks from Coffee, Hammer, Mug, Square, Threading, Stack, Stack3.
Figure D : Qualitative results on RoboTwin 2.0 tasks at a test-time control frequency of 16.7Hz. Representative examples include tasks from Adjust Bottle, Click Alarmclock, Grab Roller, Lift Pot, Place Burger Fries, Place Container Plate, Press Stapler, and Rotate QRcode.
Trajectory representation learning develops compact embeddings of ordered paths for comparing, predicting, retrieving, and acting from movement histories.
Most vision-language-action models represent future motion as fixed-rate action chunks, tying temporal resolution and prediction horizon to a fixed output budget. This pointwise representation wastes capacity on highly correlated neighboring actions, leaves temporal continuity and smoothness to be learned implicitly, and forces a tradeoff between long-horizon coverage and the local precision required for contact-rich manipulation. To address these limitations, we introduce Vela, a vision-language-action foundation model that represents future robot behavior as continuous trajectories. Vela combines a compact spline-based action representation with motion-dependent temporal support and a shared action interface for heterogeneous embodiments, allowing a fixed output budget to adapt its temporal resolution across motions. We pretrain Vela on large-scale multi-embodiment robot data and evaluate it on LIBERO-X, EBench, and two real-world long-horizon tasks, egg-cake cooking and potato shredding, obtaining promising results across simulation and physical manipulation. These results highlight the potential of continuous action representations as a foundation for future embodied foundation models. Project page and more results: https://clementine24.github.io/Vela/ .
Yifan Li, Jiaxu Wang, Dongming Wu +4
Fudan University · Shanghai Innovation Institute · The Chinese University of Hong Kong +1
We present AEM, an Action-Effect Memory pretraining framework for robot manipulation that learns compact temporal representations from vision-action history. Unlike prior robot representation pretraining methods that mainly focus on single-frame visual encoding, AEM targets the temporal nature of manipulation, where the current observation alone is often insufficient under partial observability. AEM models manipulation as an action-driven interaction process by interleaving visual and action features and applying masked modeling to recover missing content from incomplete histories, thereby learning action-conditioned state evolution. The Mamba-encoded output of the final vision token is used as a compact history representation, serving as the global context for decoding and downstream control. This design preserves a single-vector temporal bottleneck while keeping inference efficient. We evaluate AEM with Diffusion Policy and Flow Policy. AEM consistently improves manipulation performance in both simulation and real-world settings, outperforming baselines across clean scenes, cluttered and random scenes, and non-Markovian tasks. Ablation studies further show that history-aware pretraining surpasses single-frame pretraining and direct frame stacking, while reducing inference latency and computational cost.
Yijing Zhou, Qiwei Liang, Sitong Zhuang +5
Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University
Vision-language-action models have shown strong promise for robot manipulation, yet raw language is primarily needed to specify task intent rather than to be repeatedly processed during high-frequency low-level execution. Motivated by this separation, we propose a cerebello-thalamic-inspired vision-action model (CT-VAM) for efficient task-conditioned visuomotor control. CT-VAM acts as a compact local execution policy that predicts action chunks from dualview visual observations, proprioception, and a lightweight task condition, potentially enabling a practical cloud-edge paradigm in which high-level semantic reasoning can be handled by large models while fast closed-loop control runs on local hardware. To fuse heterogeneous inputs effectively, CT-VAM introduces TARS (Thalamic Action Routing Stream), a stream-separated conditional attention decoder that independently routes action, visual and task streams, preventing dense sensory tokens from overwhelming compact task-relevant conditions. With only 68M parameters, CT-VAM achieves LIBERO success rates competitive with substantially larger VLA models, while reducing inference latency. Together with flow-consistent inpainting for asynchronous chunk execution, CT-VAM supports high-frequency control and demonstrates robust realworld deployment on resource-constrained robotic platforms.
Jiacheng Li, Yize Guo, Jiabin Guo +2
AIRLab, Department of Automation University of Science and Technology of China Hefei, China