Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/
Figures & tables
Fig. 1: Overview of UniLACT’s three stages: (1) UniLARN learns modality-specific (RGB/depth) and unified discrete latent actions from pairs of RGB-D frames within a shared latent space. (2) UniLACT is pretrained with cross-modal autoregressive latent-token prediction to capture complementary priors from RGB appearance and depth geometry. (3) UniLACT is fine-tuned on action-labeled trajectories to map predicted latent actions to executable robot actions.
Model
Observation input
Tasks Completed in a row
Avg. Len ↑
T1
T2
T3
T4
T5
CALVIN(ABC) Pretraining
Diffusion Policy [ 44 ]
Static RGB
0.402
0.123
0.026
0.008
0.00
0.56
HULC [ 45 ]
Static RGB
0.418
0.165
0.057
0.019
0.011
0.67
RT-1 [ 46 ]
Static RGB
0.533
0.222
0.094
0.038
0.013
0.90
Robo-Flamingo [ 47 ]
(Static + Gripper) RGB
0.824
0.619
0.466
0.331
0.235
2.48
TABLE I: CALVIN ( ABC→D ) Benchmark results. Performance metric is the Average sequence length. Here ‘*’ represents the results are reproduced with our training settings and ‘Static RGB’,‘ Gripper RGB’ represents the static camera and gripper camera view respectively. ‘Proprio.’ denotes the robot proprioceptive state.
Figure 3
Seen Tasks
Unseen Tasks
Method
Put carrot in bowl
Move near
Put eggplant in plate
Lift block
Avg.
Moto [ 10 ]
70.0
40.0
70.0
30.0
52.5
UniLACT
90.0 (+20)
60.0 (+20)
60.0 (-10)
40.0 (+10)
62.5 (+10)
TABLE II: Real-world results. Performance comparison of baseline Moto and UniLACT on four real-world manipulation tasks, reported under seen and unseen task settings in terms of success rate(%).
Fig. 4: Illustration of Task T1: “Pick up the carrot and place it in the bowl.” Top row: Moto fails to place the carrot inside the bowl and pushes the bowl out of the workspace. Bottom row: UniLACT successfully completes the task.
Fig. 5: Illustration of Task T2: “Move the eggplant near the banana.” Top row: Moto approaches the eggplant but fails to grasp it and collides with the workspace. Bottom row: UniLACT successfully grasps the eggplant and moves it near the banana.
Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.
Mengya Liu, Baoxiong Jia, Jiangyong Huang +2
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
Latent Action Models (LAMs) can capture manipulation priors from action-free videos, but discrete latents usually impose a granularity bottleneck, limiting their ability to represent fine-grained action. Recent continuous alternatives mitigate this, but they are typically used as proxies for world modeling and require a further planning step to produce executable actions. We thus introduce UniLAP, a unified continuous latent-action policy that models the latent space via flow matching and bypasses reliance on future frames during inference. By distilling a LAM teacher with a flow-adaptive objective, UniLAP inherits the teacher's action representation while retaining the multimodal distribution of feasible behaviors. To decouple latent representations from task-irrelevant appearance, we extract a label-free motion target to isolate relevant kinematic cues from spurious visual dynamics. With the established embodiment-agnostic latent manifold, UniLAP can be easily adapted to novel platforms. Pretrained on multi-source videos at a compact 1.1B parameter scale, UniLAP outperforms state-of-the-art latent-action and Vision-Language-Action (VLA) baselines across hand pose forecasting, single-arm and humanoid manipulation, and real-world deployment.
Lei Yao, Qiao Feng, Mutian Tong +3
University of Pennsylvania · The Hong Kong Polytechnic University
Latent Action Models (LAMs) have emerged as an effective paradigm for handling heterogeneous datasets during Vision-Language-Action (VLA) model pretraining, offering a unified action space across embodiments. However, existing LAMs often rely on discrete quantization encode and decode pipelines, which can lead to trivial frame reconstruction behavior, limited representational capacity, and a lack of physically meaningful structure. We introduce RotVLA, a VLA framework built on a continuous rotational latent action representation. Latent actions are modeled as elements of SO(n), providing continuity, compositionality, and structured geometry aligned with real-world action dynamics. A triplet frame learning framework further enforces meaningful temporal dynamics while avoiding degeneration. RotVLA consists of a VLM backbone and a flow-matching action head, pretrained on large-scale cross-embodiment robotic datasets and human videos with latent-action supervision. For downstream robot control, the flow-matching head is extended into a unified action expert that jointly denoises latent and robot actions. Here, latent actions serve as a latent planner, providing high-level guidance that conditions action generation. With only 1.7B parameters and 1700+ hours of pretraining data, RotVLA achieves 98.2% on LIBERO and 89.6% / 88.5% on RoboTwin2.0 under clean and randomized settings, respectively. It also demonstrates strong real-world performance on manipulation tasks, consistently outperforming existing VLA models.
Qiwei Li, Xicheng Gong, Xinghang Li +5
Wangxuan Institute of Computer Technology, Peking University · Xiaomi Robotics · CASIA