UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Manish Kumar Govind; Dominick Reilly; Pu Wang; Srijan Das

cs.ROFeb 23, 2026

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Authors: Manish Kumar Govind, Dominick Reilly, Pu Wang, Srijan Das

Organizations: All authors are with the Department of Computer Science, University of North Carolina at Charlotte, NC, USA.

Abstract

Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of UniLACT’s three stages: (1) UniLARN learns modality-specific (RGB/depth) and unified discrete latent actions from pairs of RGB-D frames within a shared latent space. (2) UniLACT is pretrained with cross-modal autoregressive latent-token prediction to capture complementary priors from RGB appearance and depth geometry. (3) UniLACT is fine-tuned on action-labeled trajectories to map predicted latent actions to executable robot actions.

CALVIN(ABC) Pretraining
Model	Observation input	Tasks Completed in a row					Avg. Len $\uparrow$
Model	Observation input	T1	T2	T3	T4	T5	Avg. Len $\uparrow$
Diffusion Policy [ 44 ]	Static RGB	0.402	0.123	0.026	0.008	0.00	0.56
HULC [ 45 ]	Static RGB	0.418	0.165	0.057	0.019	0.011	0.67
RT-1 [ 46 ]	Static RGB	0.533	0.222	0.094	0.038	0.013	0.90
Robo-Flamingo [ 47 ]	(Static + Gripper) RGB	0.824	0.619	0.466	0.331	0.235	2.48

TABLE I: CALVIN (

ABC\rightarrow D

) Benchmark results. Performance metric is the Average sequence length. Here ‘*’ represents the results are reproduced with our training settings and ‘Static RGB’,‘ Gripper RGB’ represents the static camera and gripper camera view respectively. ‘Proprio.’ denotes the robot proprioceptive state.

Method	Put carrot in bowl	Move near	Put eggplant in plate	Lift block	Avg.
	Seen Tasks		Unseen Tasks
Moto [ 10 ]	70.0	40.0	70.0	30.0	52.5
UniLACT	90.0 (+20)	60.0 (+20)	60.0 (-10)	40.0 (+10)	62.5 (+10)

TABLE II: Real-world results. Performance comparison of baseline Moto and UniLACT on four real-world manipulation tasks, reported under seen and unseen task settings in terms of success rate(%).

Fig. 4: Illustration of Task T1: “Pick up the carrot and place it in the bowl.” Top row: Moto fails to place the carrot inside the bowl and pushes the bowl out of the workspace. Bottom row: UniLACT successfully completes the task.

Fig. 5: Illustration of Task T2: “Move the eggplant near the banana.” Top row: Moto approaches the eggplant but fails to grasp it and collides with the workspace. Bottom row: UniLACT successfully grasps the eggplant and moves it near the banana.

Table 7

Table 8

Explore similar work

LARA: Latent Action Representation Alignment for Vision-Language-Action Models
Jun 5, 2026Mengya Liu, Baoxiong Jia, Jiangyong Huang +2Latent Action Learning Robotic Manipulation
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Oct 12, 2026Lei Yao, Qiao Feng, Mutian Tong +3Latent Action Learning Robotic Manipulation
University of Pennsylvania · The Hong Kong Polytechnic University
RotVLA: Rotational Latent Action for Vision-Language-Action Model
May 13, 2026Qiwei Li, Xicheng Gong, Xinghang Li +5Latent Action Learning Vision-Language-Action Models
Wangxuan Institute of Computer Technology, Peking University · Xiaomi Robotics · CASIA

Jun 5, 2026J/K move · Enter open · S save

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

Mengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang+1

State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence

Latent Action Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.

Abstract

Explore similar work

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation

RotVLA: Rotational Latent Action for Vision-Language-Action Model