UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation

Lei Yao; Qiao Feng; Mutian Tong; Qihao Yun; Lap-Pui Chau; Lingjie Liu

cs.ROOct 12, 2026

UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation

Authors: Lei Yao, Qiao Feng, Mutian Tong, Qihao Yun, Lap-Pui Chau, Lingjie Liu

Organizations: University of Pennsylvania · The Hong Kong Polytechnic University

Abstract

Latent Action Models (LAMs) can capture manipulation priors from action-free videos, but discrete latents usually impose a granularity bottleneck, limiting their ability to represent fine-grained action. Recent continuous alternatives mitigate this, but they are typically used as proxies for world modeling and require a further planning step to produce executable actions. We thus introduce UniLAP, a unified continuous latent-action policy that models the latent space via flow matching and bypasses reliance on future frames during inference. By distilling a LAM teacher with a flow-adaptive objective, UniLAP inherits the teacher's action representation while retaining the multimodal distribution of feasible behaviors. To decouple latent representations from task-irrelevant appearance, we extract a label-free motion target to isolate relevant kinematic cues from spurious visual dynamics. With the established embodiment-agnostic latent manifold, UniLAP can be easily adapted to novel platforms. Pretrained on multi-source videos at a compact 1.1B parameter scale, UniLAP outperforms state-of-the-art latent-action and Vision-Language-Action (VLA) baselines across hand pose forecasting, single-arm and humanoid manipulation, and real-world deployment.

Figures & tables

Figure 1: UniLAP learns a continuous unified latent action policy (LAP) from action-free video. Left , prior pipelines quantize the latent into discrete codebooks. Middle , UniLAP models it as a continuous flow distilled from a frozen LAM teacher. Right , performance comparison.

Figure 2: Overview of UniLAP. Stage I pretrains a continuous flow-matching latent action policy (LAP) on action-free videos under $\mathcal{L}_{\text{FM}}$ . A frozen LAM teacher’s posterior is distilled via the flow-adaptive $\mathcal{L}_{\text{distill}}$ , and a cross-frame correspondence target supervises the motion predictor to regularize the latent. Stage II attaches a per-embodiment decoder $\phi_{\text{dec}}$ , trains it from scratch, and finetunes the latent policy on the target embodiment’s labeled actions.

Figure 3: Qualitative hand pose forecasting on EgoDex. Left overlays the predicted $1.6$ s keypoint trajectories on the egocentric frame. Right plots each keypoint’s $X$ / $Y$ / $Z$ position over the horizon.

Table 4

Models	Params	Put Spoon on Towel	Put Carrot on Plate	Stack Green Block on Yellow Block	Put Eggplant in Yellow Basket	Avg
RT-1-X	35M	0.0	4.2	0.0	0.0	1.1
DINOv2 + DP	86M	0.0	8.0	0.0	26.0	8.5
Octo-Base	93M	15.8	12.5	0.0	41.7	17.5
OpenVLA	7B	8.3	4.2	0.0	45.8	14.6
$\pi_{0}$	3B	29.1	0.0	16.6	62.5	27.1
GR00T-N1	2B	62.5	45.8	16.7	20.8	36.5

Table 3: Per-task success rate (%) on SimplerEnv ( Li et al., 2024 ) . Top : WidowX. Bottom : Google Robot. Baselines are RT-1-X ( O’Neill et al., 2023 ) , DINOv2 + DP ( Dong et al., 2026 ) , Octo-Base ( Team et al., 2024 ) , OpenVLA ( Kim et al., 2024 ) ,

\pi_{0}

( Black et al., 2024 ) , GR00T-N1 and N1.5 ( Bjorck et al., 2025 ) , RoboVLMs ( Liu et al., 2025 ) , villa-X ( Chen et al., 2025b ) , UniVLA ( Bu et al., 2025a ) , mimic-video ( Pai et al., 2025 ) , and DeFI ( Zhang et al., 2026 ) .

Figure 4: Real-world setup and results. Left , the single-camera xArm7 tabletop workspace. Right , per-task and average success rate over $50$ rollouts. See the supplementary video for details.

Figure 5: Latent multimodality.

Figure 7: Language sensitivity ( left ), motion supervision, and pretraining ( middle & right ) ablations.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t$ , $H$ , $h$	time step, chunk horizon and chunk index
$N$ , $n$	number of patch tokens per frame and patch index
$o_{t}$ , $\ell$	current observation frame and language instruction
$\tau$	flow timestep
$z_{h}\in\mathbb{R}^{d_{z}}$	latent action of the transition $(o_{t+h},o_{t+h+1})$ , sampled from the LAM teacher
$\mu_{h}$ , $\Sigma_{h}$	teacher posterior mean and diagonal covariance

Appendix

Table 5: Notation used in Sec. 3 .

Component / Hyperparameter	Value
Architecture
Latent policy: blocks / width / heads	$16$ / $1024$ / $16$
Action decoder: blocks / width / heads	$8$ / $768$ / $12$
MLP ratio	$4$
Action decoder dropout	$0.1$
Latent dimension $d_{z}$	$32$

Appendix

Table 6: Architecture and optimization settings of UniLAP.

Pretraining corpus	Domain	Episodes	Frames	Rate (Hz)
EgoDex	Human, egocentric	$338{,}234$	$90.0$ M	$30{\to}10$
BridgeData V2	WidowX, real	$53{,}192$	$1.9$ M	$5$
Fractal	Google robot, real	$87{,}212$	$3.8$ M	$3$
RoboCasa-GR1	GR1, simulation	$24{,}000$	$6.0$ M	$20$
DreamDojo-GR1	GR1, real	$22{,}209$	$6.4$ M	$20$
RoboCasa365	RoboCasa, simulation	$32{,}043$	$29.1$ M	$20$

Appendix

Table 7: Stage I pretraining corpus. Frames are native-rate totals, though we subsample only EgoDex to

10

Hz.

Benchmark	Embodiment	Action space ( $d_{a}$ )	$d_{p}$	Tasks	Metric	LAP	Steps
EgoDex	Egocentric human	$48$ -D hand delta	$48$	–	ADE / FDE ( $m$ )	frozen	$40$ k
SimplerEnv-WidowX	WidowX arm	$7$ -D EE (delta)	$8$	$4$	Success rate (%)	finetuned	$100$ k
SimplerEnv-Google	Google robot arm	$7$ -D EE (delta)	$8$	$3$	Success rate (%)	finetuned	$30$ k
RoboCasa-GR1	GR1 humanoid	$29$ -D joints	$58$	$6$	Success rate (%)	finetuned	$100$ k
Real world (ours)	xArm7 arm	$8$ -D joints	$8$	$3$	Success rate (%)	finetuned	$20$ k

Appendix

Table 8: Per-embodiment overview. The action space gives the action dimension

d_{a}

and its form,

d_{p}

is the proprioception dimension, EE denotes end-effector, LAP marks whether the latent policy is frozen or finetuned, and Steps is the Stage II adaptation budget.

Figure 8: Real-robot platform: the $7$ -DoF xArm7 with a parallel-jaw gripper, the tripod-mounted ZED 2i that supplies the fixed third-person view, and the Meta Quest controllers used for VR teleoperation during data collection.

Figure 9: Teleoperated demonstration of T1: Pineapple $\rightarrow$ Bowl in the robot’s observation view, read left to right, top to bottom. Zoom in for better viewing.

Figure 10: Teleoperated demonstration of T2: Stack Cubes in the robot’s observation view, read left to right, top to bottom. Zoom in for better viewing.

Figure 11: Teleoperated demonstration of T3: Stack Cups in the robot’s observation view, read left to right, top to bottom. Zoom in for better viewing.

Figure 12: UniLAP rollout on T1: Pineapple $\rightarrow$ Bowl , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Figure 13: UniLAP rollout on T2: Stack Cubes , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Figure 14: UniLAP rollout on T3: Stack Cups , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Figure 15: UniLAP rollout on the unseen compositional instruction pick up the red cup and put it into the bowl , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Method	Pineapple $\rightarrow$ Bowl	Stack Cubes	Stack Cups	Average
$\pi_{0.5}$	$42$	12	$14$	$22.7$
UniLAP(Ours)	46	$10$	28	28.0

Appendix

Table 9: Per-task real-robot success rate (%) over

50

rollouts, comparing against

\pi_{0.5}

( Intelligence et al., 2025 ) .

Latent source	Action $R^{2}$ $\uparrow$
	EgoDex	Bridge	Fractal
Random Proj. $f_{t}$	$0.123_{\pm 0.002}$	$0.024_{\pm 0.008}$	$0.104_{\pm 0.007}$
Teacher $\mu$ (oracle)	$0.282_{\pm 0.002}$	$0.203_{\pm 0.002}$	$0.186_{\pm 0.007}$
UniLAP(Ours)	$0.176_{\pm 0.003}$	$0.115_{\pm 0.006}$	$0.194_{\pm 0.009}$
Recover	$62.4_{\pm 0.5}\%$	$56.5_{\pm 3.3}\%$	$104.1_{\pm 3.0}\%$

Appendix

Table 10: Aggregate linear probing of

\hat{Z}

for action prediction on EgoDex, BridgeData V2 (WidowX), and Fractal (Google Robot). Recover gives the policy latent’s

R^{2}

as a fraction of the oracle’s.

Figure 16: Four finger keypoints (index, middle, ring, little) of the acting hand for the assemble disassemble furniture case of Fig. 3 , which plots only the wrist and thumb. Each panel shows one keypoint’s $X$ / $Y$ / $Z$ over the $1.6$ s horizon, with ground truth dashed and prediction solid.

Figure 17: Full per-keypoint forecast on a stack task, across all six keypoints of the acting hand. Panels follow Fig. 16 : each shows one keypoint’s $X$ / $Y$ / $Z$ over the $1.6$ s horizon, with ground truth dashed and prediction solid.

Figure 18: Full per-keypoint forecast on a vertical pick-and-place scene, across all six keypoints of the acting hand. Panels follow Fig. 16 : each shows one keypoint’s $X$ / $Y$ / $Z$ over the $1.6$ s horizon, with ground truth dashed and prediction solid.

Figure 19: Additional latent reconstructions on EgoDex, four examples. Columns are $o_{t}$ , the GT $o_{t+h+1}$ , and reconstructions decoded from the teacher, UniLAP, Uniform $\tau$ , and FM-only latents, with PSNR against the GT. Each reconstruction is a single-step extrapolation from $o_{t+h}$ , which is not shown; $o_{t}$ is given for reference.

Figure 20: Additional latent reconstructions on WidowX (BridgeData V2). Columns follow Fig. 19 .

Figure 21: Additional latent reconstructions on Google Robot (Fractal). Columns follow Fig. 19 .

Explore similar work

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models
Feb 23, 2026Manish Kumar Govind, Dominick Reilly, Pu Wang +1Vision-Language-Action Models Latent Action Learning
All authors are with the Department of Computer Science, University of North Carolina at Charlotte, NC, USA.
Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling
Sep 14, 2026Dingjie Fu, Dianxing Shi, Yangyang Xu +1Latent Action Learning Latent Dynamics Modeling
Harbin Institute of Technology (Shenzhen) · Beihang University
Learning Additively Compositional Latent Actions for Embodied AI
Apr 3, 2026Hangxing Wei, Xiaoyu Chen, Chuheng Zhang +5Latent Action Learning Embodied AI
Wuhan University · Tsinghua University · Microsoft Research

Feb 23, 2026J/K move · Enter open · S save

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Manish Kumar Govind, Dominick Reilly, Pu Wang, Srijan Das

All authors are with the Department of Computer Science, University of North Carolina at Charlotte, NC, USA.

Vision-Language-Action Models Latent Action Learning Robotic Manipulation Visual Representation Learning

Open paper AskPDF arXiv

Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/

Abstract

Explore similar work

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

Learning Additively Compositional Latent Actions for Embodied AI