UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation

Lei Yao; Qiao Feng; Mutian Tong; Qihao Yun; Lap-Pui Chau; Lingjie Liu

cs.ROOct 12, 2026

UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation

Authors: Lei Yao, Qiao Feng, Mutian Tong, Qihao Yun, Lap-Pui Chau, Lingjie Liu

Organizations: University of Pennsylvania · The Hong Kong Polytechnic University

Abstract

Latent Action Models (LAMs) can capture manipulation priors from action-free videos, but discrete latents usually impose a granularity bottleneck, limiting their ability to represent fine-grained action. Recent continuous alternatives mitigate this, but they are typically used as proxies for world modeling and require a further planning step to produce executable actions. We thus introduce UniLAP, a unified continuous latent-action policy that models the latent space via flow matching and bypasses reliance on future frames during inference. By distilling a LAM teacher with a flow-adaptive objective, UniLAP inherits the teacher's action representation while retaining the multimodal distribution of feasible behaviors. To decouple latent representations from task-irrelevant appearance, we extract a label-free motion target to isolate relevant kinematic cues from spurious visual dynamics. With the established embodiment-agnostic latent manifold, UniLAP can be easily adapted to novel platforms. Pretrained on multi-source videos at a compact 1.1B parameter scale, UniLAP outperforms state-of-the-art latent-action and Vision-Language-Action (VLA) baselines across hand pose forecasting, single-arm and humanoid manipulation, and real-world deployment.

Figures & tables

Figure 1: UniLAP learns a continuous unified latent action policy (LAP) from action-free video. Left , prior pipelines quantize the latent into discrete codebooks. Middle , UniLAP models it as a continuous flow distilled from a frozen LAM teacher. Right , performance comparison.

Figure 2: Overview of UniLAP. Stage I pretrains a continuous flow-matching latent action policy (LAP) on action-free videos under $\mathcal{L}_{\text{FM}}$ . A frozen LAM teacher’s posterior is distilled via the flow-adaptive $\mathcal{L}_{\text{distill}}$ , and a cross-frame correspondence target supervises the motion predictor to regularize the latent. Stage II attaches a per-embodiment decoder $\phi_{\text{dec}}$ , trains it from scratch, and finetunes the latent policy on the target embodiment’s labeled actions.

Figure 3: Qualitative hand pose forecasting on EgoDex. Left overlays the predicted $1.6$ s keypoint trajectories on the egocentric frame. Right plots each keypoint’s $X$ / $Y$ / $Z$ position over the horizon.

Table 4

Models	Params	Put Spoon on Towel	Put Carrot on Plate	Stack Green Block on Yellow Block	Put Eggplant in Yellow Basket	Avg
RT-1-X	35M	0.0	4.2	0.0	0.0	1.1
DINOv2 + DP	86M	0.0	8.0	0.0	26.0	8.5
Octo-Base	93M	15.8	12.5	0.0	41.7	17.5
OpenVLA	7B	8.3	4.2	0.0	45.8	14.6
$\pi_{0}$	3B	29.1	0.0	16.6	62.5	27.1
GR00T-N1	2B	62.5	45.8	16.7	20.8	36.5

Table 3: Per-task success rate (%) on SimplerEnv ( Li et al., 2024 ) . Top : WidowX. Bottom : Google Robot. Baselines are RT-1-X ( O’Neill et al., 2023 ) , DINOv2 + DP ( Dong et al., 2026 ) , Octo-Base ( Team et al., 2024 ) , OpenVLA ( Kim et al., 2024 ) ,

\pi_{0}

( Black et al., 2024 ) , GR00T-N1 and N1.5 ( Bjorck et al., 2025 ) , RoboVLMs ( Liu et al., 2025 ) , villa-X ( Chen et al., 2025b ) , UniVLA ( Bu et al., 2025a ) , mimic-video ( Pai et al., 2025 ) , and DeFI ( Zhang et al., 2026 ) .

Figure 4: Real-world setup and results. Left , the single-camera xArm7 tabletop workspace. Right , per-task and average success rate over $50$ rollouts. See the supplementary video for details.

Figure 5: Latent multimodality.

Figure 7: Language sensitivity ( left ), motion supervision, and pretraining ( middle & right ) ablations.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t$ , $H$ , $h$	time step, chunk horizon and chunk index
$N$ , $n$	number of patch tokens per frame and patch index
$o_{t}$ , $\ell$	current observation frame and language instruction
$\tau$	flow timestep
$z_{h}\in\mathbb{R}^{d_{z}}$	latent action of the transition $(o_{t+h},o_{t+h+1})$ , sampled from the LAM teacher
$\mu_{h}$ , $\Sigma_{h}$	teacher posterior mean and diagonal covariance

Appendix

Table 5: Notation used in Sec. 3 .

Component / Hyperparameter	Value
Architecture
Latent policy: blocks / width / heads	$16$ / $1024$ / $16$
Action decoder: blocks / width / heads	$8$ / $768$ / $12$
MLP ratio	$4$
Action decoder dropout	$0.1$
Latent dimension $d_{z}$	$32$

Appendix

Table 6: Architecture and optimization settings of UniLAP.

Pretraining corpus	Domain	Episodes	Frames	Rate (Hz)
EgoDex	Human, egocentric	$338{,}234$	$90.0$ M	$30{\to}10$
BridgeData V2	WidowX, real	$53{,}192$	$1.9$ M	$5$
Fractal	Google robot, real	$87{,}212$	$3.8$ M	$3$
RoboCasa-GR1	GR1, simulation	$24{,}000$	$6.0$ M	$20$
DreamDojo-GR1	GR1, real	$22{,}209$	$6.4$ M	$20$
RoboCasa365	RoboCasa, simulation	$32{,}043$	$29.1$ M	$20$

Appendix

Table 7: Stage I pretraining corpus. Frames are native-rate totals, though we subsample only EgoDex to

10

Hz.

Benchmark	Embodiment	Action space ( $d_{a}$ )	$d_{p}$	Tasks	Metric	LAP	Steps
EgoDex	Egocentric human	$48$ -D hand delta	$48$	–	ADE / FDE ( $m$ )	frozen	$40$ k
SimplerEnv-WidowX	WidowX arm	$7$ -D EE (delta)	$8$	$4$	Success rate (%)	finetuned	$100$ k
SimplerEnv-Google	Google robot arm	$7$ -D EE (delta)	$8$	$3$	Success rate (%)	finetuned	$30$ k
RoboCasa-GR1	GR1 humanoid	$29$ -D joints	$58$	$6$	Success rate (%)	finetuned	$100$ k
Real world (ours)	xArm7 arm	$8$ -D joints	$8$	$3$	Success rate (%)	finetuned	$20$ k

Appendix

Table 8: Per-embodiment overview. The action space gives the action dimension

d_{a}

and its form,

d_{p}

is the proprioception dimension, EE denotes end-effector, LAP marks whether the latent policy is frozen or finetuned, and Steps is the Stage II adaptation budget.

Figure 8: Real-robot platform: the $7$ -DoF xArm7 with a parallel-jaw gripper, the tripod-mounted ZED 2i that supplies the fixed third-person view, and the Meta Quest controllers used for VR teleoperation during data collection.

Figure 9: Teleoperated demonstration of T1: Pineapple $\rightarrow$ Bowl in the robot’s observation view, read left to right, top to bottom. Zoom in for better viewing.

Figure 10: Teleoperated demonstration of T2: Stack Cubes in the robot’s observation view, read left to right, top to bottom. Zoom in for better viewing.

Figure 11: Teleoperated demonstration of T3: Stack Cups in the robot’s observation view, read left to right, top to bottom. Zoom in for better viewing.

Figure 12: UniLAP rollout on T1: Pineapple $\rightarrow$ Bowl , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Figure 13: UniLAP rollout on T2: Stack Cubes , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Figure 14: UniLAP rollout on T3: Stack Cups , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Figure 15: UniLAP rollout on the unseen compositional instruction pick up the red cup and put it into the bowl , key frames read left to right, top to bottom. See the supplementary video for the full rollout.

Method	Pineapple $\rightarrow$ Bowl	Stack Cubes	Stack Cups	Average
$\pi_{0.5}$	$42$	12	$14$	$22.7$
UniLAP(Ours)	46	$10$	28	28.0

Appendix

Table 9: Per-task real-robot success rate (%) over

50

rollouts, comparing against

\pi_{0.5}

( Intelligence et al., 2025 ) .

Latent source	Action $R^{2}$ $\uparrow$
	EgoDex	Bridge	Fractal
Random Proj. $f_{t}$	$0.123_{\pm 0.002}$	$0.024_{\pm 0.008}$	$0.104_{\pm 0.007}$
Teacher $\mu$ (oracle)	$0.282_{\pm 0.002}$	$0.203_{\pm 0.002}$	$0.186_{\pm 0.007}$
UniLAP(Ours)	$0.176_{\pm 0.003}$	$0.115_{\pm 0.006}$	$0.194_{\pm 0.009}$
Recover	$62.4_{\pm 0.5}\%$	$56.5_{\pm 3.3}\%$	$104.1_{\pm 3.0}\%$

Appendix

Table 10: Aggregate linear probing of

\hat{Z}

for action prediction on EgoDex, BridgeData V2 (WidowX), and Fractal (Google Robot). Recover gives the policy latent’s

R^{2}

as a fraction of the oracle’s.

Figure 16: Four finger keypoints (index, middle, ring, little) of the acting hand for the assemble disassemble furniture case of Fig. 3 , which plots only the wrist and thumb. Each panel shows one keypoint’s $X$ / $Y$ / $Z$ over the $1.6$ s horizon, with ground truth dashed and prediction solid.

Figure 17: Full per-keypoint forecast on a stack task, across all six keypoints of the acting hand. Panels follow Fig. 16 : each shows one keypoint’s $X$ / $Y$ / $Z$ over the $1.6$ s horizon, with ground truth dashed and prediction solid.

Figure 18: Full per-keypoint forecast on a vertical pick-and-place scene, across all six keypoints of the acting hand. Panels follow Fig. 16 : each shows one keypoint’s $X$ / $Y$ / $Z$ over the $1.6$ s horizon, with ground truth dashed and prediction solid.

Figure 19: Additional latent reconstructions on EgoDex, four examples. Columns are $o_{t}$ , the GT $o_{t+h+1}$ , and reconstructions decoded from the teacher, UniLAP, Uniform $\tau$ , and FM-only latents, with PSNR against the GT. Each reconstruction is a single-step extrapolation from $o_{t+h}$ , which is not shown; $o_{t}$ is given for reference.

Figure 20: Additional latent reconstructions on WidowX (BridgeData V2). Columns follow Fig. 19 .

Figure 21: Additional latent reconstructions on Google Robot (Fractal). Columns follow Fig. 19 .

Explore similar work

Feb 23, 2026

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/

Manish Kumar Govind, Dominick Reilly, Pu Wang +1

All authors are with the Department of Computer Science, University of North Carolina at Charlotte, NC, USA.

Vision-Language-Action Models Latent Action Learning Robotic Manipulation Visual Representation Learning

Ask PDF

Sep 14, 2026

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing future states. However, we identify a fundamental

\textbf{reconstruction-action mismatch}

: lower reconstruction error does not necessarily yield better latent dynamics or downstream performance. We attribute this mismatch to two underconstrained aspects of reconstruction-based latent dynamics modeling: (i) the inverse dynamics model (IDM) is not explicitly encouraged to distinguish action-related transitions from nuisance appearance, and (ii) the forward dynamics model (FDM) can underutilize the inferred latent action by exploiting predictive shortcuts from the current state. To address both limitations, we propose

\textbf{ACT-LAM}

, a lightweight action-centric framework that strengthens both action extraction and action utilization. Specifically, its Action Query IDM (AQ-IDM) employs learnable action queries and gated aggregation to selectively extract rich action-related transition cues without strong information bottlenecks. And its Action Token FDM (AT-FDM) projects latent actions into action tokens that progressively interact with evolving state representations, enabling continuous state-aware action conditioning. ACT-LAM further streamlines feature processing to concentrate model capacity on latent dynamics modeling. Extensive experiments on several robotic datasets and the VP

^2

benchmark demonstrate stronger latent action consistency, forward dynamics, and downstream visual planning performance with fewer trainable parameters and lower computational overhead. In particular, ACT-LAM surpasses the previous state of the art by

\textbf{7.6%}

on the aggregated VP

^2

success rate. Codes at url.

Dingjie Fu, Dianxing Shi, Yangyang Xu +1

Harbin Institute of Technology (Shenzhen) · Beihang University

Latent Action Learning Latent Dynamics Modeling Action-Conditioned World Models Robotics

Ask PDF

Apr 3, 2026

Learning Additively Compositional Latent Actions for Embodied AI

Latent action learning infers pseudo-action labels from visual transitions, providing an approach to leverage internet-scale video for embodied AI. However, most methods learn latent actions without structural priors that encode the additive, compositional structure of physical motion. As a result, latents often entangle irrelevant scene details or information about future observations with true state changes and miscalibrate motion magnitude. We introduce Additively Compositional Latent Action Model (AC-LAM), which enforces scene-wise additive composition structure over short horizons on the latent action space. These AC constraints encourage simple algebraic structure in the latent action space~(identity, inverse, cycle consistency) and suppress information that does not compose additively. Empirically, AC-LAM learns more structured, motion-specific, and displacement-calibrated latent actions and provides stronger supervision for downstream policy learning, outperforming state-of-the-art LAMs across simulated and real-world tabletop tasks.

Hangxing Wei, Xiaoyu Chen, Chuheng Zhang +5

Wuhan University · Tsinghua University · Microsoft Research

Latent Action Learning Embodied AI Motion Representation Learning

Ask PDF