Multi-View Latent Action Pretraining for Cross-Embodiment Robot Learning from Videos

Oliver Hausdörfer; Felix Minzenmay; Mani Majd; Maximilian P. Christof; Shiyuan Zhang; Angela P. Schoellig

cs.ROOct 20, 2026

Multi-View Latent Action Pretraining for Cross-Embodiment Robot Learning from Videos

Authors: Oliver Hausdörfer, Felix Minzenmay, Mani Majd, Maximilian P. Christof, Shiyuan Zhang, Angela P. Schoellig

Organizations: Learning Systems and Robotics Lab, Technical University of Munich; and Munich Institute of Robotics and Machine Intelligence · University of Toronto

Abstract

Action-free videos are an appealing substitute for costly teleoperated data. Latent action models (LAMs) turn videos into a supervision signal by inferring abstract actions between frames. Existing LAMs observe a single camera view, which resolves 6D robot actions only ambiguously missing out-of-plane movements and potential occlusions. We propose a LAM that ingests synchronized multi-camera views using cross-attention to resolve depth and occlusions before compressing them into latent actions. The latent actions are fed into a policy in which a latent expert and an action expert are coupled by sequential flow matching around a frozen VLM backbone. We further evaluate in strict cross-embodiment settings where task-specific demonstrations come from other embodiments. We show that dual-view LAM pretraining improves the decodability of ground-truth 6D actions by 17% compared to single-view, while scaling single-view data by two orders of magnitude is not sufficient to fully recover out-of-plane motions. The downstream policy improves success rate by +13.5% compared to +6.2% for single-view LAM. A single cross-embodiment is sufficient for skill transfer, and adding a second viewpoint outperforms adding up to four further embodiments. The results transfer to hardware where demonstrations come from a human hand improving success rate by +20% over action-only policies in 850 rollouts. We show that video and robot data are complementary, and that benefits of learning from video are task-dependent. Website: https://learnsyslab.github.io/multiview_latent_action_pretraining/

Figures & tables

Fig. 1: Method : We combine latent action pretraining from in-domain cross-embodiment videos with a downstream flow-matching policy. First, latent actions are inferred from a large cross-embodiment video dataset. Second, a sequential policy is learned on the latent actions and a small set of demonstrations on the target embodiment. The LAM uses a spatio-temporal transformer to merge multiple viewpoints before computing the latents. The spatial block attends to patches of other viewpoints, yielding latent actions grounded in 3D space. Latent action labels and real robot actions are jointly used to train a hierarchical downstream policy. The policy uses sequential flow matching experts and outputs actions applied to the robot.

Latent action model ( $344$ M params)
Hidden dim $d$	$1024$
Spatial / temporal / FDM layers	$8$ / $8$ / $8$ (FDM cross-attends $z$ )
Bottleneck $B$ , latent	$1024{\rightarrow}8$ strided conv; $1{\times}8$ cont.
Input res. / offset $H$	$256{\times}256$ / $5$ ( $0.25\text{\,}\mathrm{s}$ @ $20\text{\,}\mathrm{Hz}$ )
Reconstruction loss $\rho$	Huber, $\beta{=}0.1$
Policy ( $202$ M trainable / $552$ M total)

TABLE I: Model and training hyperparameters.

Method	MimicGen	LIBERO	LIB.+	Mean
Full action budget
Ours	65.3	75.0	85.1	75.1
Low action budget
Ours (202M params)	25.7	52.6	49.8	42.7
$\pi_{0.5}$ (4.1B params)	32.0	53.0	54.0	45.0
Low action budget (our policy) + LAM

TABLE II: Rollout Success Rate (%) Across Benchmarks

Fig. 2: Scaling with Data. Policy success rate (SR) when scaling (left) the number of action-annotated demonstrations per task on the deployed embodiment Franka Panda, (middle) the fraction of LIBERO-90 used as off-task play data for latent video pretraining increasing task diversity. Top row shows mean success rate over all 40 tasks; bottom row split by in-distribution and heldout tasks (see Sec. IV ). Successes on heldout tasks prove transfer from cross-embodiment video demonstrations. (Right) : The number of cross-embodiments contained in the video dataset (1 = iiwa, 2 = +kinova3, 3 = +sawyer, 4 = +ur5e) for single- and dual-view pretraining. Dual-view pretraining outperforms adding up to four cross-embodiments.

Fig. 3: Left: Latent Action Probing . An MLP probe reconstructs ground-truth robot actions from the latents. Adding the second view improves decodability in the directions that are only ambiguously observed by single-view. Right: Cross-embodiment Learning . t-SNE showing 2D correspondence of latent actions to embodiments. Separation by color indicates separate latent per embodiment before training, while mixed colors show a latent space shared between embodiments. Shared latents already occur for a single cross-embodiment present in pretraining.

Fig. 4: Real World Tasks : We evaluate the policies and skill-transfer on six tasks consisting of novel movements, novel background, novel objects, and novel placements. For each eval task, we collect 50 human demonstrations and evaluate under settings with 0–50 robot demonstrations per task. We additionally collect an off-task play dataset across 32 tasks, totaling 1000 human demonstrations and 600 robot demonstrations.

Fig. 5: Evaluation : Each datapoint represents 25 rollouts. Left: We observe successes without using any robot data for new tasks. On average, using human demonstrations without any robot demonstrations increases SR by

+20\%

, and with 5 robot demonstrations by

+19\%

. Tasks with novel placement locations ([place]) are easiest to solve, followed by novel objects ([obj]). Middle: Multi-view video pretraining scores

46\%

against

21\%

for single-view significantly increasing SR. Right: Scaling of SR with increasing robot demonstrations for one task: video and robot demonstrations are complementary.

Fig. 6: Human to Robot Transfer. Left : We apply the latent action computed on a human frame to a robot frame from an eval rollout not seen during training: both yield the same movement to the right. Right : Motion decoded from the predicted latents for a human and a robot video. Both motions decode into similar motions in robot end-effector space. The multi-view LAM shows significantly better reconstruction of the ground-truth robot motion in $y$ -direction. See more qualitative results in the supplementary video.

Panel of Figure 9 — Fig. 8: Failure modes . Models struggle with object grasping ( $44\%$ of failure cases). The push task requires precision to align the gripper with the milk.

Robot Learning from Video

See all 49 papers

Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervision
Sep 17, 2026Maxime Alvarez, Renzo Caballero, Tatsuya Matsushima +2Latent Action Learning Robotic Manipulation
Graduate School of Engineering, The University of Tokyo
LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models
Aug 6, 2026Jiarui Yang, Jiale Zhange, Jiawei Li +5Latent Action Learning World Models for Robotics
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL) +1
Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos
Jun 17, 2026Runze Xu, Yiluo Zhang, Jian Wang +2Latent Action Learning VLM Adaptation
Department of Electronic Engineering, Tsinghua University · Tianfu Jiangxi Laboratory

Sep 17, 2026J/K move · Enter open · S save

Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervision

Maxime Alvarez, Renzo Caballero, Tatsuya Matsushima, Yusuke Iwasawa+1

Graduate School of Engineering, The University of Tokyo

Latent Action Learning Robotic Manipulation Cross-Embodiment Transfer

Open paper AskPDF arXiv

As generalist robot policies gain vision and language from web-scale pretraining, demonstrations remain costly to collect and tied to the robot that recorded them. Latent action models (LAMs) address both by learning latent actions from action-free videos that can be shared across embodiments, however, in practice, LAMs are sensitive to background visual noise, and the same motion from two different robots may be encoded with different latents. One solution to the background visual noise is to add an auxiliary loss predicting the robot action from the latent action, further associating the latent action space to the embodiment specific robot action space. We study a different use of the same labels, through action-similarity supervision. The similarity between any two latent actions is trained to match the similarity of the two ground-truth robot action sequences. The ground-truth actions are never predicted by the LAM, so the latent action does not need to encode embodiment specifics. We evaluate cross-embodiment transfer on RoboTwin 2.0 in a controlled setup, two bimanual robots demonstrate disjoint task sets, a policy is trained on all the demonstrations, and each robot is evaluated closed-loop on the tasks only the other demonstrated. With the policy architecture and its hyperparameters, the dataset, and the evaluation protocol fixed, predicting latent actions instead of ground-truth actions more than doubles cross-embodiment success. Given the same ground-truth actions, similarity supervision transfers better than an auxiliary loss that predicts the ground-truth action during the LAM training. Computing the similarities on end-effector motion rather than joint-space motion, and letting the loss compare latent actions across the two robots, gives the best approach of the study.

Abstract

Robot Learning from Video

Explore similar work

Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervision

LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models

Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos