cs.RONov 3, 2026

ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Authors: Cheng-Chun Hsu, Peter Stone

Organizations: Department of Computer Science, The University of Texas at Austin

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/

Figures & tables

Figure 1: Ela Overview. Our framework extends existing behavior cloning methods, e.g . diffusion policies, with latent actions jointly optimized for policy learning and action decoding. The approach enables training on mixtures of action-labeled and video demonstrations with little overhead, requiring only a lightweight action decoder.
Figure 2: Ela Training and Deployment. During training, image pairs are used with an inverse dynamics model (IDM) to infer latent actions, which supervise a forward dynamics model (FDM) for future frame reconstruction and an action decoder for robot action prediction. A latent action policy is then trained to output the latent actions. During deployment, the latent action policy maps image observations to latent actions, which are decoded into robot actions by the action decoder.
Figure 3: End-to-End Joint Training. Ela jointly trains the latent action model (LAM), latent action policy, and action decoder using two alternating forward passes. In the LAM-update pass , the policy is frozen and the alignment, action, and FDM losses are backpropagated to update the LAM. In the Policy-update pass , the LAM is frozen and the alignment and policy losses are used to update the policy network.
Figure 4: Few-shot Policy Learning. Comparative analysis of success rates on the LIBERO-10 benchmark. All evaluated methods were trained using NN action-labeled samples selected from 50 robot demonstrations per task. The two dashed lines at the top indicate the upper-bound performance of DP and UWM trained on the full set of 50 action-labeled demonstrations.
BaselineModelTraining Data
 Param.(# trajectories)
LAPA [ 32 ]206M1M+
Villa-X [ 4 ]124M1M+
UniVLA [ 2 ]34M †1M+
LAPA-DINO [ 20 ]102M †250
ElaDP\text{{Ela}}_{\text{DP}} (Ours)6M250
Table 1: IDM model sizes and training budgets for baselines. † DINO encoder parameters are excluded from the count.
MethodSuccess
Villa-XFS\text{Villa-X}_{\text{FS}}0.06
Villa-XFT\text{Villa-X}_{\text{FT}}0.21
Dynamo0.38
DP0.48
ElaDP\text{{Ela}}_{\text{DP}}0.70
Table 2: Real-world success under matched data access.
Figure 5: Real-world Results. (a) Comparative analysis of success rates on the real-world benchmark. (b) Ablation study of representation alignment across a subset of task suites. (c) Modelability versus Decodability for each latent action spaces.
MethodMSE
UniVLA0.043
LAPA0.042
Villa-X0.024
Villa-XFS\text{Villa-X}_{\text{FS}}0.059
Villa-XFT\text{Villa-X}_{\text{FT}}0.029
LAPA-DINO0.029
Table 3: Linear probing for action regression.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Num. of latent-action tokens kk(b) FSQ codebook
kkSuccessLevelsCodebook sizeSuccess
440.59[5,5,5,5][5,5,5,5]6256250.47
880.76[7,7,7,7][7,7,7,7]240124010.78
16†16^{\dagger}0.83[9,9,9,9]†[9,9,9,9]^{\dagger}656165610.83
32320.78[7,7,7,7,7][7,7,7,7,7]16807168070.57
Appendix
Table 4: Ablations on the latent action. We vary the number of latent-action tokens (k) (left) and the FSQ codebook configuration (right), one factor at a time while keeping all other settings fixed. The default configuration is marked with ( †\dagger ).
Figure 6: Robot Setup.
TaskUniVLALAPAVilla-XLAPA-DINODPElaDP\mathbf{\text{{Ela}}_{\text{DP}}}
place-food0.250.550.500.500.600.95
place-food-cabinet0.000.250.200.500.550.80
stack-bowl0.150.350.450.350.400.70
pour-bowl0.000.100.100.200.500.75
open-oven0.000.050.100.100.350.30
Average0.080.260.270.330.480.70
Appendix
Table 5: Success rates on the real-world benchmark (20 trials per task).
TaskDPElaDP\text{{Ela}}_{\text{DP}}UWMElaUWM\text{{Ela}}_{\text{UWM}}
N = 20
bowl-drawer0.80±0.040.80\pm{\scriptstyle 0.04}0.89±0.080.89\pm{\scriptstyle 0.08}0.91±0.060.91\pm{\scriptstyle 0.06}0.93±0.03\mathbf{0.93}\pm{\scriptstyle 0.03}
cheese-butter0.69±0.080.69\pm{\scriptstyle 0.08}0.77±0.10\mathbf{0.77}\pm{\scriptstyle 0.10}0.75±0.020.75\pm{\scriptstyle 0.02}0.74±0.000.74\pm{\scriptstyle 0.00}
moka-moka0.43±0.090.43\pm{\scriptstyle 0.09}0.26±0.070.26\pm{\scriptstyle 0.07}0.38±0.030.38\pm{\scriptstyle 0.03}0.49±0.03\mathbf{0.49}\pm{\scriptstyle 0.03}
mug-microwave0.58±0.060.58\pm{\scriptstyle 0.06}0.85±0.030.85\pm{\scriptstyle 0.03}0.66±0.000.66\pm{\scriptstyle 0.00}0.91±0.01\mathbf{0.91}\pm{\scriptstyle 0.01}
mug-mug0.36±0.040.36\pm{\scriptstyle 0.04}0.46±0.030.46\pm{\scriptstyle 0.03}0.55±0.060.55\pm{\scriptstyle 0.06}0.59±0.03\mathbf{0.59}\pm{\scriptstyle 0.03}
Appendix
Table 6: LIBERO-10 results across different numbers of action-labeled demonstrations (N) out of 50 demonstrations per task. Each entry reports mean ±\pm standard deviation over 3 runs. Best results per task are highlighted in bold.
 Phase 1Phase 2Phase 3
 LAM WarmupPretrainingPer-task Finetuning
Trained module(s)LAM ++ Action DecoderLAM ++ Action Decoder ++ PolicyPolicy
Frozen module(s)––LAM ++ Action Decoder
Steps5k5\text{k}200k200\text{k}10k10\text{k}
Batch size646464646464
OptimizerAdamWAdamWAdamW
Appendix
Table 7: Ela Training schedule
More in this topic

Latent Action Learning

Latent action learning discovers compact representations of behavior from observations or trajectories to support control, planning, and transfer.

See all 118 papers

Explore similar work

CardsList
Jul 13, 2026

WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos

Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.

Jiahao Liu, Yupeng Zheng, Zhongpu Xia +10

CASIA · Anyverse Dynamics · UCAS +2

Latent Action LearningRobot Policy LearningRobot Imitation Learning
AskPDF
Aug 20, 2026

What Matters for Latent Actions in Robot Learning

Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.

Xizhou Bu, Qingda Hu, Lei Zhou +13

College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, 200433, China · Xiaomi EV, Beijing, 100085, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China +3

Latent Action LearningRobot LearningRobotic Manipulation
AskPDF
Oct 25, 2026

RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.

Haoyu Zhao, Xingyue Zhao, Zhian Su +11

DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +3

VLA Model TrainingRobot Foundation ModelsLatent Action LearningRobot Learning from VideoSpatial Grounding for VLA ModelsFlow-Based VLA Models
AskPDF