Organizations: College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, 200433, China · Xiaomi EV, Beijing, 100085, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China · Research Center for Intelligent Robotics, School of Astronautics, Northwestern Polytechnical University, Xi’an, 710072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China · Morphi Intelligence Technology Co., Ltd., Shenzhen, 518063, China
Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.
Figures & tables
Figure 1: Overview of design choices across three dimensions for latent action learning. Design I examines the impact of latent action modeling paradigms by comparing the implicit IDM-FDM framework with the explicit CFD-AE framework. Design II investigates how different regularization methods and their regularization strengths shape latent action representations and affect downstream policy learning. Design III evaluates the necessity of latent action normalization and investigates different strategies for integrating latent actions into physical action prediction. Stage I is trained on unlabeled video data consisting only of image frames, Stage II uses labeled video data paired with latent actions, and Stage III uses robot action data paired with physical actions. Further details of the three design studies and the three-stage training pipeline are provided in Section 2 and Appendix B.2 , respectively.
Method
LIBERO
LIBERO-Plus
RoboTwin2.0
DAP
LAP
JAP
JAP-DAP
JAP-LAP
Avg.
DAP
LAP
JAP
JAP-DAP
JAP-LAP
Avg.
DAP
LAP
JAP
JAP-DAP
JAP-LAP
Avg.
Design I: Latent Action Modeling Paradigms
RAFT
0.676
0.714
0.838
0.836
0.837
0.780
0.310
0.359
0.340
0.346
0.356
0.342
0.785
0.773
0.825
0.831
0.815
0.806
SEA-RAFT
0.673
0.764
0.847
0.868
0.858
0.802
0.289
0.339
0.336
0.315
0.351
0.326
0.787
0.804
0.769
0.820
0.833
0.802
Δ RGB
0.853
0.860
0.899
0.894
0.896
0.880
0.352
0.382
0.397
0.388
0.413
0.387
0.795
0.809
0.834
0.856
0.830
0.825
Δ DINO
0.933
0.923
0.913
0.920
0.918
0.921
0.458
0.488
0.396
0.408
0.402
0.430
0.855
0.846
0.817
0.813
0.837
0.833
Table 1: Performance comparison across LIBERO, LIBERO-Plus, and RoboTwin2.0. Entries report success rates. Avg. averages over the five integration strategies; red boldface marks the highest Avg. within each design group and benchmark.
Figure 2: Effect of regularization strength λreg on latent action learning. Columns show four regularization methods, with AE as the unregularized baseline. The top and bottom rows show probe and reconstruction metrics (Section 3.5 ), respectively, on color-matched vertical axes. Solid and dashed curves denote LIBERO and RoboTwin2.0; dotted vertical lines mark selected strengths.
Figure 3: Performance comparison of different latent action integration strategies. The horizontal axis shows seven latent action modeling paradigms and five regularization methods. The vertical axis shows five latent action integration strategies. Each cell reports the mean success rate across LIBERO, LIBERO-Plus, and RoboTwin2.0.
Figure 4: Correlation between proxy metrics and downstream robotic manipulation performance. Within each panel, columns correspond to the four proxy metrics and rows to benchmark–strategy combinations (LIBERO/RoboTwin2.0 with DAP/LAP). The purple and red panels report the Pearson correlation ( r ) and Spearman correlation ( ρ ), respectively.
Figure 5: Effect of latent action dimension on downstream robotic manipulation performance. Blue (DAP) and orange (LAP) bars show success rates across latent action dimensions; the green dashed curve shows their mean. Stars mark the best dimension per benchmark; shading marks the best dimension averaged across benchmarks. Results use LAPO with VAE regularization ( 10−6 ).
Figure 6: Effect of Stage-II corpus expansion on downstream robotic manipulation performance. The left panel shows the full corpus composition; OXE includes all datasets except Robotwin and liberoplus . The right panel reports average DAP and LAP success rates by benchmark. Light and dark orange bars denote Stage-II training without and with OXE , respectively.
Figure 7: Real-World Experiments. The left panel shows the four tasks on the Franka Panda platform. The right panel plots success counts out of 20 trials against Stage-III training steps. Blue circles and red squares denote OpenVLA-OFT and OpenVLA-OFT (LA-Tuned), respectively.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Comparison of different latent action modeling paradigms. LAPO, LAOF, and CoMo follow IDM-FDM to predict future frames, while CFD-AE reconstructs consecutive-frame differences. Arrows indicate information flow, with each panel’s training losses shown at the bottom. Here, zt denotes the latent action and Δ consecutive-frame differences; hats indicate predictions.
Design I ( ⚫ : IDM-FDM ⚫ : CFD-AE)
Design II ( ⚫ : Discrete ⚫ : Continuous)
Design III ( ⚫ : Inaccessible ⚫ : Accessible)
⚫ LAPO ( Schmidt and Jiang, 2024 )
⚫ AE ( Bank et al., 2023 )
⚫ DAP
⚫ LAOF ( Bu et al., 2026 )
⚫ VQ-VAE ( Van Den Oord et al., 2017 )
⚫ LAP
⚫ CoMo ( Yang et al., 2026 )
⚫ VAE ( Kingma and Welling, 2013 )
⚫ JAP
⚫ RAFT ( Teed and Deng, 2020 ) / SEA-RAFT ( Wang et al., 2024 )
⚫ Sparsity ( Garrido et al., 2026 )
⚫ JAP-DAP
⚫ Δ RGB / Δ DINO ( Oquab et al., 2024 )
⚫ SIGReg ( Balestriero and LeCun, 2025 )
⚫ JAP-LAP
Appendix
Table 2: Design choices explored in this work. Columns list latent action modeling paradigms, learning objectives and regularization, and latent action integration strategies. Design III distinguishes whether physical action annotations are accessible during VLM backbone fine-tuning.
Hyperparameter
Stage I
Stage II
Stage III
Base LR
2.5×10−5
2.5×10−5
2.5×10−5
Action Head LR
–
1×10−4
1×10−4
Qwen3-VL-4B LR
–
1×10−5
1×10−5
LR Scheduler
Constant
Cosine with Min LR
Cosine with Min LR
Weight Decay
1×10−2
1×10−8
1×10−8
Global Batch Size
256
256
128+ / 32− / 32∗
Appendix
Table 3: Training hyperparameters for the three training stages. Stage I trains LAM, Stage II fine-tunes the VLM backbone with latent actions, and Stage III trains the downstream policy. Superscripts identify benchmark-specific settings.
Figure 9: Performance of different regularization methods throughout training. The four columns plot the proxy metrics defined in Section 3.5 against training steps, up to 150K. The top and bottom rows show LIBERO (solid lines) and RoboTwin2.0 (dashed lines), respectively. Colors distinguish AE and the four regularization methods. Higher values indicate better proxy scores.
Figure 10: Performance difference between unnormalized and normalized settings across methods and benchmarks. Angular groups identify methods, with regularization strengths in parentheses; radial position shows the success-rate gap (Unnormalized−Normalized) , averaged over DAP and LAP. Blue, orange, and green points denote LIBERO, LIBERO-Plus, and RoboTwin2.0, respectively. The white annulus denotes the zero band; the outer blue and inner orange regions indicate gains and losses without normalization, respectively.
Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.
Haoyu Zhao, Xingyue Zhao, Zhian Su +11
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +3
Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.
Mengya Liu, Baoxiong Jia, Jiangyong Huang +2
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.