What Matters for Latent Actions in Robot Learning

Xizhou Bu; Qingda Hu; Lei Zhou; Lingfeng Zhang; Yingbo Tang; Zihao Liu; Xinyi Tao; Zhiqiang Ma

cs.ROAug 20, 2026

What Matters for Latent Actions in Robot Learning

Authors: Xizhou Bu, Qingda Hu, Lei Zhou, Lingfeng Zhang, Yingbo Tang, Zihao Liu, Xinyi Tao, Zhiqiang Ma,

Organizations: College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, 200433, China · Xiaomi EV, Beijing, 100085, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China · Research Center for Intelligent Robotics, School of Astronautics, Northwestern Polytechnical University, Xi’an, 710072, China ·

Abstract

Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.

Figures & tables

Figure 1: Overview of design choices across three dimensions for latent action learning. Design I examines the impact of latent action modeling paradigms by comparing the implicit IDM-FDM framework with the explicit CFD-AE framework. Design II investigates how different regularization methods and their regularization strengths shape latent action representations and affect downstream policy learning. Design III evaluates the necessity of latent action normalization and investigates different strategies for integrating latent actions into physical action prediction. Stage I is trained on unlabeled video data consisting only of image frames, Stage II uses labeled video data paired with latent actions, and Stage III uses robot action data paired with physical actions. Further details of the three design studies and the three-stage training pipeline are provided in Section 2 and Appendix B.2 , respectively.

Design I: Latent Action Modeling Paradigms
Method	LIBERO						LIBERO-Plus						RoboTwin2.0
Method	DAP	LAP	JAP	JAP-DAP	JAP-LAP	Avg.	DAP	LAP	JAP	JAP-DAP	JAP-LAP	Avg.	DAP	LAP	JAP	JAP-DAP	JAP-LAP	Avg.
RAFT	0.676	0.714	0.838	0.836	0.837	0.780	0.310	0.359	0.340	0.346	0.356	0.342	0.785	0.773	0.825	0.831	0.815	0.806
SEA-RAFT	0.673	0.764	0.847	0.868	0.858	0.802	0.289	0.339	0.336	0.315	0.351	0.326	0.787	0.804	0.769	0.820	0.833	0.802
$\Delta$ RGB	0.853	0.860	0.899	0.894	0.896	0.880	0.352	0.382	0.397	0.388	0.413	0.387	0.795	0.809	0.834	0.856	0.830	0.825
$\Delta$ DINO	0.933	0.923	0.913	0.920	0.918	0.921	0.458	0.488	0.396	0.408	0.402	0.430	0.855	0.846	0.817	0.813	0.837	0.833

Table 1: Performance comparison across LIBERO, LIBERO-Plus, and RoboTwin2.0. Entries report success rates. Avg. averages over the five integration strategies; red boldface marks the highest Avg. within each design group and benchmark.

Figure 2: Effect of regularization strength $\lambda_{reg}$ on latent action learning. Columns show four regularization methods, with AE as the unregularized baseline. The top and bottom rows show probe and reconstruction metrics (Section 3.5 ), respectively, on color-matched vertical axes. Solid and dashed curves denote LIBERO and RoboTwin2.0; dotted vertical lines mark selected strengths.

Figure 3: Performance comparison of different latent action integration strategies. The horizontal axis shows seven latent action modeling paradigms and five regularization methods. The vertical axis shows five latent action integration strategies. Each cell reports the mean success rate across LIBERO, LIBERO-Plus, and RoboTwin2.0.

Figure 4: Correlation between proxy metrics and downstream robotic manipulation performance. Within each panel, columns correspond to the four proxy metrics and rows to benchmark–strategy combinations (LIBERO/RoboTwin2.0 with DAP/LAP). The purple and red panels report the Pearson correlation ( $r$ ) and Spearman correlation ( $\rho$ ), respectively.

Figure 5: Effect of latent action dimension on downstream robotic manipulation performance. Blue (DAP) and orange (LAP) bars show success rates across latent action dimensions; the green dashed curve shows their mean. Stars mark the best dimension per benchmark; shading marks the best dimension averaged across benchmarks. Results use LAPO with VAE regularization (

10^{-6}

Figure 6: Effect of Stage-II corpus expansion on downstream robotic manipulation performance. The left panel shows the full corpus composition; OXE includes all datasets except Robotwin and liberoplus . The right panel reports average DAP and LAP success rates by benchmark. Light and dark orange bars denote Stage-II training without and with OXE , respectively.

Figure 7: Real-World Experiments. The left panel shows the four tasks on the Franka Panda platform. The right panel plots success counts out of 20 trials against Stage-III training steps. Blue circles and red squares denote OpenVLA-OFT and OpenVLA-OFT (LA-Tuned), respectively.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Comparison of different latent action modeling paradigms. LAPO, LAOF, and CoMo follow IDM-FDM to predict future frames, while CFD-AE reconstructs consecutive-frame differences. Arrows indicate information flow, with each panel’s training losses shown at the bottom. Here, $z_{t}$ denotes the latent action and $\Delta$ consecutive-frame differences; hats indicate predictions.

Design I ( ⚫ : IDM-FDM ⚫ : CFD-AE)	Design II ( ⚫ : Discrete ⚫ : Continuous)	Design III ( ⚫ : Inaccessible ⚫ : Accessible)
⚫ LAPO ( Schmidt and Jiang, 2024 )	⚫ AE ( Bank et al., 2023 )	⚫ DAP
⚫ LAOF ( Bu et al., 2026 )	⚫ VQ-VAE ( Van Den Oord et al., 2017 )	⚫ LAP
⚫ CoMo ( Yang et al., 2026 )	⚫ VAE ( Kingma and Welling, 2013 )	⚫ JAP
⚫ RAFT ( Teed and Deng, 2020 ) / SEA-RAFT ( Wang et al., 2024 )	⚫ Sparsity ( Garrido et al., 2026 )	⚫ JAP-DAP
⚫ $\Delta$ RGB / $\Delta$ DINO ( Oquab et al., 2024 )	⚫ SIGReg ( Balestriero and LeCun, 2025 )	⚫ JAP-LAP

Appendix

Table 2: Design choices explored in this work. Columns list latent action modeling paradigms, learning objectives and regularization, and latent action integration strategies. Design III distinguishes whether physical action annotations are accessible during VLM backbone fine-tuning.

Hyperparameter	Stage I	Stage II	Stage III
Base LR	$2.5\times 10^{-5}$	$2.5\times 10^{-5}$	$2.5\times 10^{-5}$
Action Head LR	–	$1\times 10^{-4}$	$1\times 10^{-4}$
Qwen3-VL-4B LR	–	$1\times 10^{-5}$	$1\times 10^{-5}$
LR Scheduler	Constant	Cosine with Min LR	Cosine with Min LR
Weight Decay	$1\times 10^{-2}$	$1\times 10^{-8}$	$1\times 10^{-8}$
Global Batch Size	256	256	$128^{+}$ / $32^{-}$ / $32^{*}$

Appendix

Table 3: Training hyperparameters for the three training stages. Stage I trains LAM, Stage II fine-tunes the VLM backbone with latent actions, and Stage III trains the downstream policy. Superscripts identify benchmark-specific settings.

Figure 9: Performance of different regularization methods throughout training. The four columns plot the proxy metrics defined in Section 3.5 against training steps, up to 150K. The top and bottom rows show LIBERO (solid lines) and RoboTwin2.0 (dashed lines), respectively. Colors distinguish AE and the four regularization methods. Higher values indicate better proxy scores.

Figure 10: Performance difference between unnormalized and normalized settings across methods and benchmarks. Angular groups identify methods, with regularization strengths in parentheses; radial position shows the success-rate gap $(\mathrm{Unnormalized}-\mathrm{Normalized})$ , averaged over DAP and LAP. Blue, orange, and green points denote LIBERO, LIBERO-Plus, and RoboTwin2.0, respectively. The white annulus denotes the zero band; the outer blue and inner orange regions indicate gains and losses without normalization, respectively.

Latent Action Learning

Latent action learning discovers compact representations of behavior from observations or trajectories to support control, planning, and transfer.

See all 113 papers

RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions
Oct 25, 2026Haoyu Zhao, Xingyue Zhao, Zhian Su +11VLA Model Training Robot Foundation Models
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +3
LARA: Latent Action Representation Alignment for Vision-Language-Action Models
Jun 5, 2026Mengya Liu, Baoxiong Jia, Jiangyong Huang +2Latent Action Learning Robotic Manipulation
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
Jul 13, 2026Jiahao Liu, Yupeng Zheng, Zhongpu Xia +10Latent Action Learning Robot Policy Learning
CASIA · Anyverse Dynamics · UCAS +2

Oct 25, 2026J/K move · Enter open · S save

RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

Haoyu Zhao, Xingyue Zhao, Zhian Su, Bohan Hou+10

DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK+3

VLA Model Training Robot Foundation Models Latent Action Learning Robot Learning from Video Spatial Grounding for VLA Models Flow-Based VLA Models

Open paper AskPDF arXiv

Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.

Abstract

Latent Action Learning

Explore similar work

RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos