Hierarchical Latent Action Model

Hanjung Kim; Lerrel Pinto; Seon Joo Kim

cs.ROMar 6, 2026

Hierarchical Latent Action Model

Authors: Hanjung Kim, Lerrel Pinto, Seon Joo Kim

Organizations: Yonsei University · New York University

Abstract

Latent Action Models (LAMs) enable learning from actionless data for applications ranging from robotic control to interactive world models. However, existing LAMs typically focus on short-horizon frame transitions and capture low-level motion while overlooking longer-term temporal structure. In contrast, actionless videos often contain temporally extended and high-level skills. We present HiLAM, a hierarchical latent action model that discovers latent skills by modeling long-term temporal information. To capture these dependencies across long horizons, we utilize a pretrained LAM as a low-level extractor. This architecture aggregates latent action sequences, which contain the underlying dynamic patterns of the video, into high-level latent skills. Our experiments demonstrate that HiLAM improves over the baseline and exhibits robust dynamic skill discovery.

Figures & tables

Figure 1: Overview of HiLAM. (a) Overall latent skill learning pipeline. (b) Training objectives used for latent skill learning. (c) Extracting latent actions using a pretrained inverse dynamics model (IDM).

Figure 2: Latent skill extraction and policy learning. (a) Latent actions $\mathbf{z}^{l}$ are hierarchically encoded into stage-wise representations $\mathbf{z}^{s}$ and then expanded back to a per-timestep latent skill sequence $\mathbf{z}^{h}$ . (b) Overall pipeline of the hierarchical skill policy.

Panel of Figure 3 — Figure 3: LIBERO benchmark results. (a) Performance of BAKU (gray) and HiLAM (blue) on the LIBERO benchmark. (b) LIBERO-Long success rate as a function of the fraction of expert demonstrations used for fine-tuning.

Method	Pretraining dataset	Latent skill	Latent action	Success rate
BAKU	Robot	–	$\bar{z}^{0}$	$0.87$
	Robot	–	$\bar{z}^{2}$	$0.81$
	Human	–	$\bar{z}^{0}$	$0.91$
	Human	–	$\bar{z}^{2}$	$0.87$
HiLAM	–	–	–	$0.67$
	Robot	$\bar{z}^{1}$	$\bar{z}^{0}$	$0.90$

Table 1: Ablations on LIBERO-Long, effects of pretraining data and the stage used for latent skill/action conditioning.

Figure 4: Qualitative results for skill boundary prediction. Using the predicted boundary indicators $b^{s}_{t}$ , we assign each frame to a skill segment $k^{s}_{t}$ and display the segment ID for each segment.

Figure 5: Qualitative results for future frame prediction using a pretrained FDM. Given the current image $I_{t}$ and the predicted latent action $\hat{z}^{l}_{t}$ , the model predicts the future frame $\hat{I}_{t+k}$ .

Latent Action Learning

Latent action learning discovers compact representations of behavior from observations or trajectories to support control, planning, and transfer.

See all 113 papers

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling
Sep 14, 2026Dingjie Fu, Dianxing Shi, Yangyang Xu +1Latent Action Learning Latent Dynamics Modeling
Harbin Institute of Technology (Shenzhen) · Beihang University
FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning
Jun 17, 2026Takanori Yoshimoto, Yang Hu, Naruya Kondo +1Latent Action Learning
University of Tsukuba · The University of Tokyo
Learning Additively Compositional Latent Actions for Embodied AI
Apr 3, 2026Hangxing Wei, Xiaoyu Chen, Chuheng Zhang +5Latent Action Learning Embodied AI
Wuhan University · Tsinghua University · Microsoft Research

Sep 14, 2026J/K move · Enter open · S save

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

Dingjie Fu, Dianxing Shi, Yangyang Xu, Jun Yu

Harbin Institute of Technology (Shenzhen) · Beihang University

Latent Action Learning Latent Dynamics Modeling Action-Conditioned World Models Robotics

Open paper AskPDF arXiv

Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing future states. However, we identify a fundamental

\textbf{reconstruction-action mismatch}

: lower reconstruction error does not necessarily yield better latent dynamics or downstream performance. We attribute this mismatch to two underconstrained aspects of reconstruction-based latent dynamics modeling: (i) the inverse dynamics model (IDM) is not explicitly encouraged to distinguish action-related transitions from nuisance appearance, and (ii) the forward dynamics model (FDM) can underutilize the inferred latent action by exploiting predictive shortcuts from the current state. To address both limitations, we propose

\textbf{ACT-LAM}

, a lightweight action-centric framework that strengthens both action extraction and action utilization. Specifically, its Action Query IDM (AQ-IDM) employs learnable action queries and gated aggregation to selectively extract rich action-related transition cues without strong information bottlenecks. And its Action Token FDM (AT-FDM) projects latent actions into action tokens that progressively interact with evolving state representations, enabling continuous state-aware action conditioning. ACT-LAM further streamlines feature processing to concentrate model capacity on latent dynamics modeling. Extensive experiments on several robotic datasets and the VP

^2

benchmark demonstrate stronger latent action consistency, forward dynamics, and downstream visual planning performance with fewer trainable parameters and lower computational overhead. In particular, ACT-LAM surpasses the previous state of the art by

\textbf{7.6%}

on the aggregated VP

^2

success rate. Codes at url.

Abstract

Latent Action Learning

Explore similar work

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning

Learning Additively Compositional Latent Actions for Embodied AI