Latent Action Learning

Follow this topic Ask about this topic

Latent action learning studies how to represent meaningful behavior compactly, often by inferring actions from video or trajectories that lack action labels. Common approaches use inverse and forward dynamics, predictive world models, vector-quantized or continuous flow-based representations, and geometric or hierarchical structure. These representations can help vision-language-action policies plan and control robots, especially across long tasks, different embodiments, and limited demonstration data; LIBERO, CALVIN, and RoboTwin are common evaluation settings alongside real robots. A central open tension is that representations good at reconstructing visual changes may capture nuisance or environmental variation rather than controllable actions, so predictive consistency and proxy metrics do not always guarantee better downstream control.

Latest papers

113 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements
Apr 30, 2026Genki Kinoshita, Shu Nakamura, Ryo Kawahara +3Latent Action Learning Representation Learning
1Kyoto University · 2Kyoto Institute of Technology · 3RIKEN
UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling
Apr 21, 2026Boyu Chen, Yi Chen, Lu Qiu +3Latent Action Learning Humanoid Robot Control
XPENG Robotics · Tsinghua University · The University of Hong Kong
LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment
Apr 13, 2026Dujun Nie, Fengjiao Chen, Qi Lv +4Latent Action Learning Vision-Language-Action Models
Meituan, Beijing, China
From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data
Apr 4, 2026Linfang Zheng, Zikai Ouyang, Chen Wang +2Robot Learning from Video Vision-Based Robot Control
The University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China +1
Learning Additively Compositional Latent Actions for Embodied AI
Apr 3, 2026Hangxing Wei, Xiaoyu Chen, Chuheng Zhang +5Latent Action Learning Embodied AI
Wuhan University · Tsinghua University · Microsoft Research
Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversity
Mar 18, 2026Felix SchurLatent Action Learning Latent Dynamics Modeling
Department of Mathematics, ETH Zürich
One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation
Mar 15, 2026Juncheng Mu, Sizhe Yang, Hojin Bae +5Cross-Embodiment Robot Learning Cross-Embodiment Transfer
Shanghai AI Laboratory · Tsinghua University · The Chinese University of Hong Kong +2
World2Act: Latent Action Post-Training from World Model Dynamics
Mar 11, 2026An Dinh Vuong, Tuan Van Vo, Abdullah Sohail +6VLA Model Post-Training Vision-Language-Action Models
MBZUAI
Cross-Hand Latent Representation for Vision-Language-Action Models
Mar 10, 2026Guangqi Jiang, Yutong Liang, Jianglong Ye +6Robotic Manipulation Cross-Embodiment Robot Learning
UC San Diego · Amazon FAR · UC Berkeley
Hierarchical Latent Action Model
Mar 6, 2026Hanjung Kim, Lerrel Pinto, Seon Joo KimLatent Action Learning Agent Skill Learning
Yonsei University · New York University
Latent Policy Steering through One-Step Flow Policies
Mar 5, 2026Hokyun Im, Andrey Kolobov, Jianlong Fu +1RL for Robotics Latent Action Learning
Department of Artificial Intelligence, Yonsei University · Microsoft Research
Chain of World: World Model Thinking in Latent Motion
Mar 3, 2026Fuxiang Yang, Donglin Di, Lulu Tang +6Vision-Language-Action Models Reasoning for VLA Models
Harbin Institute of Technology · Li Auto · Beijing Academy of Artificial Intelligence (BAAI) +3
Spherical Latent Motion Prior for Physics-Based Simulated Humanoid Control
Mar 1, 2026Jing Tan, Weisheng Xu, Xiangrui Jiang +11Robot Motion Generation Humanoid Robot Control
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · University of Oxford, Oxford, United Kingdom
Hierarchical Action Learning for Weakly-Supervised Action Segmentation
Feb 27, 2026Junxian Huang, Ruichu Cai, Hao Zhu +5Temporal Action Segmentation Latent Action Learning
Guangdong University of Technology · Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence +1
Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild
Feb 25, 2026Hao Luo, Ye Wang, Wanpeng Zhang +5Vision-Language-Action Models Latent Action Learning
Peking University · BeingBeyond · Renmin University of China
UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models
Feb 23, 2026Manish Kumar Govind, Dominick Reilly, Pu Wang +1Vision-Language-Action Models Latent Action Learning
All authors are with the Department of Computer Science, University of North Carolina at Charlotte, NC, USA.
Temporal Action Representation Learning for Tactical Resource Control and Subsequent Maneuver Generation
Feb 21, 2026Hoseong Jung, Sungil Son, Daesol Cho +3Robot Policy Learning Latent Action Learning
Seoul National University, Republic of Korea · Seoul National University and Life Assistant Robotics, Republic of Korea · Georgia Institute of Technology, USA
In-Context Planning with Latent Temporal Abstractions
Feb 21, 2026Baiting Luo, Yunuo Zhang, Nathaniel S. Keplinger +3Model-Based Planning Offline RL
Vanderbilt University · William & Mary
Factored Latent Action World Models
Feb 18, 2026Zizhao Wang, Chang Shi, Jiaheng Hu +4Latent World Models Latent Action Learning
University of Texas at Austin · Sony AI
On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning
Feb 2, 2026Sacha Morin, Moonsub Byeon, Alexia Jolicoeur-Martineau +1Latent Action Learning Imitation Learning
Département d’informatique et de recherche opérationnelle, Université de Montréal · Mila – Quebec AI Institute · Samsung AI Lab, Montreal +1
CoLA-Flow Policy: Temporally Coherent Imitation Learning via Continuous Latent Action Flow Matching for Robotic Manipulation
Jan 30, 2026Wu Songwei, Jiang Zhiduo, Sun Wandong +4Flow Matching Latent Action Learning
Harbin Institute of Technology, Harbin 150001, China · Honor Device Co., Ltd., Shenzhen, China
Wavelet Policy: Imitation Learning in the Scale Domain with World Prior Memory
Apr 7, 2025Changchuan Yang, Haoxuan Xu, Yuhang Dong +3Latent Action Learning Visuomotor Policy Learning
Zhejiang University · The Hong Kong University of Science and Technology · Tsinghua University, DISCOVER Robotics
Synergistic Action and Style Priors for Learning Legged Locomotion
Oct 4, 2024Oliver Hausdörfer, Alexander von Rohr, Éric Lefort +1RL for Legged Locomotion Robot Policy Learning
Technical University of Munich, Germany; TUM School of Computation, Information and Technology, Department of Computer Engineering, Learning Systems and Robotics Lab; Munich Institute of Robotics and Machine Intelligence.

Apr 30, 2026J/K move · Enter open · S save

Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements

Genki Kinoshita, Shu Nakamura, Ryo Kawahara, Shohei Nobuhara+2

1Kyoto University · 2Kyoto Institute of Technology · 3RIKEN

Latent Action Learning Representation Learning Human Activity Recognition Self-Supervised Learning Hierarchical Representation Learning Human Motion Generation+2

Open paper AskPDF arXiv

Effective human behavior modeling requires a representation of the human body movement that capitalizes on its compositionality. We propose a hierarchical representation consisting of Action Atoms that capture the atomic joint movements and Action Motifs that are formed by their temporal compositions and encode similar body movements found across different overall human actions. We derive A4Mer, a nested latent Transformer to learn this hierarchical representation from human pose data in a fully self-supervised manner. A4Mer splits a 3D pose sequence into variable-length segments and represents each segment as a single latent token (Action Atoms). Through bottom-up representation learning, temporal patterns composed of these Action Atoms, which capture meaningful temporal spans of reusable, semantic segments of body movements, naturally emerge (Action Motifs). A4Mer achieves this with a unified pretext task of masked token prediction in their respective latent spaces. We also introduce Action Motif Dataset (AMD), a large-scale dataset of multi-view human behavior videos with full SMPL annotations. We introduce a novel use of cameras by mounting them on the feet to achieve their frame-wise annotations despite frequent and heavy body occlusions. Experimental results demonstrate the effectiveness of A4Mer for extracting meaningful Action Motifs, which significantly benefit human behavior modeling tasks including action recognition, motion prediction, and motion interpolation.