Latent Action Learning

Latent action learning studies how to represent meaningful behavior compactly, often by inferring actions from video or trajectories that lack action labels. Common approaches use inverse and forward dynamics, predictive world models, vector-quantized or continuous flow-based representations, and geometric or hierarchical structure. These representations can help vision-language-action policies plan and control robots, especially across long tasks, different embodiments, and limited demonstration data; LIBERO, CALVIN, and RoboTwin are common evaluation settings alongside real robots. A central open tension is that representations good at reconstructing visual changes may capture nuisance or environmental variation rather than controllable actions, so predictive consistency and proxy metrics do not always guarantee better downstream control.