Learning Additively Compositional Latent Actions for Embodied AI

Hangxing Wei; Xiaoyu Chen; Chuheng Zhang; Tim Pearce; Jianyu Chen; Alex Lamb; Li Zhao; Jiang Bian

cs.CVApr 3, 2026

Learning Additively Compositional Latent Actions for Embodied AI

Authors: Hangxing Wei, Xiaoyu Chen, Chuheng Zhang, Tim Pearce, Jianyu Chen, Alex Lamb, Li Zhao, Jiang Bian

Organizations: Wuhan University · Tsinghua University · Microsoft Research

Abstract

Latent action learning infers pseudo-action labels from visual transitions, providing an approach to leverage internet-scale video for embodied AI. However, most methods learn latent actions without structural priors that encode the additive, compositional structure of physical motion. As a result, latents often entangle irrelevant scene details or information about future observations with true state changes and miscalibrate motion magnitude. We introduce Additively Compositional Latent Action Model (AC-LAM), which enforces scene-wise additive composition structure over short horizons on the latent action space. These AC constraints encourage simple algebraic structure in the latent action space~(identity, inverse, cycle consistency) and suppress information that does not compose additively. Empirically, AC-LAM learns more structured, motion-specific, and displacement-calibrated latent actions and provides stronger supervision for downstream policy learning, outperforming state-of-the-art LAMs across simulated and real-world tabletop tasks.

Figures & tables

Figure 1: Evolution of the normalized latent action norm $\|LAM(o_{0},o_{t})\|$ over time intervals $t$ . The figure shows that, compared with baselines, our AC constraints (AC-LAM) induce displacement-calibrated latent actions, effectively capturing the magnitude of the transition from $o_{0}$ to $o_{t}$ .

Figure 2: Additively Compositional Latent Action Model (AC-LAM). For triples $(o_{i},o_{j},o_{k})$ from the same scene, scene-wise additivity encourages $z_{ik}\approx z_{ij}+z_{jk}$ , which regularizes the latent action space on top of a standard IDM–FDM architecture. The red line denotes the $(i,j)$ mapping with IDM encoder $z_{ij}=f(o_{i},o_{j})$ and FDM decoder $\hat{o}_{j}=F_{z_{ij}}(o_{i})$ . The blue and green lines depict the corresponding mappings for $(j,k)$ and $(i,k)$ , respectively.

Dataset	Method	$\mathcal{L}_{\mathrm{Norm\text{-}AC}}$ $\downarrow$	$r\big(\\|z\\|,\,\\|\Delta s\\|\big)$ $\uparrow$	$\mathcal{L}_{\mathrm{Norm\text{-}Identity}}$ $\downarrow$	$\Delta_{\mathrm{inv}}$ $\downarrow$
Fractal	AC-LAM	0.086	0.489	0.151	0.156
	Villa-X LAM	0.374	0.289	0.358	0.874
	UniVLA LAM	0.960	0.325	0.478	1.536
	LAPA LAM	0.845	0.024	0.862	1.773
Bridge	AC-LAM	0.135	0.408	0.020	0.267
	Villa-X LAM	0.386	0.225	0.326	0.875

Table 1: Comparison of structured latent metrics on Fractal, Bridge and LIBERO for AC-LAM vs baselines. Lower

\mathcal{L}_{\mathrm{Norm\text{-}AC}}

\mathcal{L}_{\mathrm{Norm\text{-}Identity}}

\Delta_{\mathrm{inv}}

indicate stronger additive/identity/inverse consistency; higher

r\big(\|z\|,\,\|\Delta s\|\big)

indicate better displacement calibration. AC-LAM generally improves latent structure across datasets.

LAM	AC-LAM	Villa-X	UniVLA	LAPA
$\mathrm{Acc}_{\mathrm{env}}^{\mathrm{mlp}}$ $\downarrow$	50.2%	63.0%	80.2%	52.0%

Table 2: Evaluation results on probe accuracy (

\mathrm{Acc}_{\mathrm{env}}^{\mathrm{mlp}}

) on four environments (fractral, bridge, kuka, droid) with different LAMs. Lower probe accuracy indicates less environment leakage.

Figure 3: Trajectory of the latent action norm $||f(o_{0},o_{t})||$ in real-world tabletop manipulation, with latent actions generated by LAPA LAM, UniVLA LAM, Villa-X LAM and AC-LAM. AC‑LAM yields the most displacement‑calibrated latents, aligning with motion magnitude.

Method	ID			Train			Val
Method	S	S/A	R	S	S/A	R	S	S/A	R
w/o LAM	22	35	0.63	26	44	0.59	19	38	0.50
LAPA LAM	19	33	0.58	7	26	0.27	9	34	0.26
UniVLA LAM	38	55	0.69	29	58	0.5	28	57	0.49
Villa-X LAM	42	53	0.79	25	52	0.48	31	58	0.53
AC-LAM	55	61	0.90	42	56	0.75	41	60	0.68

Table 3: Policy results on Emoji Table-Top (GrinningFace) across ID, Train, and Val splits. We report Success (S) (placement on the correct target), Any-Success (S/A) (placement on any target), and the Recognition Ratio (R=S/(S/A)), which approximates the model’s target recognition accuracy.

Method	ID	OOD-D	OOD-B
w/o LAM	33.3	26.7	6.7
LAPA LAM	13.3	6.7	0
UniVLA LAM	33.3	26.7	26.7
Villa-X LAM	40	20	26.7
AC-LAM	60	53.3	33.3

Table 4: Evaluation results on Real-World Tabletop Manipulation for policy learning with different LAMs.

Design	$\mathcal{L}_{\mathrm{Norm\text{-}AC}}$ $\downarrow$	$r\big(\\|z\\|,\,\\|\Delta s\\|\big)$ $\uparrow$	Stability
Default	0.102	0.256	stable
pre-VQ	0.456	0.476	stable
IDM(no sg)	-	-	collapse
sg on $z_{ik}$	0.141	0.098	stable
sg on $z_{ij}+z_{jk}$	-	-	explode

Table 5: Ablation on different design choices for AC-LAM, evaluated on Bridge.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Mix Ratio (%)
RT-1 Robot Action ( Brohan et al., 2022 )	9.70
AgiBot World Beta ( Bu et al., 2025a )	20.0
Kuka ( Kalashnikov et al., 2018 )	1.97
Bridge ( Walke et al., 2023 ; Ebert et al., 2021 )	5.47
Taco Play ( Rosete-Beas et al., 2022 ; Mees et al., 2023 )	0.76
Jaco Play ( Dass et al., 2023 )	0.12

Appendix

Table 6: Our training data mixture used in LAM training.

Panel of Figure 4 — Figure 4: Two experimental environments: (a) Emoji Table-Top (GrinningFace) simulation for controlled studies of vision–semantic generalization. A robotic arm picks a cube and places it on the instructed emoji. The viewpoint is aligned with Bridge‑v2 to leverage this large-scale dataset for knowledge transfer. The initial positions of the cube, the emojis, and the robotic arm are randomized to test robustness. (b) Real‑World Tabletop Manipulation featuring diverse pick tasks across varied objects and backgrounds; evaluations cover in‑distribution scenes, OOD distractors, and OOD backgrounds to assess robustness.

Figure 6: More trajectories of the latent action norm $\|LAM(o_{0},o_{t})\|$ in Emoji Table-Top simulation and real-world tabletop manipulation, with latent actions generated by LAPA LAM, UniVLA LAM, villa-X LAM and AC-LAM. AC‑LAM yields the most displacement‑calibrated latents, aligning with motion magnitude.

Explore similar work

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling
Sep 14, 2026Dingjie Fu, Dianxing Shi, Yangyang Xu +1Latent Action Learning Latent Dynamics Modeling
Harbin Institute of Technology (Shenzhen) · Beihang University
Why Latent Actions Fail, and How to Prevent It
May 13, 2026Jung Min Lee, Taehyun Cho, Li Zhao +1Latent Action Learning Disentangled Representation Learning
Seoul National University · Microsoft Research
UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Oct 12, 2026Lei Yao, Qiao Feng, Mutian Tong +3Latent Action Learning Robotic Manipulation
University of Pennsylvania · The Hong Kong Polytechnic University

Sep 14, 2026J/K move · Enter open · S save

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

Dingjie Fu, Dianxing Shi, Yangyang Xu, Jun Yu

Harbin Institute of Technology (Shenzhen) · Beihang University

Latent Action Learning Latent Dynamics Modeling Action-Conditioned World Models Robotics

Open paper AskPDF arXiv

Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing future states. However, we identify a fundamental

\textbf{reconstruction-action mismatch}

: lower reconstruction error does not necessarily yield better latent dynamics or downstream performance. We attribute this mismatch to two underconstrained aspects of reconstruction-based latent dynamics modeling: (i) the inverse dynamics model (IDM) is not explicitly encouraged to distinguish action-related transitions from nuisance appearance, and (ii) the forward dynamics model (FDM) can underutilize the inferred latent action by exploiting predictive shortcuts from the current state. To address both limitations, we propose

\textbf{ACT-LAM}

, a lightweight action-centric framework that strengthens both action extraction and action utilization. Specifically, its Action Query IDM (AQ-IDM) employs learnable action queries and gated aggregation to selectively extract rich action-related transition cues without strong information bottlenecks. And its Action Token FDM (AT-FDM) projects latent actions into action tokens that progressively interact with evolving state representations, enabling continuous state-aware action conditioning. ACT-LAM further streamlines feature processing to concentrate model capacity on latent dynamics modeling. Extensive experiments on several robotic datasets and the VP

^2

benchmark demonstrate stronger latent action consistency, forward dynamics, and downstream visual planning performance with fewer trainable parameters and lower computational overhead. In particular, ACT-LAM surpasses the previous state of the art by

\textbf{7.6%}

on the aggregated VP

^2

success rate. Codes at url.

Abstract

Explore similar work

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

Why Latent Actions Fail, and How to Prevent It

UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation