GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining

Jiarui Yang; Jiawei Li; Jiale Zhang; Hang Guo; Wen Huang; Maowei Hu; Tao Dai; Shu-Tao Xia

cs.ROAug 10, 2026

GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining

Authors: Jiarui Yang, Jiawei Li, Jiale Zhang, Hang Guo, Wen Huang, Maowei Hu, Tao Dai, Shu-Tao Xia

Organizations: Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL) · Shenzhen University

Abstract

Latent action models (LAMs) learn action-related representations from action-free videos via self-supervised future prediction, offering a scalable paradigm for embodied intelligence pretraining. However, existing LAMs collapse heterogeneous sources of visual change, including camera motion, object dynamics, and interaction events, into a single latent vector, resulting in entangled representations with limited semantic structure and consequently restricting world model controllability and VLA policy generalization. We introduce the Group-Disentangled Latent Action Model (GDLAM), a latent action model whose code is factorized by construction into N groups, each with an independent variational bottleneck and a spatially gated routing pathway, and trained with a set of information-geometric objectives: mutual exclusivity, group and gate sparsity, and static-dynamic orthogonality, that make the groups mutually causally distinct rather than merely decorrelated. Quantitatively, intervening on any single group changes only that group and leaves the others intact, and GDLAM improves label-free disentanglement metrics, including Modularity, MIG, and DCI, by wide margins over a strong unstructured LAM. Notably, this factorization is not at the expense of action information: across three mutual-information estimators and a linear probe, the grouped code is more informative than monolithic baselines both in- and out-of-distribution. As supporting evidence that the disentangled code is a reusable pretraining currency, we further transfer it to two downstream regimes: (1) World Modeling: World models pretrained with GDLAM achieve superior rollout fidelity and action-following capability compared with SOTA baselines. (2) VLA Policies: Pretraining with GDLAM substantially improves task success rates over previous methods across multiple simulation benchmarks and real-world robotic manipulation tasks

Figures & tables

Figure 1: GDLAM learns disentangled latent actions. (a) Cross-group response matrix under single-group intervention. A diagonal response indicates strong group independence. (b) Higher unsupervised disentanglement scores than a prior LAM. (c) Better downstream transfer.

Figure 2: Overview of GDLAM. (a) Main pipeline: a frozen DINOv2 tokenizer and a trainable Delta Encoder map two frames to latents $z_{t},z_{t+1}$ ; a Grouped Encoder reads them into $N$ latent actions $a_{t}^{1},\dots,a_{t}^{N}$ ; a Grouped Predictor transforms $z_{t}$ conditioned on the actions into the predicted future $\hat{z}_{t+1}$ . (b–e) Building blocks: the Delta Encoder, the Grouped Encoder with per-group variational bottlenecks, the Grouped Predictor, and the gated routing that forms each group’s spatial gate $\mathrm{Gate}_{t}$ .

Model	Modularity $\uparrow$	MIG $\uparrow$	DCI $\uparrow$
UniVLA	0.47	0.31	0.51
LAPA	0.51	0.33	0.55
Moto	0.45	0.28	0.49
MVP-LAM	0.58	0.37	0.60
DreamDojo	0.55	0.35	0.58
GDLAM (ours)	0.86	0.68	0.88

Table 1: Label-free disentanglement scores (higher is better for all three metrics).

Figure 3: Action-informativeness of the latent code: three MI estimators (bits, higher is better).

Figure 4: In-distribution (ID) vs. OOD linear-probe NMSE. Each model is represented by a vertical segment connecting its ID and OOD NMSE values (square marker). Shorter segments indicate better generalization, while lower positions indicate more action-informative representations.

Figure 5: Attention/gate localization on two representative manipulation scenes.

Model	Diag. $\uparrow$	Leakage $\downarrow$	Cross- $R^{2}\downarrow$
UniVLA	0.58	0.33	0.56
LAPA	0.63	0.29	0.52
Moto	0.56	0.35	0.58
MVP-LAM	0.68	0.24	0.47
DreamDojo	0.66	0.26	0.49
GDLAM (ours)	0.91	0.05	0.14

Table 2: Causal group separation: diagonal (intended) response, off-diagonal leakage, and cross-group predictability

R^{2}

. For monolithic baselines the bottleneck is split in half for reference only, since these models have no trained notion of a group.

Figure 6: Spatial gate maps for each of the four groups over the course of training. Columns show training progress; rows show groups. Distinct groups converge onto disjoint regions of the image without any spatial supervision.

Methods	Visual Quality			Motion Quality			Content Consistency			Physics Adherence		3D Accuracy		Controllability
Methods	Image Quality	Aesthetic Quality	JEPA Similarity	Dynamic Degree	Flow Score	Motion Smoothness	Subject Consistency	Background Consistency	Photometric Consistency	Interaction Quality	Trajectory Acc.	Depth Acc.	Perspectivity	Instruction Follow.	Semantic Align.	Action Follow.
GigaWorld	0.463	0.392	0.437	0.614	0.315	0.778	0.734	0.832	0.179	0.529	0.159	0.618	0.760	0.607	0.851	0.116
Genie	0.229	0.326	0.334	0.674	0.090	0.694	0.770	0.883	0.205	0.205	0.074	0.851	0.520	0.208	0.855	0.018
MIND-V	0.327	0.385	0.460	0.559	0.275	0.763	0.817	0.909	0.247	0.569	0.144	0.713	0.796	0.601	0.872	0.087
RoboMaster	0.353	0.405	0.302	0.658	0.153	0.700	0.810	0.903	0.341	0.533	0.122	0.809	0.738	0.569	0.846	0.082
Vidarc	0.416	0.411	0.559	0.281	0.145	0.771	0.748	0.808	0.226	0.528	0.192	0.783	0.755	0.580	0.863	0.047

Table 3: Comprehensive video evaluation results across 16 metrics spanning six dimensions. Best results are bolded .

Variant	DCI $\uparrow$	Leak. $\downarrow$	MI $\uparrow$
Full GDLAM	0.88	0.05	1.22
– mutual excl.	0.61	0.19	1.14
– group sparsity	0.72	0.13	1.18
– gate sparsity	0.70	0.15	1.17
– static–dyn. orth.	0.64	0.17	1.15
shared bottleneck	0.57	0.22	1.10

Table 4: Ablations across three axes relative to the full model: disentanglement (DCI), causal separation (Leakage), and action informativeness (MI-KSG, bits).

Figure 7: Average success rate of different methods across the two benchmarks.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Critic / decoder hidden width	1024
Critic / decoder depth	4
Batch size	1024
Optimizer / LR (MINE)	Adam / $1\times 10^{-4}$
Optimizer / LR (BA)	Adam / $5\times 10^{-5}$
Training steps	8000

Appendix

Table 5: Optimization hyperparameters for the two trainable MI estimators (MINE, BA).

Figure 8: Pixel-space visualization of latent future prediction.

Figure 9: Qualitative visualization of world model rollouts on the DreamDojo-HV test set (30-frame intervals).

Model	DINOv2 MSE $\downarrow$	Cosine sim. $\uparrow$
UniVLA	0.148	0.881
LAPA	0.162	0.865
Moto	0.171	0.851
MVP-LAM	0.129	0.902
DreamDojo	0.133	0.898
GDLAM (ours)	0.126	0.905

Appendix

Table 6: Next-frame prediction error in the shared, frozen DINOv2 feature space (pixel-decoding baselines are re-embedded post-hoc). Lower is better for MSE, higher is better for cosine similarity.

Hyperparameter	Value
Number of groups $N$	4 (swept: 1, 2, 4, 8)
Tokens per group $\tau$	1
Latent dim $d$	32
Encoder depth	6
Predictor depth	12
Tokenizer	DINOv2-reg ViT-L (frozen)

Appendix

Table 7: Main hyperparameters. Loss weights are annealed; see the released config for schedules.

Figure 10: Long-horizon rollout visualization results of the world model on the DreamDojo-HV test set (with frames sampled every 20 steps).

	In-lab			EgoDex			DreamDojo-HV
Conditioning code	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DreamDojo (retrained)	20.42	0.761	0.231	19.78	0.770	0.229	18.19	0.740	0.246
MVP-LAM	20.57	0.766	0.227	19.91	0.775	0.225	18.31	0.745	0.242
GDLAM (ours)	20.88	0.774	0.221	20.14	0.782	0.219	18.53	0.751	0.236

Appendix

Table 8: World-model OOD transfer with the DreamDojo generator held fixed, varying only the latent action model that supplies the conditioning code.

Hyperparameter	Value
VLM backbone	Prismatic-7B
Pretrain steps	200k
Pretrain LR	$2\times 10^{-5}$
Pretrain batch size	96
SIMPLER FT	10k steps, bs=4, accum=4
LIBERO FT	30k steps, bs=8, accum=2

Appendix

Table 9: VLA pretraining and finetuning hyperparameters.

Method	StackG2Y	Carrot2Plate	Spoon2Towel	Eggplant2Bask	AVG
UniVLA VLA	18.7	33.3	54.2	58.3	41.1
DreamDojo VLA	27.1	58.3	66.7	68.3	55.1
MVP-LAM VLA	29.2	62.5	66.7	70.8	57.3
GDLAM VLA (ours)	37.5	75.0	75.0	79.2	66.7

Appendix

Table 10: SIMPLER benchmark. Success rate (%) on a 7-DoF WidowX arm.

Method	Spatial	Object	Goal	Long	AVG
UniVLA VLA	89.7	90.2	85.1	78.3	85.8
DreamDojo VLA	94.6	94.1	90.8	85.7	91.3
MVP-LAM VLA	95.4	94.8	91.6	87.9	92.4
GDLAM VLA (ours)	96.8	96.2	93.5	91.1	94.4

Appendix

Table 11: LIBERO benchmark. Success rate (%) for VLAs pretrained on the same Bridge V2 + OXE mixture.

Figure 11: Real-robot deployment on an AgiBot G1 in the desk scene.

Split	Arm	Instruction
g1_pen	L	The left arm picks up the pen on the desk, and the left arm places the pen into the pen holder at the top of the storage box.
g1_eraser	R	The right arm picks up the eraser on the desk, and the right arm places the eraser on the third layer of the storage box from the bottom.
g1_correction	L	The left arm picks up the correction fluid on the desk, and the left arm places the correction fluid on the third layer of the storage box from the bottom.
g1_charger	R	The right arm picks up the charger on the desk, and the right arm places the charger on the third layer of the storage box from the bottom.
g1_pencil	L	The left arm picks up the pencil case on the desk, and the left arm places the pencil case on the second layer of the storage box from the bottom.
g1_stapler	R	The right arm picks up the stapler on the desk, and the right arm places the stapler on the second layer of the storage box from the bottom.

Appendix

Table 12: Language instructions for the eight desktop subtasks in g1_all_realbot . Arm denotes the acting arm (L/R).

Figure 12: Visualization results of the real-robot policy executions. The top two rows show successful task executions, while the bottom row presents examples of failed executions.

	Baseline		LAPA		Moto		UniVLA		DreamDojo		GDLAM
Subtask	S	M	S	M	S	M	S	M	S	M	S	M
Pen (L)	20	5	20	0	20	5	25	5	25	5	30	10
Eraser (R)	25	5	25	5	30	5	30	10	35	10	40	15
Correction (L)	15	0	15	0	15	0	20	5	20	5	25	10
Charger (R)	60	35	55	35	60	40	60	45	60	45	65	50
Pencil case (L)	60	45	55	45	60	50	60	55	65	55	65	60

Appendix

Table 13: Real-robot success rate (%) on the eight desktop subtasks. S: per-task policies (single-task); M: one policy post-trained on the mixture of all eight subtask splits and evaluated per subtask (multi-task). Best per column in bold .

Variant	DCI $\uparrow$	MI(KSG) $\uparrow$
Full model (default)	0.88	1.24
w/o $\mathcal{L}_{\mathrm{mx}}$	0.61	1.21
w/o $\mathcal{L}_{\mathrm{ge}}$	0.74	1.22
w/o $\mathcal{L}_{\mathrm{gs}}$	0.80	1.23
w/o $\mathcal{L}_{\mathrm{orth}}$	0.83	1.23

Appendix

Table 14: Leave-one-out ablation of the four disentanglement objectives (one term zeroed at a time; all else at default).

Variant	DCI $\uparrow$	MI(KSG) $\uparrow$
Full model (default)	0.88	1.24
shared (non-isolated) bottleneck	0.64	1.20
uniform gate (no spatial gate)	0.71	1.22
w/o anti-leak augmentation	0.82	1.06

Appendix

Table 15: Ablation of architectural mechanisms and the anti-leak augmentation.

$N$	DCI $\uparrow$	MI (KSG) $\uparrow$	Rel. step time
1 (Single-code)	0.48	0.94	0.81 $\times$
2	0.69	1.10	0.89 $\times$
4 (default)	0.88	1.24	1.00 $\times$
8	0.85	1.22	1.34 $\times$

Appendix

Table 16: Effect of the number of groups

N

on disentanglement, information content, and training cost.

Variant	DCI $\uparrow$	MI(KSG) $\uparrow$	Leakage $\downarrow$
Full model (default)	0.88	1.24	0.09
all weights $\times 0.5$	0.85	1.23	0.12
all weights $\times 2.0$	0.86	1.19	0.08
no static-group	0.86	1.23	0.19

Appendix

Table 17: Sensitivity to disentanglement-weight scaling and to the static-group designation. “Leakage” is the fraction of dynamic-group gate mass falling on the static background (

\downarrow

Model	Params (M)	Trainable (M)	PeakMem (GB)	GFLOPs	Fwd (ms)
UniVLA	287	201	2.0	132	107
LAPA	344	344	2.8	43	197
Moto	439	121	2.2	148	69
MVP-LAM	287	201	5.6	308	90
DreamDojo	710	710	5.6	225	117
GDLAM (ours)	392	285	3.1	168	101

Appendix

Table 18: Compute comparison. Params: total; PeakMem: peak forward activation memory; Fwd: mean forward latency.

Latent Action Learning

Latent action learning discovers compact representations of behavior from observations or trajectories to support control, planning, and transfer.

See all 106 papers

DiLA: Disentangled Latent Action World Models
May 15, 2026Tianqiu Zhang, Muyang Lyu, Yufan Zhang +2Action-Conditioned Video Generation Latent Action Learning
Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, IDG/McGovern Institute for Brain Research, Peking University. · Center of Quantitative Biology, Peking University. · School of Psychological and Cognitive Sciences, Key Laboratory of Machine Perception (Ministry of Education), Peking University.
DLAM: Distributional Latent Actions with Temporal Constraints
Jul 29, 2026Zuojin Tang, Feifan Luo, Haoyun Liu +10Latent Action Learning Latent Dynamics Modeling
Zhejiang University · Nanjing University · Shanghai Jiaotong University +4
GALA: Geometry-Aware Latent Action Modeling for Vision-Language-Action Model Pretraining across Embodiments
Sep 18, 2026Yichen Liu, Puzhen Yuan, Xiang Zhu +2Latent Action Learning Vision-Language-Action Models
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China

May 15, 2026J/K move · Enter open · S save

DiLA: Disentangled Latent Action World Models

Tianqiu Zhang, Muyang Lyu, Yufan Zhang, Fang Fang+1

Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, IDG/McGovern Institute for Brain Research, Peking University. · Center of Quantitative Biology, Peking University. · School of Psychological and Cognitive Sciences, Key Laboratory of Machine Perception (Ministry of Education), Peking University.

Action-Conditioned Video Generation Latent Action Learning Disentangled Representation Learning World Model Learning Action-Conditioned World Models

Open paper AskPDF arXiv

Latent Action Models (LAMs) enable the learning of world models from unlabeled video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in video generation quality, action transfer, visual planning, and manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.

Abstract

Latent Action Learning

Explore similar work

DiLA: Disentangled Latent Action World Models

DLAM: Distributional Latent Actions with Temporal Constraints

GALA: Geometry-Aware Latent Action Modeling for Vision-Language-Action Model Pretraining across Embodiments