Latent action models (LAMs) learn action-related representations from action-free videos via self-supervised future prediction, offering a scalable paradigm for embodied intelligence pretraining. However, existing LAMs collapse heterogeneous sources of visual change, including camera motion, object dynamics, and interaction events, into a single latent vector, resulting in entangled representations with limited semantic structure and consequently restricting world model controllability and VLA policy generalization. We introduce the Group-Disentangled Latent Action Model (GDLAM), a latent action model whose code is factorized by construction into N groups, each with an independent variational bottleneck and a spatially gated routing pathway, and trained with a set of information-geometric objectives: mutual exclusivity, group and gate sparsity, and static-dynamic orthogonality, that make the groups mutually causally distinct rather than merely decorrelated. Quantitatively, intervening on any single group changes only that group and leaves the others intact, and GDLAM improves label-free disentanglement metrics, including Modularity, MIG, and DCI, by wide margins over a strong unstructured LAM. Notably, this factorization is not at the expense of action information: across three mutual-information estimators and a linear probe, the grouped code is more informative than monolithic baselines both in- and out-of-distribution. As supporting evidence that the disentangled code is a reusable pretraining currency, we further transfer it to two downstream regimes: (1) World Modeling: World models pretrained with GDLAM achieve superior rollout fidelity and action-following capability compared with SOTA baselines. (2) VLA Policies: Pretraining with GDLAM substantially improves task success rates over previous methods across multiple simulation benchmarks and real-world robotic manipulation tasks
Figures & tables
Figure 1: GDLAM learns disentangled latent actions. (a) Cross-group response matrix under single-group intervention. A diagonal response indicates strong group independence. (b) Higher unsupervised disentanglement scores than a prior LAM. (c) Better downstream transfer.
Figure 2: Overview of GDLAM. (a) Main pipeline: a frozen DINOv2 tokenizer and a trainable Delta Encoder map two frames to latents zt,zt+1 ; a Grouped Encoder reads them into N latent actions at1,…,atN ; a Grouped Predictor transforms zt conditioned on the actions into the predicted future z^t+1 . (b–e) Building blocks: the Delta Encoder, the Grouped Encoder with per-group variational bottlenecks, the Grouped Predictor, and the gated routing that forms each group’s spatial gate Gatet .
Model
Modularity ↑
MIG ↑
DCI ↑
UniVLA
0.47
0.31
0.51
LAPA
0.51
0.33
0.55
Moto
0.45
0.28
0.49
MVP-LAM
0.58
0.37
0.60
DreamDojo
0.55
0.35
0.58
GDLAM (ours)
0.86
0.68
0.88
Table 1: Label-free disentanglement scores (higher is better for all three metrics).
Figure 3: Action-informativeness of the latent code: three MI estimators (bits, higher is better).
Figure 4: In-distribution (ID) vs. OOD linear-probe NMSE. Each model is represented by a vertical segment connecting its ID and OOD NMSE values (square marker). Shorter segments indicate better generalization, while lower positions indicate more action-informative representations.
Figure 5: Attention/gate localization on two representative manipulation scenes.
Model
Diag. ↑
Leakage ↓
Cross- R2↓
UniVLA
0.58
0.33
0.56
LAPA
0.63
0.29
0.52
Moto
0.56
0.35
0.58
MVP-LAM
0.68
0.24
0.47
DreamDojo
0.66
0.26
0.49
GDLAM (ours)
0.91
0.05
0.14
Table 2: Causal group separation: diagonal (intended) response, off-diagonal leakage, and cross-group predictability R2 . For monolithic baselines the bottleneck is split in half for reference only, since these models have no trained notion of a group.
Figure 6: Spatial gate maps for each of the four groups over the course of training. Columns show training progress; rows show groups. Distinct groups converge onto disjoint regions of the image without any spatial supervision.
Methods
Visual Quality
Motion Quality
Content Consistency
Physics Adherence
3D Accuracy
Controllability
Image Quality
Aesthetic Quality
JEPA Similarity
Dynamic Degree
Flow Score
Motion Smoothness
Subject Consistency
Background Consistency
Photometric Consistency
Interaction Quality
Trajectory Acc.
Depth Acc.
Perspectivity
Instruction Follow.
Semantic Align.
Action Follow.
GigaWorld
0.463
0.392
0.437
0.614
0.315
0.778
0.734
0.832
0.179
0.529
0.159
0.618
0.760
0.607
0.851
0.116
Genie
0.229
0.326
0.334
0.674
0.090
0.694
0.770
0.883
0.205
0.205
0.074
0.851
0.520
0.208
0.855
0.018
MIND-V
0.327
0.385
0.460
0.559
0.275
0.763
0.817
0.909
0.247
0.569
0.144
0.713
0.796
0.601
0.872
0.087
RoboMaster
0.353
0.405
0.302
0.658
0.153
0.700
0.810
0.903
0.341
0.533
0.122
0.809
0.738
0.569
0.846
0.082
Vidarc
0.416
0.411
0.559
0.281
0.145
0.771
0.748
0.808
0.226
0.528
0.192
0.783
0.755
0.580
0.863
0.047
Table 3: Comprehensive video evaluation results across 16 metrics spanning six dimensions. Best results are bolded .
Variant
DCI ↑
Leak. ↓
MI ↑
Full GDLAM
0.88
0.05
1.22
– mutual excl.
0.61
0.19
1.14
– group sparsity
0.72
0.13
1.18
– gate sparsity
0.70
0.15
1.17
– static–dyn. orth.
0.64
0.17
1.15
shared bottleneck
0.57
0.22
1.10
Table 4: Ablations across three axes relative to the full model: disentanglement (DCI), causal separation (Leakage), and action informativeness (MI-KSG, bits).
Figure 7: Average success rate of different methods across the two benchmarks.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Critic / decoder hidden width
1024
Critic / decoder depth
4
Batch size
1024
Optimizer / LR (MINE)
Adam / 1×10−4
Optimizer / LR (BA)
Adam / 5×10−5
Training steps
8000
Appendix
Table 5: Optimization hyperparameters for the two trainable MI estimators (MINE, BA).
Figure 8: Pixel-space visualization of latent future prediction.
Figure 9: Qualitative visualization of world model rollouts on the DreamDojo-HV test set (30-frame intervals).
Model
DINOv2 MSE ↓
Cosine sim. ↑
UniVLA
0.148
0.881
LAPA
0.162
0.865
Moto
0.171
0.851
MVP-LAM
0.129
0.902
DreamDojo
0.133
0.898
GDLAM (ours)
0.126
0.905
Appendix
Table 6: Next-frame prediction error in the shared, frozen DINOv2 feature space (pixel-decoding baselines are re-embedded post-hoc). Lower is better for MSE, higher is better for cosine similarity.
Hyperparameter
Value
Number of groups N
4 (swept: 1, 2, 4, 8)
Tokens per group τ
1
Latent dim d
32
Encoder depth
6
Predictor depth
12
Tokenizer
DINOv2-reg ViT-L (frozen)
Appendix
Table 7: Main hyperparameters. Loss weights are annealed; see the released config for schedules.
Figure 10: Long-horizon rollout visualization results of the world model on the DreamDojo-HV test set (with frames sampled every 20 steps).
In-lab
EgoDex
DreamDojo-HV
Conditioning code
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DreamDojo (retrained)
20.42
0.761
0.231
19.78
0.770
0.229
18.19
0.740
0.246
MVP-LAM
20.57
0.766
0.227
19.91
0.775
0.225
18.31
0.745
0.242
GDLAM (ours)
20.88
0.774
0.221
20.14
0.782
0.219
18.53
0.751
0.236
Appendix
Table 8: World-model OOD transfer with the DreamDojo generator held fixed, varying only the latent action model that supplies the conditioning code.
Hyperparameter
Value
VLM backbone
Prismatic-7B
Pretrain steps
200k
Pretrain LR
2×10−5
Pretrain batch size
96
SIMPLER FT
10k steps, bs=4, accum=4
LIBERO FT
30k steps, bs=8, accum=2
Appendix
Table 9: VLA pretraining and finetuning hyperparameters.
Method
StackG2Y
Carrot2Plate
Spoon2Towel
Eggplant2Bask
AVG
UniVLA VLA
18.7
33.3
54.2
58.3
41.1
DreamDojo VLA
27.1
58.3
66.7
68.3
55.1
MVP-LAM VLA
29.2
62.5
66.7
70.8
57.3
GDLAM VLA (ours)
37.5
75.0
75.0
79.2
66.7
Appendix
Table 10: SIMPLER benchmark. Success rate (%) on a 7-DoF WidowX arm.
Method
Spatial
Object
Goal
Long
AVG
UniVLA VLA
89.7
90.2
85.1
78.3
85.8
DreamDojo VLA
94.6
94.1
90.8
85.7
91.3
MVP-LAM VLA
95.4
94.8
91.6
87.9
92.4
GDLAM VLA (ours)
96.8
96.2
93.5
91.1
94.4
Appendix
Table 11: LIBERO benchmark. Success rate (%) for VLAs pretrained on the same Bridge V2 + OXE mixture.
Figure 11: Real-robot deployment on an AgiBot G1 in the desk scene.
Split
Arm
Instruction
g1_pen
L
The left arm picks up the pen on the desk, and the left arm places the pen into the pen holder at the top of the storage box.
g1_eraser
R
The right arm picks up the eraser on the desk, and the right arm places the eraser on the third layer of the storage box from the bottom.
g1_correction
L
The left arm picks up the correction fluid on the desk, and the left arm places the correction fluid on the third layer of the storage box from the bottom.
g1_charger
R
The right arm picks up the charger on the desk, and the right arm places the charger on the third layer of the storage box from the bottom.
g1_pencil
L
The left arm picks up the pencil case on the desk, and the left arm places the pencil case on the second layer of the storage box from the bottom.
g1_stapler
R
The right arm picks up the stapler on the desk, and the right arm places the stapler on the second layer of the storage box from the bottom.
Appendix
Table 12: Language instructions for the eight desktop subtasks in g1_all_realbot . Arm denotes the acting arm (L/R).
Figure 12: Visualization results of the real-robot policy executions. The top two rows show successful task executions, while the bottom row presents examples of failed executions.
Baseline
LAPA
Moto
UniVLA
DreamDojo
GDLAM
Subtask
S
M
S
M
S
M
S
M
S
M
S
M
Pen (L)
20
5
20
0
20
5
25
5
25
5
30
10
Eraser (R)
25
5
25
5
30
5
30
10
35
10
40
15
Correction (L)
15
0
15
0
15
0
20
5
20
5
25
10
Charger (R)
60
35
55
35
60
40
60
45
60
45
65
50
Pencil case (L)
60
45
55
45
60
50
60
55
65
55
65
60
Appendix
Table 13: Real-robot success rate (%) on the eight desktop subtasks. S: per-task policies (single-task); M: one policy post-trained on the mixture of all eight subtask splits and evaluated per subtask (multi-task). Best per column in bold .
Variant
DCI ↑
MI(KSG) ↑
Full model (default)
0.88
1.24
w/o Lmx
0.61
1.21
w/o Lge
0.74
1.22
w/o Lgs
0.80
1.23
w/o Lorth
0.83
1.23
Appendix
Table 14: Leave-one-out ablation of the four disentanglement objectives (one term zeroed at a time; all else at default).
Variant
DCI ↑
MI(KSG) ↑
Full model (default)
0.88
1.24
shared (non-isolated) bottleneck
0.64
1.20
uniform gate (no spatial gate)
0.71
1.22
w/o anti-leak augmentation
0.82
1.06
Appendix
Table 15: Ablation of architectural mechanisms and the anti-leak augmentation.
N
DCI ↑
MI (KSG) ↑
Rel. step time
1 (Single-code)
0.48
0.94
0.81 ×
2
0.69
1.10
0.89 ×
4 (default)
0.88
1.24
1.00 ×
8
0.85
1.22
1.34 ×
Appendix
Table 16: Effect of the number of groups N on disentanglement, information content, and training cost.
Variant
DCI ↑
MI(KSG) ↑
Leakage ↓
Full model (default)
0.88
1.24
0.09
all weights ×0.5
0.85
1.23
0.12
all weights ×2.0
0.86
1.19
0.08
no static-group
0.86
1.23
0.19
Appendix
Table 17: Sensitivity to disentanglement-weight scaling and to the static-group designation. “Leakage” is the fraction of dynamic-group gate mass falling on the static background ( ↓ ).
Latent Action Models (LAMs) enable the learning of world models from unlabeled video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in video generation quality, action transfer, visual planning, and manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.
Tianqiu Zhang, Muyang Lyu, Yufan Zhang +2
Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, IDG/McGovern Institute for Brain Research, Peking University. · Center of Quantitative Biology, Peking University. · School of Psychological and Cognitive Sciences, Key Laboratory of Machine Perception (Ministry of Education), Peking University.
Vision-language-action (VLA) models remain constrained by scarce action-labeled robot data, whereas action-free videos offer abundant observations of physical change. Latent action models can extract such priors, but reconstruction-trained codes may predict future observations without the structure required for joint generation with robot actions. Existing structured methods add temporal constraints but retain deterministic transition points, so residual errors in locally inferred transitions may propagate and compound under recursive composition. We introduce DLAM, a distributional latent-action model that represents each transition as a diagonal Gaussian. Reconstruction conditioned on the reference frame grounds the mean in observed visual change, while normalized composition and reversal over equal-gap triplets constrain both the mean and dimension-wise variance. Variance composition uses a lightweight shared-correlation coefficient to account for dependence between adjacent transitions that share an intermediate frame, whereas reversal negates the mean and preserves the variance. For downstream policy learning, we freeze the encoder and train a flow-matching policy to jointly generate mean transition sequences and robot actions. On held-out transitions, DLAM learns more temporally consistent latent dynamics than existing latent-action baselines and achieves stronger direct and cumulative reconstruction on held-out videos. Under the same controlled π0 transfer protocol, it also improves policy performance on MetaWorld MT50, LIBERO, and real-world manipulation tasks. Controlled ablations show that normalized mean constraints account for most of the reconstruction gain, while learned variance and correlation-aware composition provide complementary improvements in downstream control.
Zuojin Tang, Feifan Luo, Haoyun Liu +10
Zhejiang University · Nanjing University · Shanghai Jiaotong University +4
Learning large-scale vision-language-action (VLA) models from multi-embodiment datasets remains challenging due to heterogeneous action spaces across end effectors. Although latent action models (LAMs) can learn embodiment-agnostic action representations from diverse video data, existing image-based LAMs often fail to capture fine-grained end-effector articulation, particularly finger-level geometric changes in human and dexterous robot hands. To address this limitation, we propose GALA, a Geometry-Aware Latent-Action modeling framework that augments image-based latent actions with 3D end-effector geometric motion. However, naively incorporating point clouds yields fine-grained action representations with limited shared semantics, hindering cross-embodiment pretraining. To address this issue, we introduce the Unified End-effector Motion Representation (UEMR), which preserves fine-grained motion information while improving the cross-embodiment generalizability of latent actions. Building upon UEMR, GALA combines visual latent actions that capture scene-level dynamics with geometric latent actions that capture shared fine-grained end-effector articulation, providing effective supervision for VLA pretraining from multi-embodiment data, including action-free ego-centric human videos. Experiments on fine-grained motion probing, cross-embodiment retrieval, and downstream VLA evaluation demonstrate GALA's effectiveness in modeling generalizable fine-grained motions across embodiments, achieving 68.3% RoboCasa-GR1 success rate and 75.5% real-world success rate. Code, appendix, and demos are available at https://puzhenyuan.github.io/GALA-website/.
Yichen Liu, Puzhen Yuan, Xiang Zhu +2
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China