World models that support planning, control, and interpretable reasoning require latent representations that genuinely encode action. Yet the large-scale video data from which such models are learned overwhelmingly lack action annotations. Latent-action models infer action-like variables from observation sequences, enabling action-aware learning from unlabeled videos or trajectories. A fundamental concern, however, remains: a latent that improves future prediction may simply encode the current state, episode progress, task identity, or other predictive shortcuts. Existing evaluation practices---reconstruction loss, rollout fidelity, action recovery, or downstream performance---cannot isolate this distinction. We introduce the Latent Action Support Ratio (LASR), a dimensionless metric that measures, under offline evaluation on held-out data, whether a learned latent provides predictive support for action-related targets beyond the current state and non-action alternatives. LASR provides two complementary summaries: GeoLASR captures typical support, while WorstLASR identifies the weakest case. We instantiate the LASR benchmark with recorded actions and corresponding physical state changes as targets, episode-held-out splits, one predictor type for all comparisons, and positive and negative controls. Experiments on recorded robot manipulation data and a matched multi-task panel demonstrate that LASR detects known action information, rejects representative non-action shortcuts, and enables graded comparisons among imperfect latent representations. We further use LASR to train and audit a lightweight residual adapter, obtaining descriptively better point estimates whose uncertainty is assessed with paired comparisons. LASR thus provides a principled offline protocol for measuring action-related predictive support in world-model latents.
Figures & tables
Figure 1: Two shortcuts in offline latent-action evaluation. (a) A trajectory from the ALOHA static-battery release illustrates how the current state and task phase constrain the next motion [ 51 ] . (b) DROID-100 observations from distinct scenes, viewpoints, and object layouts illustrate how scene and episode identity may help predict typical transitions without isolating action-specific information [ 24 ] .
Method
Reported evaluation metrics
LAPA [ 44 ]
Task success and task score; qualitative latent-token and rollout analyses
UniVLA [ 5 ]
Manipulation success, step-wise real-robot score, and navigation oracle success
MVP-LAM [ 28 ]
Estimated I(Z;A) , linear action-prediction NMSE, DINOv2 feature-space reconstruction MSE under viewpoint perturbation, and task success
DiLA [ 49 ]
Rollout SSIM/LPIPS, cycle-transfer LPIPS, linear robot-action prediction MSE, and visual-planning success
LaWAM [ 7 ]
LIBERO, RoboTwin, and real-robot success, action-chunk latency, and component-ablation success
LASR (ours)
LASR scores by target, GeoLASR, WorstLASR, and paired uncertainty
Table 1: Representative metrics used to evaluate latent-action and world-action models.
Figure 2: Overview of LASR. (A) A logged transition aligns the current state St , recorded action At , and next state St+1 . (B) For each target–fold pair, separately fitted state, latent, and validation-selected control routes yield held-out risks Estate , EZ , and Ectrl . (C) LASR divides EZ by the smaller of the two reference risks. A score below one means that the latent route outperforms both references; GeoLASR and WorstLASR are the geometric mean and maximum across target–fold scores, respectively. Lower is better.
Original ALOHA panel
Matched multi-task panel
Representation
ALOHA dim.
Geo ↓
Worst ↓
Geo ↓
Worst ↓
Branch acc. (%) ↑
DiLA z
256
0.750
0.796
0.688
0.719
65.58
LaWAM μz
32
0.891
0.947
0.860
0.870
48.59
LAPA LAQ
32
0.922
0.966
0.831
0.849
50.39
UniVLA zq
512
1.009
1.077
1.010
1.049
25.44
MVP-LAM zq
512
1.095
1.157
0.988
0.993
26.75
Table 2: LASR calibration and model comparison. Original ALOHA and matched-panel results; full uncertainty and target–fold results are in Appendix Sec. H.1. Lower LASR is better; one is the operational boundary. Matched learned routes use eight-dimensional train-only projections. Bold marks the best ranked learned point estimates. † Unranked positive reference. Branch accuracy is common-decoder one-step physical replay (state-only/chance: 25%) and is not reported for the recorded-action reference. Further audits are in Appendix Secs. H.1–H.6.
(a) Representation diagnostics
Variant
Dim.
GeoLASR ↓
WorstLASR ↓
Native DiLA
256
0.750
0.796
PCA
16
0.954
0.980
Bottleneck
16
0.849
0.877
Bottleneck
8
0.913
0.949
Progress control
1
1.000
1.000
Table 3: DiLA diagnostics, adapter ablations, and world-model prediction. Identity denotes unmodified DiLA; the model in (c) uses the LASR residual adapter in (b). Arrows show preferred directions, and bold marks the best non-oracle point estimates. † Recorded-action GeoLASR is 1.19×10−7 (shown as 0).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: ALOHA data used to construct the analysis panel. (a) Four frames from a trajectory in the static-battery release illustrate task progression. (b) Synchronized observations from the high, low, and left-wrist cameras at one time step. All image-based representations evaluated in this paper are extracted from the aligned cam_high stream.
Action target
State-change target
Representation
Geo (Worst)
Geo (Worst)
DiLA
0.735 (0.759)
0.767 (0.796)
LaWAM
0.887 (0.934)
0.895 (0.947)
LAPA ∗
0.921 (0.966)
0.923 (0.956)
UniVLA
1.011 (1.077)
1.007 (1.066)
MVP-LAM
1.098 (1.157)
1.092 (1.137)
Appendix
Table 4: Target-specific summaries over five folds. Each entry is GeoLASR with WorstLASR in parentheses. The main paper combines the two targets into ten target–fold scores. ∗ LAPA retains the low-diversity warning described in the text.
Ordinary
Enriched
Representation
Geo
Worst
Geo
Worst
DiLA
0.750
0.803
0.752
0.840
LaWAM
0.882
0.908
0.897
0.981
LAPA ∗
0.920
0.942
0.924
0.999
UniVLA
1.014
1.128
1.005
1.044
MVP-LAM
1.090
1.223
1.100
1.244
Appendix
Table 5: Descriptive results for the two fixed panel halves. Predictors are still fitted on the full fold; only the held-out scoring rows are separated. These strata do not affect model selection or the main ranking.
Representation
Main
Ridge sweep
32-D check
DiLA
0.750
all <1
<1
LaWAM
0.891
all <1
<1
LAPA ∗
0.922
all <1
<1
UniVLA
1.009
all ≥1 ; †
≥1 ; †
MVP-LAM
1.095
all ≥1
≥1 ; †
Appendix
Table 6: GeoLASR sensitivity relative to the boundary at one (lower is better). Main uses ridge 10−6 ; the sweep covers {0,10−8,10−6,10−4,10−2} . The entries all <1 and all ≥1 mean that every sweep value lies on the stated side; 32-D reports the projected status. A dagger marks a finite score orders of magnitude above both one and the representation’s main score, indicating numerical instability of the evaluator under that check. It is not a new LASR threshold; only boundary status is interpreted. Scores are computed without clipping, and no check selects the main setting.
Mode
Signal
Dim.
Geo
Worst
None
Identity
256
0.750
0.796
Residual
LASR
256
0.745
0.785
Residual
MSE
256
0.759
0.810
Residual
Random
256
0.762
0.815
Residual
Shuffled
256
0.761
0.834
Replacement
LASR
256
0.770
0.855
Appendix
Table 7: Full adapter ablation. LASR denotes the LASR-inspired surrogate. Lower is better. Random and shuffled residual adapters retain the raw-latent skip connection, so they test the training signal rather than a purely random representation.
Comparator
Δ GeoLASR [95% CI]
Raw latent
−0.00563[−0.01852,0.00713]
Identity residual
−0.00563[−0.01843,0.00745]
Random residual
−0.01804[−0.02978,−0.00574]
Shuffled residual
−0.01617[−0.02810,−0.00356]
MSE residual
−0.01466[−0.02527,−0.00365]
Appendix
Table 8: Paired episode-bootstrap differences: LASR-guided residual adapter minus comparator. Negative values favor the adapter. The paired estimates are not obtained by subtracting the separately aggregated rows in Table 7 .
Model
Step
DINO ↓
SSIM ↑
LPIPS ↓
Original DiLA
1
0.389
0.818
0.302
Adapted DiLA
1
0.378
0.824
0.288
Original DiLA
15
0.521
0.828
0.339
Adapted DiLA
15
0.494
0.830
0.324
Appendix
Table 9: World-model prediction with the LASR-guided residual adapter from the main paper ablation. The world-model predictor is frozen. Bold marks the better point estimate at each horizon.
Route
GeoLASR [95% CI]
WorstLASR [95% CI]
Support cells
Branch acc. (%)
DiLA native 16-frame
0.688 [0.668,0.708]
0.719 [0.697,0.771]
10/10
65.58
DiLA pair-only
0.685 [0.665,0.704]
0.714 [0.694,0.767]
10/10
–
LAPA tokens
0.831 [0.810,0.851]
0.849 [0.831,0.908]
10/10
50.39
LaWAM
0.860 [0.838,0.883]
0.870 [0.859,0.933]
10/10
48.59
MVP-LAM
0.988 [0.981,0.995]
0.993 [0.988,1.013]
10/10
26.75
UniVLA controllable
1.010 [0.998,1.025]
1.049 [1.003,1.126]
4/10
25.44
Appendix
Table 10: Matched-panel results. Branch accuracy executes the common probe’s held-out decoded action from the exact saved source state; chance and the state-only route are 25%. The next-frame row is an outcome-leakage counterexample rather than a valid executable-action positive. † Unranked positive reference.
Task
DiLA
LAPA
LaWAM
MVP
UniVLA
PickCube
.632
.737
.769
.967
1.042
PullCube
.596
.735
.734
.970
1.008
PushCube
.597
.729
.739
.975
1.007
StackCube
.607
.749
.763
.994
.998
PegInsertionSide
1.015
1.214
1.312
1.036
.992
Appendix
Table 11: Task-stratified GeoLASR. The pooled result is not universal: PegInsertionSide is a clear failure case for DiLA, LAPA, and LaWAM.
Ridge
RFF-64
Small MLP
Route
Geo
Worst
Geo
Worst
Geo
Worst
DiLA
.688
.719
.673
.708
.671
.699
LAPA
.831
.849
.793
.815
.763
.783
LaWAM
.860
.870
.830
.859
.794
.818
MVP-LAM
.988
.993
.989
.995
.991
.998
UniVLA
1.010
1.049
196.969
2.86×1011
1.002
1.011
Appendix
Table 12: Probe-family sensitivity. UniVLA’s RFF result is a preserved numerical instability, not evidence of an intrinsic model score.
Route
Geo [95% CI]
Worst [95% CI]
Native hidden mean
.7747 [.7643,.7849]
.8418 [.8348,.8681]
Native token one-hot
1.0402 [1.0302,1.0497]
1.0765 [1.0635,1.1041]
Recorded action
2.87×10−11
.3349 [.3212,.3549]
Matched random
1.1270 [1.1161,1.1373]
1.1819 [1.1659,1.2085]
Within-episode shuffle
1.1210 [1.1102,1.1311]
1.1738 [1.1577,1.2001]
Current-frame visual
1.0093 [.9999,1.0183]
1.0439 [1.0318,1.0730]
Appendix
Table 13: UniVLA LASR on its official LIBERO-spatial rollout distribution.
Stress
Geo 0→1
Δ Geo [95% CI]
Worst 0→1
Flips
Camera
.7444 → .7625
+.0181 [ −.0136,+.0501 ]
.7769 → .8166
0/10
Outcome
.7442 → .9888
+.2446 [+.2051,+.2844]
.7739 → 1.0018
2/10
Appendix
Table 14: Registered paired interventions. The camera condition changes the rendering while holding the physical transition fixed. The outcome condition holds history, current state/frame, action, and action-induced targets fixed, then changes the realized post-transition object position before final rendering.
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with 6× less training cost and 13.9× faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to 5.5× faster inference.
Junha Song, Geonmo Gu, Jeongeun Park +5
NAVER AI Lab · Georgia Institute of Technology · Brown University +1
Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-relevant structure in their latent spaces. We study this question through a unified probe-based evaluation across diverse encoder families, including image-only self-supervision, video pretraining with and without latent prediction, reconstruction-based autoencoders, diffusion models, and shortcut-forcing dynamics models. Using a common inverse-dynamics probing objective, we find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity: models with strong pixel decoding quality can exhibit near-zero action recoverability, while video-pretrained self-supervised encoders consistently achieve the best Pareto trade-off between visual fidelity and action prediction. Comparing V-JEPA and VideoMAE further shows that most gains arise from natural-video temporal context, with feature-level latent prediction providing a smaller additional benefit. These trends transfer across robotic benchmarks, though CALVIN reveals that static-environment tasks can partially mask the importance of temporal structure by allowing strong image priors to suffice. Finally, inverse-dynamics supervision substantially improves robustness to visual corruption, suggesting that action-aware objectives regularize latent geometry beyond clean-setting performance. Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.
Jewon Yeom, Hanseul Kim, Jeongjae Park +3
Graduate School of Data Science, Seoul National University
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.