La-JEPA: Compact Robot Policies with Latent World Modeling

Mateo Clémente; George Z. Xue; Amin Abyaneh; Liheng Ma; Rui Heng Yang; Tongtong Cao; Zhanguang Zhang; Yingxue Zhang

cs.RONov 5, 2026

La-JEPA: Compact Robot Policies with Latent World Modeling

Authors: Mateo Clémente, George Z. Xue, Amin Abyaneh, Liheng Ma, Rui Heng Yang, Tongtong Cao, Zhanguang Zhang, Yingxue Zhang

Organizations: Huawei Technologies Ltd. · University of Toronto · McGill University · Mila – Quebec Artificial Intelligence Institute

Abstract

World-action models adapt pretrained video generators, and vision-language-action models build on pretrained vision-language backbones; both tie policy architecture and inference cost to large transformers designed for generation rather than control. It remains unclear how much these generation-oriented choices affect control. In this work, we introduce Latent Action JEPA (La-JEPA), a compact world-action model whose policy transformer is trained from scratch in the feature space of a pretrained V-JEPA 2.1 encoder. Following Fast-WAM, La-JEPA learns to predict future features jointly with actions during training and generates only actions at inference. Among policies without embodied pretraining, La-JEPA achieves state-of-the-art performance in clean and randomized RoboTwin 2.0 scenes and on LIBERO-Plus with correct instructions, while using far fewer parameters than the strongest competing policies. It runs substantially faster than Fast-WAM at comparable performance and outperforms the closest JEPA-based methods, LeapBot-WA and JEPA-WAM, on LIBERO-Plus and RoboTwin 2.0 without their pretrained policy backbones. On four real-robot tasks, its average success is comparable to pi0.5 and above non-embodied-pretrained baselines. Through a design study of the choices behind La-JEPA, we show that JEPA features outperform video VAE latents, most clearly out of distribution, and that training-time future prediction choices significantly impact the policy even when future prediction is skipped at inference. We correct the LIBERO-Plus instructions, as the officially released evaluator corrupts them for most variants, and re-evaluate several strong open baselines. Without a generative pretrained policy backbone, La-JEPA is better and faster than far larger policies, suggesting that robot foundation models may benefit from architectures designed for control rather than inherited from generation.

Figures & tables

Figure 1: Success against deployed parameters, including frozen encoders, on LIBERO-Plus with corrected instructions (Section 5.2 ) and on RoboTwin 2.0, where the panel reports the equally weighted mean of clean and randomized success of clean-trained policies. S, B, and L denote the La-JEPA sizes. The LIBERO-Plus panel shows the policies without embodied pretraining that we re-evaluated with corrected instructions, and the embodied-pretrained

\pi_{0.5}

; Table 1 lists all scores, including those with the released instructions.

*

: success estimated from JEPA-WAM’s 20-task results (Appendix B ).

Figure 2: La-JEPA. Left: general architecture. Right: context and future encoding with V-JEPA 2.1.

Figure 3: Gain in success (percentage points) from each design choice over its baseline: A, action-only training; B, Wan2.2 VAE latents; C, static encoding; D, no camera dropout (LIBERO: wrist-only dropout); E, no proprioception dropout; F, a plain policy without augmentation or encoder LoRA. Green: LIBERO; orange: RoboTwin 2.0; solid: in-distribution; hatched: out-of-distribution. Block A averages LIBERO-Plus gains over three training seeds. Values and evaluation budgets in Table 6 .

		Params (B)			LIBERO-Plus
Method	Transformer init.	Train.	Depl.	LIBERO	Corrected	Released
Qwen-RobotManip-scratch	VLM (supervised)	4.7	4.7	98.2	—	78.3
StarVLA	VLM (supervised)	7.8	7.8	96.9	77.6	76.7
JEPA-WAM	VLM (supervised)	0.55	1.35	96.7	77.8	77.1
Fast-WAM	video generator	6.0	12.4	97.6	70.8	51.5
Faster-WAM	video generator	5.1	11.4	98.5	—	75.1

Table 1: LIBERO and LIBERO-Plus success (%) of policies without embodied pretraining (all use pretrained perception). Parameters in billions. LIBERO-Plus: episode-weighted success over all 10,030 variants with corrected or released instructions (Section 5.2 ); corrected-instruction scores exist only for the policies we re-evaluated. Best in bold, second best underlined unless the best is tied. La-JEPA values are means over three training seeds (Appendix B ). § Scored at the upstream step limits used for all policies; with its own longer limits (400 steps, 700 for LIBERO-Long), NowWAM reports 87.7% on the released instructions.

		Params (B)		RoboTwin 2.0 success (%)
Method	Transformer init.	Train.	Depl.	Clean	Randomized	Mean
Qwen-RobotManip-scratch	VLM (supervised)	4.7	4.7	71.6	22.6	47.1
StarVLA	VLM (supervised)	7.8	7.8	58.1	10.6	34.4
Fast-WAM	video generator	6.0	12.4	77.8	1.9	39.9
Faster-WAM	video generator	5.1	11.4	68.6	2.2	35.4
La-JEPA S (ours)	scratch	0.11	0.75	85.5	33.9	59.7

Table 2: RoboTwin 2.0 success of clean-trained policies without embodied pretraining, in clean and randomized scenes. Parameters in billions, for the RoboTwin configurations. Mean: equally weighted mean of clean and randomized success. Best in bold, second best underlined. La-JEPA values are means over three training seeds, each evaluated on 5,000 episodes per condition (Appendix B ).

			Params (B)		LIBERO		RoboTwin 2.0 (20 tasks)
Method	Transformer init.	Embodied PT	Train.	Depl.	IID	Plus	C2C	C2R
LDA-1B ‡	—	yes	—	—	90.6	45.5	—	—
LeapBot-WA	video generator	yes	2.7	10.4	97.3	71.7	—	—
JEPA-WAM	VLM (supervised)	no	0.55	1.35	96.7	77.1	79.9	36.9
La-JEPA S (ours)	scratch	no	0.10	0.74	97.9	80.0	95.8	49.9
La-JEPA B (ours)	scratch	no	0.34	0.98	98.0	82.5	95.9	49.9

Table 3: Latent world-action models. Embodied PT: pretraining on robot data beyond the target demonstrations. ‡ Reported by Yan et al. (2026b) . La-JEPA LIBERO and LIBERO-Plus values are means over three training seeds (Appendix B ).

Figure 4: RoboTwin clean versus randomized success of clean-trained policies. Blue: La-JEPA B; brown: no embodied pretraining; gray: embodied pretraining; RoboManip: Qwen-RobotManip. Points above the dashed line through La-JEPA have a higher mean of the two regimes.

*

: JEPA-WAM estimate (Appendix B ); LDA-1B as reported by Yan et al. (2026b) .

Source	Camera viewpoint (LIBERO-Object)	Lighting (LIBERO-Spatial)
Training, corrected (ours)	pick up the alphabet soup and place it in the basket	pick up the black bowl between the plate and the ramekin and place it on the plate
Released	…in the basket view 0 0 100 2 4 initstate 0	…place it on the plate light 1
Flex- $\pi$	Pick the alphabet soup …	Pick the akita black bowl …

Table 4: Instruction received for two LIBERO-Plus variants. Bold: deviations from the training instruction.

Panel of Figure 5 — Figure 5: Real-robot tasks: cube placement, fruit picking, Push T, and dish manipulation.

Policy	Latency (ms)	Fruit	Cube	Push T	Dish manipulation				Average
					Plates	Utensil 1	Utensil 2	Position
With embodied pretraining
$\pi_{0.5}$	71.89	20	20	11	20	19	19	18	17.25
LingBot-VA	$\approx 4760$	18	20	2	6/10	3/10	2/10	1/10	—
Without embodied pretraining
Fast-WAM	88.09	19	19	14	20	19	15	12	16.00

Table 5: Real-robot successes out of 20 and benchmark policy-call latency in ms, not control-loop timing (Appendix C ); LingBot-VA’s is rounded to 10 ms. Position: full dish success with correct utensil placement; Average: mean over the four tasks. LingBot-VA dish: 10 trials. Bold: best.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Choice	Baseline	Benchmark	IID	OOD	Evaluation
A Future prediction
	Action + future prediction	Action only	LIBERO	$+$ 0.9	$+$ 2.1	IID: seed 42; OOD: 3 seeds
	Action + future prediction	Action only	RoboTwin	$+$ 7.0	$+$ 3.7	$5{,}000$ /cond., 120–150k
B V-JEPA features
	V-JEPA 2.1 features	Wan2.2 VAE latents	LIBERO	$+$ 2.0	$+$ 6.0	$2{,}000$ / $10{,}030$ , 150k
	V-JEPA 2.1 features	Wan2.2 VAE latents	RoboTwin	$+$ 15.9	$+$ 16.0	$5{,}000$ /cond., 150k

Appendix

Table 6: Gain in success (points) of each design choice over its baseline; comparisons hold within a block.

Seed	Selected step	Joint	Action only	Gain (pp)	Paired $p$
42	150k	76.13	74.47	$+$ 1.67	$2.3\times 10^{-4}$
1	90k	76.13	74.77	$+$ 1.37	$0.002$
2	120k	77.15	73.82	$+$ 3.33	$2.2\times 10^{-12}$
Mean	—	76.47	74.35	$+$ 2.12	—

Appendix

Table 7: Predictive supervision across training seeds on LIBERO-Plus (success in %; per-seed exact McNemar tests).

Frozen encoder, stored features
Training variant	Camera viewpoints	Sensor noise
Wrist-camera dropout only	11.4	42.0
Either-camera dropout	43.8	67.1
Online encoder, either-camera dropout
Image augmentation only	74.6	81.4
Encoder LoRA only	43.9	73.3

Appendix

Table 8: LIBERO-Plus success under camera-viewpoint and sensor-noise perturbations (%).

Policy size	S	B	L
Width / layers / heads	384 / 12 / 6	768 / 12 / 12	1024 / 24 / 16
LoRA blocks (LIBERO)	Last 8	Last 8	All 24
LoRA blocks (RoboTwin)	Last 8	Last 8	Last 8
Transformer / losses	Equal World/Action DiT widths and depths; MLP expansion 4; action and future loss weights 1 each
Optimizer	AdamW; policy LR $5\times 10^{-4}$ ; betas $(0.9,0.95)$ ; matrix weight decay 0.01; gradient clipping 1
Precision / EMA	BF16 mixed precision; EMA decay 0.9999

Appendix

Table 9: Architecture, training, and inference settings.

Operation	Probability	Parameters and scope
Unmodified window	0.3	Bypass all observation augmentation
Photometric	0.6	All cameras share brightness/contrast/saturation strength 0.3, hue 0.03 turns, gamma $[0.8,1.25]$
Corruption	0.5	Primary camera; uniform choice of Gaussian blur ( $\sigma\in[0.5,6]$ pixels), motion blur (length $[5,20]$ pixels), or fog
Fog	Within corruption	Opacity $[0.2,0.6]$ ; contrast multiplier $[0.5,0.9]$ ; grid size 4–8; gray fraction 0.5–1.0
Rotation	0.5	Primary camera; yaw/pitch $[-4,4]^{\circ}$ , roll $[-3,3]^{\circ}$ ; homography $KRK^{-1}$ , replicated borders; vertical FOV $45^{\circ}$ (LIBERO), $37^{\circ}$ (RoboTwin)

Appendix

Table 10: Observation augmentation. Probabilities are conditional on not bypassing augmentation; fog is one of three corruption types.

	LIBERO-Plus categories (revised)							LIBERO-Plus		LIBERO
Size	Camera	Robot	Language	Light	Backgr.	Noise	Layout	Revised	Orig.	Revised	Orig.
S	80.4	58.3	89.1	95.5	66.0	89.6	80.3	80.0 $\pm$ 0.4	75.2	97.9 $\pm$ 0.1	93.4
B	85.0	58.0	91.0	96.7	71.4	93.2	82.1	82.5 $\pm$ 0.4	78.9	98.0 $\pm$ 0.2	96.4
L	86.4	61.4	92.1	97.4	74.9	94.6	82.1	84.1 $\pm$ 0.4	82.3	98.1 $\pm$ 0.2	97.0

Appendix

Table 11: Revised versus original LIBERO policies (success in %). Revised: final 150k checkpoint, mean over three training seeds

\pm

sample standard deviation across seeds; LIBERO-Plus categories are three-seed means. Original: one training run at 120k updates. Both use the upstream suite-specific step limits; the original LIBERO values are counted from recorded rollouts without settling steps, whereas the revised evaluations take 10 settling steps before the policy acts.

	Clean		Randomized		Mean
Size	Revised	Orig.	Revised	Orig.	Revised	Orig.
S	85.5 $\pm$ 0.5	80.2	33.9 $\pm$ 2.2	39.0	59.7 $\pm$ 1.3	59.6
B	86.2 $\pm$ 0.5	78.8	35.4 $\pm$ 0.6	44.7	60.8 $\pm$ 0.5	61.8
L	86.0 $\pm$ 0.4	78.4	32.1 $\pm$ 2.8	39.1	59.0 $\pm$ 1.6	58.7

Appendix

Table 12: Revised versus original RoboTwin 2.0 policies (clean-trained; success in %, 5,000 episodes per condition). Revised: mean over three training seeds

\pm

sample standard deviation across seeds. Original: one training run.

Source	Example instruction	Match
LIBERO training data	pick up the black bowl between the plate and the ramekin and place it on the plate	—
Released LIBERO-Plus	pick up the black bowl between the plate and the ramekin and place it on the plate light 1	0
Flex- $\pi$ (BDDL annotation)	Pick the akita black bowl between the plate and the ramekin and place it on the plate	2,351
Ours (corrected)	pick up the black bowl between the plate and the ramekin and place it on the plate	8,493

Appendix

Table 13: Instruction sources for LIBERO-Plus. Example: the first LIBERO-Spatial task under the first lighting perturbation. Match: non-language variants whose instruction equals the training instruction, out of 8,493.

Policy	Embodied PT	Released	Corrected	$\Delta$
La-JEPA S (ours)	no	80.0	86.3	$+6.3$
La-JEPA B (ours)	no	82.5	87.3	$+4.9$
La-JEPA L (ours)	no	84.1	88.0	$+3.9$
NowWAM	no	86.2	86.2	$+0.0$
Cosmos Policy §	no	80.8	84.9	$+4.1$
ImageWAM	no	79.7	80.4	$+0.7$

Appendix

Table 14: LIBERO-Plus success (%) with the released and the corrected instructions, all evaluated in our harness on all 10,030 variants (MuJoCo 3.3.2, seed 0, first initial state, 10 settling steps, upstream per-suite step limits).

\Delta

: corrected minus released. Best in bold, second best underlined. La-JEPA values are means over three training seeds. § One variant cannot be instantiated by LIBERO-Plus, so

n=10{,}029

. ¶ Run-to-run variation of about

\pm 0.5

points.

\pi

-JEPA-WAM:

\pi_{0.5}

trained with the auxiliary JEPA objective of JEPA-WAM ( Lin et al., 2026 ) .

Policy	Steps	Optimized	Original-kernel CUDA graphs	Eager
La-JEPA S	10	17.78	34.20	99.96
La-JEPA B	10	25.16	42.35	98.21
La-JEPA L	10	59.72	88.87	167.93
Fast-WAM	10	88.09	OOM	193.44
Faster-WAM	10	50.96	OOM	62.70
$\pi_{0.5}$	10	71.89	—	—

Appendix

Table 15: Policy-call latency (ms): batch size one, median of 100 calls after 20 warm-ups on one consumer GPU. OOM: out of memory; dashes: not measured. The timed L uses last-eight-block rather than full-depth LoRA, which does not change latency.

Method	Trained components	Frozen components at inference	Trainable	Deployed
Qwen-RobotManip-scratch	Qwen3.5-4B, action head	—	4.7 e	4.7 e
StarVLA	Qwen3-VL-4B (4.44), cross-attention DiT action head (3.36)	—	7.79 m	7.79 m
Fast-WAM	Wan2.2 video DiT (5.00), action DiT (1.02)	UMT5-XXL encoder (5.68), Wan2.2 VAE (0.71)	6.02 m	12.4 m
Faster-WAM	Wan2.2 video DiT (5.00), action head (0.05)	UMT5-XXL encoder (5.68), Wan2.2 VAE (0.71)	5.05 r	11.4 e
Cosmos Policy	Cosmos-Predict2-2B DiT (1.96)	T5-11B encoder (4.87), Wan2.1 VAE (0.13)	1.96 m	6.95 m
ImageWAM	FLUX.2 4B image DiT (3.86), action DiT (0.64)	Qwen3-4B encoder (4.02), VAE (0.08)	4.52 m	8.62 m

Appendix

Table 16: Parameter inventory (billions); La-JEPA rows for LIBERO. m: measured, r: reported, e: estimated.

Figure 6: Companion to Figure 1 : in-distribution (top) and out-of-distribution (bottom) success against trainable parameters on a logarithmic axis, for LIBERO (left) and clean-trained RoboTwin 2.0 (right). Trainable counts every weight updated on the target demonstrations, adapters included. La-JEPA points are blue and connected in size order; purple denotes VLM-initialized and brown generator-initialized action transformers, while other scratch policies use muted blue. SmolVLA ( Shukor et al., 2025 ) is shown at its three sizes joined by a dotted line. Unlike Figure 1 , LIBERO-Plus uses the released instructions, for which scores of all baselines are available.

	With embodied pretraining		Without embodied pretraining
Task	$\pi_{0.5}$	LingBot-VA	Fast-WAM	Faster-WAM	La-JEPA
Fruit	24.4 $\pm$ 2.7	175.3 $\pm$ 41.0	36.7 $\pm$ 9.0	35.1 $\pm$ 9.5	21.4 $\pm$ 0.8
Cube	10.8 $\pm$ 0.5	117.8 $\pm$ 50.9	21.9 $\pm$ 7.6	18.2 $\pm$ 4.6	11.5 $\pm$ 3.2
Push T	22.8 $\pm$ 14.7	112.5 $\pm$ 9.1	24.6 $\pm$ 6.6	23.8 $\pm$ 12.1	21.1 $\pm$ 4.1
Dish	39.0 $\pm$ 3.4	269.0 †	85.0 $\pm$ 15.3	72.0 $\pm$ 8.8	58.1 $\pm$ 8.7

Appendix

Table 17: Real-robot completion time over successful trials (s; standard deviation in small type). Bold: fastest; underlined: second. † Single successful trial.

	S		B		L
Task	Clean	Rand.	Clean	Rand.	Clean	Rand.
Adjust bottle	100.0	21.0	100.0	18.7	99.7	15.7
Beat block hammer	91.3	17.7	94.7	20.7	94.0	21.0
Blocks ranking rgb	95.7	35.7	95.0	43.7	97.0	43.3
Blocks ranking size	67.3	29.3	69.3	34.0	66.7	29.7
Click alarmclock	99.7	47.7	100.0	68.0	99.7	64.3

Appendix

Table 18: Per-task RoboTwin 2.0 success of La-JEPA S, B, and L (%), mean over three training seeds with 100 episodes per task, condition, and seed.

Model	Evaluation setting	Value	Success (%)
S	Reference	Default	94.30
S	Denoising steps	3	94.50
L	Reference	Default	97.40
B	Reference	Default	97.60
L	Execution chunk	5	97.00
L	Execution chunk	8	96.30

Appendix

Table 19: Evaluation-setting sweeps at fixed checkpoints (success in %). LIBERO values use the 700-step budget in every suite (Appendix B ); B’s LIBERO reference is a re-evaluation of its default setting.

Future pooling	IID	OOD
$2\times$ head, $4\times$ wrists	61.24	15.68
$4\times$ head, $8\times$ wrists	61.96	18.56
Global spatial pooling	61.14	15.04

Appendix

Table 20: Future-only pooling on RoboTwin (success in %).

Training objective	Clean success (%)	Effective rank	Action readout $R^{2}$
Action + future prediction	66.5	137.70	0.4315
Action only	62.8	48.41	0.7937

Appendix

Table 21: Representation diagnostics with and without future prediction on RoboTwin.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 389 papers

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling
Aug 10, 2026Yihan Lin, Jiawei He, Shifeng Bao +6Joint-Embedding Predictive Architecture Robotic Manipulation
School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China +3
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent Action Learning Action-Conditioned World Models
Jilin University · Zhongguancun Academy · Nankai University +4
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Sep 29, 2026Yang Zhang, Jiangyuan Zhao, Chenyou Fan +4Visual Representation Learning Visuomotor Policy Learning
Tsinghua University · Shanghai Jiao Tong University · Fudan University +4

Aug 10, 2026J/K move · Enter open · S save

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

Yihan Lin, Jiawei He, Shifeng Bao, Chen Zhao+5

School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China+3

Joint-Embedding Predictive Architecture Robotic Manipulation Vision-Language-Action Models Latent World Models Embodied World Models

Open paper AskPDF arXiv

Robust robot control benefits from explicitly modeling state transitions, but video-generation world action models (WAMs) introduce substantial deployment cost. Existing latent WAMs avoid explicit future generation, but often compress predictive representations or separate predictive modeling from the representations used for action generation. We introduce JEPA-WAM, a latent WAM built in a pretrained V-JEPA space, which couples latent transition prediction with continuous action generation through a shared predictor. JEPA-WAM predicts a spatially structured joint current-future target that captures task-shared visual temporal structure between current and future observations, while preserving dense patch-level correspondence. Through the shared predictor, transition supervision directly shapes the backbone, from which dedicated representations are extracted for action prediction. The same design can also be instantiated in pretrained VLA policies while preserving their original perception and action pathways. On LIBERO-Plus, JEPA-WAM achieves 79.2%, the best result without large-scale robot-policy pretraining, while its pretrained

π_{0.5}

instantiation reaches 86.3%, achieving the best overall performance. Experiments on RoboTwin 2.0 and real-world bimanual manipulation further demonstrate strong generalization under visual and spatial shifts.

Abstract

Robot Policy Learning

Explore similar work

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents