Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

Xiangcheng Zhan; Zirui Chen; Yicheng Zhao; Ziteng Gao; Shuo Yang

cs.AISep 29, 2026

Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

Authors: Xiangcheng Zhan, Zirui Chen, Yicheng Zhao, Ziteng Gao, Shuo Yang

Organizations: Harbin Institute of Technology · Dalian University of Technology · Southern University of Science and Technology

Abstract

World-Action Models (WAMs) couple action generation with predictions of how physical interactions unfold. However, current post-deployment learning paradigms typically improve behavior without requiring better world predictions. Especially in dexterous manipulation, small execution errors can compound in high-dimensional action spaces, hindering policy improvement and pushing interactions beyond the world model's training distribution. Motivated by this, we propose Direct Experience World-Model Optimization (DEWO), a post-deployment learning paradigm for WAMs that, alongside action imitation, refines world representations through visual experience to better condition action generation. Specifically, it identifies interaction turning points and learns from successful and failed futures to support classifier-free guidance. An additional value head estimates task progress from video representations and activates guidance when progress stalls during inference. Across five DexJoCo tasks, DEWO improves average success across all three WAM formulations. Ablations show that visual supervision from successful and failed continuations improves both prediction and control beyond action supervision alone. On four real-world tasks across Wuji and Sharpa, 3 x 3 grid evaluations show that two rounds of deployment learning increase success from 51.0% to 71.7% in cells with at least one initial success, a gain of 20.7 percentage points. These findings support continued predictive learning for improving control through deployment experience, making world modeling an active part of WAM adaptation.

Figures & tables

Figure 1: Learning signals for post-deployment policy improvement. Direct action supervision ( top ) learns from action targets. Value-based reinforcement learning ( middle ) uses value estimates to improve the policy. DEWO ( bottom ) learns from observed visual futures and uses the learned success condition to guide actions.

Figure 2: Overview of Direct Experience World-Model Optimization (DEWO). Replay exploration locates interaction turning points and collects successful and failed continuations from matched contexts. Their visual futures train outcome-conditioned world modeling, while successful behavior supplies action targets. At deployment, a value head monitors task progress and selectively activates classifier-free guidance using the base and success-conditioned action predictions.

Model	Method	Water	Fold	Hammer	Pick	Pinch	Avg.
		Plant	Glasses	Nail	Bucket	Tongs
A. Matched post-deployment comparisons
$\pi_{0.5}$	Initial	73.3	58.0	74.7	78.7	62.7	69.5
	+ SFT	75.3	59.3	76.7	85.3	22.7	63.9
	+ RECAP	75.3	51.3	80.0	84.0	57.3	69.6
	+ DSRL	76.7	54.7	81.3	86.0	22.7	64.3

Table 1: Simulation success rates (%) over three 50-trial sets per task; Avg. weights tasks equally. A: matched adaptation comparisons. B: prediction–action formulation ablation; FastWAM repeats A. C: DEWO-S construction from pretrained components (—: no subsequent adaptation). Per-set results are in Appendix F .

Figure 3: Spatial and iterative real-world evaluation. (a) Two hands, four objects, three rounds; image labels Round #1/#2/#3 denote R0/R1/R2. (b) Pooled success for eight centers, 29 R0-nonzero cells (including centers), 43 R0-zero cells, and all 72 cells. Groups stay fixed across rounds, with ten trials per cell (720 per round). (c) Full-grid rates by object and hand; $\pi_{0.5}$ is a separate reference.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Pool or source	DEWO	DEWO-S
Expert episodes included in $D_{0}$	0	500
Collected successful episodes included in $D_{0}$	106	106
$D_{0}$ subtotal	106	606
$D_{\mathrm{scan}}$	1,396	1,396
$D_{+}$	125	125
$D_{\mathrm{fail}}$	134	134

Appendix

Table 2: Five-task training pools before window expansion. Source-entry counts differ from frame and interaction counts. The expert pool contains 100 successful episodes per task.

Setting	DEWO	DEWO-S
Initialization	Five-task FastWAM checkpoint at 55,000 updates	Pretrained VideoDiT and ActionDiT components
Trainable parameters	Text cross-attention K/V residual adapters and value head	Full video/action DiTs and MoT, proprioception/outcome encoders, and value head
Frozen components	MoT, VideoDiT, ActionDiT backbone weights	No additional freezing
Adapter	Rank 16, scale parameter 16; video and action experts	None
Maximum updates	10,000	100,000
Checkpoint save interval	2,500 updates	5,000 updates

Appendix

Table 3: Optimization settings for five-task FastWAM adaptation with DEWO and Joint WAM construction with DEWO-S.

Setting	Value
Simulation evaluation	3 sets $\times$ 50 scenes per task
Initial simulation collection	50 seeds $\times$ 4 attempts per task
Simulation / real continuations per anchor	$K=10$ / $K=4$
Simulation query start / interval	Step 96 / 24 steps
Maximum queries per failed trajectory	20 anchors
Intermediate recoverability-drop threshold	At least $4/10$

Appendix

Table 4: Collection, temporal, and deployment settings. The guidance interval includes replans 10–24 for all five FastWAM DEWO simulation tasks.

Role	Video (DEWO)	Video (DEWO-S)	Action	Value
$D_{0}$	1	1	1	1
$D_{+}$	1	1	1	1
$D_{\mathrm{scan}}$	0	0	0	1
$D_{\mathrm{fail}}$	1	1	0	1

Appendix

Table 5: Per-role supervision weights. Action and value columns apply to both methods. Value is supervised separately; failed action targets are masked.

Spatial group	Trials/round	R0	R1	R2
Center	80	64	71	71
Initially nonzero off-center	210	84	138	137
Initially zero off-center	430	0	0	0
All initially nonzero (including center)	290	148	209	208
All cells	720	148	209	208

Appendix

Table 6: Pooled spatial success counts and fixed denominators. The initially nonzero subtotal combines the center and nonzero off-center rows.

Method	Task	Seed 0	Seed 1	Seed 2	Mean
Initial	Water Plant	76	70	74	73.3
	Fold Glasses	54	62	58	58.0
	Hammer Nail	74	74	76	74.7
	Pick Bucket	80	78	78	78.7
	Pinch Tongs	66	64	58	62.7
+ SFT	Water Plant	76	74	76	75.3

Appendix

Table 7: Per-evaluation-set

\pi_{0.5}

success rates (%).

Method	Task	Seed 0	Seed 1	Seed 2	Mean
Initial	Water Plant	88	86	92	88.7
	Fold Glasses	74	70	72	72.0
	Hammer Nail	76	74	74	74.7
	Pick Bucket	94	92	90	92.0
	Pinch Tongs	78	72	78	76.0
+ SFT	Water Plant	70	74	76	73.3

Appendix

Table 8: Per-evaluation-set FastWAM success rates (%).

Method	Task	Seed 0	Seed 1	Seed 2	Mean
Initial	Water Plant	84	84	76	81.3
	Fold Glasses	78	70	76	74.7
	Hammer Nail	92	78	76	82.0
	Pick Bucket	96	86	96	92.7
	Pinch Tongs	84	76	76	78.7
+ DEWO	Water Plant	94	86	96	92.0

Appendix

Table 9: Per-evaluation-set Joint WAM success rates (%).

Method	Task	Seed 0	Seed 1	Seed 2	Mean
Initial	Water Plant	86	86	86	86.0
	Fold Glasses	66	66	72	68.0
	Hammer Nail	88	88	80	85.3
	Pick Bucket	86	90	82	86.0
	Pinch Tongs	86	86	84	85.3
+ DEWO	Water Plant	96	98	96	96.7

Appendix

Table 10: Per-evaluation-set IDM WAM success rates (%).

Task	Seed 0	Seed 1	Seed 2	Mean $\pm$ SD
Water Plant	94 (47/50)	94 (47/50)	96 (48/50)	$94.7\pm 0.9$ (142/150)
Fold Glasses	86 (43/50)	82 (41/50)	80 (40/50)	$82.7\pm 2.5$ (124/150)
Hammer Nail	82 (41/50)	74 (37/50)	74 (37/50)	$76.7\pm 3.8$ (115/150)
Pick Bucket	96 (48/50)	96 (48/50)	100 (50/50)	$97.3\pm 1.9$ (146/150)
Pinch Tongs	100 (50/50)	96 (48/50)	98 (49/50)	$98.0\pm 1.6$ (147/150)

Appendix

Table 11: Per-evaluation-set DEWO-S success rates (%). Parentheses give successful trials / total trials. The final column reports the mean and population standard deviation over the three 50-scene evaluation sets, with pooled trial counts.

Method	Task	Eval. 0	Eval. 1	Eval. 2	Mean
Initial	Water Plant	68 (34/50)	66 (33/50)	64 (32/50)	66.0 (99/150)
	Fold Glasses	76 (38/50)	64 (32/50)	64 (32/50)	68.0 (102/150)
	Hammer Nail	26 (13/50)	24 (12/50)	26 (13/50)	25.3 (38/150)
	Pick Bucket	88 (44/50)	84 (42/50)	84 (42/50)	85.3 (128/150)
	Pinch Tongs	80 (40/50)	76 (38/50)	78 (39/50)	78.0 (117/150)
	All five tasks	67.6 (169/250)	62.8 (157/250)	63.2 (158/250)	64.5 (484/750)

Appendix

Table 12: Per-evaluation-set FACT success rates (%). Each method uses one fixed checkpoint across three 50-scene evaluation sets per task. Parentheses give successful trials / total trials. The final column reports the mean over the three sets with pooled trial counts; the all-task rows aggregate 250 trials per set and 750 trials overall.

Emb.	Object	TL	TC	TR	ML	C	MR	BL	BC	BR	Overall
A. $\pi_{0.5}$ reference
Wuji	Water Bottle	0	4	1	0	10	6	0	0	0	21/90 (23.3%)
	Tape	0	0	0	4	10	0	3	9	0	26/90 (28.9%)
	Eraser	0	0	0	2	5	0	1	3	0	11/90 (12.2%)
	Tennis Ball	0	0	0	0	4	0	0	2	0	6/90 (6.7%)
Sharpa	Water Bottle	0	5	1	0	10	8	0	0	0	24/90 (26.7%)

Appendix

Table 13: Real-world spatial evaluation. Each spatial entry reports complete grasp-and-place successes out of 10 trials; Overall reports the total out of 90 trials and the corresponding success rate. Panels A–D show

\pi_{0.5}

and DEWO-S Rounds 0–2, respectively.

Explore similar work

Latent evolving World Action Model
Sep 23, 2026Xueji Fang, Boqiang Duan, Hua Wu +2Action Generation World Models
Zhejiang University · Westlake University · Baidu Inc.
Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training
Apr 23, 2026Yaxuan Li, Zhongyi Zhou, Yefei Chen +5Efficient World-Action Model Scalable Robot Learning
Current Robotics · Tsinghua University · Peking University +1
WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFT
Jun 16, 2026Zezhong Qian, Xiaowei Chi, Yu Qi +3Efficient World-Action Model Faster-Wam
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Northeastern University · Tsinghua University

Sep 23, 2026, cs.CVJ/K move · Enter open · S save

Latent evolving World Action Model

Xueji Fang, Boqiang Duan, Hua Wu, Jingdong Wang+1

Zhejiang University · Westlake University · Baidu Inc.

Action Generation World Models Joint-Embedding Predictive Architectures World Video Diffusion Models Backtranslation Augmented Direct Preference Optimization

Open paper AskPDF arXiv

World Action Models (WAMs) jointly model action generation and environment dynamics and are mostly built on pretrained Video Diffusion Models (VDMs). In VDM-based WAMs, observations are first encoded by a VAE, and the resulting compressed latents are then processed by large video diffusion backbones to extract effective features for action generation. However, this paradigm ties WAM performance and training cost to large-scale video generation pretraining, limiting WAM efficiency and scalability. In this paper, we theoretically and empirically investigate how visual representations affect action generation in WAMs. Our results show that predictive embeddings from Joint-Embedding Predictive Architecture (JEPA) encoders better support action generation than compressed VAE latents, with I-JEPA performing best in our encoder comparison. Based on these findings, we propose LeWAM, which conditions action generation on JEPA embeddings and models environment evolution by predicting future embeddings in the same space, without relying on a video diffusion backbone. We further find that imitation learning matches demonstrated actions but does not distinguish better actions from worse ones, even though small action deviations can greatly affect task success. To address this limitation without additional environment interaction or the human oversight required for resets and safety, we introduce Demonstration-Guided DPO (DemoDPO), an offline preference refinement stage that derives preference supervision directly from demonstrations. With only 0.4B trainable parameters, LeWAM achieves an average success rate of 92.28% on RoboTwin 2.0, comparable to that of state-of-the-art VLAs and WAMs, and maintains practical effectiveness on real-world manipulation tasks.

Abstract

Explore similar work

Latent evolving World Action Model

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFT