Pretrained world models can generate diverse environments, yet users often want to explore a particular scene specified by their own video. This requires learning the scene's visual identity while retaining the quality of action-conditioned generation. We introduce Personalized World Models (PWM), a framework for customizing interactive world models from short scene videos through online reinforcement learning. In PWM, the support trajectory and its associated controls provide reward feedback on continuations sampled from the current policy. In the GRPO instantiation, group-relative optimization updates a compact LoRA adapter using a unified reward for scene appearance, visual continuity, and motion, while base-policy anchoring regularizes changes to the pretrained generation prior of a frozen Yume-5B backbone. The same adaptation procedure is applied across real and rendered environments. We also instantiate PWM with DiffusionNFT as an alternative reward-guided optimization method for learning the scene-specific adapter. We also introduce PWM-Bench, comprising 150 customization tasks across Indoor, Outdoor, and Gaming, with paired evaluation on held-out continuations. The GRPO and DiffusionNFT instantiations of PWM improve customization over native Yume in 71.3% and 65.3% of the evaluated scenes, respectively, with positive mean gains across all three domains. For the GRPO instantiation, matched SFT comparisons further demonstrate higher mean customization gains and better mean image-quality scores in every domain, while retaining frame-level visual quality close to the pretrained model.
Figures & tables
Figure 1: Scene customization with PWM. A short support video trains a scene LoRA on a frozen world model. Results use DanceGRPO. Indoor and Outdoor show matched generation times; Gaming shows different steps at similar viewing distances.
Figure 2: PWM with GRPO and DiffusionNFT. Support video and controls condition fresh rollouts from frozen Yume-5B with a trainable scene LoRA. Rewards guide GRPO’s clipped objective with a base-policy anchor, or DiffusionNFT’s reward-weighted positive and negative flow-matching branches. Only LoRA parameters are updated. Eight epoch adapters are averaged for generation. Rollout icons are schematic.
Figure 3: Training scenes in PWM-Bench. Each domain contains 50 scenes; four representative training scenes are shown per domain. Each image is the full, unretouched frame at zero-based index 40 of an 80-frame support clip.
Domain
Tasks
Video source
Control source
Outdoor
50
EgoVid-5M, YouTube
VIO / visual motion
Indoor
50
EgoVid-5M, InCrowd-VI, YouTube
VIO / visual motion
Gaming
50
MineDojo
Scripted-agent logs
Table 1: Composition of the cross-domain evaluation suite. Each domain contains 50 independent per-scene customization tasks.
Figure 4: Scene appearance in a shopping-mall continuation. Rows share the controls; columns show frames 0, 21, 41, 62, 82, 103, 123, and 144. Yume replaces shop displays with window-like structures; both PWM variants retain shop displays. Boxes mark selected regions; GT denotes ground truth.
Figure 5: Object hallucination indoors. Yume adds a face to blue equipment; both PWM variants retain its appearance. Methods share the seed and controls (frame 128). Lower panels enlarge equally sized boxed crops.
Figure 6: Snow-cover retention in Gaming. Both PWM variants retain snow-covered ground. Rows share controls and use the frame indices in Figure 4 . GT denotes ground truth; boxes mark selected regions.
Figure 7: Scene adaptation outdoors. Methods share the seed, controls, and eight uniformly sampled frame indices. Yume shows a nearby wall and SFT close-ups of buildings and palms; both PWM variants retain visible passage space.
Method
Mean gain (%) ↑
Max. gain (%) ↑
Improved scenes ↑
Indoor
PWM (DanceGRPO)
5.14
49.00
36/50 (72.0%)
PWM (DiffusionNFT)
7.04
58.54
34/50 (68.0%)
Outdoor
PWM (DanceGRPO)
6.11
75.11
34/50 (68.0%)
PWM (DiffusionNFT)
2.27
46.38
30/50 (60.0%)
Table 2: PWM improves customization across all three domains. Both optimization instantiations are compared with native Yume. Gains are relative reductions in S . Mean gain includes all evaluated scenes. Maximum gain and improved-scene counts are computed after averaging three seeds within each scene.
Method
Mean gain (%) ↑
Improved scenes ↑
Indoor
SFT
−16.39
27/50 (54.0%)
PWM (DanceGRPO)
+5.14
36/50 (72.0%)
PWM (DiffusionNFT)
+7.04
34/50 (68.0%)
Outdoor
SFT
−14.86
26/50 (52.0%)
Table 3: Both PWM instantiations achieve higher mean customization gains than SFT across all three domains. All methods are compared with native Yume. Improved scenes have a lower three-seed mean S than Native. Bold indicates the best result within each domain.
Figure 8: Preserving visual quality after adaptation. SFT exhibits blur and repetitive textures indoors and mesh-like artifacts outdoors, while PWM (DanceGRPO) retains natural-looking scene appearance. Each pair uses the same seed and control sequence at frame 112. All panels show complete, unretouched frames.
Method
MUSIQ ↑
NIQE ↓
BRISQUE ↓
Indoor
Yume
58.93
5.43
44.46
SFT
47.26
5.93
45.73
PWM (DanceGRPO)
59.06
5.40
44.28
PWM (DiffusionNFT)
57.94
5.49
44.79
Outdoor
Table 4: Both PWM variants achieve better mean image-quality scores than SFT in every domain. Native Yume provides the pretrained reference. Bold indicates the best mean within each domain. MUSIQ uses the SPAQ-trained checkpoint.
Figure 9: Training reward trajectories. Each panel averages 50 scenes. Per-scene rewards average the best of four rollouts over six support windows in each epoch. Rewards are recorded under each method’s training sampling procedure.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Frozen backbone
Yume-5B
LoRA rank / scaling α
8 / 16
LoRA target modules
q, k, v, o, ffn.0, ffn.2
LoRA dropout / trainable biases
0 / none
Trainable parameters
Approximately 20.2 M
Window length
29 frames
Appendix
Table 5: PWM (DanceGRPO) adaptation configuration, fixed across scenes. Reward weights follow the order (gt,lpips,cross,dyn) .
Class
Positive prompt
Negative prompt
Person
a photo containing a person
a photo of an empty scene with no people
Vehicle
a photo containing a car or truck
a photo of a street with no vehicles
Animal
a photo containing an animal
a photo of a scene with no animals
Appendix
Table 6: Fixed CLIP prompt pairs used for entity-class scoring.