Learning visual policies requires turning interaction experience into guidance for individual decisions. Trajectory outcomes obscure action quality, while unguided exploration can revisit familiar situations. We present LightAct, an experience-learning framework that connects skill-guided exploration with policy internalization. Interaction segments are summarized into experience entries and consolidated into a game-specific skill that guides root collection toward broader state coverage. At each root, candidate actions are compared through short rollouts under a shared frozen policy. Environment-reported score gains are averaged across continuations and normalized into relative action rewards, producing reusable supervision for reinforcement learning. Policy training uses a compact dataset of just 1,024 decision roots collected in a single source game and paired with their action rewards. Across 170 GameWorld tasks spanning 34 games, LightAct improves overall progress by 23.6, 29.2, and 13.0 percentage points at the 2B, 4B, and 9B scales, respectively, with our strongest model competitive with reported proprietary-model results. Ablations show broader root-state coverage with skill guidance and consistent gains across three policy optimizers. Case studies show how rollouts distinguish actions with identical immediate gains.
Figures & tables
Figure 2: A 2048 decision by LightAct -2B. When repeated downward moves no longer change the board, the policy switches to moving right and gains 24 points.
Figure 3: Overview of LightAct . Interaction segments are summarized into experience entries, then consolidated into a source-game skill that guides root-state collection. At each root, short rollouts compare candidate actions through their subsequent score gains. These gains are normalized into action-level rewards for reinforcement learning, internalizing the collected experience in a policy evaluated across games.
Model
Arcade
Platformer
Puzzle
Runner
Simulation
Overall
SR
PG
SR
PG
SR
PG
SR
PG
SR
PG
SR
PG
Human
Novice Player
45.7
55.5
60.0
65.6
51.4
63.1
60.0
72.0
60.0
62.0
55.3
64.1
Expert Player
65.7
73.9
85.0
88.0
68.6
77.1
82.5
87.8
85.0
86.0
77.1
82.6
Generalist Multimodal Agents
Claude-Sonnet-4.6
5.7
28.3
22.5
37.0
25.7
51.5
30.0
51.9
15.0
16.6
20.6
39.3
Table 1: Main results on GameWorld. SR and PG (%) are reported by category and overall. All three LightAct models are trained with GRPO using experience collected in 2048. Qwen3.5 and LightAct results are averaged over three evaluation runs per task. Published baseline results are taken from GameWorld [ 16 ] .
Figure 4: Source-environment ablation. GameWorld SR and PG for 2B policies trained on 2048 or Tetris with GRPO and REINFORCE++.
Model
Arc.
Plat.
Puzz.
Run.
Sim.
Overall
PG
PG
PG
PG
PG
PG
Qwen3.5-2B
Base
5.6
10.6
9.4
28.4
9.1
13.3
+REINFORCE++
24.0
28.6
46.2
46.8
20.1
34.5
+GRPO
28.7
30.3
49.5
46.1
24.0
36.9
+DAPO
26.6
29.9
50.2
45.0
20.6
35.9
Table 2: Policy internalization. GameWorld PG (%) for base and trained policies across three model scales and RL algorithms.
Figure 5: Skill-guided exploration. Top: score distributions of 1,024 roots in 2048 (50-point bins). Bottom: GameWorld PG and SR for 2B policies trained with GRPO or REINFORCE++ (RF++).
Figure 6: From a root state to action rewards in 2048. Left and right gain 24 immediately but yield different outcomes. The table averages four ten-decision continuations per action and normalizes gains into rewards; endpoints are nearest each action’s mean gain.
Figure 7: 2048 exploration skill. The textual guidance used during root-state collection, with action-selection principles highlighted.
Figure S1 : A 2048 decision by LightAct -2B trained with REINFORCE++ (task 01_02, repetition 3, step 46). After a repeated downward move leaves the board unchanged, the policy selects move_right , increasing the score from 204 to 228.
Figure S2 : A Mario decision by LightAct -2B trained with REINFORCE++ on 2048 (task 17_02, repetition 2, step 40). The recorded jump action is followed by mushroom collection and task completion. Shared system instructions appear in Fig. .
Parameter
Setting
Source collection
Runs / decision budget
1 per game–task pair / 10,000 steps
Episode decision limit
100
Root-state limit
1,024 per run
Deduplication
Environment state
Visual context
Current image + 2 image–text history rounds
Table S1: Trajectory collection and reward construction.
Parameter
GRPO
REINFORCE++
DAPO
Training settings
Base model
Qwen3.5-2B/4B/9B
Qwen3.5-2B/4B/9B
Qwen3.5-2B/4B/9B
Freeze vision encoder
Yes
Yes
Yes
Batch size (prompts)
64
64
64
Minibatch size (prompts)
64
64
64
Learning rate
5×10−7
5×10−7
5×10−7
Table S2: Policy-training configuration. Settings for GRPO, REINFORCE++, and DAPO in verl.
Latent world models offer a promising way to improve Vision-Language-Action policies by capturing the consequences of actions. However, models trained primarily on expert demonstrations have limited exposure to failure outcomes and may struggle to distinguish visually similar successful and failed interactions. We propose WorldGuide, a framework that learns these distinctions in latent space and uses them to guide policy training. WorldGuide combines predictive pretraining on successful and failed trajectories with contrastive learning on matched success--failure pairs. The learned predictor then provides a differentiable reward to guide joint optimization of the policy and visual encoder. The predictor is discarded after training, so deployment requires no additional world-model inference. Extensive experiments show that WorldGuide substantially improves VLA reliability and achieves state of the art performance on LIBERO 100 and SimplerEnv, reaching 96.8% and 72.0%, respectively. Code will be publicly available.
Lin Liu, Lu Zhang, Ziying Song +6
School of Information and Communication Engineering Dalian University of Technology & Beta Infinity · School of Information and Communication, Engineering Dalian University of Technology · Nanyang Technological University +1
World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.
State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy's progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the vision-language-action model π0.5 and a diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and videos are available on the project website (https://policy-improvement-wm.pages.dev/).
Colton Crosby, Hao Wang, Mac Schwager +1
The authors are associated with the Aeronautics and Astronautics Department, Stanford University. · The author is associated with the Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California.