Learning visual policies requires turning interaction experience into guidance for individual decisions. Trajectory outcomes obscure action quality, while unguided exploration can revisit familiar situations. We present LightAct, an experience-learning framework that connects skill-guided exploration with policy internalization. Interaction segments are summarized into experience entries and consolidated into a game-specific skill that guides root collection toward broader state coverage. At each root, candidate actions are compared through short rollouts under a shared frozen policy. Environment-reported score gains are averaged across continuations and normalized into relative action rewards, producing reusable supervision for reinforcement learning. Policy training uses a compact dataset of just 1,024 decision roots collected in a single source game and paired with their action rewards. Across 170 GameWorld tasks spanning 34 games, LightAct improves overall progress by 23.6, 29.2, and 13.0 percentage points at the 2B, 4B, and 9B scales, respectively, with our strongest model competitive with reported proprietary-model results. Ablations show broader root-state coverage with skill guidance and consistent gains across three policy optimizers. Case studies show how rollouts distinguish actions with identical immediate gains.
Figures & tables
Figure 2: A 2048 decision by LightAct -2B. When repeated downward moves no longer change the board, the policy switches to moving right and gains 24 points.
Figure 3: Overview of LightAct . Interaction segments are summarized into experience entries, then consolidated into a source-game skill that guides root-state collection. At each root, short rollouts compare candidate actions through their subsequent score gains. These gains are normalized into action-level rewards for reinforcement learning, internalizing the collected experience in a policy evaluated across games.
Model
Arcade
Platformer
Puzzle
Runner
Simulation
Overall
SR
PG
SR
PG
SR
PG
SR
PG
SR
PG
SR
PG
Human
Novice Player
45.7
55.5
60.0
65.6
51.4
63.1
60.0
72.0
60.0
62.0
55.3
64.1
Expert Player
65.7
73.9
85.0
88.0
68.6
77.1
82.5
87.8
85.0
86.0
77.1
82.6
Generalist Multimodal Agents
Claude-Sonnet-4.6
5.7
28.3
22.5
37.0
25.7
51.5
30.0
51.9
15.0
16.6
20.6
39.3
Table 1: Main results on GameWorld. SR and PG (%) are reported by category and overall. All three LightAct models are trained with GRPO using experience collected in 2048. Qwen3.5 and LightAct results are averaged over three evaluation runs per task. Published baseline results are taken from GameWorld [ 16 ] .
Figure 4: Source-environment ablation. GameWorld SR and PG for 2B policies trained on 2048 or Tetris with GRPO and REINFORCE++.
Model
Arc.
Plat.
Puzz.
Run.
Sim.
Overall
PG
PG
PG
PG
PG
PG
Qwen3.5-2B
Base
5.6
10.6
9.4
28.4
9.1
13.3
+REINFORCE++
24.0
28.6
46.2
46.8
20.1
34.5
+GRPO
28.7
30.3
49.5
46.1
24.0
36.9
+DAPO
26.6
29.9
50.2
45.0
20.6
35.9
Table 2: Policy internalization. GameWorld PG (%) for base and trained policies across three model scales and RL algorithms.
Figure 5: Skill-guided exploration. Top: score distributions of 1,024 roots in 2048 (50-point bins). Bottom: GameWorld PG and SR for 2B policies trained with GRPO or REINFORCE++ (RF++).
Figure 6: From a root state to action rewards in 2048. Left and right gain 24 immediately but yield different outcomes. The table averages four ten-decision continuations per action and normalizes gains into rewards; endpoints are nearest each action’s mean gain.
Figure 7: 2048 exploration skill. The textual guidance used during root-state collection, with action-selection principles highlighted.
Figure S1 : A 2048 decision by LightAct -2B trained with REINFORCE++ (task 01_02, repetition 3, step 46). After a repeated downward move leaves the board unchanged, the policy selects move_right , increasing the score from 204 to 228.
Figure S2 : A Mario decision by LightAct -2B trained with REINFORCE++ on 2048 (task 17_02, repetition 2, step 40). The recorded jump action is followed by mushroom collection and task completion. Shared system instructions appear in Fig. .
Parameter
Setting
Source collection
Runs / decision budget
1 per game–task pair / 10,000 steps
Episode decision limit
100
Root-state limit
1,024 per run
Deduplication
Environment state
Visual context
Current image + 2 image–text history rounds
Table S1: Trajectory collection and reward construction.
Parameter
GRPO
REINFORCE++
DAPO
Training settings
Base model
Qwen3.5-2B/4B/9B
Qwen3.5-2B/4B/9B
Qwen3.5-2B/4B/9B
Freeze vision encoder
Yes
Yes
Yes
Batch size (prompts)
64
64
64
Minibatch size (prompts)
64
64
64
Learning rate
5×10−7
5×10−7
5×10−7
Table S2: Policy-training configuration. Settings for GRPO, REINFORCE++, and DAPO in verl.
School of Information and Communication Engineering Dalian University of Technology & Beta Infinity · School of Information and Communication, Engineering Dalian University of Technology · Nanyang Technological University +1
The authors are associated with the Aeronautics and Astronautics Department, Stanford University. · The author is associated with the Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California.