Think Offline, Act in Real Time: Self-Improving Visual Game Agents through Code
Organizations: Ant Group · HKUST
Abstract
Pre-trained models can read screens and reason about games, yet on an unfamiliar game, an agent still has to learn to play. We ask how an agent can self-improve on such a game from a few of its own attempts, in real-time visual games where the agent sees only the screen and the game keeps running while the agent thinks. We study two kinds of policy that can carry this improvement. In model as policy, a model selects actions during play; we adapt real-time chunking so that committed actions keep executing while the model reasons, and train a 2B controller for this interface. In code as policy, a model writes, tests, and revises a program between attempts, and the program acts during play. Across 170 GameWorld tasks, with at most eight one-minute executions per run, code as policy reaches 47.80% mean progress, against 30.90% for our trained controller, and leads in all five game categories. Improving the controller's weights on a single game, by contrast, yields only modest gains. These results favor letting a model think offline and storing what it learns in a program that acts in real time. Project repository: https://github.com/TabibitoQZP/SIGame
Figures & tables
| Condition | Tasks | Runs | Game time | Development limits |
|---|---|---|---|---|
| Model policies, subset | 31 | 4 | 180 s | — |
| RTC-2B, full benchmark | 170 | 4 | 180 s | — |
| Independent code, subset | 31 | 4 | 480 s | 8 submissions, 256K tokens |
| Independent code, full | 170 | 4 | 480 s | 8 submissions, 256K tokens |
| Collaborative code, full | 170 | 1 team | 1,920 s | 32 submissions, 1M tokens |
| Method | Arcade | Platformer | Puzzle | Runner | Simulation | Overall |
|---|---|---|---|---|---|---|
| Tasks | 7 | 7 | 6 | 8 | 3 | 31 |
| [0pt][0pt] Task progress (%) | ||||||
| GameWorld baseline | 20.21 [-1pt] [16.05, 27.17] | 10.34 [-1pt] [7.50, 15.00] | 46.56 [-1pt] [35.76, 55.48] | 29.66 [-1pt] [19.75, 35.54] | 14.72 [-1pt] [9.40, 28.38] | 24.99 [-1pt] [18.25, 32.18] |
| RTC ( s) | 13.29 [-1pt] [2.23, 22.43] | 10.73 [-1pt] [9.71, 13.63] | 45.52 [-1pt] [31.14, 66.67] | 20.89 [-1pt] [17.31, 26.60] | 22.75 [-1pt] [11.11, 47.52] | 21.83 [-1pt] [14.27, 32.51] |
| RTC ( s) | 12.06 [-1pt] [2.23, 26.50] | 23.03 [-1pt] [21.14, 25.40] | 56.67 [-1pt] [33.77, 74.59] | 23.04 [-1pt] [20.67, 26.92] | 16.79 [-1pt] [5.56, 39.15] | 26.46 [-1pt] [17.69, 36.89] |
| [0pt][0pt] Success rate (%) | ||||||
| Controller | Request (s) P50 / P95 | Release (s) P50 / P95 | Legal (%) | Empty (%) | Output tokens P50 |
|---|---|---|---|---|---|
| Qwen3.8-27B baseline | 1.40 / 2.33 | — | — | — | 91 |
| Qwen3.8-27B + RTC (3s) | 3.91 / 10.37 | 3.89 / 10.16 | 84.54 | 53.12 | 272 |
| Qwen3.8-27B + RTC (6s) | 4.28 / 11.99 | 6.00 / 11.52 | 73.88 | 42.62 | 375 |
| Qwen3.5-2B + RTC (3s) | 19.98 / 21.64 | 20.03 / 21.67 | 0.64 | 99.90 | 8,192 |
| RTC-2B (SFT), 3s | 2.83 / 5.14 | 3.00 / 4.99 | 97.86 | 23.56 | 201 |
| RTC-2B (RL), 3s | 2.57 / 5.44 | 3.00 / 5.29 | 97.48 | 24.56 | 224 |
| Method | Arcade | Platformer | Puzzle | Runner | Simulation | Overall |
|---|---|---|---|---|---|---|
| Tasks | 7 | 7 | 6 | 8 | 3 | 31 |
| [0pt][0pt] Task progress (%) | ||||||
| Qwen3.5-2B | 2.21 [-1pt] [2.13, 2.23] | 1.14 [-1pt] [1.14, 1.14] | 5.48 [-1pt] [1.50, 16.67] | 12.63 [-1pt] [11.95, 14.13] | 0.00 [-1pt] [0.00, 0.00] | 5.08 [-1pt] [4.11, 7.63] |
| RTC-2B (SFT) | 15.95 [-1pt] [10.31, 21.55] | 20.69 [-1pt] [9.71, 24.86] | 14.67 [-1pt] [9.32, 25.13] | 24.56 [-1pt] [21.78, 28.14] | 2.34 [-1pt] [0.00, 9.10] | 17.68 [-1pt] [11.94, 23.49] |
| RTC-2B (RL) | 13.21 [-1pt] [11.05, 15.53] | 21.38 [-1pt] [9.71, 27.36] | 24.13 [-1pt] [13.60, 46.94] | 26.26 [-1pt] [21.15, 34.34] | 26.45 [-1pt] [0.00, 50.26] | 21.82 [-1pt] [12.78, 32.49] |
| Qwen3.8-27B RTC ( s) | 13.29 [-1pt] [2.23, 22.43] | 10.73 [-1pt] [9.71, 13.63] | 45.52 [-1pt] [31.14, 66.67] | 20.89 [-1pt] [17.31, 26.60] | 22.75 [-1pt] [11.11, 47.52] | 21.83 [-1pt] [14.27, 32.51] |
| Controller | Environment | Games | Mean height | Completed |
| RTC-2B | Original game | 4 | 1.75 | 0/4 |
| + Stack SFT | Original game | 4 | 2.25 | 0/4 |
| + Stack RL, audited reward | Original game | 4 | 3.25 | 0/4 |
| RTC-2B + Stack SFT | Simulator | 8 | 3.50 | 0/8 |
| + Stack RL, relative advantage | Simulator | 8 | 3.75 | 0/8 |
| + Stack RL, audited reward | Simulator | 8 | 4.00 | 0/8 |
| Method | Arcade | Platformer | Puzzle | Runner | Simulation | Overall |
|---|---|---|---|---|---|---|
| Tasks | 35 | 40 | 35 | 40 | 20 | 170 |
| [0pt][0pt] Task progress (%) | ||||||
| RTC-2B (RL) | 27.40 [-1pt] [21.47, 36.62] | 22.76 [-1pt] [12.42, 37.41] | 35.74 [-1pt] [21.75, 53.10] | 43.41 [-1pt] [37.50, 51.81] | 19.81 [-1pt] [1.75, 43.35] | 30.90 [-1pt] [20.85, 44.57] |
| Code policy Independent | 42.80 [-1pt] [18.83, 63.21] | 28.76 [-1pt] [13.29, 43.90] | 70.81 [-1pt] [48.36, 87.83] | 53.22 [-1pt] [41.64, 66.77] | 43.51 [-1pt] [16.17, 68.65] | 47.80 [-1pt] [28.66, 65.21] |
| Code policy Collaborative | 53.74 [-1pt] [—, —] | 36.48 [-1pt] [—, —] | 85.07 [-1pt] [—, —] | 65.10 [-1pt] [—, —] | 62.43 [-1pt] [—, —] | 59.82 [-1pt] [—, —] |
| [0pt][0pt] Success rate (%) | ||||||
| Policy | PG (%) | SR (%) |
|---|---|---|
| RTC-2B (RL), direct (3s) | 21.82 [-1pt] [12.78, 32.49] | 7.26 [-1pt] [0.00, 12.90] |
| Qwen3.8-27B, GameWorld baseline | 24.99 [-1pt] [18.25, 32.18] | 7.26 [-1pt] [3.23, 9.68] |
| Qwen3.8-27B + RTC (3s) | 21.83 [-1pt] [14.27, 32.51] | 5.65 [-1pt] [3.23, 12.90] |
| Qwen3.8-27B + RTC (6s) | 26.46 [-1pt] [17.69, 36.89] | 11.29 [-1pt] [6.45, 19.35] |
| Qwen3.8-27B code, initial frame | 32.26 [-1pt] [18.06, 48.00] | 11.29 [-1pt] [3.23, 19.35] |
| Qwen3.8-27B code, RTC trajectory | 32.30 [-1pt] [12.87, 50.34] | 14.52 [-1pt] [3.23, 25.81] |
| Initial evidence | PG (%) | SR (%) | Submissions Mean |
|---|---|---|---|
| Initial frame | 32.26 [-1pt] [18.06, 48.00] | 11.29 [-1pt] [3.23, 19.35] | 5.98 |
| Initial RTC trajectory | 32.30 [-1pt] [12.87, 50.34] | 14.52 [-1pt] [3.23, 25.81] | 5.50 |
| Independent runs | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| Best-of- progress (%) | 47.80 | 57.84 | 62.44 | 65.21 |
| Any-success rate (%) | 27.79 | 36.76 | 41.62 | 44.71 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Controller | (s) | PG | SR |
|---|---|---|---|
| GameWorld baseline | — | 13.83 [9.81, 19.50] | 4.03 [3.23, 6.45] |
| RTC | 3 | 5.08 [4.11, 7.63] | 0.81 [0.00, 3.23] |
| Controller | Valid episodes | Complete | Released | Deadline incomplete | Other incomplete |
|---|---|---|---|---|---|
| Qwen3.8-27B baseline | 123 | 9345 | — | 79 | 0 |
| Qwen3.8-27B + RTC (3s) | 122 | 4288 | 4166 | 0 | 0 |
| Qwen3.8-27B + RTC (6s) | 123 | 3109 | 2986 | 0 | 0 |
| Qwen3.5-2B + RTC (3s) | 123 | 1257 | 1134 | 0 | 0 |
| RTC-2B (SFT), 3s | 123 | 5986 | 5863 | 0 | 0 |
| RTC-2B (RL), 3s | 122 | 5872 | 5750 | 0 | 0 |
| Code start | Direct control | PG | Max PG | SR | W / T / L |
|---|---|---|---|---|---|
| Trajectory | Qwen3.8-27B + RTC (6s) | +5.84 [-1pt] [ 1.60, +13.15] | +13.45 [-1pt] [+1.55, +25.81] | +3.23 [-1pt] [ 3.23, +9.68] | 20 / 3 / 8 |
| Trajectory | Qwen3.8-27B + RTC (3s) | +10.47 [-1pt] [+3.59, +17.34] | +17.83 [-1pt] [+5.64, +30.29] | +8.87 [-1pt] [+1.61, +17.74] | 21 / 5 / 5 |
| Trajectory | Qwen3.8-27B, GameWorld baseline | +7.31 [-1pt] [ 0.22, +15.35] | +18.16 [-1pt] [+7.54, +30.13] | +7.26 [-1pt] [+0.00, +16.13] | 17 / 3 / 11 |
| Trajectory | RTC-2B (RL), direct (3s) | +10.48 [-1pt] [+1.60, +19.80] | +17.85 [-1pt] [+5.44, +30.51] | +7.26 [-1pt] [ 1.61, +16.94] | 20 / 4 / 7 |
| Frame | Qwen3.8-27B + RTC (6s) | +5.80 [-1pt] [+0.11, +11.56] | +11.11 [-1pt] [+1.03, +20.79] | +0.00 [-1pt] [ 4.03, +4.03] | 22 / 3 / 6 |
| Frame | Qwen3.8-27B + RTC (3s) | +10.44 [-1pt] [+3.73, +17.93] | +15.49 [-1pt] [+5.18, +26.18] | +5.65 [-1pt] [ 1.61, +14.52] | 24 / 2 / 5 |
| Progress per task | Code higher | Tie | RTC higher | |
|---|---|---|---|---|
| Mean of four runs | 111 | 31 | 28 | |
| Best of four runs | 91 | 65 | 14 | |
| Solved in any of four runs | Code only | Both | RTC only | Neither |
| Number of tasks | 41 | 35 | 5 | 89 |
| Stage | Outcome | Games |
| Source snapshot | Discovered | 2,522 |
| Score analysis | Eligible / rejected | 853 / 290 |
| No score found / uncertain / failed | 923 / 167 / 289 | |
| Strict filter | Pass / reject / review | 395 / 1,667 / 460 |
| Instrumentation | Games with a host score reader | 395 |
| RL training | GameWorld evaluation | |
|---|---|---|
| Games | 395 filtered js13kgames [ 19 ] | 31-task subset or full 170-task benchmark |
| Instructions | One generic goal for all games | Official task goal and controls |
| Game clock | Virtual; paused during inference | Continuously running |
| Delay | Fixed simulated 3 s | Actual request time, with a 3 s floor |
| Budget | 32 requests, at most 960 ticks | 180 s per run |
| Policy input | RGB frames, executed actions, committed queue | Same, with task instructions |
| Controller | Heights | PG (%) | SR (%) |
|---|---|---|---|
| RTC-2B | 2, 1, 3, 1 | 5.83 [3.33, 10.00] | 0.00 [0.00, 0.00] |
| + Stack SFT | 1, 3, 3, 2 | 7.50 [3.33, 10.00] | 0.00 [0.00, 0.00] |
| + Stack RL | 2, 4, 5, 2 | 10.83 [6.67, 16.67] | 0.00 [0.00, 0.00] |
| Adaptation of RTC-2B (complete reasoning) | Mean height |
|---|---|
| RTC-2B (RL) | 2.00 |
| Full SFT, 512 mixed examples | 3.63 |
| + 36 short-reasoning examples (short-reasoning checkpoint) | 4.88 |
| + 206 speed and layout examples, 1,024 steps (visual-clock checkpoint) | 4.88 |
| Full SFT + 83 response pairs, DPO | 3.88 |
| Short-reasoning + 97 response pairs, DPO | 4.38 |
| Submission | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| A: apples | 1 | 2 | 10 | 6 | — | — | — | — |
| D: apples | 0 | 0 | 2 | 1 | 6 | 6 | 4 | 6 |
Game-Playing Agents
Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.
See all 218 papers