LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends
Abstract
Visual agents need to coordinate reasoning and action while their environment evolves during inference. We introduce LOLPlay, a benchmark in native League of Legends with 36 champion-specific tasks spanning resource management, combat, cooperation, and objectives. Agents interact through screenshots and mouse-keyboard actions; automated initialization and 82 executable subgoal checks distinguish partial progress from completion. Across 324 task evaluations of three models at three reasoning-effort settings, GPT-6 Astra achieves the highest task success rate of 25.0% at low and medium effort, and the highest subgoal success rate of 40.3% at medium effort. However, high effort improves neither metric over low effort for any model. For Astra, longer requests leave fewer interaction rounds, suggesting a cost to additional reasoning in real-time control. Trajectories reveal incomplete subgoal composition and actions executing after the observed scene changes, motivating agents that combine visual reasoning with timely action.
Figures & tables
| Benchmark | Environment | Evaluation |
|---|---|---|
| OSWorld [ 30 ] | Desktop; 369 tasks | Execution-based checks |
| OSWorld 2.0 [ 32 ] | Desktop/web; 108 workflows | Functional checkpoints; bounded judging |
| VideoGameBench [ 33 ] | 10 test games | Game progress milestones |
| GameWorld [ 19 ] | Browser; 34 games, 170 tasks | Serialized state; success and progress |
| GameVerse [ 34 ] | 15 games | VLM milestone evaluation |
| LOLPlay | Native LOL, unpaused; 36 tasks | Live Client state; event deadlines |
| Primary task demand | Tasks | Subgoals | Examples of outcomes |
|---|---|---|---|
| Economy and farming | 6 | 7 | Last hits, level and gold, purchases |
| Development | 9 | 26 | Ability allocation, attribute growth |
| Sustain | 7 | 18 | Health, mana, survival to a deadline |
| Combat | 6 | 13 | Kills, level, survival constraints |
| Cooperation | 3 | 8 | Assists and participation with bots |
| Map objectives | 5 | 10 | Dragon, turret, ordered objectives |
| Function | Description |
|---|---|
| moveTo(x, y) | Move the pointer to . |
| click(x, y) | Click at . |
| typewrite(’text’) | Type in the focused field. |
| press(’q’) | Press the Q key. |
| hotkey(’ctrl’, ’q’) | Press Ctrl and Q together. |
| scroll(200) | Scroll up by 200 units. |
| Observation | Task Instruction (Abridged) | Evaluation Script (Simplified) |
|---|---|---|
| Economy: Syndra (036) | Buy Lost Chapter within 7 minutes; finish while it is in the inventory. | player = live_state() lost_chapter = 3802 assert lost_chapter in ( player.item_ids) |
| Development: Ahri (009) | Reach 10 CS and level 4 within 7 minutes, finishing with Q2, W1, E1 and R0. | player = live_state() assert player.cs >= 10 assert player.level >= 4 assert player.qwer == (2,1,1,0) |
| Sustain: Brand (016) | At the end of 8 minutes, have at least 20 CS and 50% mana. Keep playing until the deadline. | player = deadline_state(480) assert player.cs >= 20 assert player.mana_ratio >= 0.5 |
| Combat: Darius (022) | Personally kill the enemy Soraka bot at least once within 9 minutes, with no more than one death. | player = live_state() kills = personal_kills(540) assert len(kills) >= 1 assert player.deaths <= 1 |
| Cooperation: Lulu (025) | Help allied Ashe secure a kill against Miss Fortune or Soraka within 10 minutes, with at most one death. | player = live_state() kills = allied_kills(600) assert any( player.name in kill.assisters for kill in kills) assert player.deaths <= 1 |
| Map objectives: Warwick (035) | Personally kill a Voidgrub, then destroy the enemy top outer turret within 12 minutes. | grubs = personal_grub_kills(720) turret = enemy_top_outer_kill( 720) assert grubs and turret assert (min(grubs.times) < turret.time) |
| Stage | API family | Usage in LOLPlay |
| Setup | Lobby and bots (LCU) /lol-lobby/ | Create the custom lobby; set map and mode; add and verify bots; start champion selection. |
| Setup | Champion selection (LCU) /lol-champ-select/v1/ | Select and lock the champion; configure summoner spells and the default skin. |
| Setup | Rune pages (LCU) /lol-perks/v1/ | Update and activate the prescribed rune page. |
| Setup | Input settings (LCU) /lol-game-settings/v1/ | Set Q/W/E/R to normal casting; save and verify these settings. |
| Setup | Game phase (LCU) /lol-gameflow/v1/ | Poll the game phase until InProgress . |
| Setup | Live game evidence (LCD) /liveclientdata/allgamedata | Validate champion, roster, map, and initial conditions; retain required attribute baselines. |
| Model | Effort | Task success rate (%) | Subgoal success rate (%) | Mean Turns | Mean Latency (s) |
|---|---|---|---|---|---|
| GPT-6 Astra | Low | 25.0 | 37.7 | 43.9 | 7.47 |
| Medium | 25.0 | 40.3 | 37.8 | 8.39 | |
| High | 16.7 | 33.7 | 27.1 | 16.67 | |
| GPT-6.1 Sol | Low | 13.9 | 32.1 | 42.8 | 8.90 |
| Medium | 11.1 | 31.3 | 33.2 | 11.44 | |
| High | 11.1 | 24.0 | 37.4 | 10.26 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Champion | Min. | Numbered outcome checks |
|---|---|---|---|
| 001 | Caitlyn | 10 | CS |
| 002 | Lee Sin | 10 | personally kill first elemental dragon |
| 003 | Ziggs | 10 | destroy enemy mid outer turret |
| 004 | Sion | 10 | max HP |
| 005 | Varus | 7 | CS |
| 006 | Mordekaiser | 8 | CS |
| Group | Actions | Arguments and behavior |
| Pointer | MOVE_TO , DRAG_TO | Target ; optional duration. Move the pointer or drag with the left button. |
| Click | CLICK , RIGHT_CLICK , DOUBLE_CLICK | Optional ; otherwise use the current pointer. CLICK also accepts a button and 1–3 clicks. |
| Mouse hold | MOUSE_DOWN , MOUSE_UP | Press or release a button (left by default). |
| Scroll | SCROLL | Integer horizontal and/or vertical displacement, dx / dy . |
| Keyboard | PRESS , KEY_DOWN , KEY_UP | Key name; press, hold, or release. PRESS accepts an optional duration. |
| Shortcut | HOTKEY | A list of keys forming a key combination. |
| Status | Meaning | Count |
| Runner stopping reason | ||
| done | Agent emits DONE . | 37 |
| infeasible | Agent emits FAIL . | 60 |
| execution_budget_exhausted | The wall-time/task stop condition is reached. | 129 |
| max_steps | The model-response cap is reached (100 here). | 75 |
| execution_error | An exception interrupts prediction or execution. | 22 |
| Evidence | Fields / payload group | Checker use |
|---|---|---|
| Context | gameData : gameTime , gameMode , mapNumber | Validate match context and elapsed game time against task-start time and budget. |
| Player | allPlayers : champion, identity, team , level , isDead | Match the controlled player and roster; check level and survival; attribute events to teams. |
| Scores | scores : creepScore , deaths , kills , assists | Farming and death constraints; retain score evidence. Attributed combat goals additionally use events. |
| Inventory | items : itemID , count ; activePlayer.currentGold | Check required item identities/counts and available gold. |
| Abilities | activePlayer.abilities : Q/W/E/R abilityLevel | Check learned or unlearned abilities and required ranks. |
| Attributes | activePlayer.championStats : health, resource, and target attributes | Check health/mana ratios, maximum health, and attribute gain relative to the setup baseline. |
| Model | Effort | Positive attainment | Avoidance / absence | State preservation |
|---|---|---|---|---|
| GPT-6 Astra | Low | 26/63 (41.3) | 4/13 (30.8) | 5/6 (83.3) |
| Medium | 25/63 (39.7) | 5/13 (38.5) | 5/6 (83.3) | |
| High | 21/63 (33.3) | 5/13 (38.5) | 5/6 (83.3) | |
| GPT-6.1 Sol | Low | 21/63 (33.3) | 4/13 (30.8) | 5/6 (83.3) |
| Medium | 18/63 (28.6) | 4/13 (30.8) | 6/6 (100.0) | |
| High | 17/63 (27.0) | 4/13 (30.8) | 4/6 (66.7) |
| Effort | Capped runs | Task success | Zero subgoals | Positive attainment | Avoidance / preservation |
| Low | 26 | 1 | 13 | 4/46 | 12/16 |
| Medium | 29 | 0 | 19 | 1/47 | 11/14 |
| High | 14 | 0 | 8 | 0/24 | 8/9 |
Game-Playing Agents
Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.
See all 215 papers