cs.AIOct 25, 2026

LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends

Authors: Zhaojian Yu

Abstract

Visual agents need to coordinate reasoning and action while their environment evolves during inference. We introduce LOLPlay, a benchmark in native League of Legends with 36 champion-specific tasks spanning resource management, combat, cooperation, and objectives. Agents interact through screenshots and mouse-keyboard actions; automated initialization and 82 executable subgoal checks distinguish partial progress from completion. Across 324 task evaluations of three models at three reasoning-effort settings, GPT-6 Astra achieves the highest task success rate of 25.0% at low and medium effort, and the highest subgoal success rate of 40.3% at medium effort. However, high effort improves neither metric over low effort for any model. For Astra, longer requests leave fewer interaction rounds, suggesting a cost to additional reasoning in real-time control. Trajectories reveal incomplete subgoal composition and actions executing after the observed scene changes, motivating agents that combine visual reasoning with timely action.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock

    Oct 14, 2026Ningxin Pan, Hanyu Li, Yeqing Shen +3LLM Agent Self-ImprovementLLM Decision-Making

  2. DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination

    Oct 22, 2026Ziyi Jiao, Heng Zhang, Drishti Regmi +3Multi-Agent CoordinationEmbodied Agent Benchmarks

  3. DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

    Aug 25, 2026Jintao Cheng, Weibin LiVLM EvaluationVLM Reasoning