cs.LGMar 16, 2026
SaveThe PokeAgent Challenge: Competitive and Long-Context Learning at Scale
Organizations: Princeton · UT-Austin · Team Heatz · Team Deepest ·
Abstract
We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.
Figures & tables
Figure 1: Game Benchmarks. Pokémon creates vast partially observable state spaces (see Appendix G ). Data from [ 23 , 12 , 24 , 13 , 14 , 25 , 15 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ] .
Figure 2: Pokémon Battling.
Figure 3: Baseline Performance. (Left) Agents vs. Humans : Official ratings on the Showdown ladder. Statistics from the Top 500leaderboard are provided as a frame of reference for experienced human players. (Center/Right) RL vs. RL and LLM vs. LLM : GXE is measured relative only to methods within each plot. We differentiate between prior Metamon RL policies [ 37 ] and baselines newly developed for this work; PC-Llama3.1-8B represents the original PokéChamp agent [ 36 ] .
Figure 4: Speedrunning Route (Early Game). Milestones from Littleroot Town (1) to Defeating Roxanne (15), with game frames from each waypoint. The geographic overview (right) maps key locations. Although progression appears linear, the route requires substantial exploration and backtracking—agents must revisit earlier areas, navigate branching paths, and manage nonlinear dependencies between objectives. We provide splits from the human world record as an upper bound.
Figure 5: Speedrunning Track Baseline Results. Cumulative wall-clock time, actions, tokens, and cost at each milestone for five frontier models (mean min/max range across runs). Gemini 3 Flash completes the route fastest ( 2:24 mean) but requires more actions than Gemini 3 Pro. Claude Sonnet 4.5 completes all milestones but with the highest variance and 3–4 the cost of Gemini variants. GPT-5.2 falls between the two families in both time and cost.
Figure 6: NeurIPS Battling Leaderboard. Organizer and Participant agent ratings are directly comparable. The x-axis measures disagreement between our primary rating metric and the linear trend established by GXE (a metric widely used on Showdown).
Figure 7: NeurIPS Speedrunning Leaderboard. Milestone progress vs. wall-clock time (left) and cumulative agent steps (right) for the six teams that completed all 15 milestones. Unlike most game benchmarks that pause between actions, our environment runs in real time—the game world continues while the agent reasons, making inference latency a first-class cost. We therefore report both wall-clock time (end-to-end throughput) and step count (sample efficiency). This distinction reveals that Deepest † (Judge’s Choice) completed in the fewest steps (649) despite ranking 5th by time, highlighting the tradeoff between inference speed and action efficiency.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Head-to-Head Win Rates Among Top Participants. Heatmap matrices showing pairwise win rates for Gen 1 OU (left) and Gen 9 OU (right). Each cell shows the row player’s win–loss record against the column player. Diagonal cells show each player’s Elo rating. Players are ordered by Elo (highest at top). Empty cells indicate insufficient direct head-to-head matchups.
Figure 9: Tournament Brackets. Best-of-99 single-elimination brackets for Gen 1 OU (left) and Gen 9 OU (right). Numbers indicate games won by each team; green highlighting indicates winners.
Figure 10: Rating System Comparison Across 1.6M+ Matches. Four ranking methods applied to Battling Track data. Left: Rank correlation between methods, showing Glicko-1 converges to Bradley–Terry (batch MLE) while Elo diverges for mid-ranked agents. Right: Rating trajectories with uncertainty bands for selected agents, demonstrating Glicko-1’s uncertainty quantification. Top-3 agents (shaded) show consistent rankings across methods; ranks 4+ exhibit systematic disagreement, particularly between Elo and Bradley–Terry.
Figure 11: Battling Track Qualifying Metrics. Qualifying stage results according to alternative skill metrics (Showdown’s Elo, GXE, Win Rate) and sample size (Battles Played).
Figure 12: PokeChamp Architecture. Within the battle loop, two data sources feed into the LLM pipeline: (1) historical usage statistics from Smogon (left top), providing probability distributions over abilities, EV spreads, and moves for each species; and (2) the current game state from Pokémon Showdown (left bottom). The Text Conversion + LLM module (center) converts game state to structured text, enriches it with usage statistics for opponent prediction, and performs LLM inference to select actions. The chosen move executes and updates the game state for the next turn.
Figure 13: Metamon Training Pipeline. Metamon trains RL agents on large datasets of self-play battles and human demonstrations “reconstructed” from replays gathered from Pokémon Showdown. Figure reproduced from Grigsby et al. [37] .
| Model | Size | Date | Description | Human Ladder Ratings (GXE) | ||||
|---|---|---|---|---|---|---|---|---|
| G1 | G2 | G3 | G4 | G9 | ||||
| SyntheticRLV2 | 200M | Sep 2024 | The best (public) Gen 1 OU policy during the NeurIPS competition, and therefore the basis of many of the qualifying submissions (Appendix E ). | 77% | 68% | 64% | 66% | – |
| Abra | 57M | Jul 2025 | The best (public) Gen 9 OU policy during the NeurIPS competition, and therefore the basis of many of the qualifying submissions (Appendix E ). | – | – | – | – | 50% |
| Kadabra3 | 57M | Sep 2025 | The best policy to participate in the NeurIPS competition as an organizer baseline (rank #1 in the Gen 1 OU qualifier and #2 in Gen 9 OU). | 80% | – | – | – | 64% |
| Kakuna | 142M | Dec 2025 | The best metamon model as of the launch of the PokéAgent Challenge (with results appearing in Fig. 3 ). | 82% | 70% | 63% | 64% | 71% |
Table 1: Metamon baselines referenced in Appendix results, and their estimated performance against human players. “G1” “Gen 1 OU”.
Figure 14: PokéAgent Multi-Agent Architecture. The orchestrator coordinates sub-agents and MCP tools to play the game. Data flows left-to-right from the GBA emulator (game frames, parsed state) through the Orchestrator (context management, sub-agent dispatch, stuck detection), which invokes either MCP tools (pathfinding, button inputs, knowledge retrieval) for direct game control or specialized sub-agents (battle agent, reflection agent, verification agent, gym puzzle agent) for complex reasoning tasks. A feedback loop returns sub-agent outputs and tool results to inform subsequent decisions.
Figure 15: Foul Play’s MCTS search architecture with damage roll grouping. The engine clusters damage outcomes by knockout potential rather than exploring all 32 possible values (16 rolls critical hit), enabling deeper search while preserving strategic relevance.
Figure 16: Foul Play’s set prediction pipeline. Hidden information is progressively inferred through damage calculations, priority ordering, weather duration, and move usage patterns, narrowing the space of possible opponent configurations.
Figure 17: 4thLesson’s learning architecture and training pipeline, showing the integration of the Kron optimizer and AID activation function.
Figure 18: Structure of Activation by Interval-wise Dropout (AID) used by 4thLesson to mitigate plasticity loss.
Figure 19: Team Q’s win rate performance across training iterations (v1–v6). Performance is measured against the Competitive v1 Team Set. The initial iterations (v1–v4) were trained on the v0 team set, while v6 incorporates fine-tuning on the v1 set, resulting in a performance jump to approximately 0.8.
Figure 20: Overview of Scripted Policy Distillation (SPD). The approach consists of three stages: (1) Subgoal Generation where an LLM decomposes the task into sequential subgoals with executable success conditions; (2) Scripted Policy Generation where the LLM generates policies that can invoke VLM tools and use self-directed logging for debugging; (3) Script-Guided RL where scripted policies are distilled into neural networks via supervised learning followed by RL with expert action guidance.
Figure 21: Hamburg PokéRunners’ recurrent PPO architecture, showing the CNN encoder for visual input, MLP for positional and milestone encoding, and LSTM for temporal reasoning.
Figure 22: Deepest agent architecture overview. The agent receives partial observations and milestone-specific guidance from the Guidebook. Gemini 2.5 Flash reasons and outputs actions, optionally invoking tools (Pathfinding, Memory, Thinking). Actions execute in the emulator, producing the next observation. History provides temporal context.
| System | State | Tools | Memory | Feedback | Fine-tuning |
|---|---|---|---|---|---|
| Claude Plays Pokémon | Frame + location, walkable tiles, party stats | Pathfinder, knowledge base | File-based summaries + knowledge base | ReAct loop | Zero-shot |
| GPT Plays Pokémon | Frame + location, party stats, money, tile colors | Button inputs, self-built minimap | Goal tracker + notepad | Varied | Zero-shot |
| Gemini Plays Pokémon | Frame + object positions, tile properties, navigability | Self-generated tools, code exec, pathfinder agent | Notepad + map markers + mental map | Varied | Zero-shot |
| Nunu AI | Frame + full parsed game state (map, party, items, NPCs) | Navigation, planning tools | Persistent memory store | ReAct loop + Twitch chat | Zero-shot |
| CLI Agents | Frame + map, party, bag | Game MCP tools (get state, press buttons, pathfinding) | Agent context window | ReAct loop | Zero-shot |
| PokeAgent | Frame + map, party, bag | Full MCP tool suite (pathfinding, memory, progress summary, reflection, etc) | knowledge base + Action history window | Multi-agent self-reflection | Zero-shot |
Table 2: Analysis of Pokémon RPG AI systems using the framework, characterized at the version each system was initially popularized. Heterogeneity across dimensions makes direct performance comparison methodologically unsound without standardization.
Figure 23: Speedrunning Track Evaluation Interface. The interface displays: (1) the Pokémon Emerald game screen showing a trainer battle with move selection (top left), (2) agent reasoning traces with timestamped decision logs detailing battle strategy and action rationale (right panel), (3) current party composition with sprite previews (bottom left), and (4) map overview with action history (bottom right). This visualization enables real-time debugging of agent behavior across the thousands of timesteps required for speedrunning.
| Model | Avg Input Tok/Turn | Avg Output Tok/Turn | Avg Turns | Avg Cost/Game | Avg Cost/Turn | Total Cost |
|---|---|---|---|---|---|---|
| Gemini 3.1 Pro | 1,245 | 54 | 21.1 | $0.066 | $0.0031 | $8.86 |
| GPT-5.2 | 2,472 | 4,270 | 17.8 | $1.247 | $0.0702 | $162.16 |
| Gemini 3 Flash | 1,265 | 76 | 23.7 | $0.020 | $0.0009 | $0.90 |
| GLM-5 | 1,658 | 960 | 18.3 | $0.069 | $0.0038 | $7.98 |
| Gemini 3 Pro | 1,211 | 115 | 23.9 | $0.091 | $0.0038 | $3.72 |
| Claude Opus 4.6 | 2,110 | 137 | 24.4 | $0.341 | $0.0140 | $73.59 |
Table 3: LLM baseline token usage and API cost on the Gen 9 OU Extended Timer ladder. Input/output tokens are per-turn averages. Models sorted by GXE rank (descending).
Figure 24: Speedrunning Track Task Diversity. Directed graph showing RPG subtask categories and their dependencies. Core tasks include overworld navigation, battle encounters (wild and trainer), gym puzzles, menu management, and NPC dialogue. Edges indicate how completing one task type enables or requires others—for example, navigation leads to encounters, battles yield experience for team building, and menu interactions manage resources needed for all other tasks.
| Format | Team Space (1 player) | Battle State Space | Notes |
| Gen 1 OU (RBY) | Singles; no items/abilities/natures; competitive DV constraints | ||
| Gen 9 OU | Singles; Terastallization | ||
| Gen 9 VGC | Doubles; bring 4 of 6; team preview | ||
| Chess | — | — | |
| Go | — | — |
Table 4: State space complexity across Pokémon formats and classical games. All Pokémon components are exact (E) except HP values (marked A ); status conditions and volatile statuses are exact upper bounds. Team space values are upper bounds. Battle state includes item consumption/knockout state and ability suppression/swap state; PP is omitted as it is rarely strategic in competitive play (see text). Volatile statuses are enumerated from the Pokémon Showdown source code with mutual exclusion groups (protect variants, move locks, grounding/levitation, ability-specific volatiles, self-applied one-turn effects) properly handled. Gen 9 OU and VGC share the same team pool; VGC’s larger battle state reflects 4 simultaneously active Pokémon (vs. 2 in singles), hidden team preview selection, and doubles-specific side conditions and volatiles.
| Gen 1 OU | Gen 9 OU | |
|---|---|---|
| Species in pool | 45 | 117 |
| Avg. moves per species | 13.0 | 14.0 |
| Avg. items per species | — | 6.6 |
| Avg. abilities per species | — | 1.7 |
| Avg. EV/nature spreads per species | — | 7.8 |
| Team state space ( ) | 23.5 | 37.8 |
Table 5: Effective team space on Showdown Gen 1 OU and Gen 9 OU ladders. Derived from public usage statistics (2014–2025, Glicko-1 1500). Only options appearing on at least 1% of teams (for species) or 1% of that species’ sets (for moves, items, abilities, and EV/nature spreads) are counted.
Figure 25: Empirical usage distributions in Gen 1 OU and Gen 9 OU . Derived from Smogon usage statistics (2014–2025, rank 1500). Left: cumulative species usage—the fraction of all team-slot appearances accounted for by the top- most-used species. Right: cumulative move usage per species, averaged over the top-50 species in each format.
Figure 26: Chain-of-Thought Visualization Reveals Failure Modes. Screenshots from our live ladder stream showing synchronized CoT reasoning during a Gen 9 OU battle between Gemma-3-12B (a) and Qwen-3-14B (b). In Turn 2, Gemma articulates a brief strategic assessment (“boosting Special Attack will be beneficial”) and executes Nasty Plot. By Turn 4, Qwen’s reasoning trace fills the panel with recursive damage calculations—enumerating type matchups, computing exact damage ranges, and deliberating between moves—exhibiting the “computational paralysis” described in the text. This failure mode is invisible to win/loss statistics alone.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,494 papers