The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
Authors: Seth Karten, Jake Grigsby, Tersoo Upaa, Junik Bae, Seonghun Hong, Hyunyoung Jeong, Jaeyoon Jung, Kun Kerdthaisong, Gyungbo Kim, Hyeokgi Kim, Yujin Kim, Eunju Kwon, Dongyu Liu, Patrick Mariglia, Sangyeon Park, Benedikt Schink, Xianwei Shi, Anthony Sistilli, Joseph Twin, Arian Urdu, Matin Urdu, Qiao Wang, Ling Wu, Wenli Zhang, Kunsheng Zhou, Stephanie Milani, Kiran Vodrahalli, Amy Zhang, Fei Fang, Yuke Zhu, Chi Jin
Organizations: Princeton · UT-Austin · Team Heatz · Team Deepest · Team August · Team 4thLesson · Team PA-Agent · Team FoulPlay · Team Hamburg · Team Anthonys · Team Porygon2AI · Team Q · CMU · NYU · Google DeepMind
We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.
Figures & tables
Figure 1: Game Benchmarks. Pokémon creates vast partially observable state spaces (see Appendix G ). Data from [ 23 , 12 , 24 , 13 , 14 , 25 , 15 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ] .
Figure 2: Pokémon Battling.
Figure 3: Baseline Performance. (Left) Agents vs. Humans : Official ratings on the Showdown ladder. Statistics from the Top 500leaderboard are provided as a frame of reference for experienced human players. (Center/Right) RL vs. RL and LLM vs. LLM : GXE is measured relative only to methods within each plot. We differentiate between prior Metamon RL policies [ 37 ] and baselines newly developed for this work; PC-Llama3.1-8B represents the original PokéChamp agent [ 36 ] .
Figure 4: Speedrunning Route (Early Game). Milestones from Littleroot Town (1) to Defeating Roxanne (15), with game frames from each waypoint. The geographic overview (right) maps key locations. Although progression appears linear, the route requires substantial exploration and backtracking—agents must revisit earlier areas, navigate branching paths, and manage nonlinear dependencies between objectives. We provide splits from the human world record as an upper bound.
Figure 5: Speedrunning Track Baseline Results. Cumulative wall-clock time, actions, tokens, and cost at each milestone for five frontier models (mean ± min/max range across runs). Gemini 3 Flash completes the route fastest ( ∼ 2:24 mean) but requires more actions than Gemini 3 Pro. Claude Sonnet 4.5 completes all milestones but with the highest variance and 3–4 × the cost of Gemini variants. GPT-5.2 falls between the two families in both time and cost.
Figure 6: NeurIPS Battling Leaderboard. Organizer and Participant agent ratings are directly comparable. The x-axis measures disagreement between our primary rating metric and the linear trend established by GXE (a metric widely used on Showdown).
Figure 7: NeurIPS Speedrunning Leaderboard. Milestone progress vs. wall-clock time (left) and cumulative agent steps (right) for the six teams that completed all 15 milestones. Unlike most game benchmarks that pause between actions, our environment runs in real time—the game world continues while the agent reasons, making inference latency a first-class cost. We therefore report both wall-clock time (end-to-end throughput) and step count (sample efficiency). This distinction reveals that Deepest † (Judge’s Choice) completed in the fewest steps (649) despite ranking 5th by time, highlighting the tradeoff between inference speed and action efficiency.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Head-to-Head Win Rates Among Top Participants. Heatmap matrices showing pairwise win rates for Gen 1 OU (left) and Gen 9 OU (right). Each cell shows the row player’s win–loss record against the column player. Diagonal cells show each player’s Elo rating. Players are ordered by Elo (highest at top). Empty cells indicate insufficient direct head-to-head matchups.
Figure 9: Tournament Brackets. Best-of-99 single-elimination brackets for Gen 1 OU (left) and Gen 9 OU (right). Numbers indicate games won by each team; green highlighting indicates winners.
Figure 10: Rating System Comparison Across 1.6M+ Matches. Four ranking methods applied to Battling Track data. Left: Rank correlation between methods, showing Glicko-1 converges to Bradley–Terry (batch MLE) while Elo diverges for mid-ranked agents. Right: Rating trajectories with uncertainty bands for selected agents, demonstrating Glicko-1’s uncertainty quantification. Top-3 agents (shaded) show consistent rankings across methods; ranks 4+ exhibit systematic disagreement, particularly between Elo and Bradley–Terry.
Figure 11: Battling Track Qualifying Metrics. Qualifying stage results according to alternative skill metrics (Showdown’s Elo, GXE, Win Rate) and sample size (Battles Played).
Figure 12: PokeChamp Architecture. Within the battle loop, two data sources feed into the LLM pipeline: (1) historical usage statistics from Smogon (left top), providing probability distributions over abilities, EV spreads, and moves for each species; and (2) the current game state from Pokémon Showdown (left bottom). The Text Conversion + LLM module (center) converts game state to structured text, enriches it with usage statistics for opponent prediction, and performs LLM inference to select actions. The chosen move executes and updates the game state for the next turn.
Figure 13: Metamon Training Pipeline. Metamon trains RL agents on large datasets of self-play battles and human demonstrations “reconstructed” from replays gathered from Pokémon Showdown. Figure reproduced from Grigsby et al. [37] .
Model
Size
Date
Description
Human Ladder Ratings (GXE)
G1
G2
G3
G4
G9
SyntheticRLV2
200M
Sep 2024
The best (public) Gen 1 OU policy during the NeurIPS competition, and therefore the basis of many of the qualifying submissions (Appendix E ).
77%
68%
64%
66%
–
Abra
57M
Jul 2025
The best (public) Gen 9 OU policy during the NeurIPS competition, and therefore the basis of many of the qualifying submissions (Appendix E ).
–
–
–
–
50%
Kadabra3
57M
Sep 2025
The best policy to participate in the NeurIPS competition as an organizer baseline (rank #1 in the Gen 1 OU qualifier and #2 in Gen 9 OU).
80%
–
–
–
64%
Kakuna
142M
Dec 2025
The best metamon model as of the launch of the PokéAgent Challenge (with results appearing in Fig. 3 ).
82%
70%
63%
64%
71%
Appendix
Table 1: Metamon baselines referenced in Appendix results, and their estimated performance against human players. “G1” → “Gen 1 OU”.
Figure 14: PokéAgent Multi-Agent Architecture. The orchestrator coordinates sub-agents and MCP tools to play the game. Data flows left-to-right from the GBA emulator (game frames, parsed state) through the Orchestrator (context management, sub-agent dispatch, stuck detection), which invokes either MCP tools (pathfinding, button inputs, knowledge retrieval) for direct game control or specialized sub-agents (battle agent, reflection agent, verification agent, gym puzzle agent) for complex reasoning tasks. A feedback loop returns sub-agent outputs and tool results to inform subsequent decisions.
Figure 15: Foul Play’s MCTS search architecture with damage roll grouping. The engine clusters damage outcomes by knockout potential rather than exploring all 32 possible values (16 rolls × critical hit), enabling deeper search while preserving strategic relevance.
Figure 16: Foul Play’s set prediction pipeline. Hidden information is progressively inferred through damage calculations, priority ordering, weather duration, and move usage patterns, narrowing the space of possible opponent configurations.
Figure 17: 4thLesson’s learning architecture and training pipeline, showing the integration of the Kron optimizer and AID activation function.
Figure 18: Structure of Activation by Interval-wise Dropout (AID) used by 4thLesson to mitigate plasticity loss.
Figure 19: Team Q’s win rate performance across training iterations (v1–v6). Performance is measured against the Competitive v1 Team Set. The initial iterations (v1–v4) were trained on the v0 team set, while v6 incorporates fine-tuning on the v1 set, resulting in a performance jump to approximately 0.8.
Figure 20: Overview of Scripted Policy Distillation (SPD). The approach consists of three stages: (1) Subgoal Generation where an LLM decomposes the task into sequential subgoals with executable success conditions; (2) Scripted Policy Generation where the LLM generates policies that can invoke VLM tools and use self-directed logging for debugging; (3) Script-Guided RL where scripted policies are distilled into neural networks via supervised learning followed by RL with expert action guidance.
Figure 21: Hamburg PokéRunners’ recurrent PPO architecture, showing the CNN encoder for visual input, MLP for positional and milestone encoding, and LSTM for temporal reasoning.
Figure 22: Deepest agent architecture overview. The agent receives partial observations and milestone-specific guidance from the Guidebook. Gemini 2.5 Flash reasons and outputs actions, optionally invoking tools (Pathfinding, Memory, Thinking). Actions execute in the emulator, producing the next observation. History provides temporal context.
Frame + full parsed game state (map, party, items, NPCs)
Navigation, planning tools
Persistent memory store
ReAct loop + Twitch chat
Zero-shot
CLI Agents ‡
Frame + map, party, bag
Game MCP tools (get state, press buttons, pathfinding)
Agent context window
ReAct loop
Zero-shot
PokeAgent
Frame + map, party, bag
Full MCP tool suite (pathfinding, memory, progress summary, reflection, etc)
knowledge base + Action history window
Multi-agent self-reflection
Zero-shot
Appendix
Table 2: Analysis of Pokémon RPG AI systems using the (S,T,M,F,Φ) framework, characterized at the version each system was initially popularized. Heterogeneity across dimensions makes direct performance comparison methodologically unsound without standardization.
Figure 23: Speedrunning Track Evaluation Interface. The interface displays: (1) the Pokémon Emerald game screen showing a trainer battle with move selection (top left), (2) agent reasoning traces with timestamped decision logs detailing battle strategy and action rationale (right panel), (3) current party composition with sprite previews (bottom left), and (4) map overview with action history (bottom right). This visualization enables real-time debugging of agent behavior across the thousands of timesteps required for speedrunning.
Model
Avg Input Tok/Turn
Avg Output Tok/Turn
Avg Turns
Avg Cost/Game
Avg Cost/Turn
Total Cost
Gemini 3.1 Pro
1,245
54
21.1
$0.066
$0.0031
$8.86
GPT-5.2
2,472
4,270
17.8
$1.247
$0.0702
$162.16
Gemini 3 Flash
1,265
76
23.7
$0.020
$0.0009
$0.90
GLM-5
1,658
960
18.3
$0.069
$0.0038
$7.98
Gemini 3 Pro
1,211
115
23.9
$0.091
$0.0038
$3.72
Claude Opus 4.6
2,110
137
24.4
$0.341
$0.0140
$73.59
Appendix
Table 3: LLM baseline token usage and API cost on the Gen 9 OU Extended Timer ladder. Input/output tokens are per-turn averages. Models sorted by GXE rank (descending).
Figure 24: Speedrunning Track Task Diversity. Directed graph showing RPG subtask categories and their dependencies. Core tasks include overworld navigation, battle encounters (wild and trainer), gym puzzles, menu management, and NPC dialogue. Edges indicate how completing one task type enables or requires others—for example, navigation leads to encounters, battles yield experience for team building, and menu interactions manage resources needed for all other tasks.
Format
Team Space (1 player)
Battle State Space
Notes
Gen 1 OU (RBY)
∼1057
∼10192
Singles; no items/abilities/natures; competitive DV constraints
Gen 9 OU
∼10215
∼10564
Singles; Terastallization
Gen 9 VGC
∼10215
∼10622
Doubles; bring 4 of 6; team preview
Chess
—
∼1047
—
Go
—
∼10170
—
Appendix
Table 4: State space complexity across Pokémon formats and classical games. All Pokémon components are exact (E) except HP values (marked A ); status conditions and volatile statuses are exact upper bounds. Team space values are upper bounds. Battle state includes item consumption/knockout state and ability suppression/swap state; PP is omitted as it is rarely strategic in competitive play (see text). Volatile statuses are enumerated from the Pokémon Showdown source code with mutual exclusion groups (protect variants, move locks, grounding/levitation, ability-specific volatiles, self-applied one-turn effects) properly handled. Gen 9 OU and VGC share the same team pool; VGC’s larger battle state reflects 4 simultaneously active Pokémon (vs. 2 in singles), hidden team preview selection, and doubles-specific side conditions and volatiles.
Gen 1 OU
Gen 9 OU
Species in pool
45
117
Avg. moves per species
13.0
14.0
Avg. items per species
—
6.6
Avg. abilities per species
—
1.7
Avg. EV/nature spreads per species
—
7.8
Team state space ( log10 )
23.5
37.8
Appendix
Table 5: Effective team space on Showdown Gen 1 OU and Gen 9 OU ladders. Derived from public usage statistics (2014–2025, Glicko-1 ≥ 1500). Only options appearing on at least 1% of teams (for species) or 1% of that species’ sets (for moves, items, abilities, and EV/nature spreads) are counted.
Figure 25: Empirical usage distributions in Gen 1 OU and Gen 9 OU . Derived from Smogon usage statistics (2014–2025, rank ≥ 1500). Left: cumulative species usage—the fraction of all team-slot appearances accounted for by the top- k most-used species. Right: cumulative move usage per species, averaged over the top-50 species in each format.
Figure 26: Chain-of-Thought Visualization Reveals Failure Modes. Screenshots from our live ladder stream showing synchronized CoT reasoning during a Gen 9 OU battle between Gemma-3-12B (a) and Qwen-3-14B (b). In Turn 2, Gemma articulates a brief strategic assessment (“boosting Special Attack will be beneficial”) and executes Nasty Plot. By Turn 4, Qwen’s reasoning trace fills the panel with recursive damage calculations—enumerating type matchups, computing exact damage ranges, and deliberating between moves—exhibiting the “computational paralysis” described in the text. This failure mode is invisible to win/loss statistics alone.
In this paper, we use Pokémon Battles as a case study to investigate how to improve the performance of LLM-based agents in tasks that require opponent-aware planning without additional fine-tuning. We propose Long/Short-Term Knowledge Augmented Generation (LSTKAG), a mechanism that enables LLM-based agents to leverage past states of the current task and retrieve experience summaries from similar previous task instances based on the current state. Based on LSTKAG, we design Porimon, an LLM-based agent structure for Pokémon Battles. For optimization, we introduce an external API for precise damage calculation and more detailed information about the game. We conduct tournament-like evaluation experiments comprising 15,000 battles for hyperparameter optimization, ablation studies, and performance evaluation. The results indicate that Porimon-based players with hyperparameter optimization significantly outperform players based on PokéLLMon, an LLM-based agent structure proposed in previous research, and the rule-based heuristic player. Furthermore, our ablation study shows that Porimon variants outperform the one without extension in game information retrieval, which shows the contribution of that extension. However, the current experiment results are inconclusive regarding the contribution of Long-Term KAG. These results suggest that introducing external resources, information from previous states of the current task, and experience summaries from similar previous task instances could elevate the performance of LLM-based agents designed for tasks requiring opponent-aware planning.
Dongyin Zhuo, Fengjunjie Pan, Nenad Petrovic +1
Robotics, Artificial Intelligence and Real-Time Systems School of Computation, Information and Technology Technical University of Munich Munich, Germany
Given a strategically complex board game, human players can quickly learn to devise strategies after playing a few rounds. Autonomous agents require similar capabilities in realistic interactive environments, yet existing agent benchmarks often fail to fully capture such strategic and evolving decision-making scenarios. We present PTCG-Bench, a benchmark built on the Pok'{e}mon Trading Card Game (PTCG) that evaluates LLM agents at two complementary levels: (1) their decision-making performance within a single complex environment, and (2) their ability to self-evolving through accumulated experience. We further include a modular harness ablation to better interpret agent performance without conflating it with model capability. Our experiments show that, although LLM agents can achieve non-trivial gameplay performance, sustained and stable self-evolution remains challenging, and performance is sensitive to harness design. We hope that PTCG-Bench will facilitate future research on harness-aware and self-evolving agents in realistic interactive environments.
AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present Agentick, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3-10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.
Roger Creus Castanyer, Pablo Samuel Castro, Glen Berseth
Mila Quebec AI Institute · Université de Montréal · Mila Quebec AI Institute Université de Montréal +1