We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs). In LLM-Wikirace, models must efficiently navigate Wikipedia hyperlinks step by step to reach a target page from a given source, requiring look-ahead planning and the ability to reason about how concepts are connected in the real world. We evaluate a broad set of open- and closed-source models, including Gemini-3, GPT-5, and Claude Opus 4.5, which achieve the strongest results on the easy level of the task and demonstrate superhuman performance. Despite this, performance drops sharply on hard difficulty: the best-performing model, Gemini-3, succeeds in only 23% of hard games, highlighting substantial remaining challenges for frontier models. Our analysis shows that world knowledge is a necessary ingredient for success, but only up to a point, beyond this threshold, planning and long-horizon reasoning capabilities become the dominant factors. Trajectory-level analysis further reveals that even the strongest models struggle to replan after failure, frequently entering loops rather than recovering. LLM-Wikirace is a simple benchmark that reveals clear limitations in current reasoning systems, offering an open arena where planning-capable LLMs still have much to prove. Our code and leaderboard available at https:/llmwikirace.github.io.
Figures & tables
Figure 1: (a) In LLM-WikiRace, the agent navigates the Wikipedia hyperlink graph from a source page (e.g., Banana ) to a target page (e.g., Ferrari ). At each step, it receives the current page, the titles of its outgoing links, the target, and the history of pages visited, then selects one link to follow. (b) The completion rate — the fraction of games solved — plotted against the step budget across different shortest source-to-target path lengths for Llama-3-70B-Instruct , this experiment highlights how the number of steps per game and source to target shortest path length affect model performance.
Figure 2: The LLM-WikiRace prompt, at each step of the game an LLM is prompted with the current page, the target page, a history of previously visited states and 50 possible next states.
Figure 3: The Success Rates for the best performing models across all three difficulties of the LLM-WikiRace benchmark.
Family
Model
Success Rate (%) ↑
Suboptimal Steps ↓
Usage Per Step
Easy
Medium
Hard
Easy
Medium
Hard
Tokens
Cost (cents)
GPT
GPT-5 Nano
71.5
24.7
4.0
3.4
7.6
9.2
2170
0.07
GPT-5 Mini
85.5
46.0
11.0
2.2
3.9
4.6
874
0.11
GPT-5
92.5
60.0
15.0
1.8
2.7
6.3
1826
1.51
GPT-5.2
90.0
50.7
10.0
2.2
3.2
4.0
1008
0.85
Gemini
Gemini 2.0 Flash
88.0
41.3
6.0
4.1
6.0
14.7
501
0.01
Table 1: Performance of different models across Easy, Medium, and Hard datasets. Success Rate is the percentage of games completed within the allowed 30 steps. Suboptimal Steps is the average number of suboptimal steps made in the successfully completed games. If no games were successfully completed for a given model and difficulty, we set Suboptimal Steps to "-". The best result is in bold , and the second best result is underlined for each difficulty level’s success rate.
Figure 4: LLM-WikiRace evaluates both internal world knowledge and planning, we identify a clear performance gap between the Reasoning models and Instruct tuned models despite both showing similar levels of world knowledge.
Figure 5: Raw response of Gemini 3 pro for the game instance Herstory → Warth (optimal path length =4 ), first step. The response highlights the model’s knowledge of the start and target page as well as the overall strategy of moving towards a broad topic initially.
Figure 6: Success rate vs loop frequency for different models. We say that a game contains a loop if model visited any page more than once. Each model family is displayed with different colors on the scatter plot and the best model in each family is shown with an X mark. Dashed gray line shows linear regression fitted to points.
Model
Easy
Medium
Hard
Base
22.5
1.3
0.0
FT-50
24.0
1.3
0.0
FT-100
41.0
4.0
0.0
FT-300
67.5
4.6
0.0
Table 2: Success rate across difficulty levels for the base model and DAPO-finetuned checkpoints. Number after FT− is the number of training steps model was finetuned for.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: We create easy, medium, and hard difficulty splits using the length of the shortest path between the start and target concept to determine difficulty. a) The completion rate is plotted against the number of game steps for Llama-3.3-70B-Instruct , over a variety different optimal path lengths. Across different path lengths most models consistently pass or fail by step 30. b) We analyse the performance of the easy split, when the model is provided with {30,50,70} next node options, we settle on 50 options at each step to ensure the easy split is not saturated. c) The extra steps taken for successful runs increases as the optimal path length increases, we hypothesize this is because more sub-optimal moves accumulate over longer tasks and require more steps to correct.
Model
Link Selection
Easy
Medium
Hard
GPT-5 Mini
Top-50
85.5
46.0
11.0
GPT-5 Mini
Top-1 + Random
86.5
46.98
7.00
Gemini 2.5-Flash
Top-50
91.0
53.00
10.2
Gemini 2.5-Flash
Top-1 + Random
82.0
33.33
3.00
Appendix
Table 3: Closed source success rate (%) under different link selection strategies
Model
Link Selection
Easy
Medium
Hard
Apertus 70B
Top-50
65.0
10.7
0.0
Apertus 70B
Top-1 + Random
65.0
12.0
0.0
Apertus 8B
Top-50
42.0
4.0
0.0
Apertus 8B
Top-1 + Random
41.5
4.7
0.0
Gemma 3 12B
Top-50
72.5
22.7
1.0
Gemma 3 12B
Top-1 + Random
70.5
20.7
1.0
Appendix
Table 4: Open Source success rate (%) under different link selection strategies
Figure 8: The LLM-WikiRace prompt with current page context added.
Model
Page Context
Easy
Medium
Hard
Apertus 8B
False
41.5
4.7
0.0
Apertus 8B
True
46.5
6.7
0.0
Gemma 3 12B
False
70.5
20.7
1.0
Gemma 3 12B
True
75.5
20.0
0.0
Gemma 3 27B
False
80.5
30.0
0.0
Gemma 3 27B
True
82.0
29.3
0.0
Appendix
Table 5: Results for the context prompt vs. no-context prompt ablation experiment of LLM WikiRace
Figure 9: A schematic of the prompt provided to the LLM, when asking it to select the link to follow
Hyperparameter
Value
Model & Algorithm
Base model
Qwen2.5-7B
RL algorithm
DAPO
KL penalty
low-variance KL
Optimization
Learning rate
1×10−6
Appendix
Table 6: Training hyperparameters for DAPO fine-tuning experiment on LLM-WikiRace.
Figure 10: Validation completion rate on LLM-WikiRace throughout DAPO fine-tuning of Qwen2.5-7B. The model improves rapidly during the first ∼ 30 steps, climbing from a 7.4% zero-shot baseline to over 30% , before plateauing around 33 – 35% for the remainder of training.
Figure 11: Comparison of suboptimal steps made by the most powerful models and by Human players, across the Human Gameplay Corpus games. The success rate on this corpus for was 98.5% for humans and 100% for all tested models.
Figure 12: Recovery and max visitation
Figure 13: Raw answer of Gemini 3 highlighting the model’s ability to plan ahead.
Figure 14: Raw answer reveals the model is explicitly stating possibly future paths when reasoning about the decision.
Figure 15: The model is initially stuck in a loop repeating actions in steps 9 and 15 without realising the repetition. However, in step 26 the model shows awareness and chooses an alternative options to break out of the loop. This leads to a successful completion of the task.
Figure 16: Gemini 3 fails to escape the loop and ultimately fails to complete the task, which highlights a critical limitation in terms of replanning ability.
Figure 17: Claude Opus 4.5 follows the same response pattern and overall strategy as Gemini 3 Pro.
Figure 18: This example trace showcases Claude Opus 4.5 planning ahead.
Figure 19: Claude Opus 4.5 also navigates to hub pages to traverse different topics.
Figure 20: The model is stuck on the page Beri for several steps, despite recognising the loop. After looping three times, it adapts its decision but still fails to complete the task within in the step limit.
Figure 21: Gemini 3 successfully navigates this tasks.
Figure 22: Claude 4.5 Opus fails due to incorrect world knowledge and the inability to adapt.
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Gaoling School of Artificial Intelligence, Renmin University of China · LLM Department, Hunyuan Team, Tencent · Beijing Academy of Artificial Intelligence +1