Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, even GPT-5 achieves only 27.8% accuracy on Level 3. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data are available at https://github.com/LongReasonArena/LongReasonArena.
Figures & tables
Figure 1: (a) The construction process of LongReasonArena (b) Algorithmic problems abstract key elements of long reasoning, including retrieval, backtracking, and other essential cognitive operations.
Level 1
Level 2
Level 3
# Problems
322
390
368
# Samples
696
908
769
Median execution line count
1,197
30,683
183,628
90th percentile of execution line count
7,045
79,184
544,348
Median input length
66
680
2,702
90th percentile of input length
766
8,110
23,494
Table 1: The statistics of datasets at different levels.
Reasoning Model
Level 1
Level 2
Level 3
GPT-5
✓
84.2
48.3
27.8
Claude Opus 4.5
✓
66.1
29.1
17.6
o3
✓
65.4
36.6
22.4
o1
✓
62.8
32.0
20.0
Qwen3-32B
✓
56.3
20.3
13.5
QwQ
✓
56.2
20.6
13.4
Table 2: Evaluation results of 14 models across difficulty levels.
Figure 2: Model accuracy with respect to (a) expected number of reasoning steps and (b) input length. Accuracy consistently decreases with both increasing reasoning and input difficulty. A strong log-linear relationship is observed with respect to the expected number of reasoning steps across all reasoning models.
Model
Level 1
Level 2
Level 3
GPT-5 (medium)
84.2
48.3
27.8
GPT-5 (high)
86.6
49.8
33.6
Qwen3-8B (32K)
50.4
14.9
8.5
Qwen3-8B (128K)
50.1
14.7
8.8
Table 3: Impact of reasoning effort and context length on model accuracy.
Model
AIME 2025
GPQA Diamond
LiveCodeBench
LongReasonArena
GPT-5
94.6
85.7
84.6
84.2
Claude Opus 4.5
91.3
87.0
87.1
66.1
o3
88.9
83.3
80.8
65.4
Qwen3-32B
72.9
68.4
65.7
56.3
DeepSeek-R1
70.0
71.5
63.5
42.2
Table 4: Comparison between model performance on real-world reasoning tasks and LongReasonArena . Scores on AIME 2025, GPQA Diamond, and LiveCodeBench are collected from public sources OpenAI (2025a) ; Anthropic (2025a) ; Yang et al. (2025) ; DeepSeek-AI et al. (2025) ; Artificial Analysis (2026) . The LongReasonArena column reports Level 1 accuracy.
Figure 3: Performance trends for the Two Sum and Word Search tasks. Both tasks exhibit a linear decline in accuracy as reasoning complexity increases: Two Sum with respect to the logarithm of array length, and Word Search with respect to word length.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Reasoning Models
Non-Reasoning Models
MaxNewTokens
32K
8K
System Prompt
N/A
You are a helpful assistant
Temperature
0.6
TopP
0.95
MinP
0
TopK
40
Appendix
Table 5: Settings for vLLM Inference
Seed
1
2
3
4
Level 1
48.6
49.6
49.3
48.3
Level 2
14.4
15.6
16.0
15.2
Seed
5
6
7
8
Level 1
50.6
49.3
49.0
47.8
Level 2
14.6
15.3
16.3
15.7
Appendix
Table 6: Evaluation results of Qwen3-8B on Level 1 and Level 2 under different random seeds.
Figure 4: Relationship between the expected number of reasoning steps and reasoning length of (a) QwQ and (b) DeepSeek-R1-Distill-Qwen-32B. The reasoning length of correct samples generally increases with higher reasoning complexity, but the reasoning length of incorrect samples is significantly longer than that of correct ones
Figure 5: Error type distribution for the Two Sum task. The distribution shows a transition from index errors to full errors as array length increases.
R2
p-value
GPT-5
0.964
<1e-4
QwQ
0.930
<1e-4
Claude Opus 4.5
0.964
<1e-4
o3
0.966
<1e-4
Appendix
Table 7: The significance of linear regression between model accuracy and the logarithm of expected number of reasoning steps.