Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, even GPT-5 achieves only 27.8% accuracy on Level 3. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data are available at https://github.com/LongReasonArena/LongReasonArena.
Figures & tables
Figure 1: (a) The construction process of LongReasonArena (b) Algorithmic problems abstract key elements of long reasoning, including retrieval, backtracking, and other essential cognitive operations.
Level 1
Level 2
Level 3
# Problems
322
390
368
# Samples
696
908
769
Median execution line count
1,197
30,683
183,628
90th percentile of execution line count
7,045
79,184
544,348
Median input length
66
680
2,702
90th percentile of input length
766
8,110
23,494
Table 1: The statistics of datasets at different levels.
Reasoning Model
Level 1
Level 2
Level 3
GPT-5
✓
84.2
48.3
27.8
Claude Opus 4.5
✓
66.1
29.1
17.6
o3
✓
65.4
36.6
22.4
o1
✓
62.8
32.0
20.0
Qwen3-32B
✓
56.3
20.3
13.5
QwQ
✓
56.2
20.6
13.4
Table 2: Evaluation results of 14 models across difficulty levels.
Figure 2: Model accuracy with respect to (a) expected number of reasoning steps and (b) input length. Accuracy consistently decreases with both increasing reasoning and input difficulty. A strong log-linear relationship is observed with respect to the expected number of reasoning steps across all reasoning models.
Model
Level 1
Level 2
Level 3
GPT-5 (medium)
84.2
48.3
27.8
GPT-5 (high)
86.6
49.8
33.6
Qwen3-8B (32K)
50.4
14.9
8.5
Qwen3-8B (128K)
50.1
14.7
8.8
Table 3: Impact of reasoning effort and context length on model accuracy.
Model
AIME 2025
GPQA Diamond
LiveCodeBench
LongReasonArena
GPT-5
94.6
85.7
84.6
84.2
Claude Opus 4.5
91.3
87.0
87.1
66.1
o3
88.9
83.3
80.8
65.4
Qwen3-32B
72.9
68.4
65.7
56.3
DeepSeek-R1
70.0
71.5
63.5
42.2
Table 4: Comparison between model performance on real-world reasoning tasks and LongReasonArena . Scores on AIME 2025, GPQA Diamond, and LiveCodeBench are collected from public sources OpenAI (2025a) ; Anthropic (2025a) ; Yang et al. (2025) ; DeepSeek-AI et al. (2025) ; Artificial Analysis (2026) . The LongReasonArena column reports Level 1 accuracy.
Figure 3: Performance trends for the Two Sum and Word Search tasks. Both tasks exhibit a linear decline in accuracy as reasoning complexity increases: Two Sum with respect to the logarithm of array length, and Word Search with respect to word length.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Reasoning Models
Non-Reasoning Models
MaxNewTokens
32K
8K
System Prompt
N/A
You are a helpful assistant
Temperature
0.6
TopP
0.95
MinP
0
TopK
40
Appendix
Table 5: Settings for vLLM Inference
Seed
1
2
3
4
Level 1
48.6
49.6
49.3
48.3
Level 2
14.4
15.6
16.0
15.2
Seed
5
6
7
8
Level 1
50.6
49.3
49.0
47.8
Level 2
14.6
15.3
16.3
15.7
Appendix
Table 6: Evaluation results of Qwen3-8B on Level 1 and Level 2 under different random seeds.
Figure 4: Relationship between the expected number of reasoning steps and reasoning length of (a) QwQ and (b) DeepSeek-R1-Distill-Qwen-32B. The reasoning length of correct samples generally increases with higher reasoning complexity, but the reasoning length of incorrect samples is significantly longer than that of correct ones
Figure 5: Error type distribution for the Two Sum task. The distribution shows a transition from index errors to full errors as array length increases.
R2
p-value
GPT-5
0.964
<1e-4
QwQ
0.930
<1e-4
Claude Opus 4.5
0.964
<1e-4
o3
0.966
<1e-4
Appendix
Table 7: The significance of linear regression between model accuracy and the logarithm of expected number of reasoning steps.
As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.
Sumeet Ramesh Motwani, Daniel Nichols, Charles London +17
University of Oxford · Lawrence Livermore National Laboratory (LLNL) · MBZUAI +4
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.
Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.
Shiwen Qin, Cameron Barker, Henry Gouk +4
University of Edinburgh, Edinburgh, United Kingdom