cs.LGApr 15, 2026
SaveLongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
Organizations: University of Oxford · Lawrence Livermore National Laboratory (LLNL) · MBZUAI · KAUST ·
Abstract
As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods.
Figures & tables
Figure 1: Accuracy versus token usage on LongCoT. GPT 5.2 achieves 9.83% with an average of 62K output tokens per problem.
Figure 2: LongCoT problems demand long-horizon reasoning. Each of our five domains requires constructing and traversing a computational dependency graph in a long chain-of-thought. These graphs can be DAGs, search trees, cyclic graphs, constraint graphs, or execution traces. Frontier models struggle with LongCoT problems: even the best model (GPT 5.2) achieves only 9.83% accuracy.
Table 1: Comprehensive benchmark comparison. LongCoT uniquely combines short inputs ( 6K tokens), long reasoning outputs (10K–100K+ tokens), controlled single-step difficulty, no tool dependencies, and verifiable answers across five diverse domains. Existing benchmarks either test short reasoning chains, require long inputs, depend on tools, or do not control for step-level difficulty.
Figure 3: LongCoT problem domains and reasoning structures. (Top) Distribution of subtopics across five domains. (Bottom) Example dependency graphs. Explicit templates present the graph directly in the prompt; Implicit templates require models to discover and navigate latent structure (game trees, constraint satisfaction, simulation). Actual problem graphs are significantly larger and more diverse than these schematic examples. This diversity across domains, graphs, and question types is essential for testing long-horizon reasoning: one problem might not isolate all relevant capabilities (managing context, backtracking, planning, execution), but the full benchmark does.
Figure 4: Main results on LongCoT-mini (left) and LongCoT (right). LongCoT is extremely challenging, with the best model (GPT 5.2) achieving only 9.83% and open-source models near zero. LongCoT-mini differentiates performance across a wider range of models.
Figure 5: LongCoT domain-specific results are mostly stable across all five domains for a given model. These findings comport with the design goals of LongCoT: rather than deep domain knowledge, LongCoT success demands the ability to reason over long horizons.
Figure 6: Accuracy falls as problem DAG sizes grow, inducing planning and execution difficulties before context windows saturate. Our problems increasingly differentiate model capabilities as horizon lengths grow. Under an independent error assumption (GPT 5.2 on Omni-Math), accuracy would be much higher than what we observe, highlighting issues with long-horizon reasoning.
Figure 7: RLM evals. GPT-5.2 RLM tested in reasoning-only (solid) and code simulation (dashed) settings. Without sub-agents calling tools, RLM fails to outperform GPT 5.2 alone. With code simulation, gains appear primarily on implicit domains where substantial parts of the dependency structure can be offloaded to programmatic search routines. See Appendix C for analysis.
Figure 8: Reasoning trace analysis. The distribution of reasoning spent across behaviors varies substantially by domain and model. Each cell represents 1% of the reasoning trace, read left-to-right, top-to-bottom (see Appendix C for more analysis).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Template | Difficulty | GPT5.2 | Gemini 3 Pro | Claude 4.5 Sonnet | Deepseek V3.2 | Grok 4.1 Fast | GLM | Kimi |
| BlocksWorld | Easy | 60.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Medium | 16.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Hard | 4.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Dungeon | Easy | 53.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Medium | 12.0 | 4.0 | 4.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| Hard | 0.0 | 8.0 | 0.0 | 0.0 | 0.0 | 0.0 | 8.0 |
Table 2: Model accuracy (%) on Logic tasks by template and difficulty
| Template | Difficulty | GPT5.2 | Gemini 3 Pro | Claude 4.5 Sonnet | Deepseek V3.2 | Grok 4.1 Fast | GLM | Kimi |
| HindleyMilner | Easy | 38.0 | 0.0 | 4.0 | 0.0 | 2.0 | 0.0 | 0.0 |
| Scheduling | Medium | 22.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 |
| TuringMachine | Medium | 10.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| MaxFlowMinCut | Hard | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Program Sim. | Easy | 45.0 | 45.0 | 20.0 | 30.0 | 20.0 | 25.0 | 35.0 |
| Medium | 16.7 | 13.3 | 10.0 | 0.0 | 10.0 | 0.0 | 3.3 |
Table 3: Model accuracy (%) on CS tasks by template and difficulty
| Template | Difficulty | GPT5.2 | Gemini 3 Pro | Claude 4.5 Sonnet | Deepseek V3.2 | Grok 4.1 Fast | GLM | Kimi |
| Molecules | Easy | 46.0 | 54.0 | 38.0 | 30.0 | 48.0 | 20.0 | 22.0 |
| Easy | 34.0 | 42.0 | 28.0 | 10.0 | 32.0 | 6.0 | 10.0 | |
| Medium | 14.0 | 16.0 | 4.0 | 0.0 | 12.0 | 0.0 | 2.0 | |
| Medium | 10.0 | 28.0 | 12.0 | 2.0 | 8.0 | 0.0 | 2.0 | |
| Medium | 8.0 | 16.0 | 6.0 | 2.0 | 0.0 | 0.0 | 2.0 | |
| Hard | 2.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Table 4: Model accuracy (%) on Chemistry tasks by template and difficulty
| Template | Difficulty | GPT 5.2 | Gemini 3 Pro | Claude 4.5 Sonnet | Deepseek V3.2 | Grok 4.1 Fast | GLM | Kimi |
| Template 1 | Hard | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Template 2 | Medium | 16.0 | 2.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Template 3 | Easy | 54.0 | 8.0 | 2.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Template 4 | Hard | 0.0 | 8.0 | 4.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Template 5 | Hard | 4.0 | 4.0 | 0.0 | 2.0 | 0.0 | 2.0 | 6.0 |
| Template 6 | Easy | 18.0 | 40.0 | 18.0 | 4.0 | 6.0 | 8.0 | 12.0 |
Table 5: Model accuracy (%) on Chess tasks by template and difficulty
| Template | Difficulty | GPT5.2 | Gemini 3 Pro | Claude 4.5 Sonnet | Deepseek V3.2 | Grok 4.1 Fast | GLM | Kimi |
| Linear | Easy | 65.0 | 40.0 | 20.0 | 20.0 | 25.0 | 20.0 | 15.0 |
| Medium | 36.7 | 10.0 | 13.3 | 16.7 | 16.7 | 0.0 | 13.3 | |
| Hard | 6.0 | 4.0 | 4.0 | 4.0 | 4.0 | 0.0 | 2.0 | |
| DAG | Easy | 55.0 | 25.0 | 5.0 | 25.0 | 10.0 | 5.0 | 0.0 |
| Medium | 20.0 | 20.0 | 3.3 | 10.0 | 16.7 | 0.0 | 13.3 | |
| Hard | 4.0 | 4.0 | 2.0 | 0.0 | 6.0 | 0.0 | 0.0 |
Table 6: Model accuracy (%) on Math tasks by DAG template and difficulty
Figure 9: LongCoT-mini domain-specific results provide more signal on model performance. These findings comport with the design goals of LongCoT: rather than deep domain knowledge, LongCoT success demands the ability to reason over long horizons.
Figure 10: GPT-5.2 accuracy and token usage on independent versus composed LongCoT-Math questions. Independent questions are presented in a single prompt with no inter-node dependencies and scored on the same leaf nodes. Accuracy drops sharply when questions are composed while token usage remains comparable, confirming that compositional dependency, not just output length, drives difficulty.
Figure 11: Reasoning trace structure for DeepSeek V3.2 on LongCoT-mini Chemistry Problems. Each trace is segmented into a 10×10 grid (read left-to-right, top-to-bottom ), classified into Setup, Planning, Solving, Verification, Stuck, or Backtracking. Correct traces (top) allocate more budget to setup, while incorrect traces (bottom) show visibly more backtracking (orange) and stuck (red) segments.