Too long; didn't solve

Lucía M. Cabrera; Isaac Saxton-Knight; Jocelyn D'Arcy

cs.AIApr 8, 2026

Too long; didn't solve

Authors: Lucía M. Cabrera, Isaac Saxton-Knight, Jocelyn D'Arcy

Organizations: Instituto Balseiro · Poindexter Labs

Abstract

Mathematical benchmarks consisting of a range of mathematics problems are widely used to evaluate the reasoning abilities of large language models, yet little is known about how their structural properties influence model behaviour. In this work, we investigate two structural length variables, prompt length and solution length, and analyse how they relate to model performance on a newly constructed adversarial dataset of expert-authored mathematics problems. We find that both prompt and solution lengths correlate positively with increased model failure across models. We also include a secondary, exploratory analysis of cross-model disagreement. Under a difficulty-adjusted normalised analysis, both variables retain weak negative associations with realised model separation, slightly stronger for prompt length. Overall, our main robust finding is that structural length is linked to empirical difficulty in this dataset.

Figures & tables

Model	Mean failure fraction $\mu_{m}$
GPT 5	0.416
GPT 4.1	0.860
GPT OSS 120B	0.690
Gemini 2.5 Flash	0.701
Claude Sonnet 4.5	0.680

Table 1: Mean failure fraction across the 607 evaluated problems for each model.

Figure 1: Number of problems, color-coded by level, producing mean failure rate $\mu_{i}$ across all five models.

Panel of Figure 2 — Figure 2: Failure fraction $x_{i,m}$ as a function of structural length variables. Error bars show $95\%$ bootstrap confidence intervals.

Length Variable	$\rho(\mu)$	$\rho(\widetilde{\mathrm{Var}})$
Prompt	0.28	$-0.21$
Solution	0.32	$-0.17$

Table 2: Task-level Spearman correlations between structural length variables and model behaviour. The second column reports the primary result of the paper, namely the association with mean failure. The third column reports a secondary exploratory association with difficulty-adjusted cross-model disagreement, computed on the 517 tasks with

0<\mu_{i}<1

Panel of Figure 3 — Figure 3: Predicted failure fraction $x_{i,m}$ as a function of $L_{i}=\log(1+\text{word count})$ under the fitted mixed-effects model. Solid lines show model-specific fits; the dashed line shows the global fixed effect.

Prompt Length Model
Parameter	Estimate	SE	p-value
$\beta_{1}$	0.118	0.037	0.001
$\sigma_{u}^{2}$	0.281	–	–
$\sigma_{v}^{2}$	0.006	–	–
$\sigma_{uv}$	$-0.042$	–	–
$\sigma^{2}$	0.1153	–	–

Table 3: Mixed-effects model estimates for failure fraction

x_{i,m}

as a function of log-transformed prompt length,

L_{i}^{(\text{prompt})}

Solution Length Model
Parameter	Estimate	SE	p-value
$\beta_{1}$	0.137	0.026	0.001
$\sigma_{u}^{2}$	0.225	–	–
$\sigma_{v}^{2}$	0.003	–	–
$\sigma_{uv}$	-0.025	–	–
$\sigma^{2}$	0.1128	–	–

Table 4: Mixed-effects model estimates for failure fraction

x_{i,m}

as a function of log-transformed solution length,

L_{i}^{(\text{solution})}

Explore similar work

MathDuels: Evaluating LLMs as Problem Posers and Solvers
Apr 23, 2026Zhiqiu Xu, Shibo Jin, Shreya Arya +1Mathematical Reasoning Benchmarks LLM Evaluation
Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania
Reasoning Models Don't Just Think Longer, They Move Differently
May 14, 2026Anders Gjølbye, Lars Kai Hansen, Sanmi KoyejoLLM Interpretability Representation Geometry in Language Models
Technical University of Denmark · Stanford University
Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning
Sep 26, 2026Zhuohan Wang, Haoran Ma, Tianyu Wu +3LLM Mathematical Reasoning LLM Evaluation
Harvard University · Rutgers University · The Hong Kong University of Science and Technology

Apr 23, 2026J/K move · Enter open · S save

MathDuels: Evaluating LLMs as Problem Posers and Solvers

Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik

Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania

Mathematical Reasoning Benchmarks LLM Evaluation LLM Mathematical Reasoning

Open paper AskPDF arXiv

As frontier language models attain near-ceiling performance on static mathematical benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets. We introduce MathDuels, a self-play benchmark in which models occupy dual roles: each authors math problems under adversarial prompting and solves problems authored by every other participant. Problems are produced through a three-stage generation pipeline (meta-prompting, problem generation, and difficulty amplification), and validated by an independent verifier that excludes ill-posed questions. A Rasch model (Rasch, 1993) jointly estimates solver abilities and problem difficulties; author quality is derived from the difficulties of each model's authored problems. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation reveals capability separations invisible in single-role benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the benchmark's difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. We host a public leaderboard that updates as new models are released.

Abstract

Explore similar work

MathDuels: Evaluating LLMs as Problem Posers and Solvers

Reasoning Models Don't Just Think Longer, They Move Differently

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning