Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Shiwen Qin; Cameron Barker; Henry Gouk; Elliot J. Crowley; Shay B. Cohen; Mirella Lapata; Miao Li

cs.CLOct 15, 2026

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Authors: Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley, Shay B. Cohen, Mirella Lapata, Miao Li

Organizations: University of Edinburgh, Edinburgh, United Kingdom

Abstract

Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.

Figures & tables

Figure 1: Equal input length, different difficulty due to evidence scope, dispersion and reasoning depth.

Figure 2: The dataset construction pipeline of LURCH .

Figure 3: Distributions of input context length, evidence scope, evidence dispersion, and reasoning depth. Dashed and dotted lines indicate the median and mean, respectively; zero dispersion denotes evidence contained in a single contiguous block.

Predictor	$\hat{\beta}$ (per SD)	$\Delta_{25\to 75}$ (pp)	$p_{\mathrm{adj}}$
Context length (reference)	$-0.11\ [-0.17,-0.06]$	$-1.66\ [-2.49,-0.83]$	–
Evidence scope	$-0.21\ [-0.30,-0.12]$	$-2.14\ [-3.07,-1.20]$	$3.4\times 10^{-5}$
Evidence dispersion	$-0.10\ [-0.17,-0.03]$	$-1.10\ [-1.84,-0.36]$	$4.0\times 10^{-3}$
Reasoning depth	$-0.10\ [-0.15,-0.05]$	$-1.60\ [-2.35,-0.85]$	$7.4\times 10^{-5}$

Table 1: RQ1 coefficient estimates and interquartile contrasts.

\Delta_{25\to 75}

is the interquartile contrast ( Section 5.2 ), averaged over all 6,550 questions. Brackets give 95% CIs.

p_{\mathrm{adj}}

tests

\beta_{k}=0

using the RQ1 Holm family.

Figure 4: Per-LLM interquartile contrasts, averaged over all questions as in Table 1 . Points show predicted score changes; bars give 95% CIs.

Figure 5: Predicted score at low (dashed) and high (solid) attribute values across within-task length percentiles, averaged over all questions. Bands give pointwise 95% CIs.

Attribute	Low	High	Difference (high $-$ low)	$p_{\mathrm{adj}}$
	Length contrast (pp)
Evidence scope	$-0.75$	$-1.64$	$-0.90\ [-1.43,-0.37]$	$5.5\times 10^{-3}$
Evidence dispersion	$-1.13$	$-2.01$	$-0.88\ [-1.45,-0.32]$	$1.0\times 10^{-2}$
Reasoning depth	$-1.16$	$-1.82$	$-0.66\ [-1.43,\phantom{-}0.10]$	$1.8\times 10^{-1}$

Table 2: RQ2 score contrasts (pp), averaged over all 6,550 questions. Tied attribute quartiles contribute zero to the high-minus-low difference. Brackets give 95% CIs;

p_{\mathrm{adj}}

tests a zero difference using the six-test RQ2 Holm family.

Figure 6: Mean score relative to the task–LLM mean (pp), by within-task attribute tercile. Panel (d) omits SciTrek. Residuals, confidence intervals and question counts are reported in Table 12 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Data Source	# LURCH	Context-L	Question-L	Answer-L	Metric	Thres
SciTrek	1000/5491	82,536	132	18	F1	1
HotpotQA	1663	93,580	65	5	F1	1
DocFinQA	1994	136,392	46	4	F1	1
LongMemEval	226	114,241	89	4	F1	0.8
LongBench-Pro	181	111,334	214	22	F1	0.8
$\infty$ Bench (code-debug)	356	117,695	142	1	EM	1

Appendix

Table 3: Dataset statistics of LURCH . Context, Question and Answer Length give average numbers of tokens. “Thres” is the minimum score we count as correct. We use it only for the correctness checks in zero-context screening and evidence validation; the analyses in Section 5 use raw scores. SciTrek is randomly subsampled to 1,000 instances to balance the number of instances across sources.

Figure 7: A reasoning graph of depth 2 (HotpotQA; Gemini annotation). Each release year is read from a separate evidence statement at level 1. Comparing them is one further step, and the answer depends on that step, so the depth is 2.

Measure	Exact	Gap of 1	Gap of 2 or more	Spearman’s $\rho$
Graph depth	4,299 (65.6%)	2,067 (31.6%)	184 (2.8%)	0.48
Graph width	3,906 (59.6%)	2,026 (30.9%)	618 (9.4%)	0.42
Graph steps	3,058 (46.7%)	2,427 (37.1%)	1,065 (16.3%)	0.56

Appendix

Table 4: Reasoning graph agreement between the Gemini and GPT annotators (N=6,550).

	Auditor A	Auditor B
Both acceptable	38	23
Both inadequate	0	0
Prefers Gemini	2	14
Prefers GPT	0	3

Appendix

Table 5: Graph preference of each auditor.

Figure 8: Distributions of context length and the three properties by task; numbers beside task names give question counts. For length, scope and dispersion, dots mark medians, thick bars span the interquartile range, and thin bars span the 10th–90th percentiles. Scope uses a logarithmic axis. Stacked bars show the proportions at each reasoning depth.

Figure 9: Pearson correlations of $\log_{2}$ context length, evidence scope, evidence dispersion and reasoning depth, using one observation per question ( $N=6{,}550$ ). Panel (a) pools all questions; panel (b) subtracts each task’s mean logged values before computing correlations. Zero values receive the same floor as in Section 5.1 .

Predictors	Cross-entropy	Loss reduction (%)
Constant (training mean)	0.4196	0.00
Task only	0.4006	4.51
LLM only	0.4145	1.21
Length only	0.4171	0.60
Length + three properties only	0.4165	0.74
Task + LLM	0.3954	5.77

Appendix

Table 6: Cross-validated binary cross-entropy with fractional score targets and natural logarithms (lower is better). Loss reduction is relative to the constant training-mean predictor. Every GLM includes an intercept; length and the three properties (scope, dispersion and depth) are logged.

Figure 10: Out-of-fold calibration for the eight specifications in Table 6 . Points show mean predictions and observed scores in prediction bins of width 0.05; the dashed line denotes equality. Larger markers indicate more question–LLM pairs, and hollow markers indicate fewer than 50 distinct questions. All nonempty bins are shown.

	Context length	Evidence scope	Evidence dispersion	Reasoning depth
Common tasks, Q25 $\to$ Q75	$-2.12\ [-3.19,-1.06]$	$-2.72\ [-3.90,-1.54]$	$-0.77\ [-1.28,-0.25]$	$-3.02\ [-4.44,-1.61]$
All LLMs, doubling	$-2.13\ [-3.22,-1.05]$	$-1.04\ [-1.51,-0.57]$	$-0.28\ [-0.47,-0.09]$	$-2.70\ [-4.04,-1.36]$
Gemini 3.8 Flash, doubling	$-0.20\ [-1.38,\phantom{-}0.98]$	$0.74\ [\phantom{-}0.21,\phantom{-}1.26]$	$-0.58\ [-0.85,-0.32]$	$-2.13\ [-3.71,-0.55]$
GPT-5.6 Luna, doubling	$-1.78\ [-3.08,-0.48]$	$-1.53\ [-2.08,-0.98]$	$-0.15\ [-0.38,\phantom{-}0.07]$	$-2.72\ [-4.27,-1.17]$
Qwen3.8-27B, doubling	$-1.40\ [-2.44,-0.36]$	$-0.67\ [-1.16,-0.18]$	$-0.31\ [-0.54,-0.08]$	$-0.88\ [-2.06,\phantom{-}0.30]$
Qwen3.5-4B, doubling	$-5.14\ [-6.76,-3.52]$	$-2.69\ [-3.33,-2.04]$	$-0.17\ [-0.41,\phantom{-}0.07]$	$-5.23\ [-7.11,-3.35]$

Appendix

Table 7: Supplementary RQ1 score contrasts (pp; 95% CIs). Top: interquartile contrasts on the same 3,471 questions in the nine tasks with distinct depth quartiles, using the full-data GLM. Bottom: doubling contrasts, which average the predicted score change when a predictor is doubled, over questions with positive values whose doubles remain within their task’s observed range. Coverage: context length 34%, evidence scope 91%, evidence dispersion 49%, reasoning depth 60% of questions. Per-LLM rows use separate fits.

Attribute	Link only (pp)	$\hat{\gamma}_{Lj}$	$p_{\mathrm{adj}}$
Evidence scope	$-0.16\ [-0.28,-0.04]$	$-0.070\ [-0.121,-0.019]$	$2.1\times 10^{-2}$
Evidence dispersion	$-0.10\ [-0.18,-0.02]$	$-0.059\ [-0.101,-0.018]$	$2.1\times 10^{-2}$
Reasoning depth	$-0.13\ [-0.23,-0.04]$	$-0.028\ [-0.068,\phantom{-}0.012]$	$1.8\times 10^{-1}$

Appendix

Table 8: RQ2 additive-GLM differences between length contrasts ( link only , pp) and logit interaction coefficients

\hat{\gamma}_{Lj}

from Equation 5 . Brackets give 95% CIs;

p_{\mathrm{adj}}

tests

\gamma_{Lj}=0

using the same six-test Holm family as Table 2 .

Check	Evidence scope	Evidence dispersion	Reasoning depth
Distinct depth quartiles	–	–	$-1.25\ [-2.70,\phantom{-}0.19]$
Doubling length	$-1.29\ [-2.03,-0.55]$	$-1.47\ [-2.39,-0.54]$	$-1.73\ [-3.73,\phantom{-}0.27]$
Evidence tokens	$-0.99\ [-1.50,-0.47]$	–	–
Within-task median splits	$-1.58\ [-3.94,\phantom{-}0.86]$	$-2.21\ [-4.52,\phantom{-}0.09]$	$-3.60\ [-7.54,\phantom{-}0.16]$
Gemini 3.8 Flash	$-1.25\ [-1.92,-0.59]$	$-1.50\ [-2.32,-0.67]$	$-0.33\ [-1.53,\phantom{-}0.86]$
GPT-5.6 Luna	$-0.98\ [-1.64,-0.32]$	$-1.00\ [-1.73,-0.27]$	$-0.34\ [-1.20,\phantom{-}0.51]$

Appendix

Table 9: RQ2 sensitivity checks and per-LLM differences between length contrasts (high minus low, pp). Brackets give individual, unadjusted 95% CIs. These checks are descriptive.

Any extreme	# above median	Questions	Observed	Centred	Predicted	Residual
Mean score (%)
no	0	2,212	$87.9$	$2.2\ [\phantom{-}1.3,\phantom{-}3.2]$	$87.8$	$0.1\ [-0.8,\phantom{-}1.1]$
no	1	1,504	$84.6$	$0.6\ [-0.5,\phantom{-}1.8]$	$84.2$	$0.4\ [-0.8,\phantom{-}1.6]$
no	2	1,328	$86.2$	$0.1\ [-1.0,\phantom{-}1.3]$	$84.9$	$1.3\ [\phantom{-}0.1,\phantom{-}2.5]$
no	3	164	$76.1$	$-5.4\ [-9.6,-1.3]$	$77.7$	$-1.5\ [-5.7,\phantom{-}2.6]$
yes	1	345	$81.5$	$-2.8\ [-5.4,-0.2]$	$83.8$	$-2.3\ [-4.9,\phantom{-}0.4]$

Appendix

Table 10: RQ3 configuration estimates. Observed and predicted outcomes are percentages; centred outcomes, residuals, and indicator contrasts are in pp. Brackets give individual, unadjusted 95% CIs. Definitions and estimation details are given above.

Comparison	All tasks	No SciTrek	Only SciTrek	Other omissions
Scope $\times$ dispersion	$-6.5$	$-6.2$	$-8.6$	$[-7.2,-6.1]$
Scope $\times$ depth	$-6.8$	$-3.4$	$-10.2$	$[-7.4,-6.4]$
Dispersion $\times$ depth	$-10.0$	$-4.1$	$-14.7$	$[-11.4,-9.8]$
Depth at low dispersion	$1.0$	$1.1$	$0.1$	$[0.1,3.0]$
Depth at high dispersion	$-6.7$	$-0.8$	$-12.6$	$[-7.7,-6.7]$

Appendix

Table 11: RQ3 sensitivity to task composition (pp). First three rows: high/high minus low/low centred scores. Last two: high minus low depth at the stated dispersion level. The final column is the range when omitting any single task other than SciTrek; it is not a confidence interval.

(a) Evidence scope $\times$ evidence dispersion
Horizontal	Vertical	Questions	Centred score (pp)	Residual (pp)
low	low	1,323	$3.6$	$0.9\ [-0.2,\phantom{-}2.0]$
low	mid	627	$0.9$	$-0.0\ [-1.9,\phantom{-}1.9]$
low	high	295	$-0.2$	$0.2\ [-2.6,\phantom{-}2.9]$
mid	low	612	$0.7$	$-0.7\ [-2.5,\phantom{-}1.2]$
mid	mid	792	$-0.3$	$0.0\ [-1.6,\phantom{-}1.7]$

Appendix

Table 12: Cell data for Figure 6 . Horizontal and vertical levels follow each panel’s axes. Scores are centred on the task–LLM mean; residuals are observed minus out-of-fold additive-GLM predictions. Brackets give individual, unadjusted 95% residual CIs (pp).

Long-Context Reasoning in Language Models

See all 41 papers

Jul 9, 2026

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.

Siddhartha Jain, Ameya Velingker

NVIDIA

LLM Evaluation Long-Context Language Model Evaluation Long-Context Inference

Ask PDF

Nov 4, 2025

Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the reasoning problem; and Oolong-real, a downstream setting which requires reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.

Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2

Carnegie Mellon University, Language Technologies Institute

LLM Evaluation Long-Context Language Model Evaluation Long-Context Modeling Long-Context Inference Long-Context Reasoning in Language Models

Ask PDF

Jul 2, 2026

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization. In this work, we propose Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (RECONTEXT), a training-free inference method for improving long-context reasoning. RECONTEXT uses model-internal relevance signals to construct a query-conditioned evidence pool and replays it before final generation while preserving the full original context. This recursive selection process separates evidence organization from answer generation without training, external memory, or context pruning. We also provide a theoretical analysis based on associative memory, which characterizes the context as a memory store, the question as a retrieval cue, attention as cue-trace association, and replay as trace reactivation. Experiments on eight long-context datasets with 128K context length show that RECONTEXT consistently improves evidence utilization across Qwen3-4B, Qwen3-8B, and Llama3-8B, achieving the best average rank on all three backbones. Code is available at https://github.com/Yanjun-Zhao/ReContext.

Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6

University of Illinois Urbana-Champaign

Long-Context Inference Long-Context Reasoning in Language Models

Ask PDF

Abstract

Long-Context Reasoning in Language Models

Explore similar work

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning