LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Quang Hieu Pham; Thuy Duong Nguyen; Jocelyn Qiaochu Chen; Xi Ye

cs.CLSep 29, 2026

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Authors: Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen, Xi Ye

Organizations: University of Alberta · Alberta Machine Intelligence Institute (Amii)

Abstract

Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy--cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model--harness combinations: the best reaches 68% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.

Figures & tables

Figure 1: LongHarness (ours) reveals harness differences in both accuracy and cost.

Figure 2: Examples from our benchmark. Top left: Constraint Solving Search combines evidence from documents. Top right: Equivalent Program Pair Search distinguishes equivalent programs from near matches. Bottom left: Program Execution Tracing requires reading prior outputs and matching code semantics. Bottom right: Outlier Memo Detection identifies a conflict using supporting memos.

	LongBenchV2	HELMET	LongProc	Oolong	LongHarness
Diverse, adaptive retrieval demand	$\sim$	$\sim$	$\times$	$\times$	$\checkmark$
Semantically confusable evidence	$\times$	$\checkmark$	$\sim$	$\sim$	$\checkmark$
Extensive reasoning steps	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Step-dependent retrieval	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$
Multiple strategies with different costs	$\times$	$\sim$	$\times$	$\sim$	$\checkmark$
Wide range of SoTA system accuracy	$\times$	$\times$	$\times$	$\times$	$\checkmark$

Table 1: Comparison of long-context benchmarks along properties relevant to evaluating LM harnesses.

\checkmark

: supported;

\sim

: partially supported;

\times

: not supported. Diverse, adaptive retrieval demand means choosing an appropriate way to access context, such as lexical search, semantic retrieval, or direct reading, as information needs change during reasoning. Step-dependent retrieval means that intermediate findings determine what must be retrieved next. The final two rows indicate whether model–harness configurations are widely separated in accuracy and cost.

Model	Harness	Constraint	Memos	Program	Pairs	Macro avg.
GPT-5.6-sol	Direct	58% ($0.690)	44% ($1.17)	94% ($0.648)	44% ($0.780)	60.0% ($0.822)
	RLM	44% ($0.659)	92% ($7.23)	76% ($1.47)	2% ($7.86)	53.5% ($4.30)
	OpenCode	56% ($1.72)	10% ($3.11)	96% ($1.37)	0% ($5.41)	40.5% ($2.90)
	mini-swe-agent	42% ($0.320)	100% ($0.583)	88% ($1.02)	42% ($1.34)	68.0% ($0.816)
	ReAct	32% ($1.40)	6% ($2.77)	82% ($0.866)	52% ($2.68)	43.0% ($1.93)
GLM-5.3	Direct	0% ($0.728)	0% ($1.09)	38% ($0.785)	0% ($1.19)	9.5% ($0.948)

Table 2: Exact accuracy and estimated cost across four suites. Each cell reports accuracy (%) followed by USD per-instance cost in parentheses; macro averages the four task accuracies and per-task mean costs. Appendix A gives token counts and Appendix C gives the API rates.

Figure 3: Macro-average accuracy and estimated cost across the four tasks (Table 2 ). Color denotes model; shape denotes harness. Callouts and larger markers mark each model’s best accuracy. ReAct appears where available.

Figure 4: Harness effects relative to direct inference on OOLONG-Synth, LongBench-v2, and our suites. External benchmarks use GPT-5.6-sol. The reference is zero accuracy change and

1\times

cost. Green denotes higher accuracy at lower cost; pink denotes lower accuracy at higher cost.

Harness	OOLONG-Synth	LongBench-v2
Direct	64% ($0.686)	62% ($0.532)
RLM	78% ($3.70)	62% ($0.641)
OpenCode	74% ($2.08)	60% ($0.463)
mini-swe-agent	86% ($0.217)	68% ($0.190)
ReAct	70% ($0.751)	68% ($0.354)

Table 3: GPT-5.6-sol on existing long-context benchmarks. Each cell reports accuracy and average USD per instance in parentheses.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Private specification	Controlled distractors	Gold derivation
Constraint Solving Search	Typed fact world and predicate support sets	People satisfying all but one condition	Set intersection and fact-to-passage map
Outlier Memo Detection	Typed relation graph and composed relations	Endpoint substitutions and near-twin claims	Graph-composition proofs
Program Execution Tracing	Computation graph, typed inputs, and function contracts	Same-output functions and alternate routes	Execution and contract probes
Equivalent Program Pair Search	Executable program families and test signatures	Behavior-changing syntax-tree mutations	Differential execution

Appendix

Table 5: Underlying structures used to construct the four tasks. Public contexts expose only the rendered items, not their roles or provenance.

Model	Input	Cache read	Cache write	Output
GPT-5.6-sol	4.00	0.400	5.00	20.00
GPT-5.6-sol (long)	8.00	0.800	10.00	30.00
GLM-5.3	4.86	0.972	4.86	12.15
Qwen3.8-27B	2.48	–	–	7.46
Gemini 3.8 Flash	0.75	0.075	–	3.75
Kimi-K2.6	5.15	1.03	5.15	12.81

Appendix

Table 6: API rates in USD per million tokens. A dash indicates that we do not apply a separate rate for that token category.

Explore similar work

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Jul 9, 2026Siddhartha Jain, Ameya VelingkerLLM Evaluation Long-Context Language Model Evaluation
NVIDIA
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI
ATLAS: All-round Testing of Long-context Abilities across Scales
May 27, 2026Deli Huang, Cunguang Wang, Hongyin Tang +15LLM Evaluation Benchmark Construction
Meituan · Fudan University

Jul 9, 2026, cs.AIJ/K move · Enter open · S save

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Siddhartha Jain, Ameya Velingker

NVIDIA

LLM Evaluation Long-Context Language Model Evaluation Long-Context Inference

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.

Abstract

Explore similar work

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

ATLAS: All-round Testing of Long-context Abilities across Scales