What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

Edward Lue Chee Lip; Boden Moraski; Tim Knappe; Lang Xiong; Sarvesh Gharat; Antonio Mari; Ivan Bercovich

cs.LGSep 20, 2026

What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

Authors: Edward Lue Chee Lip, Boden Moraski, Tim Knappe, Lang Xiong, Sarvesh Gharat, Antonio Mari, Ivan Bercovich

Abstract

Frontier benchmarks need tasks that current models cannot solve. But a task that no model solves is not automatically a hard task. The same zero pass rate can come from a real capability gap, but it can also come from missing context, a broken reference solution, infrastructure failure, or a verifier that can be bypassed. In this paper, we study this issue using a frozen Terminal-Bench 3 / Frontier-Bench 0.1 production record with 1,081 pull requests, 639 scored tasks, 28,801 trials, and $105,933 in logged agent spend. We ask what an all-fail task actually certifies. For the 125 tasks with no honest pass, we combine task artifacts, reference-solution runs, empty-solution controls, adversarial trials, trajectories, telemetry, and review records, and apply an ordered validity screen. Only 78 of the 125 tasks survive as certified-unsolved candidates. The remaining tasks include 14 with broken oracles, 8 dominated by infrastructure failures, 4 that are only passable through verifier bypasses, and 21 whose solvability is not certified by the available evidence. Thus, lack of saturation and genuine difficulty are not the same thing. The certified-unsolved label is also narrow: it means that the authored route passed, infrastructure did not dominate, no strict bypass was observed, and all evaluated agents failed. It does not prove intrinsic hardness, verifier completeness, or failure at the intended capability. We further analyze rejected submissions and passing tasks to show that pass rate alone cannot explain why a task is difficult. Overall, our results suggest that frontier benchmarks should report the evidence behind their all-fail tasks before using them as capability claims.

Figures & tables

Figure 1: Overview of the adjudication protocol. The audit starts from the submitted task package, joins ordinary trials with reference-solution, empty-solution, and cheat-oriented controls, and then combines these runs with telemetry, trajectory evidence, and verifier-integrity evidence. The ordered validity checks assign an evidence-qualified outcome to each task with no honest pass. The outcome counts are shown for orientation; Section 3 analyzes them.

Figure 2: One leaderboard value produces five evidence-qualified outcomes. Of the 125 tasks with no honest pass, 78 retain a working reference route and survive the declared validity screens, while 47 require repair, exclusion, or an unresolved label. Counts are computed from snapshot 3c5be84efd707da8 .

Certification requirement	Tasks
Never passed honestly	125
Observed oracle pass	85
Also infrastructure-clean at $\tau=0.5$	79
Also no strict bypass evidence	78
Also at least two oracle runs	25

Table 1: Certification strictness ladder for the all-fail set. Each row adds one requirement to the preceding row. The final row is a stronger stability cut, not evidence from an independent implementation.

Check	Result	Interpretation
Judge-panel unanimity on contested trials	48/53	Most contested trial-level decisions were stable across repeated judgments.
Panel-majority agreement with hand labels	41/53	The automatic panel gives useful but imperfect agreement with human labels.
Adversarial re-check	12/12 held	Selected labels survived a targeted attempt to overturn them.
Infrastructure-threshold sensitivity	1–3 task changes	The five-way partition is not driven by a single fragile threshold choice.
Repeated-oracle cut	25/78	A stricter certified subset remains when repeated reference evidence is required.

Table 2: Reliability and sensitivity checks for the adjudication. Each row tests a different source of uncertainty. These checks support the evidence-qualified labels, but they do not imply that every task received full domain-expert re-review.

Figure 3: Primary rejection reason for the 555 closed-unmerged Terminal-Bench 3 pull requests. The chart shows that rejected tasks are not only tasks that agents solve too easily. Many are rejected because of ambiguity, verifier problems, nondeterminism, metadata issues, duplication, or process closure. Labels are assigned from review threads by a single-pass LLM judge.

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 175 papers

What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
No More, No Less: Task Alignment in Terminal Agents
May 12, 2026Sina Mavali, David Pape, Jonathan Evertz +5Computer-Use Agent Benchmarks AI Alignment
CISPA Helmholtz Center for Information Security · ELLIS Institute Tübingen and MPI for Intelligent Systems · Tübingen AI Center
CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
Aug 6, 2026Fanzhe Meng, Guoxin Chen, Jiale Zhao +6LLM Agent Training Agent Benchmarks
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Independent Researcher · 3AweAI Team

Apr 30, 2026J/K move · Enter open · S save

What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design

Ivan Bercovich

Computer-Use Agent Benchmarks Reward Hacking Terminal-Agent Benchmarks Benchmark Design Terminal Agents AI Agent Benchmarks+1

Open paper AskPDF arXiv

Terminal-agent benchmarks have become a primary signal for measuring the coding and system-administration capabilities of large language models. As the market for evaluation environments grows, so does the pressure to ship tasks quickly, often without thorough adversarial review of the verification logic. This paper is a guideline for writing good benchmark tasks, drawn from over a year of contributing to and reviewing tasks for Terminal Bench. Most people write benchmark tasks the way they write prompts. They shouldn't. A prompt is designed to help the agent succeed; a benchmark is designed to find out if it can. We argue that good tasks are adversarial, difficult, and legible, and that a large class of common failure modes -- AI-generated instructions, over-prescriptive specifications, clerical difficulty, oracle solutions that assume hidden knowledge, tests that validate the wrong things, and reward-hackable environments -- are predictable consequences of treating task authoring as prompt authoring. We catalog these failure modes, argue that real difficulty is conceptual rather than environmental, and discuss recent empirical evidence that over 15% of tasks in popular terminal-agent benchmarks are reward-hackable. We hope this serves as a useful reference for benchmark maintainers, task contributors, and researchers using benchmark scores as evidence.

Abstract

Benchmark Validity

Explore similar work

What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design

No More, No Less: Task Alignment in Terminal Agents

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks