Production deployment of AI coding agents requires fast, reproducible evaluation signals. Existing industrial practices trade off speed and fidelity: online A/B testing takes weeks and risks user experience, shadow deployment yields signals that are not reproducible across runs, and public benchmarks diverge from production workloads in language distribution, prompt style, and codebase structure. This paper presents REAP (Relevance and Execution-Audited Pipeline), an automated curation pipeline that constructs production-derived benchmarks from real developer-agent sessions without manual labeling. Such curation, while in-distribution to production usage, runs into several challenges. Untestable prompts, misaligned tests, and test flakiness all compromise evaluation reliability. While tasks can be manually audited to ensure only high-quality tasks remain in the benchmark, this approach is infeasible in the monorepo setting: the build infrastructure state is often ephemeral in large monorepos and requires the benchmark to be continuously re-curated against the current codebase. As manual verification cannot be sustained at this cadence, REAP adds an automated verification layer using LLM-based task classification, agentic test-relevance validation, and multi-run stability checks to ensure the executable benchmark yields trustworthy signals. We use REAP to curate Harvest, a benchmark where each task feeds the coding agent a real developer prompt and verifies the resulting code change against fail-to-pass tests retrieved from production. Harvest's distribution spans more than four programming languages with a majority of tasks drawn from Hack. Model and harness evaluations reveal that solve rates range from 42.9% to 58.2% across five frontier models, surfacing capability differences that inform concrete deployment decisions.
Figures & tables
Figure 1: REAP curation pipeline overview. Dataset construction assembles candidate tasks; REAP’s three verification filters, ordered so that semantic checks narrow the candidate set before execution, produce verified benchmark tasks.
Figure 2: Data funnel for the case study, restricted to single-turn conversations: tens of thousands of candidate diffs are progressively filtered down to on the order of hundreds of benchmark tasks. Lightweight prompt-quality filters run before the more expensive backoutability and test discovery steps, reducing the input to costly stages and keeping end-to-end curation tractable at the rolling re-curation cadence.
Mean
Median
Max
Diff size (lines)
697.8
39
77,879
Files changed
3.0
2
21
Fail-to-pass tests
3.3
1
39
Total tests
8.3
3
51
Table 1: Per-task structural statistics for Harvest. Mean is reported alongside median to expose the right-tail skew driven by large agent-authored diffs.
Figure 3: Language distribution of Harvest. Hack accounts for the plurality of tasks, followed by Python, C++, Kotlin, and other languages, reflecting the polyglot nature of the host monorepo.
Figure 4: Tasks per category in Harvest, assigned by an LLM-powered task classifier based on prompt content. No single category exceeds a quarter of the benchmark.
Axis
Harvest
SWE-Bench
Prompt source
Verbatim prompts typed for an AI agent
GitHub issue descriptions written for human developers
5+ languages, incl. low-resource and industrially relevant
Python only
Curation method
LLM classifiers for testability and test relevance (automated)
90 engineers screen for test quality (Verified subset)
Contamination risk
Rolling re-curation; private codebase unseen by models
Frozen snapshot; public repos visible to model training
Task count
∼ 130 (comparable to HumanEval Chen et al. (2021) ; automated)
500 Verified Int ([n. d.]) (from 2,294 after 90-engineer screening)
Table 2: Comparison of Harvest against SWE-Bench Jimenez et al. (2024) . Both benchmarks evaluate agents at the repository level using fail-to-pass tests (no LLM judges), but diverge on provenance, scope, and sustainability.
Figure 5: Model solve rates on Harvest across five frontier models evaluated on 132 tasks with three independent runs each. Bar height shows the mean solve rate across runs; whiskers indicate the min and max of the three per-run solve rates, capturing run-to-run reproducibility.
Figure 6: Tool usage distribution across five frontier models on Harvest, shown as percentage of total tool calls per model.
Figure 7: Per-diff consistency heatmap across five frontier models on Harvest. Models are grouped by provider (Claude Opus 4.6, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3.1 Pro, GPT-5.4, top to bottom); see Figure 5 for solve-rate ranking. Diffs are sorted by difficulty (easiest on left, hardest on right). Cell colors indicate pass consistency across three independent runs: black = pass all 3 runs (consistently solvable), gray = pass 1–2 of 3 runs (non-deterministic), white = fail all 3 runs (consistently unsolvable). The top strip shows task difficulty: green = all models pass all runs, yellow = 2–4 models pass strictly, red = 0–1 models pass strictly.
Figure 8: Harness evaluation comparison on Harvest. Advanced Agent consistently outperforms Basic Agent. Basic Agent’s limited search tools lead to prolonged search operations (see Table 3 ). Advanced Agent provides better code navigation, leading to fewer searches and higher solve rates.
Tool
Basic Agent
Advanced Agent
Read File
36.8
26.5
Search Files
31.4
13.4
Shell
17.0
6.1
Todo Write
—
13.7
Edit
9.1
—
Str Replace Edit
—
9.4
Table 3: Tool usage distribution (%) across harness configurations (Opus 4.6). Basic Agent allocates significantly more calls to search, while Advanced Agent distributes effort more evenly across navigation, planning, and validation.
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.
We introduce StaminaBench, a benchmark that measures the stamina of coding agents: how many consecutive interaction turns (change requests) they can handle before failing. Unlike the prevailing fraction-of-tasks-solved metric, this matches real vibe-coding where sessions run dozens or hundreds of turns. In StaminaBench, agents implement a REST API server and modify it across a tunable number of procedurally generated follow-up change requests - 100 in our experiments, resulting in codebases of up to 6,000 lines. Tests are generated fully programmatically without LLM involvement, ensuring reproducibility and reliability; change sequences are drawn from either a hardcoded or LLM-driven sampler, both constrained to a structured action space to ensure changes are valid. The agent and the server run in an isolated environment and communicate with the benchmark through HTTP, making testing fully black-box and language-agnostic. We evaluate six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each and find that: (1) all the tested models fail within 5-6 turns, confirming that vibe-coding-style programming without thorough testing produces bugs; (2) passing test feedback back to the agent and allowing it to retry improves passed turn count by up to 12x; and (3) a good harness is required for strong performance: stronger models exhibit up to a 6x gap between their best and worst harness, while weaker models fail with any harness. We release the benchmark and the generated tasks to enable further research into multi-turn coding agent behavior. Benchmark code and data: github.com/amazon-science/StaminaBench.
Agentic coding harnesses - such as Agent-Skills, Superpowers, and Agent-Rigor - are increasingly deployed to augment underlying LLMs for real-world software engineering tasks. Existing benchmarks evaluate these agents almost exclusively on outcome correctness: whether generated code passes tests or resolves issues. We argue that this outcome-only lens is insufficient: an agent that arrives at a correct solution through reckless trial-and-error, without planning, verification, or graceful recovery, is fundamentally less reliable than one that follows sound engineering discipline. We introduce RigorBench, the first benchmark designed to measure process discipline in AI coding agents. RigorBench evaluates these harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. A composite RigorScore aggregates these dimensions into a single metric via a weighted sum. We curate a suite of 30 tasks spanning five categories - Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don't Break the Build-and evaluate leading harnesses in a controlled with/without experimental design against baseline coding assistants. Our results show that structured process discipline not only improves process quality scores by an average of 41% but also raises downstream outcome correctness by 17%, providing the first quantitative evidence that how agents code matters as much as what they produce. We release the full benchmark, scoring rubrics, and trajectory analysis tools as open-source artifacts.
Meher Bhaskar Madiraju, Meher Sai Preetam Madiraju