Production deployment of AI coding agents requires fast, reproducible evaluation signals. Existing industrial practices trade off speed and fidelity: online A/B testing takes weeks and risks user experience, shadow deployment yields signals that are not reproducible across runs, and public benchmarks diverge from production workloads in language distribution, prompt style, and codebase structure. This paper presents REAP (Relevance and Execution-Audited Pipeline), an automated curation pipeline that constructs production-derived benchmarks from real developer-agent sessions without manual labeling. Such curation, while in-distribution to production usage, runs into several challenges. Untestable prompts, misaligned tests, and test flakiness all compromise evaluation reliability. While tasks can be manually audited to ensure only high-quality tasks remain in the benchmark, this approach is infeasible in the monorepo setting: the build infrastructure state is often ephemeral in large monorepos and requires the benchmark to be continuously re-curated against the current codebase. As manual verification cannot be sustained at this cadence, REAP adds an automated verification layer using LLM-based task classification, agentic test-relevance validation, and multi-run stability checks to ensure the executable benchmark yields trustworthy signals. We use REAP to curate Harvest, a benchmark where each task feeds the coding agent a real developer prompt and verifies the resulting code change against fail-to-pass tests retrieved from production. Harvest's distribution spans more than four programming languages with a majority of tasks drawn from Hack. Model and harness evaluations reveal that solve rates range from 42.9% to 58.2% across five frontier models, surfacing capability differences that inform concrete deployment decisions.
Figures & tables
Figure 1: REAP curation pipeline overview. Dataset construction assembles candidate tasks; REAP’s three verification filters, ordered so that semantic checks narrow the candidate set before execution, produce verified benchmark tasks.
Figure 2: Data funnel for the case study, restricted to single-turn conversations: tens of thousands of candidate diffs are progressively filtered down to on the order of hundreds of benchmark tasks. Lightweight prompt-quality filters run before the more expensive backoutability and test discovery steps, reducing the input to costly stages and keeping end-to-end curation tractable at the rolling re-curation cadence.
Mean
Median
Max
Diff size (lines)
697.8
39
77,879
Files changed
3.0
2
21
Fail-to-pass tests
3.3
1
39
Total tests
8.3
3
51
Table 1: Per-task structural statistics for Harvest. Mean is reported alongside median to expose the right-tail skew driven by large agent-authored diffs.
Figure 3: Language distribution of Harvest. Hack accounts for the plurality of tasks, followed by Python, C++, Kotlin, and other languages, reflecting the polyglot nature of the host monorepo.
Figure 4: Tasks per category in Harvest, assigned by an LLM-powered task classifier based on prompt content. No single category exceeds a quarter of the benchmark.
Axis
Harvest
SWE-Bench
Prompt source
Verbatim prompts typed for an AI agent
GitHub issue descriptions written for human developers
5+ languages, incl. low-resource and industrially relevant
Python only
Curation method
LLM classifiers for testability and test relevance (automated)
90 engineers screen for test quality (Verified subset)
Contamination risk
Rolling re-curation; private codebase unseen by models
Frozen snapshot; public repos visible to model training
Task count
∼ 130 (comparable to HumanEval Chen et al. (2021) ; automated)
500 Verified Int ([n. d.]) (from 2,294 after 90-engineer screening)
Table 2: Comparison of Harvest against SWE-Bench Jimenez et al. (2024) . Both benchmarks evaluate agents at the repository level using fail-to-pass tests (no LLM judges), but diverge on provenance, scope, and sustainability.
Figure 5: Model solve rates on Harvest across five frontier models evaluated on 132 tasks with three independent runs each. Bar height shows the mean solve rate across runs; whiskers indicate the min and max of the three per-run solve rates, capturing run-to-run reproducibility.
Figure 6: Tool usage distribution across five frontier models on Harvest, shown as percentage of total tool calls per model.
Figure 7: Per-diff consistency heatmap across five frontier models on Harvest. Models are grouped by provider (Claude Opus 4.6, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3.1 Pro, GPT-5.4, top to bottom); see Figure 5 for solve-rate ranking. Diffs are sorted by difficulty (easiest on left, hardest on right). Cell colors indicate pass consistency across three independent runs: black = pass all 3 runs (consistently solvable), gray = pass 1–2 of 3 runs (non-deterministic), white = fail all 3 runs (consistently unsolvable). The top strip shows task difficulty: green = all models pass all runs, yellow = 2–4 models pass strictly, red = 0–1 models pass strictly.
Figure 8: Harness evaluation comparison on Harvest. Advanced Agent consistently outperforms Basic Agent. Basic Agent’s limited search tools lead to prolonged search operations (see Table 3 ). Advanced Agent provides better code navigation, leading to fewer searches and higher solve rates.
Tool
Basic Agent
Advanced Agent
Read File
36.8
26.5
Search Files
31.4
13.4
Shell
17.0
6.1
Todo Write
—
13.7
Edit
9.1
—
Str Replace Edit
—
9.4
Table 3: Tool usage distribution (%) across harness configurations (Opus 4.6). Basic Agent allocates significantly more calls to search, while Advanced Agent distributes effort more evenly across navigation, planning, and validation.