Benchmark Construction

Follow this topic Ask about this topic

Benchmark construction turns a capability or real-world need into evaluation tasks, reference answers or executable checks, and scoring rules. Approaches include expert annotation, mining deployment data, synthetic task generation, and interactive simulators, often paired with human review, automated verification, and reproducible release. The field increasingly emphasizes coverage, difficulty, robustness, provenance, and performance under realistic shifts rather than relying on a single aggregate score. Its central tension is scaling and refreshing benchmarks without sacrificing validity, auditability, or comparability across versions and settings.

Latest papers

68 papers8 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
Apr 2, 2026Smriti Jha, Matteo Paltenghi, Chandra Maddila +3Coding Agent Benchmarks Benchmark Construction
Meta, USA
JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Apr 1, 2026Issa Sugiura, Koki Maeda, Shuhei Kurita +3VLM Evaluation Benchmark Construction
Kyoto University · NII LLMC · Institute of Science Tokyo +2
Efficient Benchmarking of AI Agents
Mar 24, 2026Franck NdzomgaAI Agent Evaluation AI Agent Benchmarks
LiveAgentBench: Comprehensive Benchmarking of Agentic Systems Across 104 Real-World Challenges
Mar 3, 2026Hao Li, Huan Wang, Jinjie Gu +3AI Agent Evaluation AI Agent Benchmarks
Ant Group
SWE-Universe: Scale Real-World Verifiable Environments to Millions
Feb 2, 2026Mouxiang Chen, Lei Zhang, Yunlong Feng +16Software Engineering Benchmark Construction
Qwen Team, Alibaba Group · Zhejiang University
BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
Oct 18, 2025Tian Xia, Tianrun Gao, Wenhao Deng +4LLM Evaluation LLM Grounding
Westlake University, China · Uniforce AI, China
PuzzleJAX: A Benchmark for Reasoning and Learning
Aug 22, 2025Sam Earle, Graham Todd, Yuchen Li +5RL Benchmarks Game-Playing Agents
New York University · University of Malta · University of the Witwatersrand +1
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6

Apr 2, 2026J/K move · Enter open · S save

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage

Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali+2

Meta, USA

Coding Agent Benchmarks Benchmark Construction AI Agent Benchmarks AI Coding Agents Automated Evaluation

Open paper AskPDF arXiv

Production deployment of AI coding agents requires fast, reproducible evaluation signals. Existing industrial practices trade off speed and fidelity: online A/B testing takes weeks and risks user experience, shadow deployment yields signals that are not reproducible across runs, and public benchmarks diverge from production workloads in language distribution, prompt style, and codebase structure. This paper presents REAP (Relevance and Execution-Audited Pipeline), an automated curation pipeline that constructs production-derived benchmarks from real developer-agent sessions without manual labeling. Such curation, while in-distribution to production usage, runs into several challenges. Untestable prompts, misaligned tests, and test flakiness all compromise evaluation reliability. While tasks can be manually audited to ensure only high-quality tasks remain in the benchmark, this approach is infeasible in the monorepo setting: the build infrastructure state is often ephemeral in large monorepos and requires the benchmark to be continuously re-curated against the current codebase. As manual verification cannot be sustained at this cadence, REAP adds an automated verification layer using LLM-based task classification, agentic test-relevance validation, and multi-run stability checks to ensure the executable benchmark yields trustworthy signals. We use REAP to curate Harvest, a benchmark where each task feeds the coding agent a real developer prompt and verifies the resulting code change against fail-to-pass tests retrieved from production. Harvest's distribution spans more than four programming languages with a majority of tasks drawn from Hack. Model and harness evaluations reveal that solve rates range from 42.9% to 58.2% across five frontier models, surfacing capability differences that inform concrete deployment decisions.