Benchmark Construction

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 167% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 51

All topics

Deployment-complete benchmarking
May 25, 2026El Mustapha Mansouri, Keigo AraiBenchmark Auditing Benchmark Construction
School of Engineering, Institute of Science Tokyo, Yokohama, Kanagawa 226-8501, Japan
FLOATBench: A Dataset and Benchmark for Floating Offshore Wind Turbine Tower Fatigue
May 25, 2026João Alves Ribeiro, Bruno Alves Ribeiro, Francisco Pimenta +2Surrogate Modeling Benchmark Construction
Department of Mechanical Engineering Massachusetts Institute of Technology Cambridge, MA 02139 USA · CONSTRUCT, Faculty of Engineering University of Porto Porto, 4200-465 Portugal · Center for Mechanical Technology and Automation University of Aveiro Aveiro, 3810-193 Portugal +2
Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
May 21, 2026Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto +9LLM Evaluation Benchmark Construction
ONESTRUCTION Inc. Tottori, Japan · AWS GenAI Innovation Center Tokyo, Japan
Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work
May 20, 2026Haiyang Shen, Jiuzheng Wang, Taian Guo +9AI Accountability Benchmark Construction
1Peking University · 2Advanced Institute of Big Data
ShadeBench: A Benchmark Dataset for Building Shade Simulation in Sustainable Society
May 19, 2026Longchao Da, Mithun Shivakoti, Xiangrui Liu +3Benchmark Construction Synthetic Benchmark Generation
School of Computing and Augmented Intelligence, Arizona State University · Global Futures Laboratory, Arizona State University
Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech
May 17, 2026Kaavya Chaparala, Thomas Thebaud, Jesús Villalba López +3Benchmark Construction
Johns Hopkins University 3400 N. Charles St., Baltimore, MD 21218
CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean
May 17, 2026Wentao Long, Yunfei Zhang, Chenyi Li +3Mathematical Reasoning Benchmarks Benchmark Construction
Fudan University · Qingdao University · Peking University +1
GeoBuildBench: A Benchmark for Interactive and Executable Geometry Construction from Natural Language
May 13, 2026Jinwoong Kim, Rui Yang, Huishuai ZhangMathematical Reasoning Benchmarks LLM Grounding
Peking University · Wangxuan Institute of Computer Technology
SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
May 12, 2026Xiaoyuan Li, Yuzhe Wang, Moxin Li +6LLM Evaluation Benchmark Construction
University of Science and Technology of China · 2Alibaba Group · 3National University of Singapore
BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD
May 11, 2026Haozhe Zhang, Kaichen Liu, Miaomiao Chen +4Benchmark Construction Parametric CAD Modeling
University of Virginia · University of California, San Diego · Rice University
BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
May 7, 2026Jhen-Ke LinBenchmark Construction LLM Agent Evaluation
National Yang Ming Chiao Tung University
ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation
May 5, 2026Cong Liu, Milong Ren, Jiaqi Guan +4Protein Structure Prediction Benchmark Construction
AMLab, AI4Science Lab, University of Amsterdam, Amsterdam, The Netherlands · Work done at ByteDance Seed · ByteDance Seed
BadmintonGRF: A Multimodal Dataset and Benchmark for Markerless Ground Reaction Force Estimation in Badminton
May 3, 2026Kuoye Niu, Jianwei Li, Shengze Cai +6Benchmark Construction Multimodal Sensor Fusion
Beijing Sport University · Beijing, China · Zhejiang University +4
EOS-Bench: A Comprehensive Benchmark for Earth Observation Satellite Scheduling
Apr 28, 2026Qian Yin, Jiaxing Li, Jiaqi Cheng +23Benchmark Construction Benchmark Design
School of Traﬃc and Transportation Engineering, Central South University, Changsha 410083, China · School of Engineering and Materials Science, Queen Mary University of London, London E1 4NS, UK · College of Automation, Central South University, Changsha, 410083, China +20
Deja Vu at Scale: Paraphrase-Robust Detection of Duplicate Gherkin Steps in Behaviour-Driven Software Testing with Sentence-Transformer Embeddings and a 1.1M-Step Open Benchmark
Apr 22, 2026Ali Hassaan Mughal, Noor Fatima, Muhammad BilalAutomated Software Testing Benchmark Construction
Independent Researcher; Applied MBA (Data Analytics), Texas Wesleyan University, Fort Worth, TX, USA · Independent Researcher · Independent Researcher; B.E. Computer Engineering, National University of Sciences and Technology (NUST), Pakistan +1
DistortBench: Benchmarking Vision Language Models on Image Distortion Identification
Apr 21, 2026Divyanshu Goyal, Akhil Eppa, Vanya Bannihatti KumarVLM Evaluation Benchmark Construction
Adobe Inc.
How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos
Apr 16, 2026Olga Loginova, Frank KellerBenchmark Construction Egocentric Video Understanding
University of Trento · Italy · University of Edinburgh +1
SWE-Universe: Scale Real-World Verifiable Environments to Millions
Feb 2, 2026Mouxiang Chen, Lei Zhang, Yunlong Feng +16Software Engineering Benchmark Construction
Qwen Team, Alibaba Group · Zhejiang University
BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
Oct 18, 2025Tian Xia, Tianrun Gao, Wenhao Deng +4LLM Evaluation LLM Grounding
Westlake University, China · Uniforce AI, China
PuzzleJAX: A Benchmark for Reasoning and Learning
Aug 22, 2025Sam Earle, Graham Todd, Yuchen Li +5RL Benchmarks Game-Playing Agents
New York University · University of Malta · University of the Witwatersrand +1
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6

May 25, 2026, cs.LGJ/K move · Enter open · S save

Deployment-complete benchmarking

El Mustapha Mansouri, Keigo Arai

School of Engineering, Institute of Science Tokyo, Yokohama, Kanagawa 226-8501, Japan

Benchmark Auditing Benchmark Construction Benchmark Design Benchmark Validity

Open paper AskPDF arXiv

Benchmarks increasingly guide deployment, procurement and scientific screening, yet a score supports only the response it records, not necessarily the deployment action. We introduce deployment-complete benchmarking, which tests whether benchmark evidence determines a deployment action. A benchmark is complete for a claim exactly when the action is constant on each evidence fiber; mixed fibers expose missing deployment information, and completion curves quantify the evidence required to resolve ambiguity. In controlled response spaces, benchmark-channel conformal coverage of 94.98% transferred poorly to an unmeasured deployment channel (10.07%), whereas response-rank intervals achieved 94.91% coverage; even zero benchmark error certified only 45.4% of candidates at the largest residual size. Public audits revealed incompleteness, including 97.9% mixed Tox21 fibers and zero median certifiable fraction in main Matbench and JARVIS audits. In held-out replays, certify-then-acquire reduced false decisions from 1.19% to 0.027% in Tox21 and from 20.3% to 0.128% in JARVIS, while changing model choice and identifying deployment-relevant probes. Deployment-ready benchmarks should report evidence, supported actions, ambiguity and completion cost rather than scores alone.

Momentum

Related topics

Latest papers 51