Benchmark Construction

Ask about this topic

Overview

Benchmark construction turns a capability or real-world need into evaluation tasks, reference answers or executable checks, and scoring rules. Approaches include expert annotation, mining deployment data, synthetic task generation, and interactive simulators, often paired with human review, automated verification, and reproducible release. The field increasingly emphasizes coverage, difficulty, robustness, provenance, and performance under realistic shifts rather than relying on a single aggregate score. Its central tension is scaling and refreshing benchmarks without sacrificing validity, auditability, or comparability across versions and settings.

Latest papers

CardsList

68 papers8 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

  1. Evolving Self-tuning Benchmarks for Long-Horizon Tool Use

    Oct 19, 2026Hyunji Alex Nam, Michael Dennis, Natasha Jaques +1AI Agent BenchmarksTool-Use Evaluation

    Stanford University · Google Deepmind · University of Washington

  2. IndustrialCAD-1400: Benchmarking Detailed Industrial Assemblies from Text

    Oct 17, 2026Hebao Zhu, Kang An, Zhuoke Liao +4Text-to-CAD GenerationBenchmark Construction

    Chongqing University · Shanghai Jiao Tong University · SenseTime +1

  3. EvalBuilder: Can LLMs Build Efficient Benchmarks?

    Oct 14, 2026Huaisheng Zhu, Hangfan Zhang, Teng Xiao +5Benchmark ConstructionAI Agent Benchmarks

    Meituan · Pennsylvania State University · Allen Institute for AI (AI2) +2

  4. The Smart Buildings Challenge: Learning Foundation Models for Real-World Optimization at Scale

    Oct 13, 2026Judah Goldfeder, Ayush Rai, Philippe M. Wyder +11Building Energy ManagementTransfer Learning

    Columbia University · Harvard University · Distyl AI +6

  5. PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?

    Oct 12, 2026Dongyeop Woo, Nofit Segal, Nayoung Kim +3Crystal Structure PredictionMaterials Science

    Korea Advanced Institute of Science & Technology (kaist.edu) · Massachusetts Institute of Technology (mit.edu)

  6. EvalResearchBench: Can AI Agents Design Their Own Evaluations?

    Oct 3, 2026Yaolun Zhang, Tianyi Xu, Yujie Zhao +3Automated EvaluationBenchmark Construction

    Oregon State University · AG2 AI · University of California, San Diego +1

  7. AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration

    Oct 2, 2026Hy Nguyen, Nabi Rezvani, Robin VujanicBenchmark ConstructionDatabase Systems

    The University of Sydney · MongoDB Research

  8. WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

    Sep 29, 2026Haomin Qi, Xiangzhe Xu, Yiming Huang +2Software Engineering AgentsBenchmark Construction

    University of California, San Diego · Purdue University · National University of Singapore

  9. TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces

    Sep 27, 2026Dehai Min, Daoan Zhang, Yiming Zeng +13Benchmark ConstructionLLM Agent Evaluation

    ByteDance Inc., USA · University of Illinois at Chicago

  10. A comparative assessment of global building and settlement datasets across geographic and settlement contexts

    Sep 23, 2026Rufai Omowunmi Balogun, Caroline Margaux Gevaert, Capucine Riom +5Benchmark ConstructionBuilding Footprint Extraction

    The World Bank, Global Facility for Disaster Reduction and Recovery · Center for Geospatial Analytics, Clark University, Worcester, MA, USA · University of Twente, Enschede, Twente, The Netherlands +2

  11. Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

    Sep 22, 2026Wooyoung JungBenchmark ConstructionNatural Language to SPARQL

    Assistant Professor, Dept. of Civil and Architectural Engineering and Mechanics, Univ. of Arizona, Tucson, AZ 85719.

  12. The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence

    Sep 21, 2026Muhan ZhangMathematical Reasoning BenchmarksBenchmark Construction

    The University of Texas at Arlington

  13. Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

    Sep 16, 2026Guojun Zhu, Xunheng Huang, Peng Yin +3Counterfactual EvaluationBenchmark Construction

    School of Mathematical Sciences, University of Chinese Academy of Sciences, China · Department of Statistics & Data Science, National University of Singapore, Singapore · Institute of Automation, Chinese Academy of Sciences, China

  14. Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation

    Sep 14, 2026M. Ali BayramMultilingual Language Model EvaluationLLM Evaluation

    Department of Computer Engineering, Yıldız Technical University İstanbul, Turkey

  15. Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

    Sep 11, 2026Baoyang Jiang, Fengchun Zhang, Leyuan Wang +9Benchmark ConstructionEmbodied QA

    QiYuan Lab · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications +3

  16. Field-level prediction of mid-plane stress tensor fields in concrete target penetration: a cross-velocity graph neural operator surrogate

    Sep 9, 2026Wenpu Du, Peng Zhou, Yunlong Xia +5Neural Surrogate ModelingBenchmark Construction

    School of Environment and Safety Engineering, North University of China, Taiyuan, 030051, China

  17. A Large Open Multi-Energy Corpus of Soil Compaction Tests, with Machine-Learning Baselines

    Sep 3, 2026Sompote Youwai, Chana Phutthananon, Warat KongkitkulBenchmark ConstructionMaterials Property Prediction

    AI Research Group, Department of Civil Engineering King Mongkut’s University of Technology Thonburi 126 Pracha Uthit Road, Bang Mod, Thung Khru, Bangkok 10140, Thailand

  18. ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction

    Sep 2, 2026Ke Zhang, Yankang Liu, Roya Zandi +1Tool-Use EvaluationBenchmark Construction

    University of California, Riverside

  19. MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft

    Aug 28, 2026Sewoong Lee, Risham Sidhu, Julia Hockenmaier +1LLM EvaluationLLM Reliability

    Siebel School of Computing and Data Science The Grainger College of Engineering University of Illinois Urbana-Champaign · Department of Civil and Coastal Engineering University of Florida

  20. BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks

    Aug 27, 2026Jeong-Yoon Kim, Tae-Yoon LeeBenchmark ConstructionAI Agent Benchmarks

  21. A Query-Time Framework for Transient 2D Pore-Scale Flow Prediction and Generative Design

    Aug 23, 2026Yiming Wang, Jiale Zhu, Zhichen Ye +4Porous Media FlowNeural Surrogate Modeling

    School of Civil Engineering, The University of Sydney, Sydney, 2006, NSW, Australia · MOE Key Laboratory of Soft Soils and Geoenvironmental Engineering, Zhejiang University, Hangzhou, 310058, China · College of Civil Engineering and Architecture, Zhejiang University, Hangzhou, 310058, China +2

  22. BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERP

    Aug 21, 2026Haoran Sun, Klaus Marius HansenSoftware Engineering AgentsSoftware Engineering

    Microsoft

  23. CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation

    Aug 10, 2026Harmanjot Singh, Abhra Dubey, Jorge Alejandro Amador HerreraBenchmark ConstructionParametric CAD Modeling

    Mohamed Bin Zayed University of Artificial Intelligence · Abu Dhabi, UAE

  24. ArchEGraph: A Large-Scale Graph Dataset for Geometry-Topology-Physics Aligned Building Energy Modeling

    Aug 7, 2026Yihui Li, Yihui Chen, Kaidi Zha +7Graph ReconstructionBenchmark Construction

    Tsinghua University · UC Berkeley · 1Tsinghua University 2UC Berkeley 3MIT +4

  25. BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests

    Aug 3, 2026Zetong Xiong, Qiao Zhao, Jun Zhang +20Software Engineering AgentsBenchmark Construction

    Baidu Beijing, China

  26. Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms

    Aug 2, 2026Tezan Sahu, Aritra Das, Pankaj Mittal +1Benchmark ConstructionLLM Agent Evaluation

    Microsoft Hyderabad, India · Indian Institute of Technology Roorkee Roorkee, India

  27. Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

    Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM EvaluationBenchmark Auditing

    Aleph Alpha Research, Heidelberg, Germany.

  28. SafeBuild-Bench: A Temporal-Robust Construction Safety Benchmark with Graph-Enhanced Data Mining

    Jul 29, 2026Yi Cui, Zilin Wang, Yijie Xu +5Multimodal RobustnessBenchmark Construction

    Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · The Hong Kong University of Science and Technology(Guangzhou) Guangzhou, China · School of Management Northwest Polytechnical University Xi’an Xi’an, China +4

  29. HamQASBench: A Hamiltonian-Informed Diagnostic Benchmark for Evaluating Quantum Architecture Search

    Jul 6, 2026Jiayang Niu, Akib Karim, Yan Wang +5Quantum Architecture SearchBenchmark Construction

    School of Computing Technologies RMIT University · Quantum Systems School of Computing Technologies RMIT University · Data61, CSIRO

  30. MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

    Jul 2, 2026Yuanzhi Liu, Shousheng Zhao, Bo Zhou +2VLM EvaluationBenchmark Construction

    1Beijing University of Posts and Telecommunications, Beijing, China