Overview

Benchmark construction turns a capability or real-world need into evaluation tasks, reference answers or executable checks, and scoring rules. Approaches include expert annotation, mining deployment data, synthetic task generation, and interactive simulators, often paired with human review, automated verification, and reproducible release. The field increasingly emphasizes coverage, difficulty, robustness, provenance, and performance under realistic shifts rather than relying on a single aggregate score. Its central tension is scaling and refreshing benchmarks without sacrificing validity, auditability, or comparability across versions and settings.

Latest papers

CardsList

75 papers7 papers in the last four weeks, down 36% on the four weeks before. 0.1% of all new papers.

  1. Static Recovery Is Not Dynamic Stability: Dynamics-Aware Benchmarking of Protein Motif Scaffolding

    Nov 5, 2026Emile de Bruyn, Moritz Schäffler, Jannik Schneider +9Protein Structure GenerationProtein Design

  2. ToolSpace: Scaling Tool-Centric Environment Synthesis for Long-Horizon Agents

    Nov 3, 2026Guanting Dong, Zhichao Sun, Xinyu Yang +22LLM Tool UseLong-Horizon LLM Agents

  3. COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy

    Nov 3, 2026Sivaraaman Balakrishnan, Prakriti Shetty, Jeremiah Hong +2LLM EvaluationBenchmark Construction

  4. Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

    Nov 2, 2026Ao Lu, Yuanhao Zeng, Shaocheng Lan +5Coding Agent BenchmarksBenchmark Construction

  5. Evolving Self-tuning Benchmarks for Long-Horizon Tool Use

    Oct 19, 2026Hyunji Alex Nam, Michael Dennis, Natasha Jaques +1AI Agent BenchmarksTool-Use Evaluation

  6. IndustrialCAD-1400: Benchmarking Detailed Industrial Assemblies from Text

    Oct 17, 2026Hebao Zhu, Kang An, Zhuoke Liao +4Text-to-CAD GenerationBenchmark Construction

  7. EvalBuilder: Can LLMs Build Efficient Benchmarks?

    Oct 14, 2026Huaisheng Zhu, Hangfan Zhang, Teng Xiao +5Benchmark ConstructionAI Agent Benchmarks

  8. The Smart Buildings Challenge: Learning Foundation Models for Real-World Optimization at Scale

    Oct 13, 2026Judah Goldfeder, Ayush Rai, Philippe M. Wyder +11Building Energy ManagementTransfer Learning

  9. PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?

    Oct 12, 2026Dongyeop Woo, Nofit Segal, Nayoung Kim +3Crystal Structure PredictionMaterials Science

  10. EvalResearchBench: Can AI Agents Design Their Own Evaluations?

    Oct 3, 2026Yaolun Zhang, Tianyi Xu, Yujie Zhao +3Automated EvaluationBenchmark Construction

  11. WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

    Sep 29, 2026Haomin Qi, Xiangzhe Xu, Yiming Huang +2Software Engineering AgentsBenchmark Construction

  12. TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces

    Sep 27, 2026Dehai Min, Daoan Zhang, Yiming Zeng +13Benchmark ConstructionLLM Agent Evaluation

  13. A comparative assessment of global building and settlement datasets across geographic and settlement contexts

    Sep 23, 2026Rufai Omowunmi Balogun, Caroline Margaux Gevaert, Capucine Riom +5Benchmark ConstructionBuilding Footprint Extraction

  14. Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

    Sep 22, 2026Wooyoung JungBenchmark ConstructionNatural Language to SPARQL

  15. Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

    Sep 16, 2026Guojun Zhu, Xunheng Huang, Peng Yin +3Counterfactual EvaluationBenchmark Construction

  16. Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation

    Sep 14, 2026M. Ali BayramMultilingual Language Model EvaluationLLM Evaluation

  17. Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

    Sep 11, 2026Baoyang Jiang, Fengchun Zhang, Leyuan Wang +9Benchmark ConstructionEmbodied QA

  18. Field-level prediction of mid-plane stress tensor fields in concrete target penetration: a cross-velocity graph neural operator surrogate

    Sep 9, 2026Wenpu Du, Peng Zhou, Yunlong Xia +5Neural Surrogate ModelingBenchmark Construction

  19. A Large Open Multi-Energy Corpus of Soil Compaction Tests, with Machine-Learning Baselines

    Sep 3, 2026Sompote Youwai, Chana Phutthananon, Warat KongkitkulBenchmark ConstructionMaterials Property Prediction

  20. MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft

    Aug 28, 2026Sewoong Lee, Risham Sidhu, Julia Hockenmaier +1LLM EvaluationLLM Reliability

  21. A Query-Time Framework for Transient 2D Pore-Scale Flow Prediction and Generative Design

    Aug 23, 2026Yiming Wang, Jiale Zhu, Zhichen Ye +4Porous Media FlowNeural Surrogate Modeling

  22. CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation

    Aug 10, 2026Harmanjot Singh, Abhra Dubey, Jorge Alejandro Amador HerreraBenchmark ConstructionParametric CAD Modeling

  23. ArchEGraph: A Large-Scale Graph Dataset for Geometry-Topology-Physics Aligned Building Energy Modeling

    Aug 7, 2026Yihui Li, Yihui Chen, Kaidi Zha +7Graph ReconstructionBenchmark Construction