Benchmark Design

Ask about this topic

Overview

Benchmark design studies how to turn a capability or real-world job into tasks, datasets, scoring rules, and protocols that support fair, reproducible comparisons. Common approaches include expert-curated and executable tests, domain-specific suites, hidden or refreshed items, controlled splits and stress tests, and multi-metric or rubric-based scoring. Researchers also audit validity, leakage, shortcuts, uncertainty, saturation, and ranking sensitivity, while increasingly evaluating full agent workflows under realistic tools, budgets, and deployment constraints. The central tension is making evaluations realistic and current without sacrificing transparency, reproducibility, and comparability; a high score alone rarely establishes general capability or readiness for use.

Latest papers

CardsList

394 papers47 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

  1. GNBG-C: A Unified and Configurable Benchmark Generator for Constrained Continuous Optimization

    Oct 27, 2026Ali Ghasemzadeh, Erfan Babakhani, Armin Khosravi +4Synthetic Benchmark GenerationConstrained Optimization

    Department of Data and Systems Engineering, the University of Hong Kong, Hong Kong SAR, China · Independent Researcher, Isfahan, Iran · Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia +3

  2. LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

    Oct 27, 2026Sinuo Wang, Wentao Zhao, Jiacheng Lu +6Financial Language Model EvaluationBenchmark Design

    StepFun · Adelaide University · Tsinghua University +3

  3. EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

    Oct 26, 2026Yiming Sun, Yuan Zhang, Qi Cheng +4Benchmark DesignBenchmark Validity

    Rutgers University, New Brunswick · Apple Inc. · University of Alabama +1

  4. Performance of Neural Operator Baselines on 'The Well' Dataset

    Oct 25, 2026Yash Pincha, Andy Dimnaku, Andrew Chiang +3Neural OperatorsNeural Surrogate Modeling

  5. RadonBench: A Benchmark of Classical and Learning-Guided Search Techniques for Problems in Combinatorial Geometry

    Oct 23, 2026Xiaoan Xu, Ethan X. Fang, Jesus A. De Loera +2Combinatorial OptimizationBenchmark Design

    Department of Electrical and Computer Engineering, Duke University. · Department of Biostatistics and Bioinformatics, Duke University. · Department of Mathematics, University of California, Davis. +2

  6. PLE: Physical-Engineering's Last Exam

    Oct 21, 2026Rujun Han, Yufan Zhuang, Arijit Ray +10AI Agent BenchmarksBenchmark Design

    Google Cloud AI Research · Google

  7. Your Benchmark Is an Empirical Measure Over Difficulty

    Oct 21, 2026Yifan Sun, Naicheng Yu, Jiawen Gong +2Benchmark DesignBenchmark Auditing

    University of Illinois Urbana-Champaign · University of California San Diego · New York University

  8. ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

    Oct 20, 2026Akarsh Ashok, Dinil Mon Divakaran, Vivek BalachandranBenchmark DesignSoftware Engineering Benchmarks

    Singapore Institute of Technology · A*STAR Institute of Advanced Intelligence and Computing

  9. MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching

    Oct 19, 2026Yitong Li, Jiang Bian, Jingling Wang +2LLM EvaluationBenchmark Design

    Keep, Beijing, China

  10. FrontierRefactor: Benchmarking Multi-Scale Performance-Oriented Codebase Refactoring

    Oct 19, 2026Shijian Wang, Fanqing Meng, Jiajun Chen +8Automated Code RefactoringSoftware Engineering Benchmarks

    Evolvent AI · National University of Singapore · University of Southern California +2

  11. fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets

    Oct 17, 2026Sanket Mhatre, Sumit Gulwani, Vu Le +2Spreadsheet AutomationBenchmark Design

    Proximal · Microsoft, USA · Microsoft, India +1

  12. On the Selectivity of Generative Models in Structure-Based Drug Design

    Oct 16, 2026Ella Miray Rajaonson, Jungyoon Lee, William Chau +5Structure-Based Drug DesignDrug Discovery

    University of Toronto · Vector Institute · Université de Montréal +6

  13. Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

    Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark DesignSoftware Engineering Benchmarks

    Rensselaer Polytechnic Institute · IBM

  14. Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol

    Oct 15, 2026Emad Aghajanzadeh, Thomas Schwarzfischer, Moritz Thoma +5Autonomous Driving BenchmarksBenchmark Design

    BMW Group, Munich, Germany · Technical University of Munich (TUM), Munich, Germany · Vienna University of Technology (TU Wien), Vienna, Austria

  15. LOADBench: A Large Open Analog Dataset and Benchmark for Machine Learning in IC Design

    Oct 15, 2026M. Leibl, F. Azevedo, S. Uhlich +7Electronic Design AutomationBenchmark Design

    TUM / Chair of Electronic Design Automation · Instituto de Telecomunicações (IT) / IST-UL · SonyAI +4

  16. ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

    Oct 15, 2026Zichuan Xu, Yuyang Dong, Makoto Onizuka +1Tool-Use EvaluationLLM Agent Evaluation

    University of Osaka · Initial S · Nagoya University

  17. LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making

    Oct 14, 2026Jingyuan Li, Yujie Liu, Yijie Dai +18Evidence-Grounded ReasoningScientific Reasoning in Language Models

    Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +6

  18. EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents

    Oct 14, 2026Qianhong Guo, Wei Xie, Xiaofang Cai +3LLM Agent EvaluationAI Agent Benchmarks

    College of Computer Science and Technology, National University of Defense Technology · Institute of Information Engineering, Chinese Academy of Sciences

  19. Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

    Oct 13, 2026Mir Tafseer Nayeem, Davood RafieiBenchmark AuditingBenchmark Design

    University of Alberta

  20. PMO-Dock: Benchmarking Docking, Specificity, and Generalization in Molecular Optimization

    Oct 13, 2026Gor Simonyan, Tatevik Abrahamyan, Narek Abelyan +2Molecular OptimizationBenchmark Design

    Yerevan State University · University of Cambridge

  21. Beyond Equation Search: Conjecture-Guided Discovery with SCOPE and ClosureBench

    Oct 13, 2026Yuxiao Hu, Qian Li, Ding Wang +4Scientific DiscoveryAI Agents for Scientific Discovery

    The Hong Kong Polytechnic University, Hong Kong, China · Eastern Institute of Technology, Ningbo, China · Shanghai Jiao Tong University, Shanghai, China +1

  22. Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

    Oct 13, 2026Guoming Ling, Yupei Lin, Zhongzhan Huang +5LLM EvaluationBenchmark Design

    Sun Yat-sen University

  23. CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

    Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark GenerationBenchmark Design

    University of California, Berkeley

  24. How to Build a Benchmark That Actually Measures Taste

    Oct 10, 2026Xiaofan Wu, Xi Zeng, Miaoxia Chen +8Benchmark DesignBenchmark Validity

    Chance AI

  25. SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

    Oct 9, 2026Bastian Schäfer, Dominika Matus, Lennart Purucker +5Feature SelectionBenchmark Design

    Université du Littoral-Côte-d’Opale · University of Freiburg · Prior Labs +2

  26. SqueezeBench: Saturation-Resilient Evaluation of Mathematical Capability

    Oct 9, 2026Pablo Manrique, Hai Xia, Stefan SzeiderMathematical Reasoning BenchmarksBenchmark Design

    Algorithms and Complexity Group TU Wien, Vienna, Austria

  27. Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

    Oct 9, 2026Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta +9Deep Learning OptimizationBenchmark Design

    Alice · Goodfire · University of Tübingen +1

  28. SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

    Oct 8, 2026Wei Minn, Zhensu Sun, Lwin Khin Shar +1Benchmark DesignCoding Agents

    School of Computing and Information Systems Singapore Management University

  29. A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

    Oct 8, 2026Ming Chen, Rong-Xi Tan, Ke Xue +8Black-Box OptimizationAI Agent Benchmarks

    State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University

  30. How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices

    Oct 8, 2026Liulei Zhang, Dejing Zhou, Chuyue Huang +5AI-Assisted Scientific ResearchBenchmark Design

    NLP2CT Lab, Department of Artificial Intelligence University of Macau, Macau, China