Benchmark Design

Follow this topic Ask about this topic

Benchmark design studies how to turn a capability or real-world job into tasks, datasets, scoring rules, and protocols that support fair, reproducible comparisons. Common approaches include expert-curated and executable tests, domain-specific suites, hidden or refreshed items, controlled splits and stress tests, and multi-metric or rubric-based scoring. Researchers also audit validity, leakage, shortcuts, uncertainty, saturation, and ranking sensitivity, while increasingly evaluating full agent workflows under realistic tools, budgets, and deployment constraints. The central tension is making evaluations realistic and current without sacrificing transparency, reproducibility, and comparability; a high score alone rarely establishes general capability or readiness for use.

Latest papers

394 papers47 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions
Apr 13, 2026Shizheng Hou, Wenqi Pei, Nuo Chen +3Text-to-SQL Benchmark Design
National University of Singapore · Zhejiang University
WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
Apr 13, 2026Peng Yuan, Yuyang Yin, Yuxuan Cai +1Web Agent Benchmarks Benchmark Design
Tencent BAC · Tsinghua University
Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models
Apr 10, 2026Avni Mittal, Shanu Kumar, Sandipan Dandapat +1Multilingual Language Model Evaluation Benchmark Design
Microsoft Corporation, India · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Indian Institute of Technology Hyderabad
$k$ -server-bench: Automating Potential Discovery for the $k$ -Server Conjecture
Apr 8, 2026Kirill Brilliantov, Etienne Bamas, Emmanuel AbbéAI-Assisted Mathematical Discovery Autonomous Scientific Discovery
MDS Lab, EPFL
Block-Bench: A Framework for Controllable and Transparent Discrete Optimization Benchmarking
Apr 8, 2026Furong Ye, Frank Neumann, Thomas Bäck +1Benchmark Design Synthetic Benchmark Generation
LIACS, Leiden University, Leiden, the Netherlands · Optimisation and Logistics, School of Computer Science and Information Technology, Adelaide University, Adelaide, Australia
Riemann-Bench: A Benchmark for Moonshot Mathematics
Apr 8, 2026Suhaas Garre, Erik Knutsen, Sushant Mehta +1Mathematical Reasoning Benchmarks Benchmark Design
Surge AI
IndustryCode: A Benchmark for Industry Code Generation
Apr 3, 2026Puyu Zeng, Zhaoxi Wang, Zhixu Duan +7Code Generation Code Generation Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China · Tsinghua University, Beijing, China
EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild
Apr 2, 2026Yiming Fan, Jun Yeon Won, Ding Zhu +4Benchmark Design Software Security
The Ohio State University
Benchmark Problems and Benchmark Datasets for the evaluation of Machine and Deep Learning methods on Photoplethysmography signals: the D4 report from the QUMPHY project
Apr 1, 2026Urs Hackstein, Jordi Alastruey, Philip Aston +13Photoplethysmography Benchmark Design
Technische Hochschule Mittelhessen - University of Applied Sciences, Giessen, Germany · King’s College London, London, United Kingdom · National Physical Laboratory, Teddington, United Kingdom +5
YC Bench: a Live Benchmark for Forecasting Startup Outperformance in Y Combinator Batches
Apr 1, 2026Mostapha BenhendaForecasting Benchmarks Benchmark Design
BenchScope: How Many Independent Signals Does Your Benchmark Provide?
Mar 31, 2026Tommy Sha, Stella ZhaoBenchmark Auditing Benchmark Design
Stony Brook University · University of Minnesota Twin Cities
MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development
Mar 26, 2026Moshood A. Fakorede, Krishna Upadhyay, A. B. Siddique +1Software Engineering Software Engineering Benchmarks
Louisiana State University · University of Kentucky
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Mar 25, 2026Gabriel Orlanski, Devjeet Roy, Alexander Yun +7AI Coding Agents Coding Agent Benchmarks
University of Wisconsin–Madison · Washington State University · MIT
Bridging the Evaluation Gap: Standardized Benchmarks for Multi-Objective Search
Mar 25, 2026Hadar Peer, Carlos Hernandez, Sven Koenig +2Benchmark Design Multi-Objective Optimization
Technion – Israel Institute of Technology · Universidad San Sebastián · University of California, Irvine +1
GTO Wizard Benchmark
Mar 24, 2026Marc-Antoine Provost, Nejc Ilenic, Christopher Solinas +1Agent Benchmarks Game-Playing Agents
GTO Wizard
BenchBench: Benchmarking Automated Benchmark Generation
Mar 21, 2026Yandan Zheng, Haoran Luo, Zhenghong Lin +2Benchmark Design Synthetic Benchmark Generation
Nanyang Technological University Singapore, Singapore
SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits
Mar 19, 2026Edward Lin, Sahil Modi, Siva Kumar Sastry Hari +30GPU Kernel Optimization Benchmark Design
NVIDIA
AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
Mar 19, 2026An Luo, Jin Du, Xun Xian +12AI Agent Evaluation AI Agent Benchmarks
School of Statistics, University of Minnesota · AIScientists, Inc. · Data Science Institute, University of Chicago +4
Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Models
Mar 16, 2026Yujia Yang, Yuanxiang Wang, Zhenyu Guan +11Image Editing Image Editing Evaluation
University of Chinese Academy of Sciences · Tencent
s2n-bignum-bench: A practical benchmark for evaluating low-level code reasoning of LLMs
Mar 15, 2026Balaji Rao, John Harrison, Soonho Kong +2Large Language Model-Guided Theorem Proving Automated Theorem Proving
Stevens Institute of Technology Hoboken, NJ 07030 · Amazon Web Services Seattle, WA 98101
TaoBench: Do Automated Theorem Prover LLMs Generalize Beyond MathLib?
Mar 13, 2026Alexander K Taylor, Junyi Zhang, Ethan Ji +10Automated Theorem Proving Mathematical Reasoning Benchmarks
University of California, Los Angeles · University of California, Santa Barbara
SciDesignBench: Benchmarking and Improving Language Models for Scientific Inverse Design
Mar 13, 2026David van Dijk, Ivan VrkicInverse Design Large Language Model-Guided Optimization
CellType Inc.
CHIMERA-Bench: A Benchmark Dataset for Epitope-Specific Antibody Design
Mar 13, 2026Mansoor Ahmed, Nadeem Taj, Imdad Ullah Khan +2Benchmark Design Protein Binder Design
Georgia State University, Atlanta, GA, USA · Georgia Institute of Technology, Atlanta, GA, USA · University of Engineering and Technology, Lahore, Pakistan +1
Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions
Mar 11, 2026Ziwei Wang, Zhentao He, Xingyi He +6Synthetic Data Generation Brain-Computer Interfaces
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Zhongguancun Academy, Beijing, China
SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs
Mar 11, 2026Yadi Cao, Sicheng Lai, Jiahe Huang +12AI Agent Benchmarks Large Language Model-Guided Optimization
University of California San Diego · The Chinese University of Hong Kong, Shenzhen · Peking University +3
NanoBench: A Multi-Task Benchmark Dataset for Nano-Quadrotor System Identification, Control, and State Estimation
Mar 10, 2026Syed Izzat Ullah, Jose BacaBenchmark Design Nonlinear System Identification
Department of Engineering Texas A&M University-Corpus Christi, Corpus Christi, TX, USA
Leaderboard Incentives: Model Rankings under Strategic Post-Training
Mar 9, 2026Yatong Chen, Guanhua Zhang, Moritz HardtBenchmark Design Benchmark Validity
Max Planck Institute for Intelligent Systems, Tübingen and Tübingen AI Center
PathBench: Speech Intelligibility Benchmark for Automatic Pathological Speech Assessment
Mar 9, 2026Bence Mark Halpern, Thomas Tienkamp, Defne Abur +1Benchmark Design Speech Quality Assessment
1Nagoya University, Japan · Department of Neurology, Faculty of Medicine and University Hospital Cologne, University of Cologne, Germany · University of Groningen, The Netherlands
PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation
Mar 7, 2026Xin-Sheng Chen, Jiayu Zhu, Pei-lin Li +3Benchmark Design Generative AI Evaluation
Tsinghua University
Empirical Evaluation of No Free Lunch Violations in Permutation-Based Optimization
Mar 4, 2026Grzegorz SrokaBlack-Box Optimization Benchmark Design
Department of Nonlinear Analysis, Rzeszów University of Technology, Powstańców Warszawy 12, 35-959 Rzeszów, Poland

Apr 13, 2026J/K move · Enter open · S save

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

Shizheng Hou, Wenqi Pei, Nuo Chen, Quang-Trung Ta+2

National University of Singapore · Zhejiang University

Text-to-SQL Benchmark Design LLM Evaluation LLM Inference Efficiency

Open paper AskPDF arXiv

Natural Language to SQL (NL2SQL) technology empowers non-expert users to query relational databases without requiring SQL expertise. While large language models (LLMs) have greatly improved NL2SQL algorithms, their rapid development outpaces systematic evaluation, leaving a critical gap in understanding their effectiveness, efficiency, and limitations. To this end, we present NL2SQLBench, the first modular evaluation and benchmarking framework for LLM-enabled NL2SQL approaches. Specifically, we dissect NL2SQL systems into three core modules: Schema Selection, Candidate Generation, and Query Revision. For each module, we comprehensively review existing strategies and propose novel fine-grained metrics that systematically quantify module-level effectiveness and efficiency. We further implement these metrics in a flexible multi-agent framework, allowing configurable benchmarking across diverse NL2SQL approaches. Leveraging NL2SQLBench, we rigorously evaluate ten representative open-source methods on two datasets, the BIRD development set and the ScienceBenchmark development set, using two LLMs, DeepSeek-V3 and GPT-4o mini. We systematically assess each approach across the three core modules and evaluate multiple critical performance dimensions. Our evaluation reveals significant gaps in existing NL2SQL methods, highlighting not only substantial room for accuracy improvements but also the significant computational inefficiency, which severely hampers real-world adoption. Furthermore, our analysis identifies critical shortcomings in current benchmark datasets and evaluation rules, emphasizing issues such as inaccurate gold SQL annotations and limitations in existing evaluation rules. By synthesizing these insights into a unified benchmarking, our study establishes a clear reference point for fair comparison and serves as essential guidance for future targeted innovation in NL2SQL technology.