Benchmark Design

Follow this topic Ask about this topic

Benchmark design studies how to turn a capability or real-world job into tasks, datasets, scoring rules, and protocols that support fair, reproducible comparisons. Common approaches include expert-curated and executable tests, domain-specific suites, hidden or refreshed items, controlled splits and stress tests, and multi-metric or rubric-based scoring. Researchers also audit validity, leakage, shortcuts, uncertainty, saturation, and ranking sensitivity, while increasingly evaluating full agent workflows under realistic tools, budgets, and deployment constraints. The central tension is making evaluations realistic and current without sacrificing transparency, reproducibility, and comparability; a high score alone rarely establishes general capability or readiness for use.

Latest papers

394 papers47 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

Quantifying Ranking Instability Across Evaluation Protocol Axes in Gene Regulatory Network Benchmarking
Mar 3, 2026Ihor KendiukhovBenchmark Design Benchmark Validity
Department of Computer Science, University of Tübingen, Germany
Valet: A Standardized Testbed of Traditional Imperfect-Information Card Games
Mar 3, 2026Mark Goadrich, Achille Morenville, Éric PietteImperfect-Information Games AI Agent Benchmarks
Hendrix College, Conway AR 72032, USA · ICTEAM, UCLouvain, Louvain-la-Neuve, Belgium
SEED-SET: Scalable Evolving Experimental Design for System-level Ethical Testing
Mar 2, 2026Anjali Parashar, Yingke Li, Eric Yang Yu +4AI Safety Evaluation Benchmark Design
Laboratory for Information & Decision Systems (LIDS), MIT · Saab Inc.
AIDABench: AI Data Analytics Benchmark
Feb 27, 2026Yibo Yang, Fei Lei, Yixuan Sun +24LLM Evaluation Benchmark Design
SenseTime Research · Shanghai AI Laboratory · MMLab, The Chinese University of Hong Kong
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Feb 27, 2026Seungdong Yoa, Sanghyu Yoon, Suhee Yoon +4Language Model Reasoning Evaluation LLM Evaluation
LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies
FlexMS: A Unified Public Benchmark for Molecule Tandem Mass Spectrum Prediction
Feb 26, 2026Yunhua Zhong, Yixuan Tang, Yifan Li +5Benchmark Design Mass Spectrometry
The Hong Kong University of Science and Technology (Guangzhou). · The University of Hong Kong. · Yangzhou University. +3
BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Feb 25, 2026Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1Benchmark Validity Benchmark Design
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States
Qwen-BIM: developing large language model for BIM-based design with domain-specific benchmark and dataset
Feb 24, 2026Jia-Rui Lin, Yun-Hong Cai, Xiang-Rui Ni +2Benchmark Design LLM Evaluation
Department of Civil Engineering, Tsinghua University, Beijing, China, 100084 · Key Laboratory of Digital Construction and Digital Twin, Ministry of Housing and Urban-Rural Development, Beijing, China, 100084 · PetroChina Planning & Engineering Institute, Beijing, China, 100083
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Feb 24, 2026Ayush Sawarni, Jiyuan Tan, Vasilis SyrgkanisCausal Effect Estimation Benchmark Design
Stanford University
1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World
Feb 20, 2026Qiao Xu, Yipeng Yu, Chengxiao Feng +1Benchmark Design Execution-Guided Code Generation
Taobao & Tmall Group of Alibaba
Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmark
Feb 19, 2026Sourav Panda, Tanmay Ambadkar, Shreyash Kale +2RL Benchmarks Game-Playing Agents
Pennsylvania State University, PA, USA.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23
LiveClin: A Live Clinical Benchmark without Leakage
Feb 18, 2026Xidong Wang, Shuqi Guo, Yue Shen +6Clinical Language Model Evaluation Medical LLMs
The Chinese University of Hong Kong, ShenZhen · Ant Healthcare, Ant Group · The Ohio State University +1
OPBench: A Graph Benchmark to Combat the Opioid Crisis
Feb 16, 2026Tianyi Ma, Yiyang Li, Yiyue Qian +4Benchmark Design Healthcare
University of Notre Dame · Amazon · University of Connecticut
An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models
Jan 19, 2026Yuhiro Ono, Tomohiro Harada, Yukiya MiuraEvolutionary Optimization Benchmark Design
Graduate School of Systems Design, Tokyo Metropolitan University, 6-6, Asahigaoka, Hino, 1910065, Tokyo, Japan. · Graduate School of Science and Engineering, Saitama University, 2552026 Shimo-okubo, Sakura-ku, Saitama, 3388570, Saitama, Japan.
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Jan 15, 2026Yiwen Gao, Ruochen Zhao, Yang Deng +1Deep Research Agents Benchmark Design
National University of Singapore · Nanyang Technological University · Singapore Management University +1
Large-scale benchmarking of multi-objective soft-computing metaheuristics for redundancy allocation in repairable k-out-of-n systems
Dec 20, 2025Mateusz Oszczypała, David Ibehej, Jakub KudelaBenchmark Design Metaheuristic Optimization
Military University of Technology, gen. Sylwestra Kaliskiego Street 2, 00-908 Warsaw, Poland. · Institute of Automation and Computer Science, Brno University of Technology, Brno, Czech Republic.
CarBench: A Comprehensive Benchmark for Neural Surrogates on High-Fidelity 3D Car Aerodynamics
Nov 25, 2025Mohamed Elrefaie, Dule Shu, Matt Klenk +1Automotive Aerodynamics Neural Surrogate Modeling
Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA · Schwarzman College of Computing, Massachusetts Institute of Technology, Cambridge, MA, USA · Future Product Innovation, Toyota Research Institute, Los Altos, CA, USA
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Nov 8, 2025Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh +5Software Engineering Agents Benchmark Design
Harvard University · Google DeepMind · Princeton University +2
MLPerf Automotive
Oct 31, 2025Radoyeh Shojaei, Predrag Djurdjevic, Mostafa El-Khamy +7End-to-End Autonomous Driving Benchmark Design
University of California, Davis · Arm · Samsung +5
The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models
Oct 27, 2025Timo Freiesleben, Sebastian ZezulkaBenchmark Design Benchmark Validity
LMU Munich, MCMP & MCML · University of Tübingen
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Oct 9, 2025Ruiling Xu, Yifan ZhangLLM Evaluation Benchmark Design
University of Illinois Urbana-Champaign
TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing
Oct 5, 2025Jiaming Wang, Jizhuo Chen, Diwen Liu +1Benchmark Design Map-Based Localization
fev-bench: A Realistic Benchmark for Time Series Forecasting
Sep 30, 2025Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5Benchmark Design Time Series Forecasting
AWS · Keystone AI · Prior Labs +2
IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation
Sep 30, 2025Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck +57Mathematical Reasoning Benchmarks LLM Evaluation
WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning
Sep 27, 2025Xin Li, Mengbing Liu, Yiyang Zhu +4Mathematical Reasoning Benchmarks Benchmark Auditing
Nanyang Technological University
PDFBench: A Benchmark for De novo Protein Design from Function
May 25, 2025Jiahao Kuang, Nuowei Liu, Jie Wang +3Benchmark Design Molecular Generation
School of Computer Science and Technology, East China Normal University · Institute of Artificial Intelligence (TeleAI), China Telecom · College of Foreign Languages and Literatures, Fudan University
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3
BiasBench: A reproducible benchmark for tuning the biases of event cameras
Apr 25, 2025Andreas Ziegler, David Joseph, Thomas Gossard +2Event-Based Vision Benchmark Design
Cognitive Systems Group, University of Tübingen, Germany
Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Commonsense Reasoning Benchmark Auditing
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France

Mar 3, 2026J/K move · Enter open · S save

Quantifying Ranking Instability Across Evaluation Protocol Axes in Gene Regulatory Network Benchmarking

Ihor Kendiukhov

Department of Computer Science, University of Tübingen, Germany

Benchmark Design Benchmark Validity Pairwise Ranking Gene Regulatory Network Inference

Open paper AskPDF arXiv

Benchmark rankings are routinely used to justify scientific claims about method quality in gene regulatory network (GRN) inference, yet the stability of these rankings under plausible evaluation protocol choices is rarely examined. We present a systematic diagnostic framework for measuring ranking instability under protocol shift, including decomposition tools that separate base rate effects from discrimination effects. Using existing single cell GRN benchmark outputs across three human tissues and six inference methods, we quantify pairwise reversal rates across four protocol axes: candidate set restriction (16.3 percent, 95 percent CI 11.0 to 23.4 percent), tissue context (19.3 percent), reference network choice (32.1 percent), and symbol mapping policy (0.0 percent). A permutation null confirms that observed reversal rates are far below random order expectations (0.163 versus null mean 0.500), indicating partially stable but non invariant ranking structure. Our decomposition reveals that reversals are driven by changes in the relative discrimination ability of methods rather than by base rate inflation, a finding that challenges a common implicit assumption in GRN benchmarking. We propose concrete reporting practices for stability aware evaluation and provide a diagnostic toolkit for identifying method pairs at risk of reversal.