Benchmark Design

Follow this topic Ask about this topic

Benchmark design studies how to turn a capability or real-world job into tasks, datasets, scoring rules, and protocols that support fair, reproducible comparisons. Common approaches include expert-curated and executable tests, domain-specific suites, hidden or refreshed items, controlled splits and stress tests, and multi-metric or rubric-based scoring. Researchers also audit validity, leakage, shortcuts, uncertainty, saturation, and ranking sensitivity, while increasingly evaluating full agent workflows under realistic tools, budgets, and deployment constraints. The central tension is making evaluations realistic and current without sacrificing transparency, reproducibility, and comparability; a high score alone rarely establishes general capability or readiness for use.

Latest papers

394 papers47 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarking
May 5, 2026Susmit Agrawal, Jannis Hollman, Matthias KümmererBenchmark Design Eye Tracking
Tübingen AI Center, University of Tübingen, Germany
ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation
May 5, 2026Cong Liu, Milong Ren, Jiaqi Guan +4Protein Structure Prediction Benchmark Construction
AMLab, AI4Science Lab, University of Amsterdam, Amsterdam, The Netherlands · Work done at ByteDance Seed · ByteDance Seed
LUMINA: A Grid Foundation Model for Benchmarking AC Optimal Power Flow Surrogate Learning
May 4, 2026Hongwei Jin, Keunju Song, Zeeshan Memon +5Neural Surrogate Modeling Benchmark Design
Mathematics and Computer Science Division, Argonne National Laboratory · Department of Electronic Engineering, Sogang University · Department of Computer Science, Emory University +1
RenCon 2025: Revival of the Expressive Performance Rendering Competition
May 3, 2026Huan Zhang, Taegyun Kwon, Anders Friberg +7Benchmark Design
Queen Mary University of London London, UK · Korea Advanced Institute of Science and Technology Daejeon, Korea · KTH Royal Institute of Technology Stockholm, Sweden +5
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
May 3, 2026Mario Koddenbrock, Christoph Lange, Robin Legner +6Benchmark Design Scientific ML
HTW Berlin · TU Berlin · KWS SAAT, Einbeck +4
PHBench: A Benchmark for Predicting Startup Series A Funding from Product Hunt Launch Signals
May 3, 2026Yagiz Ihlamur, Ben Griffin, Rick ChenBenchmark Design Finance
Amazon, Seattle, WA, USA · University of Oxford, Oxford, UK
PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction
May 3, 2026Zhiwen Yang, Pan Liu, Yifan Li +2Benchmark Design Benchmark Contamination
The Hong Kong University of Science and Technology (Guangzhou). · Yangzhou University. · The Hong Kong University of Science and Technology.
StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
May 3, 2026Yongrui Chen, Yangyang Ma, Xiaoying Huang +4LLM Evaluation Benchmark Design
Southeast University · Hangzhou Dianzi University · Zhejiang University +1
VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learning
May 3, 2026Zi-Yi Jia, Zi-Jian Cheng, Xin-Yue Zhang +4Benchmark Design Multimodal Model Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China
ECG-biometrics-bench: A Unified Framework for Reproducible Benchmarking of ECG Biometrics
May 2, 2026Milad ParvanBiometric Identification Benchmark Design
Independent Researcher · Milan, Italy
Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
May 1, 2026Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger +4Mathematical Reasoning Benchmarks LLM Evaluation
ETH Zurich · INSAIT · Sofia University "St. Kliment Ohridski"
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
Apr 28, 2026Leon Kogler, Stefan Hangler, Maximilian Ehrhart +3Automated Software Testing Benchmark Design
CASABLANCA hotelsoftware GmbH Schönwies, Austria · University of Innsbruck Innsbruck, Austria · Technical University of Munich Munich, Germany +1
PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators
Apr 28, 2026Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu +4Benchmark Design Clinical Language Model Evaluation
University of Illinois Urbana-Champaign · VinUniversity
TrialCalibre: A Fully Automated Causal Engine for RCT Benchmarking and Observational Trial Calibration
Apr 28, 2026Amir Habibdoust, Xing SongCausal Effect Estimation Benchmark Design
Institute for Data Science and Informatics, University of Missouri, Columbia, MO, USA. · Department of Biomedical Informatics, Biostatistics, and Medical Epidemiology, University of Missouri, Columbia, MO, USA.
EOS-Bench: A Comprehensive Benchmark for Earth Observation Satellite Scheduling
Apr 28, 2026Qian Yin, Jiaxing Li, Jiaqi Cheng +23Benchmark Construction Benchmark Design
School of Traﬃc and Transportation Engineering, Central South University, Changsha 410083, China · School of Engineering and Materials Science, Queen Mary University of London, London E1 4NS, UK · College of Automation, Central South University, Changsha, 410083, China +20
Benchmarking bandgap prediction in semiconductors under experimental and realistic evaluation settings
Apr 28, 2026Haolin Wang, Xianyuan Liu, Anna Jungbluth +3Benchmark Design Interpretable ML
Centre for Machine Intelligence, University of Sheffield, Sheffield, S1 3JD, UK · School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Climate Office, European Space Agency, Harwell, OX11 0FD, UK +2
Benchmarking Stopping Criteria for Evolutionary Multi-objective Optimization
Apr 28, 2026Kenji Kitamura, Ryoji TanabeBenchmark Design Multi-Objective Evolutionary Optimization
Yokohama National University Yokohama, Kanagawa, Japan
Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting
Apr 27, 2026Max Kleinebrahm, Jonathan Berrisch, Philipp Eiser +11Benchmark Design Time Series Forecasting
Karlsruhe Institute of Technology, Institute for Industrial Production, Karlsruhe, Germany · University of Duisburg-Essen, House of Energy Markets and Finance, Essen, Germany · Karlsruhe Institute of Technology, Institute for Automation and Applied Informatics, Eggenstein-Leopoldshafen, Germany +5
On Benchmark Hacking in ML Contests: Modeling, Insights and Design
Apr 24, 2026Xiaoyun Qiu, Yang Yu, Haifeng XuBenchmark Design
Department of Economics, Dartmouth College. · Sloan & CSAIL, Massachusetts Institute of Technology. · Department of Computer Science, University of Chicago.
OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving
Apr 23, 2026Xinyu Zhang, Boxuan Zhang, Yuchen Wan +5LLM Evaluation Benchmark Design
School of Computer Science and Technology, Xi’an Jiaotong University · 2Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China · 4Lenovo Research +1
Performance Anomaly Detection in Athletics: A Benchmarking System with Visual Analytics
Apr 23, 2026Blessed Madukoma, Prasenjit MitraBenchmark Design Time-Series Anomaly Detection
Carnegie Mellon University Africa · Kigali, Rwanda
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Apr 20, 2026Taylor Lundy, Narun K. Raman, Kevin Leyton-BrownLLM Evaluation Bayesian Optimization
Department of Computer Science University of British Columbia Vancouver, BC
R&F-Inventory: A Large-Scale Dataset for Monotonic Inventory Estimation in Reach and Frequency Advertising
Apr 18, 2026Yunshan Peng, Ji Wu, Wentao Bai +6Benchmark Design Online Advertising
Kuaishou Technology, Beijing, China
In Search of Lost DNA Sequence Pretraining
Apr 17, 2026Zhijiang Tang, Jiaxin Qi, Yan Cui +3Language Model Pretraining Masked Language Modeling
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China
MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events
Apr 16, 2026Raunak Agarwal, Markus Wenzel, Simon Baur +3Multi-Label Classification Healthcare
Department of Artificial Intelligence Fraunhofer Heinrich Hertz Institute Berlin, 10587, Germany
An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics
Apr 16, 2026Miri Liu, ChengXiang ZhaiAI for Science Benchmark Design
Department of Computer Science University of Illinois at Urbana-Champaign (UIUC)
FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems
Apr 16, 2026Wenhao Wang, Haoting Shi, Mengying Yuan +7Benchmark Design GUI Agents
Zhejiang University · Shanghai Jiao Tong University · Multi-Agent Governance & Intelligence Crew (MAGIC) +2
Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex
Apr 16, 2026Zhonghao Yang, Yu Li, Yanxu Zhu +6Benchmark Design Software Engineering Benchmarks
Shanghai Artificial Intelligence Laboratory
Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Apr 14, 2026Eliya Habba, Itay Itzhak, Asaf Yehudai +5LLM Evaluation Benchmark Design
The Hebrew University of Jerusalem · Technion · IBM Research +2

May 5, 2026J/K move · Enter open · S save

Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarking

Susmit Agrawal, Jannis Hollman, Matthias Kümmerer

Tübingen AI Center, University of Tübingen, Germany

Benchmark Design Eye Tracking Saliency Map Evaluation Kernel Density Estimation Visual Saliency Density Estimation

Open paper AskPDF arXiv

Empirical fixation densities, spatial distributions estimated from human eye-tracking data, are foundational to saliency benchmarking. They directly shape benchmark conclusions, leaderboard rankings, failure case analyses, and scientific claims about human visual behavior. Yet the standard estimation method, fixed-bandwidth isotropic Gaussian KDE, has gone essentially unchanged for decades. This matters now more than ever: as the field shifts toward sample-level evaluation (failure case analysis, inverse benchmarking, per-image model comparison), reliable per-image density estimates become critical. We propose a principled mixture model that combines an adaptive-bandwidth KDE based on Abramson's method, center bias and uniform components, and a state-of-the-art saliency model, to capture different spatial and semantic types of interobserver consistency, and optimize all parameters per image via leave-one-subject-out cross-validation. Our method yields substantially higher interobserver consistency estimates across multiple benchmarks, with median per-image gains of 5-15% in log-likelihood and up to 2 percentage points in AUC. For the most affected images -- precisely those most relevant to failure case analysis -- improvements exceed 25%. We leverage these improved estimates to identify and analyze remaining failure cases of state-of-the-art saliency models, demonstrating that significant headroom for model improvement remains. More broadly, our findings highlight that empirical fixation densities should not be treated as fixed ground truths but as evolving estimates that improve with better methodology.