Overview

Benchmark design studies how to turn a capability or real-world job into tasks, datasets, scoring rules, and protocols that support fair, reproducible comparisons. Common approaches include expert-curated and executable tests, domain-specific suites, hidden or refreshed items, controlled splits and stress tests, and multi-metric or rubric-based scoring. Researchers also audit validity, leakage, shortcuts, uncertainty, saturation, and ranking sensitivity, while increasingly evaluating full agent workflows under realistic tools, budgets, and deployment constraints. The central tension is making evaluations realistic and current without sacrificing transparency, reproducibility, and comparability; a high score alone rarely establishes general capability or readiness for use.

Latest papers

CardsList

413 papers26 papers in the last four weeks, down 49% on the four weeks before. 0.2% of all new papers.

  1. Bookkeeping Is Not Judgement: EJBench, an Engineering Judgement Benchmark Built from a Conjugate Heat Transfer Design Campaign

    Nov 16, 2026James Sadler, Rotimi Alabi, Kotub UddinBenchmark Design

  2. RSLiveBench-UAV: A Continually Evolving Vision-Language Benchmark for Aerial Spatial Intelligence

    Nov 11, 2026Yue Zhou, Jiawei Liu, Jue Chen +9Continual Learning BenchmarkingVLM Evaluation

  3. TrialDesignBench: Benchmarking Language Models for Phase-I Trial Design Against Clinical Knowledge

    Nov 9, 2026Yihan Hank Tang, Peng Yang, Jiayi Xin +5Benchmark DesignClinical Language Model Evaluation

  4. ICLayout-Bench: Benchmarking LLM Agents for Custom Circuit Layout Design

    Nov 8, 2026Zhangmai Li, Tianrui Ma, Zheng Weng +8Electronic Design AutomationLLM Agent Evaluation

  5. GlobalTap: Globally Diverse Beat Tracking Benchmark via Crowdsourced Tapping

    Nov 8, 2026William T. Botticelli-Wells, Manuel Anglada-Tort, Elif Celen +4Benchmark DesignMusic Information Retrieval

  6. Receiver-Resolution APS-PDP Maps for Wireless Channels: Benchmarking and Coarse-to-Fine Modeling

    Nov 7, 2026Peilin Zheng, Haozhe Jia, Xiucheng Wang +2Radio Propagation ModelingBenchmark Design

  7. BO-Arena: An Evolving Benchmark for High-Dimensional Bayesian Optimisation

    Nov 6, 2026Colin Doumont, Thomas Christie, Henry Moss +1Bayesian OptimizationBenchmark Design

  8. PINNeval: A Comprehensive Evaluation Standard for Physics-Informed Neural Networks

    Nov 5, 2026Kevin von Bargen, Hamid Ebrahimy, Tim RömerPhysics-Informed MLBenchmark Design

  9. TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research

    Nov 5, 2026Yifan Luo, Zhengying Liu, Yang Yuan +2Language Model Reasoning EvaluationAI Agent Benchmarks

  10. TestBED: An Open-Source Toolkit for Bayesian Experimental Design

    Nov 4, 2026Gavin Kerrigan, Marcel Hedman, Sahel Iqbal +1Bayesian Experimental DesignBenchmark Reproducibility

  11. ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

    Nov 2, 2026Kevin Dick, Katie LonswayHyperparameter OptimizationFitness Landscape Analysis

  12. BEND 2.0: Benchmarking the Capabilities Needed to Design DNA

    Nov 2, 2026Andreas Bjerregaard, Rachael DeVries, Dionysia Danai Brilli +6Genomic Foundation ModelsGenomics

  13. Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking

    Nov 2, 2026Victor Barres, Aden Barton, Nicolai J. Foss +1AI Agent BenchmarksAutomated Agent Evaluation

  14. Data-Scarce, Multi-Granular, Multimodal: Benchmark Design for Biosynthetic Gene Clusters

    Nov 1, 2026Hanlin Xiao, Eriko Takano, Mauricio A. Álvarez +1Benchmark DesignBioinformatics

  15. Simple Features Beat Most AI Control Monitors

    Oct 30, 2026Shubhrangshu Debsarkar, Valmik NahataBackdoor DetectionLLM Safety Monitoring

  16. GNBG-C: A Unified and Configurable Benchmark Generator for Constrained Continuous Optimization

    Oct 27, 2026Ali Ghasemzadeh, Erfan Babakhani, Armin Khosravi +4Synthetic Benchmark GenerationConstrained Optimization

  17. EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

    Oct 26, 2026Yiming Sun, Yuan Zhang, Qi Cheng +4Benchmark DesignBenchmark Validity

  18. RadonBench: A Benchmark of Classical and Learning-Guided Search Techniques for Problems in Combinatorial Geometry

    Oct 23, 2026Xiaoan Xu, Jesus A. De Loera, Ethan X. Fang +2Combinatorial OptimizationBenchmark Design

  19. PLE: Physical-Engineering's Last Exam

    Oct 21, 2026Rujun Han, Yufan Zhuang, Arijit Ray +10AI Agent BenchmarksBenchmark Design

  20. Your Benchmark Is an Empirical Measure Over Difficulty

    Oct 21, 2026Yifan Sun, Naicheng Yu, Jiawen Gong +2Benchmark DesignBenchmark Auditing

  21. ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

    Oct 20, 2026Akarsh Ashok, Dinil Mon Divakaran, Vivek BalachandranBenchmark DesignSoftware Engineering Benchmarks

  22. FrontierRefactor: Benchmarking Multi-Scale Performance-Oriented Codebase Refactoring

    Oct 19, 2026Shijian Wang, Fanqing Meng, Jiajun Chen +8Automated Code RefactoringSoftware Engineering Benchmarks

  23. fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets

    Oct 17, 2026Sanket Mhatre, Sumit Gulwani, Vu Le +2Spreadsheet AutomationBenchmark Design

  24. On the Selectivity of Generative Models in Structure-Based Drug Design

    Oct 16, 2026Ella Miray Rajaonson, Jungyoon Lee, William Chau +5Structure-Based Drug DesignDrug Discovery

  25. Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

    Oct 16, 2026Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3Benchmark DesignSoftware Engineering Benchmarks