Benchmark Design

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 65% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 285

All topics

Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions
May 21, 2026Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio +2Healthcare Benchmark Design
New York University
HyperBench: Standardizing and Scaling Synthetic Evaluation for Hyperspectral Super-Resolution
May 20, 2026Ritik Shah, Marco F. DuarteBenchmark Design Remote Sensing Image Super-Resolution
University of Massachusetts Amherst, MA 01003
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
May 20, 2026Bole Ma, Jan Eitzinger, Harald Koestler +1Expert Parallelism Expert Load Balancing
ShapeBench: A Scalable Benchmark and Diagnostic Suite for Standardized Evaluation in Aerodynamic Shape Optimization
May 20, 2026Shaghayegh Fazliani, Krissh Chawla, Jack Guo +3Benchmark Design
Stanford University. · Spinoza Labs.
Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation
May 18, 2026Liang Wang, Heng Meng, Zekai Xiang +4Benchmark Design Parametric CAD Modeling
School of Computer Science, Wuhan University, Wuhan 430000, Hubei, China · Spatial Design Intelligence Lab, BitInf Ltd., Shanghai 200003, China · College of Computer and Information Engineering, Nanjing Tech University, Nanjing 211800, Jiangsu, China +1
BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting
May 18, 2026Zhensheng Wang, Wenmian Yang, Qingtai Wu +3Quantitative Finance Benchmark Design
Beijing Normal University Beijing, China · Beijing Normal University Zhuhai, China · Elmleaf Ltd. Shanghai, China
MagBridge-Battery: A Synthetic Bridge Dataset for Li-ion Magnetometry and State-of-Health Diagnostics
May 17, 2026Sakthi Prabhu Gunasekar, Prasanna Kumar RangarajanBenchmark Design Synthetic Data Generation
Dept. of Computer Science & Engineering Amrita School of Computing Amrita Vishwa Vidyapeetham, India
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
May 16, 2026Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1Bayesian Optimization Benchmark Design
National University of Singapore
UniER: A Unified Benchmark for Item-level and Path-level Exercise Recommendation
May 16, 2026Xinghe Cheng, Guiyong Zhuang, Yusheng Xie +5Benchmark Design Recommender Systems
Jinan University, Guangzhou, China · Beijing University of Technology, Beijing, China · Griffith University, Gold Coast, Australia
ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents
May 15, 2026Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu +5Benchmark Design Web Agent Benchmarks
North Carolina State University · Shopify
FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models
May 14, 2026Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov +4LLM Evaluation Quantitative Finance
Lime FinTech
Breakeven complexity: A new perspective on neural partial differential equation solvers
May 14, 2026Yijing Zhang, Nicholas Roberts, Tanya Marwah +1Neural PDE Solvers Benchmark Design
University of Wisconsin–Madison · Google DeepMind
TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
May 14, 2026Pei Yang, Wanyi Chen, Tongyun Yang +15Benchmark Design LLM Agent Evaluation
Gradient · Soochow University · Independent Researcher +7
The Evaluation Trap: Benchmark Design as Theoretical Commitment
May 13, 2026Theodore J KalaitzidisBenchmark Auditing Benchmark Design
Brown University
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
May 13, 2026Stefan Baack, Christo Buschek, Maty BohacekLLM Evaluation Benchmark Design
Independent Researcher · Mozilla Fellowship · Stanford University
A Benchmark for Early-stage Parkinson's Disease Detection from Speech
May 13, 2026Terry Yi Zhong, Cristian Tejedor-Garcia, Khiet P. Truong +3Benchmark Design Speech Processing
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands · Center of Expertise for Parkinson and Movement Disorders, Radboud University Medical Center, Nijmegen, the Netherlands
Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
May 13, 2026Moritz Firsching, Paul Lezeau, Salvatore Mercuri +8Mathematical Reasoning Benchmarks Automated Theorem Proving
Google DeepMind · Imperial College London · Stockholms universitet +1
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmark Design High-Performance Computing
HKUST · NYU · SWUPL
Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models
May 12, 2026Mohammed Saidul Islam, Negin Baghbanzadeh, Farnaz Kohankhaki +5LLM Evaluation Benchmark Design
1Vector Institute · 2York University
MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces
May 11, 2026Srinivas Sridharan, Theodor-Adrian Badea, Andy Balogh +26Benchmark Design Hardware-Software Co-Design
1NVIDIA · 2Keysight · 3AMD +7
The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains
May 11, 2026Jung Min KangBenchmark Design Benchmark Validity
Independent Researcher · Seoul, South Korea
V4FinBench: Benchmarking Tabular Foundation Models, LLMs, and Standard Methods on Corporate Bankruptcy Prediction
May 11, 2026Marcin Kostrzewa, Sebastian Tomczak, Roman Furman +5Class-Imbalanced Learning Tabular Foundation Models
Department of Artificial Intelligence, Wrocław University of Science and Technology, Poland · Opera, Wrocław, Poland · Tooploox, Wrocław, Poland
BEACON: A Multimodal Dataset for Learning Behavioral Fingerprints from Gameplay Data
May 11, 2026Ishpuneet Singh, Gursmeep Kaur, Uday Pratap Singh Atwal +3Biometric Identification Benchmark Design
Computer Science and Engineering Department Thapar Institute of Engineering and Technology Patiala, India · David R Cheriton School of Computer Science University of Waterloo, Canada
Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation
May 11, 2026Shanshan Gao, Liyi ZhouComputer-Use Agent Benchmarks Benchmark Design
The University of Sydney
PlantMarkerBench: A Multi-Species Benchmark for Evidence-Grounded Plant Marker Reasoning
May 11, 2026Sajib Acharjee Dip, Song Li, Liqing ZhangLLM Evaluation Evidence-Grounded Reasoning
Department of Computer Science, Virginia Tech · School of Plant and Environmental Sciences, Virginia Tech · 3Health Sciences, Virginia Tech +2
An Executable Benchmarking Suite for Tool-Using Agents
May 10, 2026Zhiqing Zhong, Zhijing Ye, Jiamin Wang +1Computer-Use Agent Benchmarks Tool-Use Evaluation
Stevens Institute of Technology
Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon
May 10, 2026Víctor GallegoGPU Kernel Optimization Benchmark Design
Komorebi AI Technologies, Madrid, Spain
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation
May 10, 2026Zhen Hang, Yushan Yashengjiang, Junhui Li +21PDE Solving Benchmark Design
University of Science and Technology of China · Beijing University of Posts and Telecommunications · Tencent +10
TIDE-Bench: Task-Aware and Diagnostic Evaluation of Tool-Integrated Reasoning
May 10, 2026Yize Li, Junzhi Li, Jason Song +3LLM Evaluation Tool-Use Evaluation
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences
Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
May 9, 2026Guijin Son, Seungone Kim, Catherine Arnett +73Mathematical Reasoning Benchmarks Benchmark Design

May 21, 2026, cs.CYJ/K move · Enter open · S save

Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions

Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio, Fei Fang+1

New York University

Healthcare Benchmark Design Human-in-the-Loop Evaluation Benchmark Validity Clinical Language Model Evaluation

Open paper AskPDF arXiv

Benchmarks are necessary for healthcare evaluation, but are not sufficient for predicting deployment performance. Our position is that the evaluation--deployment gap arises not because of poorly designed benchmarks, but from implicit assumptions about how users interact with models that cannot be surfaced from benchmarks alone. To make this precise, we propose a classification of assumptions into two categories: task, which can be tested from conversation data alone, and outcome, which requires outcome data and behavioral studies for testing. Critically, outcome assumptions depend on human behavior, something that even well-designed benchmarks cannot directly observe. To demonstrate the operationality of this framework, we retrospectively analyze a healthcare RCT as a case study and find that the gap naturally separates into task and outcome gaps of roughly equal size. To address this, we make two contributions: first, we propose BenchmarkCards, an artifact that documents assumptions, and second, we propose staged evaluation, a procedure that systematically tests assumptions and evaluates performance.

Momentum

Related topics

Latest papers 285