Benchmark Design

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 65% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 285

All topics

PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators
Apr 28, 2026Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu +4Benchmark Design Clinical Language Model Evaluation
University of Illinois Urbana-Champaign · VinUniversity
TrialCalibre: A Fully Automated Causal Engine for RCT Benchmarking and Observational Trial Calibration
Apr 28, 2026Amir Habibdoust, Xing SongCausal Effect Estimation Benchmark Design
Institute for Data Science and Informatics, University of Missouri, Columbia, MO, USA. · Department of Biomedical Informatics, Biostatistics, and Medical Epidemiology, University of Missouri, Columbia, MO, USA.
EOS-Bench: A Comprehensive Benchmark for Earth Observation Satellite Scheduling
Apr 28, 2026Qian Yin, Jiaxing Li, Jiaqi Cheng +23Benchmark Construction Benchmark Design
School of Traﬃc and Transportation Engineering, Central South University, Changsha 410083, China · School of Engineering and Materials Science, Queen Mary University of London, London E1 4NS, UK · College of Automation, Central South University, Changsha, 410083, China +20
Benchmarking bandgap prediction in semiconductors under experimental and realistic evaluation settings
Apr 28, 2026Haolin Wang, Xianyuan Liu, Anna Jungbluth +3Benchmark Design Interpretable ML
Centre for Machine Intelligence, University of Sheffield, Sheffield, S1 3JD, UK · School of Computer Science, University of Sheffield, Sheffield, S1 4DP, UK · Climate Office, European Space Agency, Harwell, OX11 0FD, UK +2
Benchmarking Stopping Criteria for Evolutionary Multi-objective Optimization
Apr 28, 2026Kenji Kitamura, Ryoji TanabeBenchmark Design Multi-Objective Evolutionary Optimization
Yokohama National University Yokohama, Kanagawa, Japan
Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting
Apr 27, 2026Max Kleinebrahm, Jonathan Berrisch, Philipp Eiser +11Benchmark Design Time Series Forecasting
Karlsruhe Institute of Technology, Institute for Industrial Production, Karlsruhe, Germany · University of Duisburg-Essen, House of Energy Markets and Finance, Essen, Germany · Karlsruhe Institute of Technology, Institute for Automation and Applied Informatics, Eggenstein-Leopoldshafen, Germany +5
On Benchmark Hacking in ML Contests: Modeling, Insights and Design
Apr 24, 2026Xiaoyun Qiu, Yang Yu, Haifeng XuBenchmark Design
Department of Economics, Dartmouth College. · Sloan & CSAIL, Massachusetts Institute of Technology. · Department of Computer Science, University of Chicago.
OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving
Apr 23, 2026Xinyu Zhang, Boxuan Zhang, Yuchen Wan +5LLM Evaluation Benchmark Design
School of Computer Science and Technology, Xi’an Jiaotong University · 2Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China · 4Lenovo Research +1
Performance Anomaly Detection in Athletics: A Benchmarking System with Visual Analytics
Apr 23, 2026Blessed Madukoma, Prasenjit MitraBenchmark Design Time-Series Anomaly Detection
Carnegie Mellon University Africa · Kigali, Rwanda
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Apr 20, 2026Taylor Lundy, Narun K. Raman, Kevin Leyton-BrownLLM Evaluation Bayesian Optimization
Department of Computer Science University of British Columbia Vancouver, BC
R&F-Inventory: A Large-Scale Dataset for Monotonic Inventory Estimation in Reach and Frequency Advertising
Apr 18, 2026Yunshan Peng, Ji Wu, Wentao Bai +6Benchmark Design Online Advertising
Kuaishou Technology, Beijing, China
In Search of Lost DNA Sequence Pretraining
Apr 17, 2026Zhijiang Tang, Jiaxin Qi, Yan Cui +3Language Model Pretraining Masked Language Modeling
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China
MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events
Apr 16, 2026Raunak Agarwal, Markus Wenzel, Simon Baur +3Multi-Label Classification Healthcare
Department of Artificial Intelligence Fraunhofer Heinrich Hertz Institute Berlin, 10587, Germany
An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics
Apr 16, 2026Miri Liu, ChengXiang ZhaiAI for Science Benchmark Design
Department of Computer Science University of Illinois at Urbana-Champaign (UIUC)
FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems
Apr 16, 2026Wenhao Wang, Haoting Shi, Mengying Yuan +7Benchmark Design GUI Agents
Zhejiang University · Shanghai Jiao Tong University · Multi-Agent Governance & Intelligence Crew (MAGIC) +2
Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex
Apr 16, 2026Zhonghao Yang, Yu Li, Yanxu Zhu +6Benchmark Design Software Engineering Benchmarks
Shanghai Artificial Intelligence Laboratory
Riemann-Bench: A Benchmark for Moonshot Mathematics
Apr 8, 2026Suhaas Garre, Erik Knutsen, Sushant Mehta +1Mathematical Reasoning Benchmarks Benchmark Design
Surge AI
EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild
Apr 2, 2026Yiming Fan, Jun Yeon Won, Ding Zhu +4Benchmark Design Software Security
The Ohio State University
NanoBench: A Multi-Task Benchmark Dataset for Nano-Quadrotor System Identification, Control, and State Estimation
Mar 10, 2026Syed Izzat Ullah, Jose BacaBenchmark Design Nonlinear System Identification
Department of Engineering Texas A&M University-Corpus Christi, Corpus Christi, TX, USA
PathBench: Speech Intelligibility Benchmark for Automatic Pathological Speech Assessment
Mar 9, 2026Bence Mark Halpern, Thomas Tienkamp, Defne Abur +1Benchmark Design Speech Quality Assessment
1Nagoya University, Japan · Department of Neurology, Faculty of Medicine and University Hospital Cologne, University of Cologne, Germany · University of Groningen, The Netherlands
FlexMS: A Unified Public Benchmark for Molecule Tandem Mass Spectrum Prediction
Feb 26, 2026Yunhua Zhong, Yixuan Tang, Yifan Li +5Benchmark Design Mass Spectrometry
The Hong Kong University of Science and Technology (Guangzhou). · The University of Hong Kong. · Yangzhou University. +3
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Feb 24, 2026Ayush Sawarni, Jiyuan Tan, Vasilis SyrgkanisCausal Effect Estimation Benchmark Design
Stanford University
1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World
Feb 20, 2026Qiao Xu, Yipeng Yu, Chengxiao Feng +1Benchmark Design Execution-Guided Code Generation
Taobao & Tmall Group of Alibaba
Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmark
Feb 19, 2026Sourav Panda, Tanmay Ambadkar, Shreyash Kale +2RL Benchmarks Game-Playing Agents
Pennsylvania State University, PA, USA.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23
An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models
Jan 19, 2026Yuhiro Ono, Tomohiro Harada, Yukiya MiuraEvolutionary Optimization Benchmark Design
Graduate School of Systems Design, Tokyo Metropolitan University, 6-6, Asahigaoka, Hino, 1910065, Tokyo, Japan. · Graduate School of Science and Engineering, Saitama University, 2552026 Shimo-okubo, Sakura-ku, Saitama, 3388570, Saitama, Japan.
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Jan 15, 2026Yiwen Gao, Ruochen Zhao, Yang Deng +1Deep Research Agents Benchmark Design
National University of Singapore · Nanyang Technological University · Singapore Management University +1
Large-scale benchmarking of multi-objective soft-computing metaheuristics for redundancy allocation in repairable k-out-of-n systems
Dec 20, 2025Mateusz Oszczypała, David Ibehej, Jakub KudelaBenchmark Design Metaheuristic Optimization
Military University of Technology, gen. Sylwestra Kaliskiego Street 2, 00-908 Warsaw, Poland. · Institute of Automation and Computer Science, Brno University of Technology, Brno, Czech Republic.
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Nov 8, 2025Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh +5Software Engineering Agents Benchmark Design
Harvard University · Google DeepMind · Princeton University +2
MLPerf Automotive
Oct 31, 2025Radoyeh Shojaei, Predrag Djurdjevic, Mostafa El-Khamy +7End-to-End Autonomous Driving Benchmark Design
University of California, Davis · Arm · Samsung +5
The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models
Oct 27, 2025Timo Freiesleben, Sebastian ZezulkaBenchmark Design Benchmark Validity
LMU Munich, MCMP & MCML · University of Tübingen
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Oct 9, 2025Ruiling Xu, Yifan ZhangLLM Evaluation Benchmark Design
University of Illinois Urbana-Champaign
TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing
Oct 5, 2025Jiaming Wang, Jizhuo Chen, Diwen Liu +1Benchmark Design Map-Based Localization
fev-bench: A Realistic Benchmark for Time Series Forecasting
Sep 30, 2025Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5Benchmark Design Time Series Forecasting
AWS · Keystone AI · Prior Labs +2
IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation
Sep 30, 2025Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck +57Mathematical Reasoning Benchmarks LLM Evaluation
WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning
Sep 27, 2025Xin Li, Mengbing Liu, Yiyang Zhu +4Mathematical Reasoning Benchmarks Benchmark Auditing
Nanyang Technological University
PDFBench: A Benchmark for De novo Protein Design from Function
May 25, 2025Jiahao Kuang, Nuowei Liu, Jie Wang +3Benchmark Design Molecular Generation
School of Computer Science and Technology, East China Normal University · Institute of Artificial Intelligence (TeleAI), China Telecom · College of Foreign Languages and Literatures, Fudan University
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3
BiasBench: A reproducible benchmark for tuning the biases of event cameras
Apr 25, 2025Andreas Ziegler, David Joseph, Thomas Gossard +2Event-Based Vision Benchmark Design
Cognitive Systems Group, University of Tübingen, Germany
Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Benchmark Auditing Reasoning Evaluation
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France
BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?
Mar 19, 2025Pierre Chambon, Baptiste Roziere, Benoit Sagot +1Benchmark Design Code Generation
FAIR at Meta · Inria · Work done at Meta, now working at Mistral AI
Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models
Mar 9, 2025Batu Guan, Xiao Wu, Yuanyuan Yuan +1Benchmark Design Data Contamination in Language Models
The Chinese University of Hong Kong · Huazhong University of Science and Technology · ETH Zurich
GraphChase: A Platform and Benchmark for Urban Network Security Games
Jan 29, 2025Shuxin Zhuang, Shuxin Li, Tianji Yang +4Benchmark Design AI Agent Benchmarks
City University of Hong Kong CAIR, Hong Kong Institute of Science & Innovation, CAS Hong Kong, Hong Kong · Nanyang Technological University Singapore, Singapore · Georgia Institute of Technology Atlanta, Georgia, United States +2
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6
What does the model actually see? Evaluation protocols and input availability in data-driven prediction of room acoustic parameters
Date pendingAk\in OktavBenchmark Design Room Acoustics
Vibration and Acoustics Laboratory (VAL) and Department of Mechanical Engineering, Alanya Alaaddin Keykubat University, 07425, Antalya, Türkiye

Apr 28, 2026, cs.CLJ/K move · Enter open · S save

PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators

Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu, Yi-Jyun Sun+3

University of Illinois Urbana-Champaign · VinUniversity

Benchmark Design Clinical Language Model Evaluation User Simulation

Open paper AskPDF arXiv

Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically meaningful diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, and move through therapeutic stages and toward positive valence too quickly. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our benchmark is strongly aligned with judgments of mental health professionals. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible benchmark to guide future simulator design and evaluation.

Momentum

Related topics

Latest papers 285