Benchmark Design

Ask about this topic

Momentum

28 papers in the last four weeks, up 65% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 285

All topics
CardsList

  1. IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

    Sep 9, 2026Yiling Ma, Yilun Zhao, Sihong Wu +2Benchmark DesignClarification Question Generation

    Yale University · TCS Research

  2. PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us

    Sep 5, 2026Gal Sapir, Alon Diament, Adva Wolf +7LLM EvaluationBenchmark Design

    Pheno.AI, Tel Aviv, Israel · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel · Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel +1

  3. RealCADBench: Benchmarking Parametric CAD Modeling from Industrial Design Intents

    Sep 3, 2026JoyIndustrial VisCAD Team, Linxin Cai, Qiuhe Hong +10Benchmark DesignParametric CAD Modeling

    JoyIndustrial-VisCAD Team

  4. ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction

    Sep 2, 2026Ke Zhang, Yankang Liu, Roya Zandi +1Tool-Use EvaluationBenchmark Construction

    University of California, Riverside

  5. What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal

    Sep 1, 2026Radin Shayanfar, Keheliya Gallaba, Ahmed E. HassanSoftware EngineeringBenchmark Design

    Queen’s University

  6. StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?

    Sep 1, 2026Yinghao Chen, Zixi Chen, Bingxiang He +7Benchmark DesignLanguage Model Self-Improvement

    Tsinghua University · Zhejiang University

  7. Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

    Aug 31, 2026Junyi Yao, Baichuan Li, Zihao Zheng +1Benchmark DesignLLM Agent Evaluation

    Washington University in St. Louis, USA · Southern Methodist University, USA

  8. Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation

    Aug 30, 2026Jhen-Ke LinLLM EvaluationBenchmark Design

    National Yang Ming Chiao Tung University

  9. A Browser-Native Digital Test Range for Benchmarking 4D Ocean-Glider Planning Algorithms

    Aug 13, 2026Edward Holmberg, Elias Ioup, Mahdi AbdelguerfiBenchmark DesignUnderwater Robotics

    Gulf States Center for Environmental Informatics · LSU New Orleans, LA, USA · Ocean Sciences Division +2

  10. Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks

    Aug 13, 2026Wojciech Zarzecki, Jarosław ArabasEvolutionary OptimizationBenchmark Design

    Warsaw University of Technology, Institute of Computer Science

  11. LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation

    Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM EvaluationBenchmark Design

    1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

  12. CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility

    Aug 13, 2026Akanta Das, Al Amin Farhad, Mrinmoy Sarkar Anto +3Benchmark DesignSynthetic Data Generation

    Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · School of Medicine, Stanford University, Stanford, California, United States

  13. DiG-bench: Discovery in Games

    Aug 12, 2026Ruairidh M. Battleday, Kai Sandbrink, Jimi Cullen-Drohan +13Benchmark DesignAI Agent Evaluation

    1Thinking About Thinking · 2Independent · University of Oxford +5

  14. When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide

    Aug 12, 2026Binshuang LiInverse Probability WeightingCausal Effect Estimation

    Independent Researcher

  15. NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

    Aug 12, 2026Jiarui Ma, Jianghan Wang, Yuheng Ma +2LLM EvaluationBenchmark Design

    Southern University of Science and Technology School of Microelectronics Shenzhen, China

  16. How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models

    Aug 12, 2026Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics +4Protein Structure PredictionInference-Time Optimization

    InstaDeep Ltd · University of Oxford · Lancaster University

  17. Large-scale AI-Ready Data for Anti-Cancer Drug Response Modeling

    Aug 11, 2026Vincent Lavelle, Yitan Zhu, Kaitlyn Marlor +2Benchmark DesignOOD Generalization

    Division of Data Science and Learning, Argonne National Laboratory · Computing, Environment, and Life Sciences, Argonne National Laboratory · Department of Computer Science, University of Chicago

  18. ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons

    Aug 10, 2026Adrian Li, Kelong Mao, Yudong Guo +7Benchmark DesignLLM Agent Evaluation

  19. Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure

    Aug 9, 2026Víctor GallegoBenchmark DesignLarge Language Model-Guided Optimization

    Komorebi AI Technologies

  20. IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

    Aug 5, 2026Shahd Gaben, Heba Sbahi, Samer Rashwani +5LLM EvaluationBenchmark Design

    Hamad Bin Khalifa University, Doha, Qatar. · Nazarbayev University, Astana, Kazakhstan.

  21. Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think

    Aug 5, 2026Simon Klüttermann, Jérôme Rutinowski, Frederik Polachowski +1Benchmark DesignML Reproducibility

    Carnegie Mellon University, Pittsburgh, USA · TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +2

  22. OmniRouting: A Semantic-Coupled Multimodal Benchmark for Constraint-Aware Spatial Reasoning in PCB Routing

    Aug 5, 2026Taiting Lu, Kaiyuan Lin, Ziwei Dong +18Benchmark DesignElectronic Design Automation

    1Pennsylvania State University · 2Independent Researcher · 3Shanghai Jiao Tong University +3

  23. Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

    Aug 4, 2026William Bolton, Philip TorrAI Agents for Scientific DiscoveryBenchmark Design

    Department of Engineering Science, University of Oxford, Oxford, United Kingdom.

  24. Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

    Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4AI for ScienceBenchmark Design

    Westlake University · Zhejiang University · Nanjing University of Science and Technology +1

  25. Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

    Aug 4, 2026Mohsen ArjmandiLong-Context Language Model EvaluationBenchmark Design

    evolutionID GmbH

  26. FinVerse: Financial Time-Series Benchmark

    Aug 4, 2026Jaehoon Lee, Jun Seo, Seunghan Lee +9Benchmark DesignFinancial Forecasting

    LG AI Research

  27. On the missing benchmarks layer and a potential solution

    Aug 4, 2026Francis F Daniel, Mauro Ibañez, Francis Perelman +1LLM EvaluationBenchmark Auditing

    1SURUS · 2Independent

  28. onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

    Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety BenchmarksLLM Evaluation

    Onepot AI, Inc.

  29. From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

    Aug 3, 2026Can Wang, Haoran Chen, Haowen Gao +3LLM EvaluationBenchmark Design

    Shandong Key Laboratory of Digital Service Computing Technology and Systems · Alibaba Token Hub · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences +1

  30. How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

    Aug 2, 2026Lorenzo Guerra, Thomas Chapuis, Guillaume Duc +2Data ProvenanceBenchmark Design

    LTCI, T´el´ecom Paris, Institut Polytechnique de Paris · Ampere Software Technology