Benchmark Design

Ask about this topic

Momentum

28 papers in the last four weeks, up 65% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 285

All topics
CardsList

  1. CANDO: Cooperative Agentic Network for Layout Design Optimization

    Oct 7, 2026Athanasios Masouris, Zheng Jing, Benjamin Sam Chandler +1Benchmark DesignCombinatorial Optimization

    Shell Information Technology International · Shell China Limited · Delft University of Technology (TU Delft)

  2. The AI Evaluation Ecosystem

    Oct 7, 2026Yash Dave, Sang T. Truong, Serena Wang +1AI Agent EvaluationBenchmark Design

    Stanford University · University of British Columbia

  3. Do Higher-Order Models Win for Higher-Order Reasons? Rethinking Performance Gains in Hypergraph Learning

    Oct 6, 2026Fanchen Bu, Fan Li, Geon Lee +4Benchmark DesignHypergraph Neural Networks

    KAIST · UNSW Sydney · University of Oxford

  4. TabJoinBench: A Benchmark for Joinable Table Discovery

    Sep 30, 2026Sandipan De, Jin Wang, Vivek GuptaBenchmark Design

    Arizona State University

  5. PhysicsMate: A Curriculum-Grounded Bengali Benchmark for Secondary Physics QA with Small-Model Adaptation

    Sep 30, 2026Rashid Azraf Jahin, Saadman Sajid, Khan Raiyan Ibne Reza +1Benchmark DesignScientific QA

    Dept. of Electrical and Computer Engineering North South University Dhaka, Bangladesh

  6. cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

    Sep 30, 2026Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck +3Computer-Use Agent BenchmarksBenchmark Design

    Carnegie Mellon University

  7. A 3GPP-Compliant Benchmark Dataset for RIS-Aided Beyond 5G Networks

    Sep 30, 2026Pujitha Mamillapalli, Pankaj Singh Rathour, Abhinav KumarReconfigurable Intelligent SurfacesBenchmark Design

    Department of Artificial Intelligence Indian Institute of Technology Hyderabad, India · Department of Electrical Engineering Indian Institute of Technology Hyderabad, India

  8. PDE-OBS: Controlled Evaluation Across Observation Patterns

    Sep 29, 2026Ruichen Xu, Siyao Wang, Fang Wan +7Benchmark DesignPDE Surrogate Modeling

    Stony Brook University · University of California, Davis · Independent Research +3

  9. GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation

    Sep 28, 2026Yuchen Sun, Jinjin He, Sinan Wang +1GPU Kernel OptimizationBenchmark Design

    Georgia Institute of Technology

  10. AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

    Sep 28, 2026Ignacio Iacobacci, Faroq Altam, Zhaozhi Qian +1LLM EvaluationArabic NLP

    Elm Company

  11. SymbolicArena: A Unified Infrastructure for Benchmark Distillation and Dynamic Evaluation in Symbolic Regression

    Sep 28, 2026Ziwen Zhang, Xiju Wu, Yuheng Jing +9Benchmark DesignAutomated Evaluation

    School of Artificial Intelligence, University of Chinese Academy of Sciences · C2DL, Institute of Automation, Chinese Academy of Sciences · Nanjing University of Science and Technology +1

  12. MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

    Sep 27, 2026Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu +3LLM EvaluationBenchmark Design

    National Taiwan University · NTU AI-Core · Queen Mary University of London

  13. Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks

    Sep 27, 2026Analog Design Bench TeamBenchmark DesignElectronic Design Automation

  14. Artificial Societies Benchmark: A Validation Framework for Synthetic Research

    Sep 24, 2026Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis +1Benchmark DesignComputational Social Science

    Artificial Societies University of Oxford, UK

  15. TopU-LBVS: A Realistic Multi Target Benchmark for Ligand Based Virtual Screening

    Sep 24, 2026Surbhi Kumar, Yuhe Zhou, Varun Shiralkar +2Benchmark DesignVirtual Screening

    UT Dallas, Mathematical Sciences Richardson, TX, USA

  16. Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation

    Sep 24, 2026Yezhou Cheng, Runjia Du, Zeming Liu +5Confidence Region EstimationUncertainty Quantification

    Independent · Northwestern University · Pinterest, Inc.

  17. Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure

    Sep 23, 2026Divyansh SinghCausal Effect EstimationBenchmark Design

    University of Florida

  18. WPBench: A Comprehensive Benchmark for Wind Power Forecasting

    Sep 21, 2026Yuhan Zhu, Jilin Hu, Xinying Cai +7Multivariate Time Series ForecastingBenchmark Design

    School of Data Science & Engineering, East China Normal University, Shanghai, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Hangzhou Hikvision Digital Technology Co., Ltd., Hangzhou, China

  19. VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

    Sep 20, 2026Liyang Fan, Yingcheng Shi, Yongbin Li +7Benchmark DesignSoftware Engineering Benchmarks

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · SUAT · Alibaba Token Hub, Alibaba Group +1

  20. Are Coreset Selection Methods Worth Their Cost?

    Sep 19, 2026Yangze Liu, Zhongyi HanBenchmark DesignTraining Data Selection

    Shandong University

  21. GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

    Sep 18, 2026Xinyu Che, Yunfei Ge, Shihao Li +9AI Coding AgentsBenchmark Design

    Nanjing University

  22. DeltaSelect: Affordable A/B Testing for Coding Agents

    Sep 17, 2026Nicholas J. ConnAI Coding AgentsBenchmark Design

    Conn Castle Studios

  23. TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

    Sep 16, 2026Giorgia Adorni, Michela Papandrea, Battista Rimoldi +1Text-to-Music GenerationMusic Generation Evaluation

    Institute of Information Systems and Networking (ISIN), SUPSI, Lugano, Switzerland · t2b AG, Basel, Switzerland

  24. Behavior2Value: Benchmarking and Empowering LLMs for Consumer Value Measurement from E-commerce Behaviors

    Sep 16, 2026Peixuan Hou, Bin Chen, Li He +4Benchmark DesignUser Preference Modeling

    State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Alibaba Group

  25. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

    Sep 16, 2026Li ChenGPU Kernel OptimizationBenchmark Design

  26. KnowBench: Effort Reduction as a Unified, Deployment-Grounded Benchmark for Clinical AI

    Sep 14, 2026Jocelyn Kang, Caroline ZhangBenchmark DesignWorkflow Automation

    Knowtex Inc.

  27. HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications

    Sep 14, 2026Inesh Chakrabarti, Zejun Xiong, Pragya Sharma +1Cyber-Physical SystemsEfficient Neural Network Inference

    University of California, Los Angeles Los Angeles, California, USA

  28. IBBench-Light: A Paired Evaluation of Task-Conditioned Responses to External Directives

    Sep 12, 2026Kainan Zhou, Zhaoyi Li, Janet Sung +2LLM EvaluationBenchmark Design

    Google LLC Mountain View, CA, USA

  29. Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens

    Sep 12, 2026Carl Edwards, Edward De Brouwer, Xiner Li +5Benchmark DesignAdaptive Sampling

    Genentech, South San Francisco, CA, USA

  30. Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

    Sep 10, 2026Koutian Wu, Junjie Zhou, Ergan Shang +6Benchmark AuditingBenchmark Design

    Earth-Space-AI · Tacite AI · Hangzhou Dianzi University +6

  31. IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

    Sep 9, 2026Yiling Ma, Yilun Zhao, Sihong Wu +2Benchmark DesignClarification Question Generation

    Yale University · TCS Research

  32. PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us

    Sep 5, 2026Gal Sapir, Alon Diament, Adva Wolf +7LLM EvaluationBenchmark Design

    Pheno.AI, Tel Aviv, Israel · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel · Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel +1

  33. RealCADBench: Benchmarking Parametric CAD Modeling from Industrial Design Intents

    Sep 3, 2026JoyIndustrial VisCAD Team, Linxin Cai, Qiuhe Hong +10Benchmark DesignParametric CAD Modeling

    JoyIndustrial-VisCAD Team

  34. ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction

    Sep 2, 2026Ke Zhang, Yankang Liu, Roya Zandi +1Tool-Use EvaluationBenchmark Construction

    University of California, Riverside

  35. What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal

    Sep 1, 2026Radin Shayanfar, Keheliya Gallaba, Ahmed E. HassanSoftware EngineeringBenchmark Design

    Queen’s University

  36. StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?

    Sep 1, 2026Yinghao Chen, Zixi Chen, Bingxiang He +7Benchmark DesignLanguage Model Self-Improvement

    Tsinghua University · Zhejiang University

  37. Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

    Aug 31, 2026Junyi Yao, Baichuan Li, Zihao Zheng +1Benchmark DesignLLM Agent Evaluation

    Washington University in St. Louis, USA · Southern Methodist University, USA

  38. Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation

    Aug 30, 2026Jhen-Ke LinLLM EvaluationBenchmark Design

    National Yang Ming Chiao Tung University

  39. A Browser-Native Digital Test Range for Benchmarking 4D Ocean-Glider Planning Algorithms

    Aug 13, 2026Edward Holmberg, Elias Ioup, Mahdi AbdelguerfiBenchmark DesignUnderwater Robotics

    Gulf States Center for Environmental Informatics · LSU New Orleans, LA, USA · Ocean Sciences Division +2

  40. Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks

    Aug 13, 2026Wojciech Zarzecki, Jarosław ArabasEvolutionary OptimizationBenchmark Design

    Warsaw University of Technology, Institute of Computer Science

  41. LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation

    Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM EvaluationBenchmark Design

    1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

  42. CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility

    Aug 13, 2026Akanta Das, Al Amin Farhad, Mrinmoy Sarkar Anto +3Benchmark DesignSynthetic Data Generation

    Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · School of Medicine, Stanford University, Stanford, California, United States

  43. DiG-bench: Discovery in Games

    Aug 12, 2026Ruairidh M. Battleday, Kai Sandbrink, Jimi Cullen-Drohan +13Benchmark DesignAI Agent Evaluation

    1Thinking About Thinking · 2Independent · University of Oxford +5

  44. When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide

    Aug 12, 2026Binshuang LiInverse Probability WeightingCausal Effect Estimation

    Independent Researcher

  45. NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

    Aug 12, 2026Jiarui Ma, Jianghan Wang, Yuheng Ma +2LLM EvaluationBenchmark Design

    Southern University of Science and Technology School of Microelectronics Shenzhen, China

  46. How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models

    Aug 12, 2026Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics +4Protein Structure PredictionInference-Time Optimization

    InstaDeep Ltd · University of Oxford · Lancaster University

  47. Large-scale AI-Ready Data for Anti-Cancer Drug Response Modeling

    Aug 11, 2026Vincent Lavelle, Yitan Zhu, Kaitlyn Marlor +2Benchmark DesignOOD Generalization

    Division of Data Science and Learning, Argonne National Laboratory · Computing, Environment, and Life Sciences, Argonne National Laboratory · Department of Computer Science, University of Chicago

  48. ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons

    Aug 10, 2026Adrian Li, Kelong Mao, Yudong Guo +7Benchmark DesignLLM Agent Evaluation

  49. Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure

    Aug 9, 2026Víctor GallegoBenchmark DesignLarge Language Model-Guided Optimization

    Komorebi AI Technologies

  50. IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

    Aug 5, 2026Shahd Gaben, Heba Sbahi, Samer Rashwani +5LLM EvaluationBenchmark Design

    Hamad Bin Khalifa University, Doha, Qatar. · Nazarbayev University, Astana, Kazakhstan.

  51. Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think

    Aug 5, 2026Simon Klüttermann, Jérôme Rutinowski, Frederik Polachowski +1Benchmark DesignML Reproducibility

    Carnegie Mellon University, Pittsburgh, USA · TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +2

  52. OmniRouting: A Semantic-Coupled Multimodal Benchmark for Constraint-Aware Spatial Reasoning in PCB Routing

    Aug 5, 2026Taiting Lu, Kaiyuan Lin, Ziwei Dong +18Benchmark DesignElectronic Design Automation

    1Pennsylvania State University · 2Independent Researcher · 3Shanghai Jiao Tong University +3

  53. Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

    Aug 4, 2026William Bolton, Philip TorrAI Agents for Scientific DiscoveryBenchmark Design

    Department of Engineering Science, University of Oxford, Oxford, United Kingdom.

  54. Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

    Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4AI for ScienceBenchmark Design

    Westlake University · Zhejiang University · Nanjing University of Science and Technology +1

  55. Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

    Aug 4, 2026Mohsen ArjmandiLong-Context Language Model EvaluationBenchmark Design

    evolutionID GmbH

  56. FinVerse: Financial Time-Series Benchmark

    Aug 4, 2026Jaehoon Lee, Jun Seo, Seunghan Lee +9Benchmark DesignFinancial Forecasting

    LG AI Research

  57. On the missing benchmarks layer and a potential solution

    Aug 4, 2026Francis F Daniel, Mauro Ibañez, Francis Perelman +1LLM EvaluationBenchmark Auditing

    1SURUS · 2Independent

  58. onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

    Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety BenchmarksLLM Evaluation

    Onepot AI, Inc.

  59. From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

    Aug 3, 2026Can Wang, Haoran Chen, Haowen Gao +3LLM EvaluationBenchmark Design

    Shandong Key Laboratory of Digital Service Computing Technology and Systems · Alibaba Token Hub · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences +1

  60. How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

    Aug 2, 2026Lorenzo Guerra, Thomas Chapuis, Guillaume Duc +2Data ProvenanceBenchmark Design

    LTCI, T´el´ecom Paris, Institut Polytechnique de Paris · Ampere Software Technology