Benchmark Design

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 65% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 285

All topics

Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
May 8, 2026Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula +4Competitive Analysis Multi-Agent Orchestration
1IBM · 2Artificial Intelligence Institute at University of South Carolina · 3Steel Authority of India Limited +2
NeuralBench: A Unifying Framework to Benchmark NeuroAI Models
May 8, 2026Hubert Banville, Stéphane d'Ascoli, Simon Dahan +12Benchmark Design EEG Decoding
Lucy · Brain & AI team, Meta FAIR · MIND, Inria +2
Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare
May 8, 2026Prasanna Desikan, Harshit Rajgarhia, Shivali Dalmia +1Healthcare Benchmark Design
Centific AI Research Redmond, WA, USA
PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism
May 8, 2026Yotam Perlitz, Elad Venezian, Corentin Royer +2Benchmark Design Text-to-SQL
1IBM Research AI · 2ETH Zürich
Have Graph -- Will Lift? The Case for Higher-Order Benchmarks
May 8, 2026Bastian RieckBenchmark Design
AIDOS Lab, University of Fribourg, Switzerland · Institute of AI for Health, Helmholtz Munich, Germany
Benchmarking Fairness in Spiking Neural Networks: Data Bias, Spurious Features, and Hardware Effects
May 8, 2026Hudi He, Fukun Wang, Zhe Wang +7Benchmark Design Algorithmic Fairness
Jilin University · Nanyang Technological University · RMIT University
An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation
May 8, 2026Haoyu Han, Li Ma, Hanbing Wang +9Sequential Recommendation Benchmark Design
Michigan State University · Airbnb, Inc.
GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations
May 8, 2026Jyotika Singh, Fang Tu, Aziza Mirsaidova +11Mathematical Reasoning Benchmarks Benchmark Design
Oracle AI
DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models
May 7, 2026Juntong Wang, Jiarui Wang, Huiyu Duan +3Learning to Rank Benchmark Design
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China
iPhoneBlur: A Difficulty-Stratified Benchmark for Consumer Device Motion Deblurring
May 7, 2026Abdullah Al Shafi, Kazi Saeed AlamImage Deblurring Benchmark Design
Department of Computer Science and Engineering Khulna University of Engineering & Technology, Bangladesh
Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
May 7, 2026Yang Xu, Jiefu Zhang, Haixiang Sun +3LLM Evaluation Selective Inference
Purdue University · Johns Hopkins University · Purdue University, USA
AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting
May 7, 2026Xing Xu, Xu Wang, Yudong Zhang +3Benchmark Design Time Series Forecasting
University of Science and Technology of China (USTC), Hefei, Anhui, China · 2Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, China · 3The Hong Kong Polytechnic University, Hong Kong, China
The First Controllable Bokeh Rendering Challenge at NTIRE 2026
May 6, 2026Tim Seizinger, Florin-Alexandru Vasluianu, Jeffrey Chen +23Benchmark Design Neural Rendering
NTIRE 2026 challenge organizers
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
May 6, 2026Han Wang, Jintao Zhang, Kai Jiang +3GPU Kernel Optimization GPU Acceleration
∗Equal contribution.
Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization
May 6, 2026Noel ThomasCausal Effect Estimation Conditional Average Treatment Effect Estimation
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE
BenCSSmark: Making the Social Sciences Count in LLM Research
May 6, 2026Arnault Chatelain, Étienne Ollion, Qianwen Guan +7LLM Evaluation Benchmark Design
CREST (École Polytechnique, ENSAE, CNRS), 5 avenue Le Chatelier, 91120 Palaiseau, France · LLF (Université Paris Cité and CNRS), UFRL Olympe de Gouges, 13 place Paul Ricoeur, 75013 Paris, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France +1
Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games
May 5, 2026Connacher MurphyGame-Playing Agents Benchmark Design
Stanford University
Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarking
May 5, 2026Susmit Agrawal, Jannis Hollman, Matthias KümmererBenchmark Design Eye Tracking
Tübingen AI Center, University of Tübingen, Germany
ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation
May 5, 2026Cong Liu, Milong Ren, Jiaqi Guan +4Protein Structure Prediction Benchmark Construction
AMLab, AI4Science Lab, University of Amsterdam, Amsterdam, The Netherlands · Work done at ByteDance Seed · ByteDance Seed
LUMINA: A Grid Foundation Model for Benchmarking AC Optimal Power Flow Surrogate Learning
May 4, 2026Hongwei Jin, Keunju Song, Zeeshan Memon +5Neural Surrogate Modeling Benchmark Design
Mathematics and Computer Science Division, Argonne National Laboratory · Department of Electronic Engineering, Sogang University · Department of Computer Science, Emory University +1
RenCon 2025: Revival of the Expressive Performance Rendering Competition
May 3, 2026Huan Zhang, Taegyun Kwon, Anders Friberg +7Benchmark Design
Queen Mary University of London London, UK · Korea Advanced Institute of Science and Technology Daejeon, Korea · KTH Royal Institute of Technology Stockholm, Sweden +5
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
May 3, 2026Mario Koddenbrock, Christoph Lange, Robin Legner +6Benchmark Design Scientific ML
HTW Berlin · TU Berlin · KWS SAAT, Einbeck +5
PHBench: A Benchmark for Predicting Startup Series A Funding from Product Hunt Launch Signals
May 3, 2026Yagiz Ihlamur, Ben Griffin, Rick ChenBenchmark Design Finance
Amazon, Seattle, WA, USA · University of Oxford, Oxford, UK
PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction
May 3, 2026Zhiwen Yang, Pan Liu, Yifan Li +2Benchmark Design Benchmark Contamination
The Hong Kong University of Science and Technology (Guangzhou). · Yangzhou University. · The Hong Kong University of Science and Technology.
StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
May 3, 2026Yongrui Chen, Yangyang Ma, Xiaoying Huang +4LLM Evaluation Benchmark Design
Southeast University · Hangzhou Dianzi University · Zhejiang University +1
VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learning
May 3, 2026Zi-Yi Jia, Zi-Jian Cheng, Xin-Yue Zhang +4Benchmark Design Multimodal Model Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China
ECG-biometrics-bench: A Unified Framework for Reproducible Benchmarking of ECG Biometrics
May 2, 2026Milad ParvanBiometric Identification Benchmark Design
Independent Researcher · Milan, Italy
Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
May 1, 2026Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger +4Mathematical Reasoning Benchmarks LLM Evaluation
ETH Zurich · INSAIT · Sofia University "St. Kliment Ohridski"
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
Apr 28, 2026Leon Kogler, Stefan Hangler, Maximilian Ehrhart +3Automated Software Testing Benchmark Design
CASABLANCA hotelsoftware GmbH Schönwies, Austria · University of Innsbruck Innsbruck, Austria · Technical University of Munich Munich, Germany +1

May 8, 2026, cs.AIJ/K move · Enter open · S save

Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge

Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula, Ling Yue+3

1IBM · 2Artificial Intelligence Institute at University of South Carolina · 3Steel Authority of India Limited+2

Competitive Analysis Multi-Agent Orchestration Benchmark Design AI Agent Evaluation AI Agent Benchmarks Multi-Agent Planning

Open paper AskPDF arXiv

Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design patterns were rewarded. We revisit the CODS 2025 \assetopslive{} challenge, a privacy-aware Codabench competition on industrial multi-agent orchestration built on \assetops{}. We combine final rank sheets, a 300-submission server log, 149-team registrations, best-submission exports, the organizer winners report, the companion \assetopslive{} system paper, and verified planning-track source trees. Five results stand out. First, the public planning leaderboard saturates at 72.73%, and richer prompts do not improve that peak. Second, hidden evaluation changes the story: public and private scores correlate moderately in planning (

r{=}0.69

) but negatively in execution (

r{=}{-}0.13

), with several 45.45% public execution systems reaching 63.64% on the hidden set. Third, the \tmatch{} term is numerically almost inert in the official composite -- combined on a 0--1 scale with 0--100 percentage scores, it contributes at most 0.05 points per track, and rescaling would swap the top two teams. Fourth, the competition is operationally account-based but substantively team-based: 149 registered teams reduce to 24 with non-zero public scores and 11 fully ranked, while 52.3% of deduplicated registrations list multiple usernames. Fifth, successful execution methods mostly improve guardrails -- response selection, contamination cleanup, fallback, and context control -- rather than novel agent architectures. These findings identify which behaviors the evaluation rewarded, and motivate scale-aware composites, skill-level diagnostics, and versioned artifact release.

Momentum

Related topics

Latest papers 285