Benchmark Design

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 65% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 285

All topics

REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation
Jun 17, 2026Tengjie Lin, Yutao Sun, Jingwei Ni +7VLM Evaluation Visual Question Answering
Zhejiang University · ETH Zürich · Nanjing University of Science and Technology +3
MetaboNet-Bench: A Multi-modal Benchmark for Glucose Forecasting in Type 1 Diabetes
Jun 17, 2026Nathaniel Jeffries, Miriam Wolff, Sam Royston +5Multivariate Time Series Forecasting Type 1 Diabetes
Department of Genetics, Stanford University School of Medicine, Stanford, CA, USA · Replica Health, New York, NY, USA · Boston Children’s Hospital, Harvard Medical School, Boston, MA, USA +1
SP-TransientBench: A Real-Captured Single Photon Perception Benchmark
Jun 16, 2026Hongzhou Dong, Zili Zhang, Ziting Wen +12Benchmark Design Depth Estimation
Shanghai University, Shanghai, China · Southern University of Science and Technology, Shenzhen, China · The University of Sydney, Sydney, Australia
Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice
Jun 16, 2026Andrew Lou, David ShinBenchmark Design Language Model Robustness
Yale Law School, USA
Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
Jun 16, 2026Maria I. Gorinova, Macey Baker, Amy Heineike +3AI Coding Agents Benchmark Design
Tessl London, UK
RecourseBench: A Modular Framework for Reproducible Algorithmic Recourse Evaluation
Jun 15, 2026Zahra Khotanlou, Hashir Ahmed, Chenghao Tan +2Benchmark Design Automated Evaluation
University of Waterloo
FinBalance: A Multi-Document Accounting Reconciliation Benchmark
Jun 14, 2026Sasank Tumpati, Devansh Agarwal, Ayush Kedia +6Financial Services LLM Evaluation
BITS Pilani · KIIT Bhubaneswar · University of Oxford
MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis
Jun 11, 2026Lushi Pu, Weiming Zhang, Xinheng Xie +6Mathematical Reasoning Benchmarks Automated Theorem Proving
ModelBest Inc. · Tsinghua University
EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments
Jun 11, 2026Jundong Xu, Qingchuan Li, Jiaying Wu +11Benchmark Design Long-Horizon Agent Evaluation
National University of Singapore · Singapore Management University · University of Washington +5
How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
Jun 11, 2026Chase M. Fensore, Kaustubh Dhole, Jason Fan +2Benchmark Design Synthetic Data Generation
Department of Computer Science Emory University Atlanta, USA
Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends
Jun 10, 2026Jinshan Lai, Jianwei Hu, Baoyang Jiang +7Benchmark Construction Benchmark Design
University of Electronic Science and Technology of China, Chengdu, China · Qiyuan Lab, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China +2
ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics
Jun 9, 2026Shunkai Zhang, Haoran Zhang, Yun Luo +15Mathematical Reasoning Benchmarks Benchmark Design
2Peking University · 1Shanghai AI Laboratory · 3Shanghai Jiao Tong University +2
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
Jun 8, 2026QuocViet Pham, Elvir Karimov, Andrey Galichin +1Mathematical Reasoning Benchmarks LLM Evaluation
Skolkovo Institute of Science and Technology · HSE University · Sberbank +1
ERBench: A Benchmark and Testsuite for Equation Discovery Algorithms
Jun 8, 2026Paul Kahlmeyer, Henrik Voigt, Michael Habeck +1Benchmark Design Scientific Discovery
Department of Computer Science University of Jena Inselplatz 5, 07743 Jena, Germany
RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
Jun 8, 2026Jing Wang, Shang Liu, Wenji Fang +3LLM Evaluation Benchmark Design
Hong Kong University of Science and Technology
PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems
Jun 7, 2026Suraj Ranganath, Anish RaghavendraBenchmark Design Text-to-Cypher
Halıcıoğlu School of Data Science and Computing University of California, San Diego · Independent Researcher
Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies
Jun 5, 2026Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov +8Algorithm Selection Pairwise Comparison
HSE University Moscow, Russian Federation
RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents
Jun 5, 2026Ameya Joshi, Joon Kim, Gus Eggert +5Benchmark Design Document Information Extraction
Extend AI
Decision-Aware Evaluation of Physics-Informed Surrogates
Jun 5, 2026Daniel Cieślak, Andrzej CzyżewskiSurrogate Modeling Benchmark Design
Gdańsk University of Technology, Gdańsk, Poland
ShallowBench: Benchmarking Generative Drug Design Models on Shallow-Pocket Targets
Jun 4, 2026Saket Reddy, Shiwei LiuBenchmark Design Structure-Based Drug Design
University of Illinois - Urbana-Champaign
Benchmark Everything Everywhere All at Once
Jun 4, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5LLM Evaluation Benchmark Design
MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen +3
TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
Jun 4, 2026Bobby Yan, Fredrik KjolstadAI Coding Agents Benchmark Design
Department of Computer Science Stanford University Stanford, CA 94305
GENEB: Why Genomic Models Are Hard to Compare
Jun 3, 2026Daria Ledneva, Mikhail Nuridinov, Denis KuznetsovBenchmark Design Genomics
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · Moscow State Institute of Steel and Alloys, Moscow, Russia
Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models
Jun 3, 2026Julian Skirzynski, Harry Cheon, Shreyas Kadekodi +2Concept Bottleneck Models Benchmark Design
University of California, San Diego
What Are We Actually Benchmarking in Robot Manipulation?
Jun 2, 2026Tianchong Jiang, Xiangshan Tan, Samuel Wheeler +3Benchmark Design Benchmark Contamination
Toyota Technological Institute at Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA · University of Chicago, Chicago, IL, USA
HighTide: An Agent-Curated Open-Source VLSI Benchmark Suite
Jun 2, 2026Benjamin Goldblatt, Paolo Pedroso, Farhad Modaresi +2Benchmark Design Electronic Design Automation
University of California, Santa Cruz
BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents
Jun 2, 2026Alex Wang, Georg Meinhardt, Jacob Katz +4Benchmark Design Financial QA
1Rogo · 2OpenAI · †Work done while at Rogo.
CityTrajBench: A Unified Benchmark for City-Scale Vehicle Trajectory Generation
Jun 1, 2026Shibo Zhu, Xiaodan Shi, Dayin Chen +4Trajectory Generation Benchmark Design
Department of Building Environment and Energy Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Eastern Institute for Advanced Study, Eastern Institute of Technology, Ningbo, China · International Centre of Urban Energy Nexus, The Hong Kong Polytechnic University, Hong Kong SAR, China +5
Graph Instance Landscapes: When Structural Similarity Does (Not) Reflect Shortest-Path Performance
Jun 1, 2026Maryam Gholami Shiri, Ivana Krminac, Marko Djukanović +3Benchmark Design
Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · Faculty of Natural Science and Mathematics, University of Banja Luka, Banja Luka, Bosnia and Herzegovina +3
OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models
May 30, 2026Ziling Lu, Zongsheng Li, Xinke Shen +11Brain-Computer Interfaces Benchmark Design
Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China · School of Computer Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China · Omni-Intelligence, Shenzhen, China +1

Jun 17, 2026, cs.CVJ/K move · Enter open · S save

REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation

Tengjie Lin, Yutao Sun, Jingwei Ni, Shuhan Ge+6

Zhejiang University · ETH Zürich · Nanjing University of Science and Technology+3

VLM Evaluation Visual Question Answering Benchmark Design Benchmark Contamination

Open paper AskPDF arXiv

Static visual question answering (VQA) benchmarks age quickly: Once the items leak into training corpora, scores can reflect memorization rather than genuine visual ability, thus obscuring real progress. Rebuilding high-quality benchmarks such as VBench requires substantial human annotation, yet each static release can quickly become another leaked artifact. We propose ReKey, a live benchmark protocol that randomly regenerates the answer-bearing local detail, or visual key, in real images at evaluation time. Using human-validated edit slots, ReKey samples fresh instances with new answers, construction-grounded labels, and controlled visual-search difficulty. On VBench, the ReKey regenerated benchmark reveals a sharp score jump across eight frontier vision-language models (VLMs): The original items score 9.5--18.8 percentage points higher than the regenerated variants. By making the visual key renewable, ReKey keeps evaluation fresh as models and training data evolve.

Momentum

Related topics

Latest papers 285