Benchmark Validity

Ask about this topic

Momentum

15 papers in the last four weeks, up 114% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 117

All topics
CardsList

  1. On Hyperparameter Tuning on the Test Set

    Oct 5, 2026Matteo Fregonara, Tom Viering, Jan van GemertModel SelectionBenchmark Validity

    Delft University of Technology

  2. What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

    Oct 4, 2026Bowen Xu, Boyu ChenLLM EvaluationBenchmark Validity

    Department of Biology, Stanford University, Stanford, CA, USA · Institute of Health Informatics, University College London, London, UK

  3. SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

    Sep 30, 2026Liang You, Hengyu Shi, Dongwen OuGroup-Conditional Conformal PredictionConformal Prediction

    University of Pittsburgh · Duke University

  4. Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

    Sep 30, 2026Anson Y. Lam, Shuqing Li, Michael R. LyuVLM EvaluationBenchmark Validity

    Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China

  5. The Last Mile Is the File: OfficeEditBench for Preservation-Aware Office Editing

    Sep 28, 2026Zhiwen Wu, Chengxu WuBenchmark ValiditySpreadsheet Automation

    Peking University

  6. Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

    Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Reward HackingBenchmark Auditing

    Scale AI

  7. Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

    Sep 28, 2026Jiapeng LiLLM-as-a-JudgeAI Agent Auditing

  8. TopU-LBVS: A Realistic Multi Target Benchmark for Ligand Based Virtual Screening

    Sep 24, 2026Surbhi Kumar, Yuhe Zhou, Varun Shiralkar +2Benchmark DesignVirtual Screening

    UT Dallas, Mathematical Sciences Richardson, TX, USA

  9. Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

    Sep 22, 2026Jiamiao Liu, Dewen Qiao, Yu Zhang +1Probability CalibrationBenchmark Auditing

    Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China

  10. Auditing Proxy-Based Validation Across Text Spans

    Sep 22, 2026Daein Weon, Dong Ho KangBenchmark ValidityAutomated Evaluation

    Kookmin University · UStechlab

  11. Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

    Sep 16, 2026Navyansh Singh, Animesh Pathak, Aarav SinghArgument MiningComputational Argumentation

    IIIT Naya Raipur

  12. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

    Sep 14, 2026Ali Ansari, Haoran Sun, Andy Zeyi Liu +48LLM EvaluationPhysical Reasoning

  13. Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints

    Sep 12, 2026Junlong Shen Xingyu LiBenchmark AuditingBenchmark Validity

    University of Alberta Edmonton, Canada

  14. Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets

    Sep 12, 2026Jia Huang, Yankai Wan, Yangjun OuBenchmark ValidityData Contamination

    Guanghua School of Management, Peking University · College of Artificial Intelligence, Jilin University · School of Mathematical Sciences, Peking University

  15. What Fixed-Rollout pass@k Evaluations Can Identify

    Sep 10, 2026Pranav Singh, Prashant SinghConfidence Region EstimationPartial Identification

    Department of Mathematics Indian Institute of Technology Ropar Ropar, Punjab 140001, India

  16. SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

    Sep 8, 2026Pujun Zheng, Zixin Shang, Shufan Jiang +5Reward HackingSoftware Engineering Agents

    East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University

  17. Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

    Aug 31, 2026Qiaoyuan Zheng, Yiqu YangLLM EvaluationPairwise Ranking

    ETH Zurich · Zurich, Switzerland

  18. Three Ways Classical Test Theory Can Mislead About LLM Judges

    Aug 31, 2026Louis Yiven ZhuLLM-as-a-JudgePsychometric Validation

    University of Oxford

  19. Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation

    Aug 30, 2026Jhen-Ke LinLLM EvaluationBenchmark Design

    National Yang Ming Chiao Tung University

  20. Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation

    Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM EvaluationLLM Auditing

    School of Statistics and Data Science, Southwestern University of Finance and Economics · School of Statistics and Data Science, Southwestern University of Finance and Economics Chengdu, China

  21. Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation

    Aug 13, 2026Rana Muhammad Ahmed, Sabahat AbbasMCP SecurityLLM Agent Security

    Department of Computer Science, Bahria University, Islamabad, Pakistan

  22. FormStruct-Bench:A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recognition

    Aug 11, 2026Lujie Ban, Jiangtao Zhu, Yuanheng Yu +2Document UnderstandingTable Structure Recognition

    The Chinese University of Hong Kong, Shenzhen · University of Washington

  23. CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation

    Aug 10, 2026Harmanjot Singh, Abhra Dubey, Jorge Alejandro Amador HerreraBenchmark ConstructionParametric CAD Modeling

    Mohamed Bin Zayed University of Artificial Intelligence · Abu Dhabi, UAE

  24. Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints

    Aug 6, 2026Omid Bazgir, Md Nasir, Jacob Hoffman +6Synthetic Benchmark GenerationBenchmark Validity

    Oracle Health and Life Sciences United States

  25. SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

    Aug 5, 2026Sihan Hu, Lyuhan Huang, Youjin Deng +1LLM EvaluationScientific Code Generation

    Hefei National Laboratory College of Mechanical and Electrical Engineering University of Science and Technology of China · Hefei National Laboratory Institute of Theoretical Physics University of Science and Technology of China Chinese Academy of Sciences Hefei 230026, China · Chinese Academy of Sciences Beijing 100190, China

  26. NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap

    Aug 5, 2026Dasol Choi, Joonyong Park, Daegon Yu +3Multilingual Language Model EvaluationBenchmark Validity

    AIM Intelligence · HAERAE LAB · KT Corp. +3

  27. Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks

    Aug 4, 2026Ine Gevers, Walter DaelemansLLM EvaluationReasoning Evaluation

    CLiPS, University of Antwerp

  28. Reassessing the Feasibility of PPG-Based Non-Invasive Blood Glucose Level Estimation

    Aug 3, 2026Supraja Ramesh, Markus Neufeld, Michael Küttner +2Data LeakageOOD Generalization

    Karlsruhe Institute of Technology, Karlsruhe, Germany

  29. UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation

    Aug 2, 2026Binshuang LiConditional Average Treatment Effect EstimationBenchmark Design

    Independent Researcher USA

  30. How Benchmarks Mis-Score Computer-Use Agents

    Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure AnalysisComputer-Use Agent Benchmarks

    Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3

  31. Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

    Jul 30, 2026Youting Wang, Xiao Han, Dingyan Shang +2LLM Safety EvaluationBenchmark Validity

  32. Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

    Jul 29, 2026Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6Benchmark AuditingAI Agent Benchmarks

    Independent · UK AI Security Institute · MARS +4

  33. When benchmark inferences do not compose: Projectibility in AI evaluation

    Jul 28, 2026Brett ReynoldsBenchmark DesignBenchmark Validity

    Humber Polytechnic & University of Toronto

  34. Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

    Jul 27, 2026Jingkun Luo, Da-Tian PengData ProvenanceBenchmark Auditing

    Northwestern Polytechnical University

  35. Charging Phase Health Indicators for Battery State-of-Health Estimation: A Systematic Comparison of CC, CV, and Combined Approaches under Cross-Battery Validation

    Jul 26, 2026Huy Hoang Le, Kim-Anh NguyenBenchmark ValidityLithium-Ion Batteries

    a R&D Department, PowerMore Ltd., Viet Nam · b Faculty of Electrical Engineering, University of Science and Technology, The University of Danang, Viet Nam

  36. What AI Red-Team Evaluations Can and Cannot Prove

    Jul 23, 2026Bandana KaurLLM Safety BenchmarksLLM Red Teaming

    APIsec Research Labs

  37. Good Benchmarks

    Jul 13, 2026Ivan BercovichBenchmark DesignSoftware Engineering Benchmarks

  38. When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

    Jul 9, 2026Zongyou Yang, Yinghan Hou, Xiaokun YangLLM-as-a-JudgeLLM Auditing

    Dyson School of Design Engineering, Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom

  39. From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings

    Jul 8, 2026Shi-Ting Chen, Jinsong ChenText Embedding EvaluationBenchmark Validity

    Faculty of Education, The University of Hong Kong

  40. Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation

    Jul 3, 2026Saeed Mahmoudpour, Mylene C. Q. Farias, Gi-Mun Um +6Pairwise ComparisonBenchmark Validity

    Dept. of Electronics and Informatics, Vrije Universiteit Brussel, Belgium · IMEC, Kapeldreef 75, Leuven, B-3001, Belgium

  41. Meta-Benchmarks for Financial-Services LLM Evaluation

    Jul 2, 2026Blair HudsonFinancial ServicesLLM Evaluation

    Commonwealth Bank of Australia

  42. Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

    Jul 1, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangAI AssuranceBenchmark Auditing

    Northeastern University, USA · University of Illinois Urbana-Champaign, USA · Southern Methodist University, USA

  43. Validating Causal Abstraction Metrics on Simulated Complex Systems

    Jun 30, 2026Maxime Méloux, Tiago Pimentel, François Portet +1Causal AbstractionBenchmark Validity

    Université Grenoble Alpes, CNRS, Grenoble INP, LIG · ETH Zürich

  44. Exploring Differences Between Tabular Enterprise Data and Public Benchmarks

    Jun 29, 2026Myung Jun Kim, Maximilian Schambach, Frank Essenberger +2Benchmark DesignTabular ML

    SAP SE, Germany.

  45. EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

    Jun 29, 2026Buğra Alperen Uluırmak, Rifat KurbanAI AlignmentLLM Safety Alignment

    Erciyes University, Türkiye. · Abdullah Gül University, Türkiye.

  46. Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

    Jun 28, 2026Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella BidermanBenchmark DesignBenchmark Validity

    Eleuther AI · Cambridge

  47. Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

    Jun 28, 2026Lining Hu, Ting Liu, Yuzhuo FuPairwise ComparisonAlgorithmic Auditing

    School of Electronic, Information and Electrical Engineering Shanghai Jiao Tong University Shanghai, China

  48. Building to the Test: Coding Agents Deliver What You Check, Not What You Requested

    Jun 26, 2026Yanuo Ma, Ben Kereopa-Yorke, Ben SchultzReward HackingAI Coding Agents

    Microsoft

  49. Life After Benchmark Saturation: A Case Study of CORE-Bench

    Jun 23, 2026Nitya Nadgir, Sayash Kapoor, Kangheng Liu +11OOD GeneralizationAI Agent Evaluation

    1Independent · 2Princeton University · 3UC Berkeley +1

  50. One Ruler: A Same-Hands Re-Evaluation of Bivariate Causal Direction on Tuebingen, with a Parameter-Free Compression Baseline

    Jun 22, 2026Wietse StienstraCausal DiscoveryBenchmark Validity

    State of the Art — Compression Lab

  51. Computational references are not experiments: pre-registered validation of machine-learned sodium-cathode voltages

    Jun 19, 2026Krishna Teja VepaMaterials Property PredictionVirtual Screening

    Independent Researcher†

  52. The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

    Jun 19, 2026David NordforsLLM EvaluationLLM-as-a-Judge

  53. Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

    Jun 18, 2026Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin +58OOD GeneralizationLLM Agent Evaluation

  54. MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery

    Jun 17, 2026Hongxuan Liu, Roman Bushuiev, Ivy Lightheart +12Benchmark DesignBenchmark Contamination

    1Massachusetts Institute of Technology · Institute of Organic Chemistry and Biochemistry of the Czech Academy of Sciences · 3Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague +2

  55. Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

    Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater ReliabilityLLM Evaluation

    UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708

  56. LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation

    Jun 14, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1LLM EvaluationLLM-as-a-Judge

    Department of Computer Science, Graduate School of Engineering, Chubu University, Kasugai, Aichi 487-8501, Japan · Mitsubishi Heavy Industries, Ltd., Research & Innovation Center, Heat Transfer Research Department, Takasago, Hyogo 676-8686, Japan

  57. Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well

    Jun 10, 2026Célestin Eve, Gaël Varoquaux, Thomas MoreauBenchmark ValidityVariance Reduction

    MIND Team, Université Paris-Saclay, Inria, CEA, Palaiseau, France · SODA Team, Inria, Palaiseau, France · Probabl

  58. ERBench: A Benchmark and Testsuite for Equation Discovery Algorithms

    Jun 8, 2026Paul Kahlmeyer, Henrik Voigt, Michael Habeck +1Benchmark DesignScientific Discovery

    Department of Computer Science University of Jena Inselplatz 5, 07743 Jena, Germany

  59. SC3: The Multi-Solvent Solubility Challenge and Benchmark

    Jun 3, 2026Vansh Ramani, Har Ashish Arora, Dhairya Kuchhal +4Benchmark ConstructionBenchmark Validity

    Indian Institute of Technology Delhi, India · Kurnakov Institute of General and Inorganic Chemistry RAS, Russia

  60. What Are We Actually Benchmarking in Robot Manipulation?

    Jun 2, 2026Tianchong Jiang, Xiangshan Tan, Samuel Wheeler +3Benchmark DesignBenchmark Contamination

    Toyota Technological Institute at Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA · University of Chicago, Chicago, IL, USA