Benchmark Auditing

Ask about this topic

Momentum

17 papers in the last four weeks, up 325% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 61

All topics
CardsList

  1. Finding Blind Spots in AppWorld and WorkArena Task Verifiers

    Oct 6, 2026Richard AbrichBenchmark AuditingAgent Benchmarks

    OpenAdapt.AI (MLDSAI Inc.)

  2. Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

    Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark AuditingAI Agent Auditing

    Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1

  3. Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

    Oct 5, 2026Hang Xiao, Janet Sung, Zhaoyi Li +2Benchmark AuditingAnomaly Detection

    Fortinet, Inc. Sunnyvale, USA · Google LLC Mountain View, USA · Independent Researcher Mountain View, USA +1

  4. SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output

    Oct 4, 2026Shiyuan Zhou, Ashwin Gerard Colaco, Sainyam Galhotra +1Benchmark AuditingText-to-SQL

    University of California, Irvine, Irvine, California, USA · Cornell University, Ithaca, New York, USA

  5. Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves

    Sep 30, 2026Sohail, Sarkar, Shakuntala BaichooBenchmark AuditingTest-Time Scaling

    Neel · PMCC AI Lab, Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada

  6. Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

    Sep 29, 2026Rohith Reddy Bellibatlu, Zichong Wang, Wenbin ZhangTool-Use EvaluationAutomated Software Testing

    Florida International University, Miami, FL, USA

  7. From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents

    Sep 28, 2026Tianyu Liu, Dingyuan Dai, Yufan Du +1Software Engineering AgentsBenchmark Auditing

    Tsinghua University · UCLA

  8. Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

    Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Reward HackingBenchmark Auditing

    Scale AI

  9. The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

    Sep 27, 2026Farhan Shahriyar Hossain, Taufikur Rahman Fuad, Md Abrar Jahin +1Few-Shot Anomaly DetectionBenchmark Auditing

    Islamic University of Technology · University of Southern California · Qatar Computing Research Institute (QCRI)

  10. Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

    Sep 23, 2026Makar Ulesov, Vladislav Smirnov, Omar Ibrahim +1Benchmark AuditingLLM Auditing

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · School of Computing and Information, University of Pittsburgh

  11. Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

    Sep 22, 2026Imad BuljićGraph Neural NetworksBenchmark Auditing

    Polytechnic Faculty, University of Zenica

  12. Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

    Sep 22, 2026Jiamiao Liu, Dewen Qiao, Yu Zhang +1Probability CalibrationBenchmark Auditing

    Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China

  13. ReSTI: A Source-Grounded Audit and Repair of STI-Bench

    Sep 21, 2026Pengzhan Sun, Ramanathan Rajaraman, Shiu-Hong Kao +2Spatial Reasoning BenchmarksSpatiotemporal Reasoning

    National University of Singapore · University of Science and Technology of China

  14. ETH-TraceBench: A Large-Scale Event-Stream Benchmark for Ethereum DeFi under Temporal, Protocol, and Contract Shift

    Sep 20, 2026Kemal Kirtac, Carsten MapleDistribution Shift RobustnessBenchmark Auditing

    Department of Computer Science, University College London 66–72 Gower Street, London WC1E 6EA, United Kingdom · Department of Computer Science, University of Warwick Coventry CV4 7AL, United Kingdom · WMG, University of Warwick Coventry CV4 7AL, United Kingdom

  15. Auditing Bayesian Graph Alignment: Diagnostic Comparisons and Reference Failure

    Sep 19, 2026Melika Gorgi, Kourosh MirsohiMarkov Chain Monte CarloBenchmark Auditing

    Center for Complex Biological Systems University of California, Irvine Irvine, CA 92697, USA · Department of Computer Science University of California, Irvine Irvine, CA 92697, USA

  16. DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

    Sep 14, 2026Hongye Yang, Zhihao Xie, Shengjun Xiong +1Benchmark AuditingText-to-CAD Generation

    College of Computing Georgia Institute of Technology · Independent Researcher

  17. When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

    Sep 14, 2026Stefan G. Creadore, Peyton WoakzAlgorithmic AuditingBenchmark Auditing

    Tampa, FL, USA

  18. Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints

    Sep 12, 2026Junlong Shen Xingyu LiBenchmark AuditingBenchmark Validity

    University of Alberta Edmonton, Canada

  19. Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

    Sep 10, 2026Koutian Wu, Junjie Zhou, Ergan Shang +6Benchmark AuditingBenchmark Design

    Earth-Space-AI · Tacite AI · Hangzhou Dianzi University +6

  20. IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier

    Sep 9, 2026Blake Stenstrom, Charangan Vasantharajan, Brian SathianathanAI Agent ReliabilityBenchmark Auditing

  21. SoK: Where Do Flow Labels Come From? Auditing Label Provenance in Encrypted Traffic Benchmarks

    Sep 2, 2026Sizhe Huang, Shujie YangBenchmark Auditing

    arXiv:2609.02140v1

  22. Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions

    Aug 31, 2026Ahmed El Kady, Aravind Narayanan, Rehana Noorani +2LLM EvaluationEnergy-Efficient ML

    Vector Institute · Independent researcher · University of Calgary

  23. Auditing MCQA Benchmarks through Probability Landscapes

    Aug 31, 2026Minsoo Song, Chanjun ParkBenchmark AuditingLLM Auditing

    Soongsil University

  24. CatchBench: When Can an Agent Failure Be Caught?

    Aug 24, 2026Yue Zhao, Mengyuan Li, Ruolin Li +5Agent Failure AnalysisBenchmark Auditing

    University of Southern California · Carnegie Mellon University · AWS Agentic AI +3

  25. When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits

    Aug 8, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaStatistical Power AnalysisBenchmark Auditing

    Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

  26. Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons

    Aug 7, 2026Guilin Zhang, Kai ZhaoBenchmark AuditingAutomated Evaluation

    Independent Researcher

  27. On the missing benchmarks layer and a potential solution

    Aug 4, 2026Francis F Daniel, Mauro Ibañez, Francis Perelman +1LLM EvaluationBenchmark Auditing

    1SURUS · 2Independent

  28. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    Aug 3, 2026Shuyang Xie, Shuxiao Xie, Feng Zhu +2Automated Software TestingBenchmark Auditing

    Harbin Institute of Technology · Fudan University · Sun Yat-sen University

  29. FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

    Jul 31, 2026Martin LukkBenchmark AuditingLLM Auditing

    University of Toronto

  30. Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

    Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM EvaluationBenchmark Auditing

    Aleph Alpha Research, Heidelberg, Germany.