Benchmark Auditing

Ask about this topic

Momentum

17 papers in the last four weeks, up 325% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 60

All topics
CardsList

  1. Finding Blind Spots in AppWorld and WorkArena Task Verifiers

    Oct 6, 2026Richard AbrichBenchmark AuditingAgent Benchmarks

    OpenAdapt.AI (MLDSAI Inc.)

  2. Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

    Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark AuditingAI Agent Auditing

    Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1

  3. Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

    Oct 5, 2026Hang Xiao, Janet Sung, Zhaoyi Li +2Benchmark AuditingAnomaly Detection

    Fortinet, Inc. Sunnyvale, USA · Google LLC Mountain View, USA · Independent Researcher Mountain View, USA +1

  4. SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output

    Oct 4, 2026Shiyuan Zhou, Ashwin Gerard Colaco, Sainyam Galhotra +1Benchmark AuditingText-to-SQL

    University of California, Irvine, Irvine, California, USA · Cornell University, Ithaca, New York, USA

  5. Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves

    Sep 30, 2026Sohail, Sarkar, Shakuntala BaichooBenchmark AuditingTest-Time Scaling

    Neel · PMCC AI Lab, Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada

  6. Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

    Sep 29, 2026Rohith Reddy Bellibatlu, Zichong Wang, Wenbin ZhangTool-Use EvaluationAutomated Software Testing

    Florida International University, Miami, FL, USA

  7. From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents

    Sep 28, 2026Tianyu Liu, Dingyuan Dai, Yufan Du +1Software Engineering AgentsBenchmark Auditing

    Tsinghua University · UCLA

  8. Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

    Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Reward HackingBenchmark Auditing

    Scale AI

  9. The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

    Sep 27, 2026Farhan Shahriyar Hossain, Taufikur Rahman Fuad, Md Abrar Jahin +1Few-Shot Anomaly DetectionBenchmark Auditing

    Islamic University of Technology · University of Southern California · Qatar Computing Research Institute (QCRI)

  10. Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

    Sep 23, 2026Makar Ulesov, Vladislav Smirnov, Omar Ibrahim +1Benchmark AuditingLLM Auditing

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · School of Computing and Information, University of Pittsburgh

  11. Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

    Sep 22, 2026Imad BuljićGraph Neural NetworksBenchmark Auditing

    Polytechnic Faculty, University of Zenica

  12. Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

    Sep 22, 2026Jiamiao Liu, Dewen Qiao, Yu Zhang +1Probability CalibrationBenchmark Auditing

    Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China

  13. ReSTI: A Source-Grounded Audit and Repair of STI-Bench

    Sep 21, 2026Pengzhan Sun, Ramanathan Rajaraman, Shiu-Hong Kao +2Spatial Reasoning BenchmarksSpatiotemporal Reasoning

    National University of Singapore · University of Science and Technology of China

  14. ETH-TraceBench: A Large-Scale Event-Stream Benchmark for Ethereum DeFi under Temporal, Protocol, and Contract Shift

    Sep 20, 2026Kemal Kirtac, Carsten MapleDistribution Shift RobustnessBenchmark Auditing

    Department of Computer Science, University College London 66–72 Gower Street, London WC1E 6EA, United Kingdom · Department of Computer Science, University of Warwick Coventry CV4 7AL, United Kingdom · WMG, University of Warwick Coventry CV4 7AL, United Kingdom

  15. Auditing Bayesian Graph Alignment: Diagnostic Comparisons and Reference Failure

    Sep 19, 2026Melika Gorgi, Kourosh MirsohiMarkov Chain Monte CarloBenchmark Auditing

    Center for Complex Biological Systems University of California, Irvine Irvine, CA 92697, USA · Department of Computer Science University of California, Irvine Irvine, CA 92697, USA

  16. DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

    Sep 14, 2026Hongye Yang, Zhihao Xie, Shengjun Xiong +1Benchmark AuditingText-to-CAD Generation

    College of Computing Georgia Institute of Technology · Independent Researcher

  17. When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

    Sep 14, 2026Stefan G. Creadore, Peyton WoakzAlgorithmic AuditingBenchmark Auditing

    Tampa, FL, USA

  18. Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints

    Sep 12, 2026Junlong Shen Xingyu LiBenchmark AuditingBenchmark Validity

    University of Alberta Edmonton, Canada

  19. Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

    Sep 10, 2026Koutian Wu, Junjie Zhou, Ergan Shang +6Benchmark AuditingBenchmark Design

    Earth-Space-AI · Tacite AI · Hangzhou Dianzi University +6

  20. IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier

    Sep 9, 2026Blake Stenstrom, Charangan Vasantharajan, Brian SathianathanAI Agent ReliabilityBenchmark Auditing

  21. SoK: Where Do Flow Labels Come From? Auditing Label Provenance in Encrypted Traffic Benchmarks

    Sep 2, 2026Sizhe Huang, Shujie YangBenchmark Auditing

    arXiv:2609.02140v1

  22. Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions

    Aug 31, 2026Ahmed El Kady, Aravind Narayanan, Rehana Noorani +2LLM EvaluationEnergy-Efficient ML

    Vector Institute · Independent researcher · University of Calgary

  23. Auditing MCQA Benchmarks through Probability Landscapes

    Aug 31, 2026Minsoo Song, Chanjun ParkBenchmark AuditingLLM Auditing

    Soongsil University

  24. CatchBench: When Can an Agent Failure Be Caught?

    Aug 24, 2026Yue Zhao, Mengyuan Li, Ruolin Li +5Agent Failure AnalysisBenchmark Auditing

    University of Southern California · Carnegie Mellon University · AWS Agentic AI +3

  25. When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits

    Aug 8, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaStatistical Power AnalysisBenchmark Auditing

    Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

  26. Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons

    Aug 7, 2026Guilin Zhang, Kai ZhaoBenchmark AuditingAutomated Evaluation

    Independent Researcher

  27. On the missing benchmarks layer and a potential solution

    Aug 4, 2026Francis F Daniel, Mauro Ibañez, Francis Perelman +1LLM EvaluationBenchmark Auditing

    1SURUS · 2Independent

  28. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    Aug 3, 2026Shuyang Xie, Shuxiao Xie, Feng Zhu +2Automated Software TestingBenchmark Auditing

    Harbin Institute of Technology · Fudan University · Sun Yat-sen University

  29. FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

    Jul 31, 2026Martin LukkBenchmark AuditingLLM Auditing

    University of Toronto

  30. Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

    Jul 30, 2026Philipp D. Siedler, Jordan SassoonLLM EvaluationBenchmark Auditing

    Aleph Alpha Research, Heidelberg, Germany.

  31. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

    Jul 30, 2026Manyi Wang, Junjielong Xu, Pinjia HeBenchmark AuditingSoftware Engineering Benchmarks

    The Chinese University of Hong Kong, Shenzhen

  32. How Benchmarks Mis-Score Computer-Use Agents

    Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure AnalysisComputer-Use Agent Benchmarks

    Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3

  33. Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

    Jul 29, 2026Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6Benchmark AuditingAI Agent Benchmarks

    Independent · UK AI Security Institute · MARS +4

  34. Visual Credit Audit for Multimodal Spatial Reasoning

    Jul 29, 2026Feixiang Liu, Qiang Qiu, Lanbo Sun +3VLM EvaluationSpatial Reasoning Benchmarks

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  35. Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

    Jul 27, 2026Jingkun Luo, Da-Tian PengData ProvenanceBenchmark Auditing

    Northwestern Polytechnical University

  36. Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI

    Jul 24, 2026Jiaqi Shao, Hanck Chen, Wei Zhang +2Computer-Use Agent BenchmarksReward Hacking

    1Hunyuan Team, Tencent · 2The Hong Kong University of Science and Technology · 3Duke Kunshan University

  37. Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

    Jul 1, 2026Zhi Chen, Zhensu Sun, Yuling Shi +2Benchmark AuditingBenchmark Design

    Singapore Management University · Singapore, Singapore · Shanghai Jiao Tong University +1

  38. Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

    Jul 1, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangAI AssuranceBenchmark Auditing

    Northeastern University, USA · University of Illinois Urbana-Champaign, USA · Southern Methodist University, USA

  39. Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

    Jun 30, 2026Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic +1Benchmark AuditingOOD Generalization

    Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

  40. Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

    Jun 28, 2026Lining Hu, Ting Liu, Yuzhuo FuPairwise ComparisonAlgorithmic Auditing

    School of Electronic, Information and Electrical Engineering Shanghai Jiao Tong University Shanghai, China

  41. BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

    Jun 21, 2026Shreyas KCMultilingual Language Model EvaluationLLM-as-a-Judge

  42. Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations

    Jun 15, 2026Yanan LongLLM EvaluationBenchmark Auditing

    StickFlux Labs

  43. The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

    Jun 2, 2026Wojciech Zarzecki, Jan Dubiński, Sebastian CygertBenchmark AuditingLLM Auditing

    NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland · Gdańsk University of Technology, Gdańsk, Poland

  44. Auditing LLM Benchmarks with Item Response Theory

    May 28, 2026Sander Land, Daniel M. BikelBenchmark AuditingLLM Auditing

    Writer, Inc.

  45. FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks

    May 27, 2026Nishal Thomas, Noel ThomasMathematical Reasoning BenchmarksBenchmark Auditing

    Independent Researcher · Mohamed Bin Zayed University of Artificial Intelligence

  46. Automated Benchmark Auditing for AI Agents and Large Language Models

    May 25, 2026Junlin Wang, Federico Bianchi, Shang Zhu +4LLM EvaluationBenchmark Auditing

    Duke University · Together AI · Stanford University

  47. Deployment-complete benchmarking

    May 25, 2026El Mustapha Mansouri, Keigo AraiBenchmark AuditingBenchmark Construction

    School of Engineering, Institute of Science Tokyo, Yokohama, Kanagawa 226-8501, Japan

  48. Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit

    May 25, 2026Zexin Zhuang, Yanhang Li, Zhichao FanLLM QuantizationStatistical Power Analysis

    1Southern Methodist University · 2Northeastern University · University of Illinois Urbana-Champaign

  49. Metadata Predictability Is Not Evidence Dependence: An Intervention-Based Audit for Weak-Label Benchmarks

    May 22, 2026Kan ShaoCounterfactual EvaluationBenchmark Auditing

    Jinglue Technology Development (Nanjing) Co., Ltd., Nanjing, China

  50. What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

    May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark AuditingLLM Auditing

    BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2

  51. AgentAtlas: Beyond Outcome Leaderboards for LLM Agents

    May 19, 2026Parsa Mazaheri, Kasra MazaheriAgent Failure AnalysisComputer-Use Agent Benchmarks

    University of California, Santa Cruz · Massachusetts Institute of Technology

  52. Are Sparse Autoencoder Benchmarks Reliable?

    May 18, 2026David ChaninBenchmark AuditingLLM Interpretability

    Decode Research, MATS, UCL

  53. Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

    May 14, 2026Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray +1LLM Safety BenchmarksBenchmark Auditing

    Palo Alto Networks

  54. Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

    May 14, 2026Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8Benchmark AuditingImage Retrieval

    Politecnico di Bari

  55. The Evaluation Trap: Benchmark Design as Theoretical Commitment

    May 13, 2026Theodore J KalaitzidisBenchmark AuditingBenchmark Design

    Brown University

  56. Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

    May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent BenchmarksReward Hacking

    UC Berkeley

  57. BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

    Apr 27, 2026Xinming Tu, Tianze Wang, Yingzhou +4Benchmark AuditingAI Agent Benchmarks

    Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1

  58. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

    Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark AuditingAI Agent Evaluation

    Anthropic Fellows Program · Anthropic

  59. WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning

    Sep 27, 2025Xin Li, Mengbing Liu, Yiyang Zhu +4Mathematical Reasoning BenchmarksBenchmark Auditing

    Nanyang Technological University

  60. Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

    Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Benchmark AuditingReasoning Evaluation

    1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France