Benchmark Auditing

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 325% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 61

All topics

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
Jul 30, 2026Manyi Wang, Junjielong Xu, Pinjia HeBenchmark Auditing Software Engineering Benchmarks
The Chinese University of Hong Kong, Shenzhen
How Benchmarks Mis-Score Computer-Use Agents
Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure Analysis Computer-Use Agent Benchmarks
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks
Jul 29, 2026Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6Benchmark Auditing AI Agent Benchmarks
Independent · UK AI Security Institute · MARS +4
Visual Credit Audit for Multimodal Spatial Reasoning
Jul 29, 2026Feixiang Liu, Qiang Qiu, Lanbo Sun +3VLM Evaluation Spatial Reasoning Benchmarks
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation
Jul 27, 2026Jingkun Luo, Da-Tian PengData Provenance Benchmark Auditing
Northwestern Polytechnical University
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
Jul 24, 2026Jiaqi Shao, Hanck Chen, Wei Zhang +2Computer-Use Agent Benchmarks Reward Hacking
1Hunyuan Team, Tencent · 2The Hong Kong University of Science and Technology · 3Duke Kunshan University
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Jul 1, 2026Zhi Chen, Zhensu Sun, Yuling Shi +2Benchmark Auditing Benchmark Design
Singapore Management University · Singapore, Singapore · Shanghai Jiao Tong University +1
Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits
Jul 1, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangAI Assurance Benchmark Auditing
Northeastern University, USA · University of Illinois Urbana-Champaign, USA · Southern Methodist University, USA
Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit
Jun 30, 2026Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic +1Benchmark Auditing OOD Generalization
Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks
Jun 28, 2026Lining Hu, Ting Liu, Yuzhuo FuPairwise Comparison Algorithmic Auditing
School of Electronic, Information and Electrical Engineering Shanghai Jiao Tong University Shanghai, China
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
Jun 15, 2026Yanan LongLLM Evaluation Benchmark Auditing
StickFlux Labs
The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection
Jun 2, 2026Wojciech Zarzecki, Jan Dubiński, Sebastian CygertBenchmark Auditing LLM Auditing
NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland · Gdańsk University of Technology, Gdańsk, Poland
Auditing LLM Benchmarks with Item Response Theory
May 28, 2026Sander Land, Daniel M. BikelBenchmark Auditing LLM Auditing
Writer, Inc.
FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
May 27, 2026Nishal Thomas, Noel ThomasMathematical Reasoning Benchmarks Benchmark Auditing
Independent Researcher · Mohamed Bin Zayed University of Artificial Intelligence
Automated Benchmark Auditing for AI Agents and Large Language Models
May 25, 2026Junlin Wang, Federico Bianchi, Shang Zhu +4LLM Evaluation Benchmark Auditing
Duke University · Together AI · Stanford University
Deployment-complete benchmarking
May 25, 2026El Mustapha Mansouri, Keigo AraiBenchmark Auditing Benchmark Construction
School of Engineering, Institute of Science Tokyo, Yokohama, Kanagawa 226-8501, Japan
Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit
May 25, 2026Zexin Zhuang, Yanhang Li, Zhichao FanLLM Quantization Statistical Power Analysis
1Southern Methodist University · 2Northeastern University · University of Illinois Urbana-Champaign
Metadata Predictability Is Not Evidence Dependence: An Intervention-Based Audit for Weak-Label Benchmarks
May 22, 2026Kan ShaoCounterfactual Evaluation Benchmark Auditing
Jinglue Technology Development (Nanjing) Co., Ltd., Nanjing, China
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
May 19, 2026Parsa Mazaheri, Kasra MazaheriAgent Failure Analysis Computer-Use Agent Benchmarks
University of California, Santa Cruz · Massachusetts Institute of Technology
Are Sparse Autoencoder Benchmarks Reliable?
May 18, 2026David ChaninBenchmark Auditing LLM Interpretability
Decode Research, MATS, UCL
Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks
May 14, 2026Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray +1LLM Safety Benchmarks Benchmark Auditing
Palo Alto Networks
Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
May 14, 2026Matteo Attimonelli, Alessandro De Bellis, Aryo Pradipta Gema +8Benchmark Auditing Image Retrieval
Politecnico di Bari
The Evaluation Trap: Benchmark Design as Theoretical Commitment
May 13, 2026Theodore J KalaitzidisBenchmark Auditing Benchmark Design
Brown University
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
Apr 27, 2026Xinming Tu, Tianze Wang, Yingzhou +4Benchmark Auditing AI Agent Benchmarks
Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1
AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark Auditing AI Agent Evaluation
Anthropic Fellows Program · Anthropic
Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
Nov 6, 2025Ellis Brown, Jihan Yang, Shusheng Yang +2Benchmark Auditing VLM Evaluation
New York University
WirelessMathBench-XL: An Auditable Benchmark for Wireless Mathematical Reasoning
Sep 27, 2025Xin Li, Mengbing Liu, Yiyang Zhu +4Mathematical Reasoning Benchmarks Benchmark Auditing
Nanyang Technological University

Jul 30, 2026, cs.SEJ/K move · Enter open · S save

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Manyi Wang, Junjielong Xu, Pinjia He

The Chinese University of Hong Kong, Shenzhen

Benchmark Auditing Software Engineering Benchmarks

Open paper AskPDF arXiv

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

Momentum

Related topics

Latest papers 61