Pass-Rate Evaluation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 18

All topics

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Pass-Rate Evaluation Agentic Benchmarks
Scale AI
Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks
Sep 27, 2026Analog Design Bench TeamAnalog Design Analog
The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability
Sep 17, 2026Michael Hernandez, Tian ZhaoPass-Rate Evaluation Code Generation
University of Wisconsin–Milwaukee
What Fixed-Rollout pass@k Evaluations Can Identify
Sep 10, 2026Pranav Singh, Prashant SinghPass-Rate Evaluation Extrapolation
Department of Mathematics Indian Institute of Technology Ropar Ropar, Punjab 140001, India
Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse
Jul 5, 2026Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3Large Language Models Fail Instruction-Tuned Models
IIIT Delhi · Microsoft Research India · IIT Kanpur
Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
Jun 7, 2026Sayed Erfan ArefinCode Generation Pass-Rate Evaluation
Department of Computer Science University of Dayton Dayton, OH 45469, USA
Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
May 30, 2026Yongxi Zhou, Lai Yun Choi, Jiaxi Wen +1Pass-Rate Evaluation High-Fidelity Conditional Generation
Northeastern University, Massachusetts, USA · University of Southern California, California, USA
Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study
May 29, 2026Xiaonan Xu, Wenjing WuSkills Large Language Model Agents
Computer Information Technology, Northern Arizona University, Flagstaff, AZ, USA · Computer Science, University of Colorado Boulder, Boulder, CO, USA
Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
May 27, 2026Linxin Song, Jiefeng Chen, Yue Huang +5Codebases Pass-Rate Evaluation
University of Southern California · Google Cloud AI Research · University of Notre Dame +1
Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation
May 27, 2026Le Bronnec Florian, Alexandre Verine, Rio Yokota +1Code Generation Code Optimization
RIKEN Center for Computational Science, Tokyo, Japan · École Normale Supérieure Paris, PSL University, Paris, France · LAMSADE, CNRS, Université Paris-Dauphine-PSL, Paris, France
Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work
May 20, 2026Haiyang Shen, Jiuzheng Wang, Taian Guo +9Artificial Intelligence Education Artificial Intelligence Benchmarks
1Peking University · 2Advanced Institute of Big Data
PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
May 19, 2026Qiran Zhang, Yuheng Wang, Runde Yang +9Procedural Spatio-Temporal Reasoning
School of Artificial Intelligence, Shanghai Jiao Tong University
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
May 13, 2026Priyam Sahoo, Gaurav Mittal, Xiaomin Li +4Swe-Bench Verified Pass-Rate Evaluation
University of Illinois, Urbana-Champaign, IL, USA · Microsoft, Redmond, WA, USA
Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
May 8, 2026Hugh Xuechen Liu, Kıvanç TatarPass-Rate Evaluation Large Language Model Generation
Chalmers University of Technology and University of Gothenburg
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
May 7, 2026Haoxiang Wang, Da Yu, Huishuai ZhangLarge Language Model Evaluation Pass-Rate Evaluation
Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence
Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime
May 6, 2026Tianshu Zhu, Wenyu Zhang, Xiaoying Zuo +8Rollout Pass-Rate Evaluation
Baidu
Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
May 1, 2026Xin-Ye Li, Ren-Biao Liu, Yun-Ji Zhang +3Pass-Rate Evaluation Code Generation
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Test Generation Code Generation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

Weijun Luo, Kelvin Luu, Xinyi Liu, Guangze Luo+5

Scale AI

Pass-Rate Evaluation Agentic Benchmarks Security Evaluation Adversarial Evaluation Remediation

Open paper AskPDF arXiv

Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes constitute unearned passes; their proportion among all passes defines the integrity gap. As agents improve, benchmark surfaces that once seemed harmless can become exploitable, making benchmark validity an ongoing maintenance problem. We introduce a process-verification framework that audits passing trajectories, distinguishes evidenced reward hacking from verifier weakness, and localizes exploitable surfaces for repair. Across 3,810 passing trajectories from 29 model-benchmark cohorts, confirmed violations often increase with model generation but not monotonically. On SWEBench Pro V1.0, confirmed violation rates rise from 24% to 73% between Opus 4.7 and Fable 5 on matched tasks; later cohorts fall to 11% for Fable 5.1 and 0% for GPT-6 Astra. These comparisons are descriptive: configurations were not normalized, and the latest models also pass fewer exploitable tasks. Violations concentrate around a small set of recurring surfaces, especially unintended access to reference solutions through git history. Three repair case studies across two benchmarks show why blocking a recorded exploit is insufficient: the same protected information can remain accessible through another route. Therefore, we combine minimal patches with exploit replay and fresh agent evaluation, auditing new passes under the original standard. No evaluated attempt against the final patches reached the protected channel, and every post-patch pass was judged legitimate. Benchmark integrity requires ongoing maintenance: audit passing behavior, repair the enabling surface, and re-evaluate both exploit access and legitimate solvability.

Momentum

Related topics

Latest papers 18