Software Engineering Benchmarks

Overview

Software engineering benchmarks measure whether AI systems can perform tasks such as fixing repository-level bugs, writing tests, reviewing code, and building or evolving applications. They commonly use real repositories, executable environments, hidden tests, and simulated user interactions, with newer evaluations also examining agent trajectories, tool use, and coordination. The field is moving beyond one-shot code generation toward long-running workflows and broader measures of quality, including performance, security, design constraints, and maintainability. A central challenge is making results reliable and representative: tests can miss valid solutions or accept flawed ones, and scores can depend heavily on task selection, evaluation rules, and agent setup.

Latest papers

CardsList

173 papers25 papers in the last four weeks, down 4% on the four weeks before. 0.2% of all new papers.

  1. BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

    Nov 18, 2026John Wu, Rushi Qiang, Yuchen Zhuang +10Coding Agent BenchmarksSoftware Engineering Benchmarks

  2. CratesBump: A Version-Bump Dataset for the Rust Ecosystem

    Nov 18, 2026Muhammad HassnainMining Software RepositoriesSoftware Engineering Benchmarks

  3. Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

    Nov 16, 2026Bowen Liu, Yuhan Li, Nuo Chen +7Coding Agent BenchmarksSoftware Engineering Benchmarks

  4. Robo-SWE: An Executable Issue-Resolution Dataset Mined from Robot Software Repositories

    Nov 11, 2026Vahid Haratian, Vahid Ansari, Mohammadhossein Malmir +1Software Engineering BenchmarksSoftware Engineering Agents

  5. Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

    Nov 6, 2026Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander GolubevSoftware Engineering BenchmarksSoftware Engineering

  6. BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization

    Nov 6, 2026Yuhao Tan, Hanyun Jiang, Fangkai Yang +10Agent Harness OptimizationAgent Harness Self-Evolution

  7. XStack: A Benchmark for ML Repo Migration

    Nov 5, 2026Linrong Cai, Taiming Lu, Haozhe Chen +1Coding Agent BenchmarksSoftware Engineering Benchmarks

  8. Take It From Here: Evaluating Coding Agents On Project Takeover

    Nov 5, 2026Dingxian Wang, Zhe Fu, Jiaqi Deng +9Coding Agent BenchmarksSoftware Engineering Benchmarks

  9. DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

    Nov 2, 2026Yuxuan Jiang, Chenglin Liu, John Rose +6Coding Agent BenchmarksLLM Agent Evaluation

  10. Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

    Nov 2, 2026Ao Lu, Yuanhao Zeng, Shaocheng Lan +5Coding Agent BenchmarksBenchmark Construction

  11. VerusWeave: A Benchmark for Proof Repair in Concurrent Verus Programs

    Oct 30, 2026Pedro Ribeiro, Martim Afonso, André Vasconcelos +1Software Engineering BenchmarksCoding Agent Benchmarks

  12. SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code

    Oct 30, 2026Omid Saghatchian, Ali Mahzoun, Khashayar Etemadi +1Semantic Change DetectionCode Language Models

  13. Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

    Oct 29, 2026Zhonghao Jiang, Xuyang Zheng, Yanfu Yan +3LLM-Assisted AnnotationSoftware Engineering Benchmarks

  14. Repairing Broken Access Control in Web Applications with LLMs: An Empirical Benchmark Study

    Oct 25, 2026Mingrui Liu, Liang He, Xinchu Wang +4Access ControlAutomated Program Repair

  15. SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

    Oct 23, 2026Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7Coding AgentsSoftware Engineering Benchmarks

  16. SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?

    Oct 22, 2026Jialei Li, Dingxian Wang, Xiaoyan Zhao +11Coding AgentsSoftware Engineering Benchmarks

  17. ProductWebBench: Evaluating Website Continuity Beyond Task Completion

    Oct 22, 2026Jiajun Wu, Jian Yang, Yaxin Du +9Software Engineering BenchmarksCode Generation Evaluation

  18. ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

    Oct 20, 2026Akarsh Ashok, Dinil Mon Divakaran, Vivek BalachandranBenchmark DesignSoftware Engineering Benchmarks

  19. MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories

    Oct 20, 2026Guorui Chen, Tao Liu, Zhenzhen Huang +9Memory Agent BenchmarksMemory-Augmented Agents

  20. When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories

    Oct 19, 2026Tianqi Liu, Yandong Guan, Songze Li +3AI Coding AgentsLLM Agent Evaluation

  21. SWE-Residual: Evaluating Starting-State Robustness of Coding Agents

    Oct 19, 2026Jinze Li, Weijie Liu, Jinfeng Xu +6AI Coding AgentsSoftware Engineering Benchmarks

  22. FrontierRefactor: Benchmarking Multi-Scale Performance-Oriented Codebase Refactoring

    Oct 19, 2026Shijian Wang, Fanqing Meng, Jiajun Chen +8Automated Code RefactoringSoftware Engineering Benchmarks

  23. Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

    Oct 17, 2026Boyi Liu, Yite Wang, Xinyu Lian +6Long-Horizon RLRL for LLM Agents

  24. HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

    Oct 17, 2026Wanyi Chen, Wanqin Shen, Yuanchen Shi +2AI Agent EvaluationActive Learning