Software Engineering Benchmarks

Ask about this topic

Overview

Software engineering benchmarks measure whether AI systems can perform tasks such as fixing repository-level bugs, writing tests, reviewing code, and building or evolving applications. They commonly use real repositories, executable environments, hidden tests, and simulated user interactions, with newer evaluations also examining agent trajectories, tool use, and coordination. The field is moving beyond one-shot code generation toward long-running workflows and broader measures of quality, including performance, security, design constraints, and maintainability. A central challenge is making results reliable and representative: tests can miss valid solutions or accept flawed ones, and scores can depend heavily on task selection, evaluation rules, and agent setup.

Latest papers

CardsList

157 papers32 papers in the last four weeks, up 113% on the four weeks before. 0.2% of all new papers.

  1. Neurosymbolic Repo-level Code Localization

    Apr 17, 2026Xiufeng Xu, Xiufeng Wu, Zejun Zhang +1Software Engineering BenchmarksLLM-Based Program Synthesis

    Nanyang Technological University, Singapore

  2. Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex

    Apr 16, 2026Zhonghao Yang, Yu Li, Yanxu Zhu +6Benchmark DesignSoftware Engineering Benchmarks

    Shanghai Artificial Intelligence Laboratory

  3. LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB

    Apr 15, 2026Vekil Bekmyradov, Noah C. Pütz, Thomas Bartz-BeielsteinShortcut Learning in Language ModelsLLM Evaluation

    THK-AI Research Cluster, TH Köln

  4. LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software

    Apr 14, 2026Syed Md Mukit Rashid, Abdullah Al Ishtiaq, Kai Tu +7Automated Program RepairSoftware Security

    The Pennsylvania State University

  5. CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation

    Apr 14, 2026Zaoyu Chen, Jianbo Dai, Boyu Zhu +6LLM EvaluationSoftware Engineering Benchmarks

    The Hong Kong Polytechnic University · University of Edinburgh · UCL +5

  6. ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization

    Apr 12, 2026Suyoung Bae, CheolWon Na, Jaehoon Lee +3Factual Consistency EvaluationSummarization Evaluation

    College of Computing and Informatics, Sungkyunkwan University, South Korea

  7. ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents

    Apr 9, 2026Kenan Li, Qirui Jin, Liao Zhu +13Software Engineering AgentsLLM Agent Evaluation

    Microsoft · Georgia Institute of Technology

  8. Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development

    Apr 8, 2026Xinchen Wang, Ruida Hu, Cuiyun Gao +2Software Engineering BenchmarksAI-Assisted Software Development

    Harbin Institute of Technology, Shenzhen, China · Independent Researcher, China

  9. Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution

    Apr 7, 2026Kai Yu, Zhenhao Zhou, Junhao Zeng +8LLM Agent EvaluationSoftware Engineering Benchmarks

    Fudan University, China · Nanyang Technological University, Singapore

  10. Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks

    Apr 6, 2026Amir M. Ebrahimi, Gopi Krishnan RajbahadurSoftware Engineering BenchmarksBenchmark Validity

    Queen’s University, Kingston, Canada

  11. SW-A2A^2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web

    Apr 5, 2026Linyao Chen, Bo Huang, Qinlao Zhao +13Agentic Code GenerationWeb Agents

    The University of Tokyo · Shanghai Jiao Tong University · Shanghai Innovation Institute +6

  12. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

    Apr 3, 2026KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara +2Code Generation EvaluationSoftware Engineering

    Fujitsu Research

  13. In harmony with gpt-oss

    Apr 1, 2026Borislav MavrinAI Agent BenchmarksTool-Use Evaluation

  14. ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair

    Mar 28, 2026Jia Li, Zeyang Zhuang, Zhuangbin Chen +3Automated Program RepairSoftware Engineering Benchmarks

    The Chinese University of Hong Kong, Hong Kong, China · School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Sun Yat-sen University, Guangzhou, China

  15. Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering

    Mar 27, 2026Yoseph Berhanu Alebachew, Hunter Leary, Swanand Vaishampayan +1LLM EvaluationSoftware Engineering Benchmarks

    Department of Computer Science, Virginia Tech Blacksburg, Virginia, USA

  16. ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation

    Mar 27, 2026Xianpeng, Sun, Haonan Sun +5Software Engineering BenchmarksTemporal Data Leakage

    Simon

  17. MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development

    Mar 26, 2026Moshood A. Fakorede, Krishna Upadhyay, A. B. Siddique +1Software EngineeringSoftware Engineering Benchmarks

    Louisiana State University · University of Kentucky

  18. Composer 2 Technical Report

    Mar 25, 2026Cursor Research, :, Aaron Chan +53Code Language ModelsSoftware Engineering Agents

  19. Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

    Mar 25, 2026Myeongsoo Kim, Dingmin Wang, Siwei Cui +6Agent Failure AnalysisAI Coding Agents

    AWS AI Labs · Monash University

  20. Code Review Agent Benchmark

    Mar 24, 2026Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf +3Automated Code ReviewSoftware Engineering Benchmarks

    National University of Singapore · Zhejiang University · SonarSource

  21. Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics

    Mar 23, 2026Tianxiang Xu, Xiaoyan Zhu, Xin Lai +4Software Engineering BenchmarksScientific Reproducibility

    Xi’an Jiaotong University

  22. Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis

    Mar 23, 2026Tae-Eun SongBenchmark ContaminationData Contamination in Language Models

    Daejeon Jungang Cheonggua Co., Ltd.

  23. SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding

    Mar 17, 2026Songcheng Cai, Zhiheng Lyu, Yuansheng Ni +13AI Agent BenchmarksSoftware Engineering Benchmarks

    University of Waterloo · University of Toronto · The Hong Kong University of Science and Technology +2

  24. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

    Mar 16, 2026Tingxu Han, Yi Zhang, Wei Song +4Software Engineering AgentsSoftware Engineering Benchmarks

    Nanjing University · Mohamed bin Zayed University of Artificial Intelligence · South China University of Technology +1

  25. daVinci-Env: Open SWE Environment Synthesis at Scale

    Mar 13, 2026Dayuan Fu, Shenyu Wu, Yunze Wu +11Software EngineeringLLM Agent Training

    SII · GAIR · SJTU

  26. SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution

    Mar 13, 2026Gangda Deng, Zhaoling Chen, Zhongming Yu +11Software EngineeringSoftware Engineering Benchmarks

    USC · UCR · UCSD +5

  27. SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

    Mar 4, 2026Jialong Chen, Xander Xu, Hu Wei +2Software EngineeringSoftware Engineering Benchmarks

    Sun Yat-sen University · Skylenage · Alibaba Group

  28. BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

    Mar 3, 2026Guoxin Chen, Fanzhe Meng, Jiale Zhao +12Retrieval-Augmented Code GenerationSoftware Engineering Benchmarks

  29. SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks

    Feb 28, 2026Yucheng Zeng, Shupeng Li, Daxiang Dong +11Synthetic Benchmark GenerationSoftware Engineering Benchmarks

    Qianfan Team, Baidu Inc.

  30. SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

    Feb 27, 2026Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov +1Software Engineering BenchmarksSoftware Engineering Agents

    Nebius, Amsterdam, Netherlands.