Software Engineering Benchmarks

Ask about this topic

Momentum

12 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 89

All topics
CardsList

  1. Evaluating Exact Output and Checkpoint-State Prediction in Real Programs

    Oct 8, 2026Xiaohong Chen, David Bucur, Chenglong Ma +4LLM EvaluationSoftware Engineering Benchmarks

    Intent Computing Org · University of Illinois Urbana-Champaign · Georgia Institute of Technology

  2. SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

    Oct 8, 2026Hexuan Deng, Yue Wang, Wenyu Jiang +13Coding AgentsAI Agent Benchmarks

    Tencent Hy AI Data · Beijing Zhongguancun Academy

  3. Coding-Agent Benchmarks Should Match Their Users' Task Flows

    Oct 7, 2026Igor Slinko, Yaroslav Golubev, Sergey TitovLLM Agent EvaluationAI Agent Benchmarks

    JetBrains Research

  4. Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

    Oct 5, 2026Hai Dang Truong, Rayner Goh, Thanh Le-Cong +1AI Agent AuditingCoding Agents

    Singapore Management University · Singapore University of Technology and Design

  5. EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses

    Sep 30, 2026Zhixuan Tan, Pengjie Gu, Zhao Li +4Software Engineering BenchmarksLong-Horizon Agent Evaluation

    The Chinese University of Hong Kong, Shenzhen · MemoraX AI

  6. Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

    Sep 29, 2026Sabrina Kaniewski, Tim Krämer, Julius Bächle +3Software Vulnerability DetectionSoftware Engineering Benchmarks

    Institute for Secure Networked Systems, Esslingen University, Germany · Esslingen University, Germany · Institute for Intelligent Systems, Esslingen University, Germany +1

  7. WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

    Sep 29, 2026Haomin Qi, Xiangzhe Xu, Yiming Huang +2Software Engineering AgentsBenchmark Construction

    University of California, San Diego · Purdue University · National University of Singapore

  8. From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents

    Sep 28, 2026Tianyu Liu, Dingyuan Dai, Yufan Du +1Software Engineering AgentsBenchmark Auditing

    Tsinghua University · UCLA

  9. SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories

    Sep 24, 2026Jiajun Wu, Leixin Sun, Zihan Tan +9Software EngineeringSoftware Engineering Agents

    CosmosMind · ModCraft · Peking University +2

  10. Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

    Sep 23, 2026Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband +3LLM EvaluationSoftware Engineering Benchmarks

    York University Toronto, Canada · The University of Texas at Dallas Texas, USA

  11. The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

    Sep 22, 2026Wenbo Pan, Zhichao Liu, Shujie Liu +6Software Engineering BenchmarksLong-Horizon Agent Evaluation

    City University of Hong Kong · Microsoft · Independent Researcher

  12. VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

    Sep 20, 2026Liyang Fan, Yingcheng Shi, Yongbin Li +7Benchmark DesignSoftware Engineering Benchmarks

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · SUAT · Alibaba Token Hub, Alibaba Group +1

  13. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

    Sep 16, 2026Jeonghye Kim, Minseon Kim, Young Jin Kim +5Coding AgentsSoftware Engineering Benchmarks

    KAIST · Microsoft Research Montréal · Microsoft AI

  14. Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents

    Sep 14, 2026Mykhailo Kozyrev, Andrei Kozyrev, Anton PodkopaevSoftware Engineering AgentsSoftware Engineering Benchmarks

    Technical University of Munich, JetBrains Research Munich, Germany · JetBrains Research Berlin, Germany · Constructor University Bremen, JetBrains Research Amsterdam, Netherlands

  15. SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

    Sep 10, 2026Qibai Chen, Zeming LiuSoftware EngineeringLLM Evaluation

    Independent Researcher United States · Brown University United States

  16. SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

    Sep 8, 2026Pujun Zheng, Zixin Shang, Shufan Jiang +5Reward HackingSoftware Engineering Agents

    East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University

  17. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

    Sep 3, 2026Xin He, Yanlin Wang, Mingwei Liu +3Software Engineering AgentsAutomated Program Repair

    School of Software Engineering, Sun Yat-sen University · College of Computer Science and Technology, Zhejiang University · School of Big Data and Software Engineering, Chongqing University +1

  18. Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

    Sep 1, 2026Kefeng Duan, Dewu Zheng, Yanlin Wang +7Software Engineering BenchmarksLLM Agent Evaluation

    School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Huawei Cloud Computing Technologies Co., Ltd., China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  19. What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal

    Sep 1, 2026Radin Shayanfar, Keheliya Gallaba, Ahmed E. HassanSoftware EngineeringBenchmark Design

    Queen’s University

  20. RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

    Aug 28, 2026Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim +2LLM PromptingCoding Agents

    Sungkyunkwan University

  21. SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

    Aug 20, 2026Zhipeng Xu, Jiahao Lu, Yining Zheng +2Software EngineeringAutomated Program Repair

    Shanghai Innovation Institute · Fudan University

  22. VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

    Aug 12, 2026Jin Lu, Xuening Han, Yang Zhong +4Software Vulnerability DetectionSoftware Engineering Benchmarks

    Amazon Web Services · University of Pittsburgh · Purdue University

  23. SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

    Aug 10, 2026Yuling Shi, Jinghan Xu, Kelin Fu +12AI Coding AgentsSoftware Engineering Benchmarks

    Shanghai Jiao Tong University · Peking University · Douyin Group +4

  24. RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

    Aug 5, 2026Yuexi Yang, Alyssa Wu, Ji Luo +4Software EngineeringLLM Evaluation

    Zhejiang University · Hunyuan, Tencent · Ningbo Global Innovation Center, Zhejiang University

  25. Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

    Aug 5, 2026Haobin Li, Ping Deng, Weizhong Qian +4Automated Program RepairAI Coding Agents

    Sichuan University · University of Electronic Science and Technology of China · Independent Researchers

  26. SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

    Aug 3, 2026Yuqiao Tan, Jinxiang Meng, Fangyu Lei +4Coding AgentsSoftware Engineering Benchmarks

    Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  27. BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests

    Aug 3, 2026Zetong Xiong, Qiao Zhao, Jun Zhang +20Software Engineering AgentsBenchmark Construction

    Baidu Beijing, China

  28. LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

    Jul 31, 2026Han Li, Zhemin Fang, Rili Feng +8AI Coding AgentsSoftware Engineering Benchmarks

    Microsoft · Nanjing University · Shanghai Jiao Tong University +1

  29. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

    Jul 30, 2026Manyi Wang, Junjielong Xu, Pinjia HeBenchmark AuditingSoftware Engineering Benchmarks

    The Chinese University of Hong Kong, Shenzhen

  30. An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

    Jul 30, 2026Yanyu Ren, Yunfeng Bai, Xizheng Wang +2Multi-Agent CoordinationMulti-Agent Orchestration

    1Tsinghua University · 2Zhongguancun Laboratory