Software Engineering Benchmarks

Follow this topic Ask about this topic

Software engineering benchmarks measure whether AI systems can perform tasks such as fixing repository-level bugs, writing tests, reviewing code, and building or evolving applications. They commonly use real repositories, executable environments, hidden tests, and simulated user interactions, with newer evaluations also examining agent trajectories, tool use, and coordination. The field is moving beyond one-shot code generation toward long-running workflows and broader measures of quality, including performance, security, design constraints, and maintainability. A central challenge is making results reliable and representative: tests can miss valid solutions or accept flawed ones, and scores can depend heavily on task selection, evaluation rules, and agent setup.

Latest papers

157 papers32 papers in the last four weeks, up 113% on the four weeks before. 0.2% of all new papers.

SWE-Universe: Scale Real-World Verifiable Environments to Millions
Feb 2, 2026Mouxiang Chen, Lei Zhang, Yunlong Feng +16Software Engineering Benchmark Construction
Qwen Team, Alibaba Group · Zhejiang University
SpIDER: Spatially Informed Dense Embedding Retrieval for Software Issue Localization
Dec 18, 2025Shravan Chaudhari, Rahul Thomas Jacob, Jiajun Cao +3Software Engineering Software Engineering Benchmarks
Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Work done while interning at AWS · AWS AI Labs, Seattle, WA, USA
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Nov 8, 2025Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh +5Software Engineering Agents Benchmark Design
Harvard University · Google DeepMind · Princeton University +2
BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
Sep 27, 2025Zehua Zhang, Ati Priya Bajaj, Divij Handa +14Software Engineering Agents Software Engineering Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs
Jun 16, 2025Anh Ho, Thanh Le-Cong, Bach Le +1Automated Program Repair Software Engineering Benchmarks
The University of Melbourne, Australia · Singapore University of Technology and Design, Singapore
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
May 23, 2025Haoyu Sun, Huichen Will Wang, Jiawei Gu +2Software Engineering Web Application Generation
Tongji University · Shanghai AI Laboratory · University of Washington +1
Real Faults in Deep Learning Fault Benchmarks: How Real Are They?
Dec 20, 2024Gunel Jahangirova, Nargiz Humbatova, Jinhan Kim +2Software Engineering Benchmarks Benchmark Reproducibility
Department of Informatics, King’s College London, London, UK · Software Institute, Università della Svizzera italiana (USI), Lugano, Switzerland · School of Computing, KAIST, Daejeon, Republic of Korea

Feb 2, 2026J/K move · Enter open · S save

SWE-Universe: Scale Real-World Verifiable Environments to Millions

Mouxiang Chen, Lei Zhang, Yunlong Feng, Xuwu Wang+15

Qwen Team, Alibaba Group · Zhejiang University

Software Engineering Benchmark Construction Software Engineering Benchmarks

Open paper AskPDF arXiv

We propose SWE-Universe, a scalable and efficient framework for automatically constructing real-world software engineering (SWE) verifiable environments from GitHub pull requests (PRs). To overcome the prevalent challenges of automatic building, such as low production yield, weak verifiers, and prohibitive cost, our framework utilizes a building agent powered by an efficient custom-trained model. This agent employs iterative self-verification and in-loop hacking detection to ensure the reliable generation of high-fidelity, verifiable tasks. Using this method, we scale the number of real-world multilingual SWE environments to a million scale (807,693). We demonstrate the profound value of our environments through large-scale agentic mid-training and reinforcement learning. Finally, we applied this technique to Qwen3-Max-Thinking and achieved a score of 75.3% on SWE-Bench Verified. Our work provides both a critical resource and a robust methodology to advance the next generation of coding agents.