Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose SWE-CI, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability. The key insight is simple: Maintainability can be revealed by tracking how functional correctness changes over time. The benchmark comprises 100 tasks, each deriving from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.
Figures & tables
Figure 1: Unlike previous benchmarks, SWE-CI proposes an evolution-based evaluation. The red and blue arrows represent the actions of functions require and code , respectively. Dashed lines indicate processes that are unknown to the user.
Figure 2: Data curation process of SWE-CI.
Figure 3: SWE-CI uses an architect-programmer dual-agent workflow to model the continuous integration cycle of professional software teams in the real world.
Figure 4: The EvoScore variation of state-of-the-art models from 8 providers on SWE-CI.
Figure 5: The model’s EvoScore ranking changes as γ increases. When γ>1 , higher-ranking models indicate better codebase maintenance.
Figure 6: All models are sorted from smallest to largest by the zero regression rate.
Figure 7: Pearson correlation of regression rate (left) and magnitude (right) with number of iterations.
Figure 8: Win/loss rate of LLM-generated code against human oracle code in terms of MI score (left) and Pylint score (right) on successfully solved problems.
Figure 9: Scatter plots comparing changed lines between LLM-generated and human oracle solutions across successfully solved problems.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: The solved rate of all evaluated models, sorted in descending order.
Figure 11: The average number of turns consumed per task across all evaluated models, sorted in ascending order.
Figure 12: Cumulative maximum relative change over binned turn intervals, grouped by provider.