cs.SEMar 13, 2026
SaveSWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
Organizations: USC · UCR · UCSD · Army Research Office ·
Abstract
Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to 38.03% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.
Figures & tables
Figure 1: Milestone-level task granularity optimally balances functional coherence and evolutionary awareness for benchmarking continuous software evolution.
| Category | Benchmark | Language | Task Properties | Cross-task Dependency | Task Collection | ||
| Granularity | Avg LoC | Additional Context | |||||
| Function Completion | HumanEval ( Chen et al., 2021 ) | Python | Function-level | 6.8 | – | – | Manual |
| Issue Resolution | SWE-bench ( Jimenez et al., 2024 ) | Python | Commit-level | 32.8 | Codebase | – | Rule-based |
| SWE-rebench ( Badertdinov et al., 2025 ) | Python | Commit-level | 142 | Codebase | – | Rule-based | |
| Multi-SWE-bench ( Zan et al., 2025 ) | Multi | Commit-level | 246 | Codebase | – | Rule-based | |
| SWE-bench Pro ( Deng et al., 2025 ) | Multi | Commit-level | 107 | Codebase | – | Rule-based | |
Table 1: Representative software engineering benchmarks for LLMs. Unlike other categories of benchmarks that evaluate agents on isolated snapshots or against ground-truth states at each step, Repository Evolution requires agents to continuously build upon their own accumulated development history, exposing them to error propagation across tasks. SWE-Milestone adopts Milestone-level granularity: each milestone is a functionally coherent group of commits that collectively advances a development objective, avoiding the noise of individual commits and the excessive scope of full releases. Dev History denotes the agent’s own development trace accumulated from preceding tasks.
Figure 2: The DeepCommit pipeline architecture. Phase 1 extracts structured data from commit history through static analysis, including source filtering, commit extraction, PR/Issues, releases, commit DAG, code metrics, and symbol changes. Phase 2 employs an LLM agent to construct a Milestone DAG via four iterative stages: seed discovery, milestone consolidation, dependency inference, and milestone decomposition. Phase 3 resolves runtime dependencies through testbed construction and test collection, with DAG refinement and fallback patches as repair strategies, followed by flaky test filtering to produce an executable testbed. Quality Assurance validates outputs at textual, compilation, and test collection levels. See Appendix B.4 for all DAG visualizations.
Figure 3: Illustration of the evaluation pipelines. (a) In the independent task evaluation workflow, the environment resets after each task. (b) In the continuous task evaluation workflow, tasks are organized as a dependency graph. The agent continuously evolves the codebase from a base snapshot. Upon completing a task (e.g., M1 & M2), the repository is snapshotted for isolated evaluation while the planner unlocks subsequent tasks (e.g., M3) for the agent to fetch, ensuring a continuous and stateful development loop.
Figure 4: Dataset statistics and characteristics of SWE-Milestone.
| Agent | Model | Score ⋆ (%) | Precision (%) | Recall (%) | Resolve (%) | Cost ($) | Out Tok. (K) | Time (h) | Turns |
| Claude Code | Claude Sonnet 4.5 | 15.16 | 18.88 | 28.50 | 5.49 | 27.02 | 243 | 2.06 | 770 |
| Claude Opus 4.5 | 25.85 | 28.04 | 40.80 | 6.28 | 71.10 | 309 | 2.35 | 999 | |
| Claude Sonnet 4.6 | 29.58 | 29.62 | 47.63 | 5.88 | 68.88 | 852 | 4.41 | 1538 | |
| Claude Opus 4.6 | 36.29 | 37.84 | 56.32 | 11.57 | 88.22 | 578 | 2.73 | 1891 | |
| Codex CLI | GPT 5.2-Codex | 13.46 | 12.65 | 26.65 | 4.78 | 38.11 | 701 | 3.56 | 1259 |
| GPT 5.2 | 23.30 | 20.89 | 45.76 | 8.18 | 56.90 | 814 | 5.35 | 1717 |
Table 2: Performance of coding agents on SWE-Milestone under continuous task evaluation. All metrics are per-evolution-range averages. Out Tok. (K): total generated tokens in thousands, including reasoning where applicable. ⋆ Primary metric (shaded). Bold : best in column.
Figure 5: Per-repository score comparison under two evaluation modes. High independent-task performance across all repositories confirms that milestones are individually solvable.
Figure 6: Overall Score vs. cost trade-off across all repositories.
Figure 7: Task complexity effects on Score (top) and Resolve Rate (bottom), binned by code size, specification length, execution order, and DAG layer. Dashed lines are per-bin averages across models.
Figure 8: Evolution dynamics across models. (Left) Multi-window extrapolation of saturation curves fitted with , showing projected ceilings beyond the observed window. Legend annotations report (marginal efficiency at the onset of the sequence) and (fraction of efficiency preserved after each observation window). (Middle) Continuous vs. Independent comparison for GPT 5.3-Codex (better retain) and Gemini 3.1 Pro (better init). (Right) Continuous vs. Independent comparison for the Claude model family.
Figure 9: Per-model cumulative Recall (solid) and Precision (dotted) over evolution progress. Stronger models achieve near-linear Recall growth, yet Precision saturates across all evaluated configurations.
Figure 10: Propagation type analysis for Opus 4.6 . Left : Selected error chain patterns across repositories, where each column is an error chain and each row a milestone. Right : Distribution of propagation event types across milestone progress bins (averaged over all repositories), showing how inherited failures (P1) and infrastructure effects (PX) increasingly dominate in later stages.
Figure 11: Root Cause Type Propagation Type heatmap. Each cell shows the macro-averaged event proportion across all repositories.
Figure 12: Continuous-to-Independent turns ratio across normalized execution progress (10 bins). Values above indicate the agent expends more effort under continuous evaluation than on the same milestone independently. The ratio remains near or below for most of the sequence but rises sharply in the final bin. This reflects increased rework and error-recovery effort as accumulated technical debt compounds.
Figure 13: Exploration ratio and context wave for Claude Code (powered by Opus 4.6 ) on element-web . Top plot shows milestone progression (M1 to M18). Middle plot shows per-minute exploration ratio (read/search vs. write/execute). Bottom plot shows total context token usage over active time, with compaction and eviction events marked.
Figure 14: Exploration tool-call count aggregated across all repositories, grouped by agent framework and sorted by count within each cluster. Diamond markers show average Score. Higher-performing agents consistently devote greater effort to exploration (e.g., reading and searching).
Figure 15: Average verification tool-call count per milestone progress bin (10 bins), for representative agent configurations. Verification effort generally increases with progress. It peaks around 70% to 80% completion before declining in the final bin.
Figure 16: Score landscape in the edit-thrashing (repeat edit ratio) vs. verification (test execution frequency) space, aggregated across all repositories. A sweet spot of moderate thrashing and moderate verification yields the highest scores. The blind thrashing quadrant (high thrash, low verify) produces the worst outcomes.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17: Excerpt of the SRS for milestone M001 of the zeromicro/go-zero repository (v1.6.0 v1.9.3) in SWE-Milestone, showing the Overview, two of the seven Feature Requirements (FR1 and FR2), and the trailing Environment Dependency section. Each Feature Requirement specifies a Problem , a Requirements block, and an Acceptance clause. Full SRS instances for all milestones are available at https://huggingface.co/datasets/DeepCommit-ai/SWE-Milestone-data .
Figure 18: System prompt provided to every agent under the Continuous Task Evaluation setting in SWE-Milestone. The prompt is framework-agnostic and is loaded as the agent’s system message before the first SRS task is read. Variable placeholders such as {src_dirs} and {milestone_id} are filled in per repository and milestone at runtime.
| Codebase | Release Range | Milestone DAG | ||||||
| Org/Repo | #Files | #LoC | Start | End | LoC | #M | #Deps | LoC CV |
| zeromicro/go-zero | 1,021 | 110K | v1.6.0 | v1.9.3 | 6,403 | 23 | 25 | 1.29 |
| apache/dubbo | 4,279 | 350K | 3.3.3 | 3.3.6 | 4,154 | 12 | 9 | 0.76 |
| BurntSushi/ripgrep | 159 | 48K | 14.1.1 | 15.0.0 | 1,474 | 11 | 12 | 0.83 |
| nushell/nushell | 1,727 | 264K | 0.106.0 | 0.108.0 | 15,520 | 13 | 28 | 1.10 |
| element-hq/element-web | 2,430 | 476K | v1.11.95 | v1.11.97 | 7,657 | 18 | 12 | 0.87 |
Table 3: Detailed per-repository statistics for SWE-Milestone. HumanDT and HumanWT are average human-estimated development time and SRS writing time, respectively. #M counts graded milestones; the 3 non-graded context milestones are excluded.
Figure 19: Milestone DAGs for SWE-Milestone repositories (Part 1/4).
Figure 20: Milestone DAGs for SWE-Milestone repositories (Part 2/4).
Figure 21: Milestone DAGs for SWE-Milestone repositories (Part 3/4).
Figure 22: Milestone DAGs for SWE-Milestone repositories (Part 4/4).
Figure 23: Responsibility boundary misplacement in apache/dubbo (M004). The ground truth checks stream parameters in resolveValue() , preserving the resolver chain contract. The agent checks in accept() , ejecting the parameter from the terminal resolver and causing 3 P2P regressions.
Figure 24: Shotgun fix in nushell . The gold patch uses a whitelist strategy: compile_break() returns NotInALoop as required by the SRS, and only the known catch -block false positive is exempted in parse_internal_call . The agent uses a blacklist strategy: it suppresses all NotInALoop errors across every block expression and deletes the is_in_loop() API. Tests pass only because push_break() has an internal fallback.
Figure 25: API signature degradation in nushell ’s type checker. The agent’s group-by-input algorithm forces OutputMismatch.expected from a structured Type enum to an opaque String , losing pattern-match capability. The agent also omits workaround cleanup and utility deduplication (2 vs. 7 files modified).
Figure 26: Continuous vs. independent evaluation on element-web . The shaded area highlights how error accumulation in continuous mode causes a cost-effective model ( Gemini 3 Flash , independent) to outperform frontier models in continuous mode.
Figure 27: Human-annotated and DeepCommit milestone decompositions for scikit-learn v1.5.2–v1.6.0. Top: milestones grouped by human analyzers (M1–M6). Bottom: milestones grouped by DeepCommit (G1–G6). Each block summarizes representative commits, commit count, LoC, and time span.
Figure 28: Structural comparison of Human and DeepCommit Milestone DAGs. The Human DAG contains 14 milestones and 19 dependencies; the DeepCommit DAG contains 12 milestones and 14 dependencies. Edges denote functional or process-level dependencies with strong/weak strength.
| Dimension | Human DAG | DeepCommit DAG |
| Leaf milestones | 14 | 12 |
| Groups | 6 | 6 |
| Covered commits | 373 | 201 |
| DAG edges | 19 | 14 |
| Edge types | FUNC (14) + NFR (5) | FUNC (13) + NFR (1) |
| Strong edges | 2 | 8 |
Table 4: Structural overview of the DeepCommit Milestone DAG and the Human-annotated Milestone DAG.
Figure 29: Contingency matrix of commit assignments (Human DeepCommit) for the 201 shared commits. The dominant Human M6.1 DeepCommit M6.1 block (71 commits) reflects high agreement on documentation, while the dispersed pattern across DeepCommit M1.1–M2.3 reveals how phase-based partitioning fragments intent-based Human milestones.
More in this topic
Software Engineering
Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.
See all 270 papers