Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits T0 and evaluating 100 post-T0 pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity (+15.6% to +86.2% relative F1 lift, p<10−4), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline F1 of the next-generation model across all evaluated families.
Uses 100% MIT-licensed Tree-sitter AST for flat symbol indexing and introduces ServiceExplorer ’s deterministic 2-level hierarchy (Level-1 system/package containers and Level-2 architectural modules).
D5. Cross-Generation Model Scaling
Single-model evaluations cannot reveal whether repository graphs substitute for or amplify foundation model scale.
Evaluates six models across the MAI-Code Flash, Claude, and GPT-Sol families ( 7,200 runs), testing whether a lower-tier or prior-generation model with ServiceExplorer surpasses the next-generation Baseline.
Table 1: Methodological Design of the Three-Condition Time-Consistent Benchmark.
Repo
Granularity
Baseline
Tree-sitter AST
ServiceExplorer
SE vs. BL
SE vs. AST
F1
F1 (MIT)
F1 (2-Level)
Gain (%)
Gain (%)
DragonFly
Minimal
0.3436
0.3536
0.4014
+16.8%
+13.5%
DragonFly
Concise
0.4389
0.4818
0.5365
+22.2%
+11.4%
DragonFly
Contextual
0.4348
0.4646
0.5549
+27.6%
+19.4%
DragonFly
Guided
0.5406
0.5419
0.5494
+1.6%
+1.4%
React
Minimal
0.2065
0.1946
0.2316
+12.2%
+19.0%
Table 2: Three-Condition F1 Progression Across Prompt Granularities ( Claude-Opus-4.6 , N=50 PRs per row).
Figure 1: Relative F1 Lift (%) of ServiceExplorer (2-Level) and Tree-sitter AST (MIT) over the Zero-Leakage Baseline across prompt granularities on DragonFly and React .
Figure 2: DragonFly (C++): Zero-leakage Recall, Precision, and F1 trajectories across prompt granularities for Baseline , Tree-sitter AST (MIT), and ServiceExplorer ( N=200 evaluations). ServiceExplorer dominates across all metrics and granularities.
Figure 3: React (JS/TS): Zero-leakage Recall, Precision, and F1 trajectories across prompt granularities for Baseline , Tree-sitter AST (MIT), and ServiceExplorer ( N=200 evaluations). Flat AST symbol expansion without package-level containment degrades Precision on React , whereas ServiceExplorer simultaneously improves both Recall and Precision across all four granularities.
Figure 4: Paired ΔF1 lift over Baseline across prompt granularities with 95% bootstrap confidence intervals on DragonFly and React .
Evaluation of repository-aware software engineering systems is often confounded by synthetic task design, prompt leakage, and temporal contamination between repository knowledge and future code changes. We present a time-consistent benchmark methodology that snapshots a repository at time T0, constructs repository-derived code knowledge using only artifacts available before T0, and evaluates on engineering tasks derived from pull requests merged in the future interval (T0, T1]. Each historical pull request is transformed into a natural-language task through an LLM-assisted prompt-generation pipeline, and the benchmark is formalized as a matched A/B comparison in which the same software engineering agent is evaluated with and without repository-derived code knowledge while all other variables are held constant. We also report a baseline characterization study on two open-source repositories, DragonFly and React, using three Claude-family models and four prompt granularities. Across both repositories, file-level F1 increases monotonically from minimal to guided prompts, reaching 0.8081 on DragonFly and 0.8078 on React for the strongest tested model. These results show that prompt construction is a first-order benchmark variable. More broadly, the benchmark highlights that temporal consistency and prompt control are core validity requirements for repository-aware software engineering evaluation.
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.
Shaoqiu Zhang, Yuhang Wang, Jialiang Liang +8
1Shanghai Jiao Tong University · 2Xinjiang University · 4Independent Researcher +2