Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits T0 and evaluating 100 post-T0 pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity (+15.6% to +86.2% relative F1 lift, p<10−4), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline F1 of the next-generation model across all evaluated families.
Uses 100% MIT-licensed Tree-sitter AST for flat symbol indexing and introduces ServiceExplorer ’s deterministic 2-level hierarchy (Level-1 system/package containers and Level-2 architectural modules).
D5. Cross-Generation Model Scaling
Single-model evaluations cannot reveal whether repository graphs substitute for or amplify foundation model scale.
Evaluates six models across the MAI-Code Flash, Claude, and GPT-Sol families ( 7,200 runs), testing whether a lower-tier or prior-generation model with ServiceExplorer surpasses the next-generation Baseline.
Table 1: Methodological Design of the Three-Condition Time-Consistent Benchmark.
Repo
Granularity
Baseline
Tree-sitter AST
ServiceExplorer
SE vs. BL
SE vs. AST
F1
F1 (MIT)
F1 (2-Level)
Gain (%)
Gain (%)
DragonFly
Minimal
0.3436
0.3536
0.4014
+16.8%
+13.5%
DragonFly
Concise
0.4389
0.4818
0.5365
+22.2%
+11.4%
DragonFly
Contextual
0.4348
0.4646
0.5549
+27.6%
+19.4%
DragonFly
Guided
0.5406
0.5419
0.5494
+1.6%
+1.4%
React
Minimal
0.2065
0.1946
0.2316
+12.2%
+19.0%
Table 2: Three-Condition F1 Progression Across Prompt Granularities ( Claude-Opus-4.6 , N=50 PRs per row).
Figure 1: Relative F1 Lift (%) of ServiceExplorer (2-Level) and Tree-sitter AST (MIT) over the Zero-Leakage Baseline across prompt granularities on DragonFly and React .
Figure 2: DragonFly (C++): Zero-leakage Recall, Precision, and F1 trajectories across prompt granularities for Baseline , Tree-sitter AST (MIT), and ServiceExplorer ( N=200 evaluations). ServiceExplorer dominates across all metrics and granularities.
Figure 3: React (JS/TS): Zero-leakage Recall, Precision, and F1 trajectories across prompt granularities for Baseline , Tree-sitter AST (MIT), and ServiceExplorer ( N=200 evaluations). Flat AST symbol expansion without package-level containment degrades Precision on React , whereas ServiceExplorer simultaneously improves both Recall and Precision across all four granularities.
Figure 4: Paired ΔF1 lift over Baseline across prompt granularities with 95% bootstrap confidence intervals on DragonFly and React .