cs.SENov 12, 2026

Time-Consistent Evaluation of Hierarchical Code Graphs for SWE Agents

Authors: Xianpeng, Sun, Tian Yu, Haonan Sun, Sheng Ma, Qincheng Zhang, Lifei Rao, Jie Li, 

Organizations: Simon · Microsoft · Ken

Abstract

Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits T0T_0 and evaluating 100 post-T0T_0 pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity (+15.6%+15.6\% to +86.2%+86.2\% relative F1F_1 lift, p<10−4p < 10^{-4}), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline F1F_1 of the next-generation model across all evaluated families.

Figures & tables

Explore similar work

CardsList
  1. SWE-Explore: Benchmarking How Coding Agents Explore Repositories

    Jun 5, 2026Shaoqiu Zhang, Yuhang Wang, Jialiang Liang +8AI Coding AgentsAgentic Search