cs.SEAug 14, 2026
SaveEvaluating Agentic Code Repair Capabilities in Distributed Systems
Organizations: University of Southern California
Abstract
LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified. Distributed-system debugging, however, remains an under-explored regime: bugs span processes, nodes, and protocol interactions, with root causes rarely recoverable from source alone and brute-force exploration intractable across non-deterministic interleavings. This leaves two gaps in LLM and agent evaluation: no code-repair benchmark targets distributed-system bugs, and no controlled study isolates how much externally provided debugging context changes agent success on them. We introduce DDBench, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems, partitioned into three difficulty tiers. DDBench evaluates every case under two matched conditions: a symptom-only condition where the agent receives only the bug symptom and repository, and a context-augmented condition where it additionally receives a bounded debugging context (logs, traces, runtime state, and targeted code-investigation notes), isolating the effect of debugging context from model capability. The evaluation of ten LLMs on DDBench reveals several findings. First, distributed debugging exercises a reasoning dimension that single-process benchmarks do not surface: models' pass rates span 61 pp, and pairwise bootstrap separates 9 of 15 top-tier model pairs at p < 0.05 on DDBench's hardest case-set. Second, bounded debugging context lifts aggregate pass rate by +18.1 pp, and the lift is asymmetric: weaker models gain pass rate, while stronger models gain efficiency. Third, debugging context requires careful curation, as even faithful debugging context can sometimes mislead LLMs.
Figures & tables
Figure 1: DDBench sharply differentiates models on distributed debugging. Under the symptom-only condition, six models span 58 pp on DDBench tier-1 ( ), compared with only 7 pp on SWE-bench Verified ( ).
Figure 2: Examples of debug-context files shipped in DDBench .
Figure 3: The flow of evaluating a DDBench bug case. The buggy codebase and agent are containerized together, and the agent can access the codebase and a Markdown symptom description. In the context-augmented condition, the agent also has access to a debug-context bundle (green). After the agent concludes its patch, a hidden oracle runs against the codebase and returns PASS/FAIL as the final evaluation.
Figure 4: The DDBench curation pipeline, where an LLM agent performs first-pass mining and artifact synthesis under tight specifications, and human researchers perform final sanitization, validation, and acceptance.
Figure 5: Bug-family composition of DDBench . Left : per-tier breakdown where tier-1 concentrates protocol/recovery and replication/consistency bugs; tier-3 absorbs all input-triggered crashes, which can be localized within a single process; tier-2 bridges the two. Right : each family’s overall share of the benchmark.
| Model | Thinking | Form | Total Cost ($) | Pass Rate % | ||||
| Sym. | +Context | (pp) | Sym. | +Context | ||||
| Claude Opus 4.6 | Adaptive High | API | 59.4 | 23.6 | - 60.3% | 67.7 | 67.7 | — |
| Claude Sonnet 4.6 | Adaptive High | API | 68.4 | 34.0 | - 50.3% | 41.9 | 61.3 | + 19.4 |
| GPT 5.4 | High | API | 25.0 | 21.4 | - 14.3% | 41.9 | 67.7 | + 25.8 |
| GPT 5.4 mini | High | API | 7.9 | 6.1 | - 23.5% | 45.2 | 61.3 | + 16.1 |
| GLM 5.1 (FP8) | Enabled | Open, Self-Host | 23.1 | 14.6 | - 36.7% | 32.3 | 50.0 | + 17.7 |
Table 1: Per-model pass rates (%) on tier-1 (31 cases) under symptom-only and context-augmented conditions. denotes the percentage-point improvement. Nine of ten models improved; one tied; none regressed.
Figure 6: Debug-context reshapes both the capability–cost Pareto and the capability ranking on tier-1. (a) Each arrow traces a model from symptom-only (hollow) to context-augmented (solid). Weaker models’ arrows point up (capability gain) and stronger models’ arrows point left (cost reduction at unchanged pass rate). (b) Solid lines denote rank gains, dashed lines no change, and lines with a soft red halo denote losses; ties share a rank with vertical jitter. The presence of debug-context reshuffles the leaderboard.
| Bug Category | Sym. | +Ctx. | (pp) |
| Concurrency hazards (n=4) | 32.5% | 75.0% | +42.5 |
| Replication & consistency (n=9) | 38.9% | 57.8% | +18.9 |
| Protocol & recovery (n=13) | 27.7% | 39.5% | +11.8 |
| Resource & delivery (n=5) | 34.0% | 48.0% | +14.0 |
Table 2: Pass-rate lift by bug category on tier-1, pooled across 10 models. “Input-triggered crashes” is omitted because its cases reside exclusively in tier-3.
| Metric | Sym. | +Ctx. | |
| Mean steps | 56.7 | 42.7 | 13.9 |
| Mean first-edit step | 32.2 | 20.5 | 11.7 |
| Mean tokens | 1.69M | 0.93M | 0.75M |
| Mean read actions | 23.9 | 17.3 | 6.6 |
| Mean explore actions | 10.7 | 6.7 | 3.9 |
| Mean edit count | 3.0 | 3.2 | +0.1 |
Table 3: Behavioral metrics across 310 paired runs on tier-1. Debug-context reduces exploration effort consistently while edit count remains stable.
Figure 7: Pass rates across tiers 1–3: debug-context lift on pass-rate attenuates as the diagnostic search space shrinks. Dots show per-model pass rates.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Project | Language | System family | Number of Cases | Tier-1 | Tier-2 | Tier-3 |
| Dragonfly | C++ | kv-store | 14 | 1 | 1 | 12 |
| NATS | Go | messaging | 9 | 5 | 3 | 1 |
| Kvrocks | C++ | kv-store | 8 | 4 | 3 | 1 |
| etcd | Go | kv-store | 7 | 5 | 2 | 0 |
| hashicorp/raft | Go | consensus | 6 | 4 | 2 | 0 |
| Consul | Go | service-mesh | 3 | 2 | 1 | 0 |
Table 4: Per-project coverage of DDBench . DDBench spans 13 open-source distributed systems across five languages and six system families, partitioned into three difficulty tiers (tier-1, tier-2, tier-3).
| Project | License | Link |
| Dragonfly | BSL 1.1 | https://github.com/dragonflydb/dragonfly |
| NATS | Apache-2.0 | https://github.com/nats-io/nats-server |
| Kvrocks | Apache-2.0 | https://github.com/apache/kvrocks |
| etcd | Apache-2.0 | https://github.com/etcd-io/etcd |
| hashicorp/raft | MPL-2.0 | https://github.com/hashicorp/raft |
| Consul | BSL 1.1 | https://github.com/hashicorp/consul |
Table 5: Per-project license and upstream repository for the 13 open-source systems in DDBench .
Figure 8: Fix-patch size vs. cross-model pass rate on tier-1 ( ). Each marker is a case; LoC is the upstream production-file diff size (lines added removed, tests/mocks excluded), sourced from cases-solution/*/solution.md (packaged with DDBench ) or from the upstream GitHub PR diff when not annotated directly in DDBench , with the smallest passing agent patch as a fallback (the executable oracle accepts only correct fixes, so passing-patch size faithfully bounds upstream scope). Red markers denote cases unsolved by every model. Patch size correlates only moderately with difficulty ( ); within the -LoC bucket alone, cross-model pass rate ranges from to , confirming that reasoning load, not the surface size of the fix, drives difficulty.
Figure 9: Per-case debug-context bundle size across all 60 cases. Red dashed lines mark the median; the horizontal grey line is the 50th percentile. The channel is bounded across every case: median 321 estimated tokens, max 1,403 tokens, 3 markdown files per case.
| Category | n | Share |
| Code investigation | 33 | 55.0% |
| Test output | 25 | 41.7% |
| Crash output | 12 | 20.0% |
| Runtime state | 3 | 5.0% |
| Race/sanitizer report | 3 | 5.0% |
| Debugger trace (GDB) | 2 | 3.3% |
Table 6: Debug-context-bundle file-type composition (60 cases). Categories are not mutually exclusive, so shares do not sum to 100%.
| Benchmark | Cross-process | Code fix | Executable oracle | Debug-context channel | Languages |
| SWE-bench Verified [ 12 ] | ✗ | ✓ | ✓(unit tests) | ✗ | 1 (Python) |
| Multi-SWE-Bench [ 37 ] | ✗ | ✓ | ✓(unit tests) | ✗ | 8 |
| DebugBench [ 22 ] | ✗ | ✓ | ✓(unit tests) | ✗ | 3 |
| AIOpsLab [ 5 ] | ✓ | ✗ | ✓(telemetry) | ✓(telemetry) | — |
| CloudOpsBench [ 25 ] | ✓ | ✗ | ✓(telemetry) | ✓(telemetry) | — |
| DDBench (ours) | ✓ | ✓ | ✓(E2E & unit tests) | ✓(controlled) | 5 |
Table 7: DDBench versus existing code-repair and IT-operations benchmarks. Existing code-repair benchmarks supply an executable oracle but restrict to single-process workloads; IT-operations benchmarks exercise cross-service reasoning but ask agents to diagnose from telemetry rather than repair source. DDBench is the only benchmark that combines an executable repair oracle, cross-process bugs, and a controllable runtime-evidence channel.
Figure 10: Pairwise bootstrap tests on the six models with published SWE-bench Verified scores (10,000 resamples each). DDBench Tier-1 uses a paired bootstrap over 31 per-case outcomes; SWE-bench Verified uses an unpaired bootstrap over 500 simulated Bernoulli trials per model at vendor-reported rates (as per-case data not publicly released). Cells show the two-sided per pair; color encodes clipped to , with a red mark at ( , , ). On the same six models, DDBench Tier-1 separates 9 of 15 pairs at , compared with 2 on SWE-bench Verified. The contrast is consistent with DDBench exercising a reasoning dimension that single-process code repair does not.
| Model | Thinking | Form | Total Cost ($) | Pass Rate % | ||||
| Base | +Evidence | (pp) | Base | +Evidence | ||||
| Claude Opus 4.6 | Adaptive High | API | 28.46 | 22.12 | - 22.3% | 73.3 | 80.0 | + 6.7 |
| Claude Sonnet 4.6 | Adaptive High | API | 27.47 | 21.42 | - 22.0% | 73.3 | 73.3 | — |
| GPT 5.4 | High | API | 10.75 | 10.35 | - 3.7% | 60.0 | 86.7 | + 26.7 |
| GPT 5.4 mini | High | API | 2.70 | 2.25 | - 16.7% | 66.7 | 73.3 | + 6.7 |
| GLM 5.1 (FP8) | Enabled | Open, Self-Host | 7.12 | 5.33 | - 25.1% | 40.0 | 46.7 | + 6.7 |
Table 8: Per-model pass rates (%) on tier-2 (15 cases) under symptom-only and evidence-augmented conditions. denotes the percentage-point improvement. Eight of ten models improved; two tied (Claude Sonnet 4.6, GPT OSS 120B); none regressed.
| Model | Thinking | Form | Total Cost ($) | Pass Rate % | ||||
| Base | +Evidence | (pp) | Base | +Evidence | ||||
| Claude Opus 4.6 | Adaptive High | API | 30.56 | 14.61 | - 52.2% | 85.71 | 92.86 | + 7.14 |
| Claude Sonnet 4.6 | Adaptive High | API | 21.32 | 10.47 | - 50.9% | 85.71 | 92.86 | + 7.14 |
| GPT 5.4 | High | API | 13.17 | 9.50 | - 27.9% | 78.57 | 92.86 | + 14.29 |
| GPT 5.4 mini | High | API | 2.37 | 1.75 | - 26.2% | 85.71 | 92.86 | + 7.14 |
| GLM 5.1 (FP8) | Enabled | Open, Self-Host | 9.47 | 10.41 | + 9.9% | 57.14 | 50.00 | - 7.14 |
Table 9: Per-model pass rates (%) on tier-3 (14 cases) under symptom-only and evidence-augmented conditions. denotes the percentage-point improvement. Six of ten models improved; two tied (GLM 4.7, GLM 5); two regressed.
Figure 11: Per-case total-token CDFs on tier-1–tier-3. Each panel overlays the symptom-only (orange) and context-augmented (blue) distributions, pooling all evaluated models. Medians shift from 0.88M to 0.47M on tier-1 ( 47%), 0.93M to 0.47M on tier-2 ( 49%), and 0.95M to 0.68M on tier-3 ( 28%). The context-augmented CDF lies to the left of the symptom-only CDF at every percentile on tier-1 and tier-2; on tier-3 the two curves converge in the upper tail.
Figure 12: Per-case agent-step CDFs on tier-1–tier-3. Medians drop from 44 to 34 on tier-1 ( 23%), 46 to 33 on tier-2 ( 28%), and 52 to 40 on tier-3 ( 23%). Unlike the token distributions, the step CDFs separate roughly uniformly across the support, so the savings are not driven by a few long-running outliers.
Figure 13: Per-case agent-step distribution for Claude Opus 4.6 across tier-1–tier-3, restricted to runs that terminated under Submitted . The median drops from 55 to 30 steps under context augmentation ( 45% reduction). The right tail extends past 180 steps in both conditions—the longest in the cohort—indicating that even the strongest model in our setup encounters cases where the cross-process search dominates the trajectory regardless of debug context.
Figure 14: Per-case agent-step distribution for Claude Sonnet 4.6 across tier-1–tier-3. The median drops from 42 to 34 steps. The base count ( ) is lower than the context-augmented count ( ): seven additional cases reached Submitted under context augmentation. The recovered cases are dominated by runs that produced no exit status under the symptom-only condition (most plausibly wall-clock timeouts or framework aborts) rather than by step-cap exceptions; under either reading, the asymmetry is consistent with debug context shortening the path to a candidate patch and is not visible in the median alone.
Figure 15: Per-case agent-step distribution for GPT 5.4 across tier-1–tier-3. The median drops from 29 to 25 steps. Both distributions concentrate tightly below 85 steps with negligible right-tail mass, indicating that GPT 5.4 either converges on a candidate patch quickly or abandons the case, rather than continuing to explore.
Figure 16: Per-case agent-step distribution for GPT 5.4 mini across tier-1–tier-3. The median drops from 30 to 24 steps. The shape mirrors GPT 5.4: short-tailed and tightly concentrated. The smaller absolute reduction ( 6 steps) is consistent with the already-compact base distribution—there are fewer wasted exploratory steps for debug context to remove.
Figure 17: Per-case agent-step distribution for GLM 4.7 across tier-1–tier-3. The median drops from 77 to 54 steps—the highest medians in this subset under either condition. The right tail extends past 190 steps in both conditions, reflecting GLM 4.7’s tendency to run a long exploratory phase on hard cases. The base count ( ) is lower than the context-augmented count ( ): all seven cases that did not terminate under Submitted in the symptom-only condition (one step-cap exception and six runs with no recorded exit status) reached Submitted when given debug context, so every GLM 4.7 case terminated normally under context augmentation.
Figure 18: Per-case paired outcomes across all 10 models on tier-1 (rows: models, ordered by symptom-only pass rate; columns: the 31 cases, grouped by bug category and ordered within each group by cross-model win count). Green cells are gains (FAIL PASS), red cells are regressions (PASS FAIL). Weak models (bottom rows) are dominated by green—GLM 4.7 flips 13 of 31 cases—while strong models (top rows) are dominated by blue tie-passes. Losses are often cases where the runtime symptom misdirects. For instance, dragonfly-6687 (second column of Resource & delivery) is one of the highlighted examples in Appendix § G.6 .
More in this topic
AI Coding Agents
AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.
See all 327 papers