DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?
Organizations: University of Michigan
Abstract
LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents' ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope. The benchmark and evaluator are available at https://github.com/OrderLab/distdebug-bench, with agent traces and a leaderboard at https://distdebug.com.
Figures & tables
| Configuration | Localization coverage | Explanation | Instability | Effort | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Harness | File | Func. | Line | Block | Block prec. | Mention | Flaky bugs | Blocks/diag. | Calls/run | Tokens/run |
| Mini-SWE-Agent model sweep | |||||||||||
| GPT-6 Astra | Mini-SWE-Agent | 86.1 | 59.8 | 58.2 | 72.6 [65.7, 79.3] | 46.0 | 83.3 | 14 (12%) | 3.3 | 9.0 | 130K |
| Opus 5.5 | Mini-SWE-Agent | 84.5 | 61.8 | 58.9 | 71.8 [65.2, 78.2] | 49.0 | 82.1 | 19 (16%) | 2.8 | 8.8 | 149K |
| GPT-5.5 | Mini-SWE-Agent | 86.2 | 57.4 | 57.7 | 70.0 [63.1, 76.7] | 37.3 | 76.0 | 23 (19%) | 3.6 | 22.0 | 474K |
| GPT-5.4 | Mini-SWE-Agent | 81.0 | 54.6 | 53.6 | 66.6 [59.7, 73.4] | 41.6 | 70.4 | 23 (19%) | 2.9 | 25.0 | 687K |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Issues | Selection step |
|---|---|---|
| Public issue pool | 4,789 | Historical ZooKeeper issues considered by the collection pipeline. |
| Automatic candidates | 99 | Closed fixed bugs that pass metadata, report, and fixing-change filters. |
| Manual inspection sample | 50 | Candidate issues sampled for human audit and root-cause labelability checks. |
| Released ZooKeeper incidents | 18 | Accepted incidents with diagnosable issue-side context and reconciled source-level labels. |
| Category | Definition | Count |
|---|---|---|
| Method-named | A non-generic gold function name appears in the issue text | 38 |
| File/class-named | A gold file stem or class name appears, but no gold method | 32 |
| Opaque | No extracted gold method, file, or class identifier appears | 50 |
| Category | Definition | Count |
|---|---|---|
| Co-located | First project-internal stack frame is in a gold file | 13 |
| Adjacent | Crash frame and gold file are in nearby package structure | 6 |
| Cross-component | Stack trace exists, but the first internal frame is elsewhere | 26 |
| No stack trace | No usable project-internal stack frame is found | 75 |
| Signal present in report or attached evidence | Incidents |
|---|---|
| At least one code identifier | 96 |
| Prior analysis or causal explanation | 37 |
| Stack trace | 28 |
| Logs | 20 |
| Cross-linked issue reference | 22 |
| Attached artifacts | 7 |
| Root-cause type | Incidents |
|---|---|
| Logic | 56 |
| Concurrency | 28 |
| Error handling | 26 |
| Performance | 4 |
| Resource leak | 3 |
| Communication | 2 |
| Granularity | Measure | Agreement |
|---|---|---|
| File | Krippendorff’s | 0.919 |
| Pairwise Jaccard | 0.902 | |
| Normalized function | Krippendorff’s | 0.866 |
| Pairwise Jaccard | 0.837 | |
| Block (five-line tolerance) | Coverage | 0.861 |
| F1 | 0.848 |
| Construction method | GT blocks | Pred. blocks | Recall | Precision | F1 |
|---|---|---|---|---|---|
| LLM-assisted label generation | 22 | 15 | 54.5% | 73.3% | 62.6% |
| Non-LLM extraction pipeline | 22 | 52 | 77.3% | 34.6% | 47.8% |
| Configuration | File | Function | Line | Block | Block precision | Mention |
|---|---|---|---|---|---|---|
| MiniSWE GPT-6 Astra | 86.1 [80.8, 91.0] | 59.8 [51.9, 67.8] | 58.2 [51.4, 65.2] | 72.6 [65.7, 79.3] | 46.0 [40.1, 52.1] | 83.3 [77.4, 88.8] |
| MiniSWE Claude Opus 5.5 | 84.5 [79.1, 89.6] | 61.8 [54.3, 69.2] | 58.9 [52.2, 65.6] | 71.8 [65.2, 78.2] | 49.0 [43.2, 54.7] | 82.1 [76.2, 87.5] |
| MiniSWE GPT-5.5 | 86.2 [80.8, 91.0] | 57.4 [49.5, 65.1] | 57.7 [50.9, 64.5] | 70.0 [63.1, 76.7] | 37.3 [32.4, 42.2] | 76.0 [69.4, 82.2] |
| MiniSWE GPT-5.4 | 81.0 [75.3, 86.2] | 54.6 [46.9, 62.3] | 53.6 [46.8, 60.4] | 66.6 [59.7, 73.4] | 41.6 [36.5, 46.8] | 70.4 [63.6, 76.8] |
| MiniSWE GPT-5.3-Codex | 79.9 [73.6, 85.7] | 54.1 [46.1, 61.9] | 48.1 [41.4, 54.9] | 63.3 [55.9, 70.4] | 42.7 [37.2, 48.2] | 69.9 [62.9, 76.6] |
| MiniSWE MiMo v2.5 Pro | 80.5 [74.3, 86.2] | 60.6 [53.1, 68.1] | 46.7 [39.7, 53.5] | 62.0 [54.5, 69.3] | 40.2 [34.5, 46.0] | 73.5 [66.6, 80.0] |
| Candidate | Reference | Difference | 95% CI |
|---|---|---|---|
| MiniSWE GPT-6 Astra | MiniSWE GPT-5.2-Codex | ||
| MiniSWE Claude Opus 5.5 | MiniSWE GPT-5.2-Codex | ||
| MiniSWE GPT-5.5 | MiniSWE GPT-5.2-Codex | ||
| MiniSWE GPT-5.4 | MiniSWE GPT-5.2-Codex | ||
| MiniSWE GPT-5.3-Codex | MiniSWE GPT-5.2-Codex | ||
| MiniSWE MiMo v2.5 Pro | MiniSWE GPT-5.2-Codex |
| System | Bugs | Block coverage [95% CI] |
|---|---|---|
| Overall | 120 | 66.6 [59.7, 73.4] |
| Elasticsearch | 19 | 70.1 [53.3, 85.4] |
| etcd | 8 | 78.1 [61.5, 92.7] |
| HBase | 5 | 40.0 [13.3, 73.3] |
| HDFS | 10 | 52.7 [26.0, 79.3] |
| Ignite | 3 | 61.1 [0.0, 100.0] |
| Omitted system | Bugs left | File | Block [95% CI] | File–block gap | |
|---|---|---|---|---|---|
| None | 120 | 81.0 | 66.6 [59.7, 73.4] | 14.5 | |
| Elasticsearch | 101 | 79.9 | 65.9 [58.1, 73.3] | 14.0 | |
| etcd | 112 | 80.7 | 65.7 [58.3, 72.8] | 15.0 | |
| HBase | 115 | 82.1 | 67.7 [60.4, 74.5] | 14.4 | |
| HDFS | 110 | 80.9 | 67.8 [60.8, 74.7] | 13.1 | |
| Ignite | 117 | 81.0 | 66.7 [59.7, 73.5] | 14.3 |
| Harness | Runs with early scope | Late recoveries | Rate |
|---|---|---|---|
| Mini-SWE-Agent | 21 | 7 | 33.3% |
| OpenCode | 103 | 19 | 18.4% |
| Combined | 124 | 26 | 21.0% |
| Model | Bugs with different first-five commands | Bugs with different first source file |
|---|---|---|
| GPT-5.4 | 120/120 | 29/120 |
| GPT-5.3-Codex | 120/120 | 27/120 |
| GPT-5.2-Codex | 120/120 | 32/120 |
| Case | Lower-scoring tool path | Higher-scoring tool path |
|---|---|---|
| ZooKeeper-3769, Mini-SWE HY3 | Greps the node logs for election states and notifications, then follows the connection-management branch into QuorumCnxManager and haveDelivered . The final diagnosis blames connection delivery and has 0 block coverage. | Greps the same logs for ERROR , Exception , and WorkerReceiver , then opens the receive loop in FastLeaderElection.java . The final diagnosis covers WorkerReceiver.run . |
| ZooKeeper-2845, Mini-SWE GPT-5.3-Codex | Searches retain , ZKDatabase , QuorumPeer , Learner , and DIFF / TRUNC . The run inspects database loading and sync code, then diagnoses quorum synchronization; it misses ZooKeeperServer.shutdown . | Starts from similar retain-database searches, but adds clear / shutdown queries and opens the ZooKeeperServer shutdown window. The final diagnosis covers the lifecycle block that retains stale database state. |
| Model | Mean block | Best-attempt block | Gain | Bugs with better attempt |
|---|---|---|---|---|
| GPT-6 Astra | 72.6% | 76.5% | 3.8 | 16/120 |
| Claude Opus 5.5 | 71.8% | 78.4% | 6.5 | 21/120 |
| GPT-5.5 | 70.0% | 76.7% | 6.7 | 25/120 |
| GPT-5.4 | 66.6% | 73.7% | 7.1 | 25/120 |
| GPT-5.3-Codex | 63.2% | 69.8% | 6.5 | 26/120 |
| GPT-5.2-Codex | 59.5% | 64.3% | 4.8 | 21/120 |
| Model | Harness | Subset | Block cov. | Block prec. | Mention cov. | Cov. std. | Flaky bugs | Flaky delta | Tokens/run |
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V4 Pro | Mini-SWE | 67/10 | 64.8 (+5.6) | 41.6 (-3.5) | 74.0 (+11.8) | 8.8 (+4.7) | 17 | +8 | 1.4M (2.5 ) |
| Qwen3-Coder-Next | Mini-SWE | 55/8 | 59.9 (+1.0) | 47.4 (+3.2) | 66.5 (+4.7) | 17.1 (+12.8) | 25 | +18 | 4.4M (7.7 ) |
| Qwen3.6 Max | Mini-SWE | 35/7 | 61.8 (+6.4) | 48.7 (+10.5) | 70.2 (+11.8) | 14.2 (+9.6) | 15 | +10 | 1.1M (1.9 ) |
| Qwen3.6 Max | OpenCode | 63/10 | 53.6 (-4.0) | 46.7 (+4.2) | 63.8 (+3.0) | 8.2 (+3.5) | 15 | +6 | 0.6M (0.9 ) |
| Outcome | Diagnoses |
|---|---|
| Complete causal chain, different plausible intervention point | 4 |
| Incorrect hypothesis | 9 |
| Missing the decisive implementation link | 2 |
| Superficial or incomplete causal account | 5 |
| Total | 20 |
| Counting rule | False negatives among missed blocks | Block coverage gain (points) |
|---|---|---|
| Both reviewers agree | 4.5% [1.6, 8.3] | 2.3 [0.7, 4.3] |
| Claude Opus 5.5 alone | 5.6% [2.3, 9.8] | 3.1 [1.2, 5.5] |
| GPT-6 Astra alone | 16.3% [10.0, 23.9] | 7.2 [4.2, 10.6] |
| Set | Diagnosis | Patch | File touch | Block touch | Calls/run | Cost |
|---|---|---|---|---|---|---|
| High | None | 100.0 | 86.7 | 56.7 | 34.6 | $1.44 |
| High | High coverage | 100.0 | 90.0 | 60.0 | 30.1 | $1.02 |
| Low | None | 93.3 | 26.7 | 6.7 | 40.4 | $2.00 |
| Low | Low coverage | 100.0 | 20.0 | 0.0 | 40.6 | $1.74 |
| Diagnosis | Attempts | File touch | Block touch | Calls/run | Cost |
|---|---|---|---|---|---|
| None | 18 | 44.4 (8/18) | 11.1 (2/18) | 40.9 | $2.02 |
| Low coverage | 18 | 33.3 (6/18) | 0.0 (0/18) | 37.9 | $1.54 |
| High coverage | 18 | 61.1 (11/18) | 50.0 (9/18) | 31.6 | $1.34 |
| Phase | Model | Flagged (%) | Reasoning | Tool output | Tool input |
|---|---|---|---|---|---|
| Before | GPT-5.4 | 102 (28.3) | 0 | 1,007 | 16 |
| After | GPT-5.4 | 4 (1.1) | 0 | 22 | 0 |
| After | GPT-5.5 | 4 (1.1) | 0 | 22 | 2 |
| After | GPT-6 Astra | 3 (0.8) | 0 | 10 | 2 |
| After | Claude Opus 5.5 | 0 (0.0) | 0 | 0 | 0 |
| Before | GPT-5.3-Codex | 8 (2.2) | 0 | 88 | 0 |
| Model | Pearson | Spearman | Pearson (%) |
|---|---|---|---|
| GPT-6 Astra | 0.00 | ||
| Claude Opus 5.5 | 0.08 | ||
| GPT-5.5 | 0.26 | ||
| GPT-5.4 | 0.07 | ||
| GPT-5.3-Codex | 0.19 | ||
| MiMo v2.5 Pro | 0.39 |
| Condition | File | Function | Line | Block |
|---|---|---|---|---|
| Repository-assisted | 76.0 | 46.7 | 46.9 | 58.3 |
| Closed book: identifiable report | 43.9 | 17.1 | 0.0 | 0.0 |
| Closed book: anonymous report | 42.7 | 21.1 | 0.0 | 0.0 |
| Closed book: issue ID only | 0.0 | 0.0 | 0.0 | 0.0 |