Inquest: Log-Guided Root-Cause Code Localization with a Staged Agent
Organizations: Department of Computer Science, Brock University, St. Catharines, Ontario, Canada · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Ontario, Canada
Abstract
When a production issue is reported, engineers get a description and some logs, but no failing test and no way to reproduce the fault. The logs show where the failure was noticed, not where it was caused: in 49% of the real reports we study, the text names none of the files that the fix later changed. We built a benchmark for this setting from two public datasets and 27 open-source projects (Apache, Red Hat, MongoDB, MariaDB, Qt and Sakai). It holds 38,001 resolved issue reports with their pasted logs, the code as it was while the bug was still present, and the files the fix changed. We present INQUEST, an agent that finds the faulty code using a 27B open-weights model. Code, not the model, controls the procedure. It first gathers the evidence, including the files fixed in similar past issues that were resolved before the report was filed. The model then makes only three bounded decisions: which files to open, what role each file plays in the failure, and how to rank them. On 26,544 evaluation reports, INQUEST ranks a changed file first for 68.6% and in the top five for 85.9%, names the correct module for 90.2%, and ranks a changed function first for 53.8%. On the reports that name no faulty file, it reaches 52.8% and 73.6%. A free agent loop with the same model and tools finds files equally well, but names the faulty function 14 points less often. On a 1,000-report sample, INQUEST beats our reimplementations of two published agents by 7 and 20 points of top-1 accuracy (4 and 16 counting only the reports those agents answered), and the same model used as a re-ranker by 6. We report the cost of every system and the results that go against us, and we release the benchmark, both systems and every run transcript.
Figures & tables
| log-bearing fixed bugs, uncertainty | 70,999 |
| no fix commit found | 18,661 |
| more than ten fix commits | 55 |
| no non-test source file modified | 14,207 |
| gold over fifty files (bulk edit) | 75 |
| instances | 38,001 |
| gold entirely test code (scoring-time exclusion) | 101 |
| file, all | file, hard | module | region | chain | cost | ||||||
| Acc@1 | Acc@5 | Acc@10 | MRR | Acc@1 | Acc@5 | Acc@1 | Acc@1 | Acc@5 | cites | calls / tokens | |
| evaluation split, 26,544 instances (12,799 hard) | |||||||||||
| retrieval layer (no model) | 41.2 | 70.3 | 78.3 | 0.539 | 20.3 | 48.7 | 76.5 | 21.9 | 42.1 | – | 0 / 0 |
| Inquest | 68.6 | 85.9 | 88.1 | 0.761 | 52.8 | 73.6 | 90.9 | 53.8 | 69.9 | 57.3 | 15.7 / 106k |
| stratified sample, 1,000 instances (483 hard) | |||||||||||
| retrieval layer (no model) | 40.6 | 69.4 | 77.6 | 0.533 | 20.7 | 48.7 | 78.2 | – | – | – | 0 / 0 |
| file, all | file, hard | region | chain | cost | ||||
| Acc@1 | Acc@5 | Acc@1 | Acc@1 | Acc@5 | Acc@10 | cites | calls / tokens | |
| unconstrained loop | 67.9 | 86.8 | 53.3 | 39.8 | 50.7 | 50.7 | 0.0 | 13.6 / 99k |
| Inquest | 68.6 | 85.9 | 52.8 | 53.8 | 69.9 | 74.8 | 57.3 | 15.7 / 106k |
| Inquest minus loop, paired | +0.7 [+0.2, +1.2] | -0.9 [-1.2, -0.6] | -0.5 [-1.3, +0.3] | +14.1 [+13.5, +14.7] | +19.2 [+18.6, +19.9] | +24.1 [+23.5, +24.8] | +57.3 | – |
| all | hard | easy | ||||
|---|---|---|---|---|---|---|
| Acc@1 | Acc@5 | MRR | Acc@1 | Acc@5 | Acc@5 | |
| S1 trace mentions | 26.0 | 43.2 | 0.333 | 0.0 | 0.0 | 83.4 |
| S2 path BM25 | 19.7 | 38.6 | 0.287 | 6.0 | 17.8 | 58.0 |
| S3 content BM25 | 32.3 | 61.5 | 0.456 | 24.4 | 50.2 | 72.1 |
| S4 incident memory | 22.8 | 42.1 | 0.317 | 21.0 | 40.2 | 43.8 |
| fusion, no memory | 35.5 | 59.7 | 0.464 | 7.4 | 26.4 | 90.6 |
| eligible | Acc@1 | Acc@5 | Acc@10 | MRR | oracle | |
|---|---|---|---|---|---|---|
| BM25 over the layer’s top-10 files | 24,820 | 21.9 | 42.1 | 51.1 | 0.313 | 77.0 |
| unconstrained loop (ablation) | 24,847 | 39.8 | 50.7 | 50.7 | 0.446 | – |
| Inquest | 24,847 | 53.8 | 69.9 | 74.8 | 0.600 | – |
| Acc@1 | Acc@5 | |
| random module ranking (expected) | 5.7 | 25.8 |
| development-split defect frequency | 49.3 | 76.8 |
| the retrieval layer (module of the top file) | 76.5 | 96.7 |
| unconstrained loop (module of the top file) | 90.8 | – |
| Inquest (module named by the agent) | 90.2 | – |
Software Engineering
Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.
See all 291 papers