Resolving Java Code Repository Issues with iSWE Agent
Organizations: IBM
Abstract
Resolving issues on code repositories is an important part of software engineering. Various recent systems automatically resolve issues using large language models and agents, often with impressive performance. Unfortunately, most of these models and agents focus primarily on Python, and their performance on other programming languages is lower. In particular, a lot of enterprise software is written in Java, yet automated issue resolution for Java is under-explored. This paper introduces iSWE Agent, an automated issue resolver with an emphasis on Java. It consists of two sub-agents, one for localization and the other for editing. Both have access to novel tools based on rule-based Java static analysis and transformation. Using this approach, iSWE achieves state-of-the-art issue resolution rates across the Java splits of both Multi-SWE-bench and SWE-PolyBench. More generally, we hope that by combining the best of rule-based and model-based techniques, this paper contributes towards improving enterprise software development.
Figures & tables
| Bench mark | Base LLM | % Resolved | $ Cost Total (Avg.) | Avg. # Tokens | Edit Correct Location | |||
| File | Node | |||||||
| Recall | Precision | Recall | Precision | |||||
| iSWE-Agent (Frontier Models) | ||||||||
| Multi-SWE-Bench (Java) | Claude-4.5-Sonnet | 43 (33.6%) | 237.89 (1.86) | 598k | 62% | 82% | 44% | 68% |
| Claude-3.7-Sonnet | 32 (25.0%) | 154.51 (1.21) | 386k | 50% | 72% | 32% | 63% | |
| Leaderboard submissions (top other agents for comparison) | ||||||||
| Benchmark | Base LLM | Avg. # Turns | Avg. Cost | ||||
| Loc. | Editing | Total | Loc. | Editing | Total | ||
| Multi-SWE- Bench (Java) | Claude-4.5-Sonnet (Loc & Edit) | 31.5 | 1.7 | 33.3 | $1.77 | $0.09 | $1.86 |
| DeepSeek-V3-2 (Loc & Edit) | 41.6 | 2.5 | 44.2 | $0.24 | $0.01 | $0.25 | |
| SWE-PolyBench (Java) | Claude-4.5-Opus (Loc & Edit) | 34.1 | 2.2 | 36.3 | $2.41 | $0.14 | $2.55 |
| DeepSeek-V3-2 (Loc & Edit) | 54.8 | 3.3 | 58.0 | $0.27 | $0.01 | $0.28 | |
| Benchmark | Base LLM | Localization Correct Location | Edit Correct Location | ||||||
| File | Node | File | Node | ||||||
| Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | ||
| iSWE-Agent (Frontier Models) | |||||||||
| Multi-SWE-Bench (Java) | Claude-4.5-Sonnet | 63% | 83% | 36% | 59% | 62% | 82% | 44% | 68% |
| Claude-3.7-Sonnet | 56% | 77% | 28% | 53% | 50% | 72% | 32% | 63% | |
| iSWE-Agent (Open Source Models) | |||||||||
| Bench mark | Base LLM | Tool Usage (Number of Calls) | Total | ||||||
| get file info | get class info | get method info | get symbol info | get inheritance hierarchy | get function callers | get call chain | |||
| Frontier Models | |||||||||
| Multi-SWE-Bench (Java) | Claude-4.5-Sonnet | 87 | 124 | 403 | 181 | 2 | 19 | 33 | 849 |
| Claude-3.7-Sonnet | 163 | 136 | 265 | 75 | 1 | 7 | 5 | 652 | |
| Open Source Models | |||||||||
| DeepSeek-V3-2 | 497 | 549 | 1803 | 1113 | 22 | 93 | 49 | 4126 | |
| Tool Set | $ Cost | Avg. | Turns | File Retrieval | Node Retrieval | ||
| Total (Avg.) | #Tokens | Recall | Precision | Recall | Precision | ||
| iSWE-tools | 226 (1.77) | 575k | 4037 | 63% | 83% | 36% | 59% |
| iSWE-tools & view-file | 229 (1.79) | 584k | 3777 | 63% | 83% | 37% | 57% |
| Bash | 263 (2.06) | 667k | 5815 | 62% | 82% | 37% | 57% |
| All-Tools (iSWE-tools & Bash & swea) | 336 (2.63) | 855k | 5820 | 62% | 81% | 37% | 58% |
| Bench mark | Complexity | # Instances | Base LLM | % Resolved | Avg. Cost | File Retrieval | Node Retrieval | ||
| Recall | Precision | Recall | Precision | ||||||
| Multi-SWE- Bench (Java) | Easy | 27 | Claude-4.5-Sonnet | 16 (59.2%) | $1.65 | 65% | 80% | 42% | 49% |
| DeepSeek-V3-2 | 9 (33.3%) | $0.18 | 58% | 74% | 39% | 47% | |||
| Medium | 65 | Claude-4.5-Sonnet | 26 (40.0%) | $1.71 | 66% | 86% | 39% | 58% | |
| DeepSeek-V3-2 | 18 (27.6%) | $0.21 | 61% | 70% | 37% | 47% | |||
| Hard | 36 | Claude-4.5-Sonnet | 1 (2.7%) | $2.26 | 55% | 80% | 22% | 65% | |
| Bench mark | Modification Type | # Instances | Base LLM | % Resolved | Avg. Cost | File Retrieval | Node Retrieval | ||
| Recall | Precision | Recall | Precision | ||||||
| Multi-SWE-Bench (Java) | None | 3 | Claude-4.5-Sonnet | 0 (0.0%) | $2.27 | 36% | 100% | 36% | 100% |
| DeepSeek-V3-2 | 0 (0.0%) | $0.14 | 25% | 100% | 25% | 100% | |||
| Function Only Single All | 33 | Claude-4.5-Sonnet | 20 (60.6%) | $1.75 | 72% | 87% | 58% | 70% | |
| DeepSeek-V3-2 | 10 (30.3%) | $0.23 | 56% | 69% | 50% | 61% | |||
| 44 | Claude-4.5-Sonnet | 27 (61.3%) | $1.63 | 73% | 90% | 59% | 74% | ||
| # Hints Provided | # Instances | % Resolved | File Retrieval | Node Retrieval | ||
| Recall | Precision | Recall | Precision | |||
| 0 | 79 | 44% | 68.33% | 84.99% | 52.03% | 73.89% |
| 1 | 23 | 30% | 59.66% | 86.96% | 35.98% | 59.50% |
| 2 | 4 | 25% | 50.00% | 100.00% | 43.15% | 95.00% |
| 3 | 2 | 0% | 50.00% | 62.50% | 50.00% | 66.67% |
| 4 | 8 | 0% | 62.35% | 75.00% | 18.31% | 68.84% |
Software Engineering Agents
Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.
See all 344 papers