Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency
Organizations: Colorado State University, United States
Abstract
Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have shown promise, prior studies primarily focus on overall repair effectiveness. The effects of bug complexity, fault localization, reasoning settings, and repair cost-effectiveness remain insufficiently explored. Aims: This study presents a comprehensive empirical analysis of LLM-based APR, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs. Method: We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs (DeepSeek, GPT, and Llama), and examine their performance across diverse levels of bug complexity and localization strategies through a multi-dimensional empirical framework and statistical analysis. Results: Although structurally complex bugs and imprecise fault localization make repair more challenging, LLM-based APR techniques still achieve competitive repair effectiveness. Imprecise fault localization can substantially enlarge the performance gap between APR techniques. Furthermore, higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, revealing a nontrivial trade-off between repair effectiveness and computational cost. Conclusions: Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. The repair effectiveness gap between APR techniques becomes larger as fault localization becomes less precise. GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively; whereas the total repair cost of DeepSeek-V3.2 shows the best cost-efficiency performance.
Figures & tables
| SHSL vs SHML | SHSL vs MHSL | SHSL vs MHML |
| Type Fixed Unfixed SHSL 197(78.5%) 54 SHML 113(50.0%) 113 | Type Fixed Unfixed SHSL 197(78.5%) 54 MHSL 39(58.2%) 28 | Type Fixed Unfixed SHSL 197(78.5%) 54 MHML 48(50.0%) 48 |
| SHML vs MHSL | SHML vs MHML | MHSL vs MHML |
| Type Fixed Unfixed SHML 113(50.0%) 113 MHSL 39(58.2%) 28 | Type Fixed Unfixed SHML 113(50.0%) 113 MHML 48(50.0%) 48 | Type Fixed Unfixed MHSL 39(58.2%) 28 MHML 48(50.0%) 48 |
| SHSL | SHML | MHSL | MHML | |
| SHSL | – | |||
| SHML | – | – | ||
| MHSL | – | – | – | |
| MHML | – | – | – | – |
Automated Program Repair
Automated program repair develops methods that locate software faults and produce validated fixes with limited human effort.
See all 158 papers