IssueLighter: An Issue Type-Aware LLM Agent for Issue Report Validity Assessment
Organizations: School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China
Abstract
Invalid issue reports are prevalent in open source software (OSS) projects, imposing substantial burdens on maintainers who already face a high risk of burnout. Prior research has primarily focused on identifying invalid reports for specific issue categories (e.g., bug and wontfix), leaving the broader landscape of OSS issue types largely unexplored and limiting the generalizability of existing approaches. Moreover, existing methods provide little support for explaining or validating their predictions, reducing their practical usefulness for maintainers. With the rise of LLM-based agents, it remains unclear whether a comprehensive understanding of the causes of invalidity across all issue types can improve validity assessment performance while producing well-grounded rationales. To this end, we first manually annotated the validity status of 520 issue reports from three OSS projects, i.e., pandas, scikit-learn, and mypy, resulting in 257 invalid reports and 263 valid ones. Then, we conducted a thematic analysis on these invalid reports and identified 13 causes of invalidity that are closely specific to issue types. Based on these insights, we propose IssueLighter, an agent-based framework with a type-aware validity assessment strategy that dynamically executes commands. Experimental results show that IssueLighter consistently and substantially outperforms all baseline methods, with an average F1 improvement of over 20 percentage points. Ablation studies confirm that both the type-aware strategy and the taxonomy of causes of invalidity contribute to the performance gains. Qualitative analysis further demonstrates that IssueLighter generates rationales grounded in accurately identified causes of invalidity and supported by evidence. Moreover, leveraging these rationales, we challenged maintainers' original decisions in four disputed cases, and all received approval.
Figures & tables
| Project | Total | Valid | Invalid | Invalid Ratio |
|---|---|---|---|---|
| pandas | 209 | 117 | 92 | 44.0% |
| scikit-learn | 118 | 56 | 62 | 52.5% |
| mypy | 193 | 90 | 103 | 53.4% |
| Total | 520 | 263 | 257 | 49.4% |
| Cause | Explanation | pd | sk | mp |
| User misunderstanding | The issue stems from user misunderstanding of the project’s expected behavior, such as intentional code behavior, correct documentation, or reasonable project configuration. | 26 | 15 | 56 |
| Unable reproduction | The issue cannot be reproduced based on the information provided in the report. | 11 | 5 | 10 |
| Misplaced report | The issue stems from a third-party project, not in the project itself. | 12 | 4 | 8 |
| Out of date | The issue has been fixed in subsequent versions. | 4 | 1 | 8 |
| Inherent limitation | Due to inherent constraints in the project’s technology stack, the issue cannot be resolved. | 1 | 2 | 5 |
| Existing alternative | The request can already be achieved through other means. | 13 | 9 | 2 |
| Model | Method | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|---|
| ChatGPT | IssueLighter | 96.2% | 98.6% | 93.6% | 96.0% |
| Enhanced zero-shot | 78.5% | 89.7% | 63.8% | 74.5% | |
| Few-shot | 78.3% | 84.4% | 69.7% | 76.4% | |
| Grok | IssueLighter | 88.0% | 96.6% | 78.4% | 86.6% |
| Enhanced zero-shot | 68.8% | 93.5% | 39.4% | 55.5% | |
| Few-shot | 68.3% | 84.2% | 44.0% | 57.8% |
| Type | N | ChatGPT | Grok | Qwen | |||
|---|---|---|---|---|---|---|---|
| Acc. | F1 | Acc. | F1 | Acc. | F1 | ||
| Bug | 296 | 96.6% | 96.9% | 86.5% | 88.5% | 87.2% | 89.1% |
| Enh. | 65 | 93.8% | 89.5% | 90.8% | 85.0% | 81.5% | 73.9% |
| Doc. | 58 | 96.6% | 97.7% | 89.7% | 93.0% | 77.6% | 85.4% |
| CI | 6 | 100% | 100% | 83.3% | 90.9% | 83.3% | 90.9% |
| Inst. | 5 | 80.0% | 66.7% | 100% | 100% | 80.0% | 66.7% |
| Model | Variant | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|---|
| ChatGPT | w/o Strategy | 82.5% | 84.4% | 79.4% | 81.8% |
| w/o Strategy & Taxonomy | 76.5% | 95.2% | 55.0% | 69.8% | |
| Grok | w/o Strategy | 73.7% | 82.3% | 59.6% | 69.1% |
| w/o Strategy & Taxonomy | 67.4% | 99.9% | 33.9% | 47.8% | |
| Qwen | w/o Strategy | 74.8% | 85.9% | 58.7% | 69.8% |
| w/o Strategy & Taxonomy | 70.4% | 93.1% | 43.1% | 58.9% |
| Method | ChatGPT | Grok | Qwen | |||
|---|---|---|---|---|---|---|
| cost ($) | runtime (s) | cost ($) | runtime (s) | cost ($) | runtime (s) | |
| IssueLighter | 0.045 | 119.4 | 0.042 | 96.3 | 0.007 | 106.4 |
| Enhanced zero-shot | 0.019 | 23.3 | 0.005 | 5.8 | 0.001 | 4.6 |
| Few-shot | 0.021 | 23.6 | 0.010 | 6.0 | 0.001 | 5.0 |
| w/o Strategy | 0.044 | 110.1 | 0.039 | 82.4 | 0.006 | 102.5 |
| w/o Strategy & Taxonomy | 0.045 | 95.1 | 0.040 | 78.9 | 0.006 | 98.7 |
Software Engineering
Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.
See all 290 papers