Invalid issue reports are prevalent in open source software (OSS) projects, imposing substantial burdens on maintainers who already face a high risk of burnout. Prior research has primarily focused on identifying invalid reports for specific issue categories (e.g., bug and wontfix), leaving the broader landscape of OSS issue types largely unexplored and limiting the generalizability of existing approaches. Moreover, existing methods provide little support for explaining or validating their predictions, reducing their practical usefulness for maintainers. With the rise of LLM-based agents, it remains unclear whether a comprehensive understanding of the causes of invalidity across all issue types can improve validity assessment performance while producing well-grounded rationales. To this end, we first manually annotated the validity status of 520 issue reports from three OSS projects, i.e., pandas, scikit-learn, and mypy, resulting in 257 invalid reports and 263 valid ones. Then, we conducted a thematic analysis on these invalid reports and identified 13 causes of invalidity that are closely specific to issue types. Based on these insights, we propose IssueLighter, an agent-based framework with a type-aware validity assessment strategy that dynamically executes commands. Experimental results show that IssueLighter consistently and substantially outperforms all baseline methods, with an average F1 improvement of over 20 percentage points. Ablation studies confirm that both the type-aware strategy and the taxonomy of causes of invalidity contribute to the performance gains. Qualitative analysis further demonstrates that IssueLighter generates rationales grounded in accurately identified causes of invalidity and supported by evidence. Moreover, leveraging these rationales, we challenged maintainers' original decisions in four disputed cases, and all received approval.
Figures & tables
Project
Total
Valid
Invalid
Invalid Ratio
pandas
209
117
92
44.0%
scikit-learn
118
56
62
52.5%
mypy
193
90
103
53.4%
Total
520
263
257
49.4%
TABLE I: Dataset Overview
Cause
Explanation
pd
sk
mp
User misunderstanding
The issue stems from user misunderstanding of the project’s expected behavior, such as intentional code behavior, correct documentation, or reasonable project configuration.
26
15
56
Unable reproduction
The issue cannot be reproduced based on the information provided in the report.
11
5
10
Misplaced report
The issue stems from a third-party project, not in the project itself.
12
4
8
Out of date
The issue has been fixed in subsequent versions.
4
1
8
Inherent limitation
Due to inherent constraints in the project’s technology stack, the issue cannot be resolved.
1
2
5
Existing alternative
The request can already be achieved through other means.
13
9
2
TABLE II: Root Cause Taxonomy and Project Distribution of Invalid Issue Reports
Fig. 1: Correspondence Between Causes of Invalidity and Issue Types
Fig. 2: Overall Architecture of IssueLighter
Model
Method
Acc.
Prec.
Rec.
F1
ChatGPT
IssueLighter
96.2%
98.6%
93.6%
96.0%
Enhanced zero-shot
78.5%
89.7%
63.8%
74.5%
Few-shot
78.3%
84.4%
69.7%
76.4%
Grok
IssueLighter
88.0%
96.6%
78.4%
86.6%
Enhanced zero-shot
68.8%
93.5%
39.4%
55.5%
Few-shot
68.3%
84.2%
44.0%
57.8%
TABLE III: Performance Comparison of IssueLighter and Baseline Methods
Type
N
ChatGPT
Grok
Qwen
Acc.
F1
Acc.
F1
Acc.
F1
Bug
296
96.6%
96.9%
86.5%
88.5%
87.2%
89.1%
Enh.
65
93.8%
89.5%
90.8%
85.0%
81.5%
73.9%
Doc.
58
96.6%
97.7%
89.7%
93.0%
77.6%
85.4%
CI
6
100%
100%
83.3%
90.9%
83.3%
90.9%
Inst.
5
80.0%
66.7%
100%
100%
80.0%
66.7%
TABLE IV: Performance by Issue Types Across Models
Model
Variant
Acc.
Prec.
Rec.
F1
ChatGPT
w/o Strategy
82.5%
84.4%
79.4%
81.8%
w/o Strategy & Taxonomy
76.5%
95.2%
55.0%
69.8%
Grok
w/o Strategy
73.7%
82.3%
59.6%
69.1%
w/o Strategy & Taxonomy
67.4%
99.9%
33.9%
47.8%
Qwen
w/o Strategy
74.8%
85.9%
58.7%
69.8%
w/o Strategy & Taxonomy
70.4%
93.1%
43.1%
58.9%
TABLE V: Ablation Study Results
Method
ChatGPT
Grok
Qwen
cost ($)
runtime (s)
cost ($)
runtime (s)
cost ($)
runtime (s)
IssueLighter
0.045
119.4
0.042
96.3
0.007
106.4
Enhanced zero-shot
0.019
23.3
0.005
5.8
0.001
4.6
Few-shot
0.021
23.6
0.010
6.0
0.001
5.0
w/o Strategy
0.044
110.1
0.039
82.4
0.006
102.5
w/o Strategy & Taxonomy
0.045
95.1
0.040
78.9
0.006
98.7
TABLE VI: Average Cost ($) and Runtime (s) per Issue Report
Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes, and are resolved with a no-code fix. Manually determining the root cause of the invalid bug reports and providing actionable resolutions by the customer support causes a serious waste of resources. Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification, and perform experiments to test the accuracy of various approaches on invalid subclassification and no-code fix generation. We study how different configurations perform on a gold-standard benchmark we have created. Using a manually curated benchmark for higher quality analysis, we experimented with vanilla LLMs, Retrieval Augmented Generation, and agentic web search to identify invalid subclasses and generate no-code fixes. We evaluated the results against manually labeled ground truth data that includes the invalid subclass and no-code fixes from the original bug reports. We measured subclass detection performance with weighted F1-Score, and assessed no-code fix suggestions using BERTScore and Judge LLM success rates. For subclassification, retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64. At the subclass level, performance peaks at 0.85 F1 for Non-reproducibility and 0.79 for Feature Request and Question, while Wrong Version remains the most challenging with scores between 0.00 and 0.29. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.9%, compared to 64.4% for RAG applications and 64.9% for vanilla LLMs, with subclass-level peaks of 87.4% for Working as Designed and 72.2% for Question.
Mahmut Furkan Gon, Emre Dinc, Tevfik Emre Sungur +1
All authors are with the Department of Computer Engineering, Bilkent University, Ankara, Türkiye.
Repository-level issue resolution benchmarks have become a standard testbed for evaluating LLM-based agents, yet success is still predominantly measured by test pass rates. In practice, however, acceptable patches must also comply with project-specific design constraints, such as architectural conventions, error-handling policies, and maintainability requirements, which are rarely encoded in tests and are often documented only implicitly in code review discussions. This paper introduces design-aware issue resolution and presents \bench{}, a benchmark that makes such implicit design constraints explicit and measurable. \bench{} is constructed by mining and validating design constraints from real-world pull requests, linking them to issue instances, and automatically checking patch compliance using an LLM-based verifier, yielding 495 issues and 1,787 validated constraints across six repositories, aligned with SWE-bench-Verified and SWE-bench-Pro. Experiments with state-of-the-art agents show that test-based correctness substantially overestimates patch quality: fewer than half of resolved issues are fully design-satisfying, design violations are widespread, and functional correctness exhibits negligible statistical association with design satisfaction. While providing issue-specific design guidance reduces violations, substantial non-compliance remains, highlighting a fundamental gap in current agent capabilities and motivating design-aware evaluation beyond functional correctness.
Kai Yu, Zhenhao Zhou, Junhao Zeng +8
Fudan University, China · Nanyang Technological University, Singapore
Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descriptions enhances developer productivity by simplifying root cause analysis, promotes test-driven development -- "test first, write code later", and can be used for improving the effectiveness of automated issue resolution systems like coding agents. Existing methods proposed for this task predominantly rely on closed-source LLMs, with limited exploration of open models. To address this, we propose SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests. First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories and use it to train LLMs of varying sizes and families. The fine-tuned models achieve absolute improvements of up to 10% in success rate and 21% in change coverage on SWT-Bench Verified. Further analysis shows consistent improvements with increased inference-time compute, more data, and larger models. These results highlight the effectiveness of our framework for advancing open-source LLMs in this domain.
Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava +2
Carnegie Mellon University · Nutanix · Stanford University