The sprint-based iterative approach in the Agile software development method allows continuous feedback and adaptation. One of the crucial Agile software development activities is the sprint planning session where developers estimate the effort required to complete tasks through a consensus-based estimation technique such as Planning Poker. In the Agile software development method, a common unit of measuring development effort is Story Point (SP) which is assigned to tasks to understand the complexity and development time needed to complete them. Despite the benefits of this process, it is an extremely time-consuming manual process. To mitigate this issue, in this study, we investigated if this manual process can be automated using Retrieval Augmented Generation (RAG) which comprises a "Retriever" and a "Generator". We applied two embedding models - bge-large-en-v1.5, and Sentence-Transformers' all-mpnet-base-v2 on 23 open-source software projects of varying sizes and examined four key aspects: 1) how retrieval hyper-parameters influence the performance, 2) whether estimation accuracy differs across different sizes of the projects, 3) whether embedding model choice affects accuracy, and 4) how the RAG-based approach compares to the existing baselines. Although the RAG-based approach outperformed the baseline models in several occasions, our results did not exhibit statistically significant differences in performance across the projects or across the embedding models. This highlights the need for further studies and refinement of the RAG, and model adaptation strategies for better accuracy in automatically estimating user stories.
Figures & tables
Figure 1. Methodology
Project Group
Top_k
Temperature
Small
2
0.1
Mid
2
0.1
Large
4
0.0
Table 1. Best parameter combination across project size groups with BAAI.
Project Group
Top_k
Temperature
Small
4
0.1
Mid
2
0.1
Large
4
0.2
Table 2. Best parameter combination across project size groups with SBERT.
Size
Project Name
# Tasks
Small
Daemon
205
Sawtooth
206
Confluence Cloud
234
Alloy
241
Compass
260
CLI
293
Table 3. Project sizes and task count
Project Group
Mean MAE
Standard Deviation
Number of Projects
Small
1.99
1.36
12
Mid
1.67
0.62
6
Large
1.90
0.75
5
Table 4. Performance summary of BAAI across project size groups.
Project Group
Mean MAE
Standard Deviation
Number of Projects
Small
1.90
1.26
12
Mid
1.61
0.57
6
Large
1.86
0.73
5
Table 5. Performance summary of SBERT across project size groups.
Method
Alloy
CLI
Daemon
Clover
Confluence Cloud
MAE
MdAE
MAE
MdAE
MAE
MdAE
MAE
MdAE
MAE
MdAE
LHC-SE
1.84
2.00
1.87
2.00
2.81
3.00
3.88
2.00
1.34
1.00
LHC TC -SE
2.28
2.00
1.76
2.00
2.74
3.00
4.23
1.50
1.37
1.00
Deep-SE
1.51
1.28
1.76
1.30
3.29
2.00
3.78
1.05
1.48
0.93
TF-IDF
1.44
2.00
2.98
3.00
2.74
3.00
4.04
1.00
1.33
1.00
RAG-SBERT
1.76
1.00
1.48
1.00
3.17
3.00
3.82
1.25
1.30
1.00
Table 6. Test Results for LHC-SE, LHC TC -SE, Deep-SE, TF-IDF, and RAG on Tawosi dataset
Method
Small
Mid-sized
Large
LHC-SE
LHC TC -SE
Deep-SE
TF-IDF
LHC-SE
LHC TC -SE
Deep-SE
TF-IDF
LHC-SE
LHC TC -SE
Deep-SE
TF-IDF
RAG-SBERT
7
5
5
4
3
3
3
4
1
1
1
1
RAG-BAAI
4
4
5
3
3
3
2
4
1
1
1
1
Table 7. Number of times RAG performed better compared to baseline methods
Baseline Method
Small Projects
Medium Projects
Large Projects
LHC-SE
0.7402
0.4219
0.9062
LHCtc-SE
0.8970
0.4219
0.8438
Deep-SE
0.8833
0.5781
0.9062
TF-IDF
0.9119
0.1562
0.7812
Table 8. Wilcoxon Signed-Rank Test Results: RAG-BAAI vs Baseline Methods
Baseline Method
Small Projects
Medium Projects
Large Projects
LHC-SE
0.6499
0.2812
0.9375
LHCtc-SE
0.8494
0.2812
0.8438
Deep-SE
0.6963
0.3438
0.9375
TF-IDF
0.9126
0.1562
0.7812
Table 9. Wilcoxon Signed-Rank Test Results: RAG-SBERT vs Baseline Methods
Department of Artificial Intelligence, Faculty of Information and Communication Technology, Wrocław University of Science and Technology, Wrocław 50-370, Poland