Agile Story-Point Estimation: Is RAG a Better Way to Go?

Lamyea Maha; Tajmilur Rahman; Chanchal Roy

cs.SEApr 3, 2026

Agile Story-Point Estimation: Is RAG a Better Way to Go?

Authors: Lamyea Maha, Tajmilur Rahman, Chanchal Roy

Organizations: University of Saskatchewan, Saskatoon, SK, Canada · Gannon University, Erie, PA, USA

Abstract

The sprint-based iterative approach in the Agile software development method allows continuous feedback and adaptation. One of the crucial Agile software development activities is the sprint planning session where developers estimate the effort required to complete tasks through a consensus-based estimation technique such as Planning Poker. In the Agile software development method, a common unit of measuring development effort is Story Point (SP) which is assigned to tasks to understand the complexity and development time needed to complete them. Despite the benefits of this process, it is an extremely time-consuming manual process. To mitigate this issue, in this study, we investigated if this manual process can be automated using Retrieval Augmented Generation (RAG) which comprises a "Retriever" and a "Generator". We applied two embedding models - bge-large-en-v1.5, and Sentence-Transformers' all-mpnet-base-v2 on 23 open-source software projects of varying sizes and examined four key aspects: 1) how retrieval hyper-parameters influence the performance, 2) whether estimation accuracy differs across different sizes of the projects, 3) whether embedding model choice affects accuracy, and 4) how the RAG-based approach compares to the existing baselines. Although the RAG-based approach outperformed the baseline models in several occasions, our results did not exhibit statistically significant differences in performance across the projects or across the embedding models. This highlights the need for further studies and refinement of the RAG, and model adaptation strategies for better accuracy in automatically estimating user stories.

Figures & tables

Project Group	Top_k	Temperature
Small	2	0.1
Mid	2	0.1
Large	4	0.0

Table 1. Best parameter combination across project size groups with BAAI.

Project Group	Top_k	Temperature
Small	4	0.1
Mid	2	0.1
Large	4	0.2

Table 2. Best parameter combination across project size groups with SBERT.

Size	Project Name	# Tasks
Small	Daemon	205
	Sawtooth	206
	Confluence Cloud	234
	Alloy	241
	Compass	260
	CLI	293

Table 3. Project sizes and task count

Project Group	Mean MAE	Standard Deviation	Number of Projects
Small	1.99	1.36	12
Mid	1.67	0.62	6
Large	1.90	0.75	5

Table 4. Performance summary of BAAI across project size groups.

Project Group	Mean MAE	Standard Deviation	Number of Projects
Small	1.90	1.26	12
Mid	1.61	0.57	6
Large	1.86	0.73	5

Table 5. Performance summary of SBERT across project size groups.

Method	Alloy		CLI		Daemon		Clover		Confluence Cloud
Method	MAE	MdAE	MAE	MdAE	MAE	MdAE	MAE	MdAE	MAE	MdAE
LHC-SE	1.84	2.00	1.87	2.00	2.81	3.00	3.88	2.00	1.34	1.00
LHC ${}_{\text{TC}}$ -SE	2.28	2.00	1.76	2.00	2.74	3.00	4.23	1.50	1.37	1.00
Deep-SE	1.51	1.28	1.76	1.30	3.29	2.00	3.78	1.05	1.48	0.93
TF-IDF	1.44	2.00	2.98	3.00	2.74	3.00	4.04	1.00	1.33	1.00
RAG-SBERT	1.76	1.00	1.48	1.00	3.17	3.00	3.82	1.25	1.30	1.00

Table 6. Test Results for LHC-SE, LHC

{}_{\text{TC}}

-SE, Deep-SE, TF-IDF, and RAG on Tawosi dataset

Method	Small				Mid-sized				Large
Method	LHC-SE	LHC ${}_{\text{TC}}$ -SE	Deep-SE	TF-IDF	LHC-SE	LHC ${}_{\text{TC}}$ -SE	Deep-SE	TF-IDF	LHC-SE	LHC ${}_{\text{TC}}$ -SE	Deep-SE	TF-IDF
RAG-SBERT	7	5	5	4	3	3	3	4	1	1	1	1
RAG-BAAI	4	4	5	3	3	3	2	4	1	1	1	1

Table 7. Number of times RAG performed better compared to baseline methods

Baseline Method	Small Projects	Medium Projects	Large Projects
LHC-SE	0.7402	0.4219	0.9062
LHCtc-SE	0.8970	0.4219	0.8438
Deep-SE	0.8833	0.5781	0.9062
TF-IDF	0.9119	0.1562	0.7812

Table 8. Wilcoxon Signed-Rank Test Results: RAG-BAAI vs Baseline Methods

Baseline Method	Small Projects	Medium Projects	Large Projects
LHC-SE	0.6499	0.2812	0.9375
LHCtc-SE	0.8494	0.2812	0.8438
Deep-SE	0.6963	0.3438	0.9375
TF-IDF	0.9126	0.1562	0.7812

Table 9. Wilcoxon Signed-Rank Test Results: RAG-SBERT vs Baseline Methods

Explore similar work

Apr 15, 2026

Cognitive Offloading in Agile Teams: How Artificial Intelligence Reshapes Risk Assessment and Planning Quality

Recent advances in artificial intelligence (AI) have shown promise in automating key aspects of Agile project management, yet their impact on team cognition remains underexplored. In this work, we investigate cognitive offloading in Agile sprint planning by conducting a controlled, three-condition experiment comparing AI-only, human-only, and hybrid planning models on a live client deliverable at a mid-sized digital agency. Using quantitative metrics -- including estimation accuracy, rework rates, and scope change recovery time -- alongside qualitative indicators of planning robustness, we evaluate each model's effectiveness beyond raw efficiency. We find that while AI-only planning minimizes time and cost, it significantly degrades risk capture rates and increases rework due to unstated assumptions, whereas human-only planning excels at adaptability but incurs substantial overhead. Drawing on these findings, we propose a theoretical framework for hybrid AI-human sprint planning that assigns algorithmic tools to estimation and backlog formatting while mandating human deliberation for risk assessment and ambiguity resolution. Our results challenge the assumption that efficiency equates to effectiveness, offering actionable governance strategies for organizations seeking to augment rather than erode team cognition.

Adriana Caraeni, Alexander Shick, Andrew Lan

University of Massachusetts Amherst

Cognitive Offloading Human-AI Collaboration Human-AI Decision Making Software Engineering AI Risk Management

Ask PDF

May 8, 2026

The AI-Native Large-Scale Agile Software Development Manifesto

Despite the widespread adoption of agile methods, achieving true agility at scale remains elusive. Large-scale agile frameworks remain largely human-centric and manual, relying on coordination meetings, artifact synchronization, and role-based handoffs that inhibit real-time adaptation. Meanwhile, rapid advances in AI, particularly large language models, have begun transforming software engineering, yet their potential for organizational-level agility remains underexplored. We present the AI-Native Large-Scale Agile Software Development Manifesto: a set of values and principles that redefine how large-scale software development is organized when AI becomes a first-class participant rather than a peripheral tool. The manifesto is grounded in six principles, parallel processes, intent-driven teams, living knowledge, verification-first assurance, orchestrated agent workforces, and reusable blueprints, that together shift development from a meeting-driven, document-heavy, sequential process to an intelligent, adaptive, continuously learning system.

Ricardo Britto, Fredrik Palmgren, Nishrith Saini +1

Ericsson, Sweden · Blekinge Institute of Technology, Sweden

Software Engineering Software Engineering Agents Human-AI Collaboration AI-Assisted Software Development

Ask PDF

May 30, 2026

Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitations

Retrieval-Augmented Generation (RAG) has demonstrated significant capabilities in enhancing the performance of Large Language Models (LLMs). One of the key tasks in RAG systems is the chunking process. Traditionally, fixed-size chunking and semantic chunking have been the standard approaches. However, interest in chunking strategies has been increasing, leading to a growing number of proposed methods that often claim improved performance over these conventional techniques. Many of these approaches are tailored to specific use cases and data types, with limited evidence of their effectiveness across diverse scenarios. As a result, it remains challenging to directly compare different techniques and assess their relative strengths. To the best of our knowledge, this study is the first to systematically evaluate the effectiveness of a wide range of chunking methods and emphasize the underlying challenges of chunking strategies in RAG systems. While chunking is commonly treated as a simple preprocessing step, we show that it introduces a range of impactful and often overlooked issues.

Mateusz Śmigielski, Michał Rajkowski, Mateusz Zbrocki +5

Department of Artificial Intelligence, Faculty of Information and Communication Technology, Wrocław University of Science and Technology, Wrocław 50-370, Poland

Document Chunking Retrieval-Augmented Generation RAG Evaluation

Ask PDF