App store reviews provide a constant flow of real user feedback that can help improve software requirements. However, these reviews are often messy, informal, and difficult to analyze manually at scale. Although automated techniques exist, many do not perform well when replicated and often fail to produce clean, backlog-ready user stories for agile projects. In this study, we evaluate how well large language models (LLMs) such as GPT-3.5 Turbo, Gemini 2.0 Flash, and Mistral 7B Instruct can generate usable user stories directly from raw app reviews. Using the Mini-BAR dataset of 1,000+ health app reviews, we tested zero-shot, one-shot, and two-shot prompting methods. We evaluated the generated user stories using both human judgment (via the RUST framework) and a RoBERTa classifier fine-tuned on UStAI to assess their overall quality. Our results show that LLMs can match or even outperform humans in writing fluent, well-formatted user stories, especially when few-shot prompts are used. However, they still struggle to produce independent and unique user stories, which are essential for building a strong agile backlog. Overall, our findings show how LLMs can reliably turn unstructured app reviews into actionable software requirements, providing developers with clear guidance to turn user feedback into meaningful improvements.
Figures & tables
Figure 1: Overview of the proposed methodology.
Evaluator
Role
Agile Experience
Region
E1
Senior Project Manager
6+
Europe
E2
DevOps Engineer
3+
Asia
E3
Systems Engineer
3+
Europe
Table 1: Demographic Information of Human Evaluators
Figure 2: Canonical and non-canonical user stories produced by different LLMs under zero-, one-, and two-shot prompting.
Quality Criterion
Human
Gemini 2.0 Flash
GPT-3.5 Turbo
Mistral 7B Instruct
Two-Shot
One-Shot
Zero-Shot
Two-Shot
One-Shot
Zero-Shot
Two-Shot
One-Shot
Zero-Shot
Atomic
0.76
0.83
0.82
0.82
0.82
0.82
0.82
0.82
0.82
0.82
Minimal
0.83
0.94
0.94
0.94
0.94
0.94
0.94
0.94
0.94
0.94
Conceptually Sound
0.72
0.81
0.81
0.80
0.82
0.82
0.81
0.81
0.81
0.82
Problem-Oriented
0.69
0.78
0.78
0.77
0.79
0.78
0.78
0.77
0.77
0.78
Unambiguous
0.56
0.60
0.60
0.58
0.59
0.60
0.59
0.60
0.60
0.57
Table 2: Automated Quality Evaluation Scores Across Models and Prompt Strategies
Figure 3: Overall user-story quality scores for different LLMs under varying prompt strategies.
Sample1 - Sample2
Test Statistic
Std. Error
Std. Test Statistic ( z )
Sig.
Adj. Sig.
Zero-Shot vs One-Shot
-1,860.06
67.086
-27.727
< 0.0001
< 0.0001
Zero-Shot vs Two-Shot
-2,138.46
67.086
-31.877
< 0.0001
< 0.0001
One-Shot vs Two-Shot
-278.396
67.086
-4.15
< 0.0001
< 0.0001
Table 3: Pairwise Comparison of Prompting Strategies
Source
Mean Score
Std. Dev.
N Ratings
Gemini 2.0 Flash (Two-Shot)
4.5265
0.6359
400
GPT-3.5 Turbo (Two-Shot)
4.4903
0.5949
400
GPT-3.5 Turbo (One-Shot)
4.4492
0.6384
400
Agile Expert
4.3879
0.6963
400
Table 4: Overall RUST Evaluation Scores by Source
Figure 4: Dimension-wise RUST evaluation scores by source.
User stories are one of the most widely used artifacts in the software industry to define functional requirements. In parallel, the use of high-fidelity mockups facilitates end-user participation in defining their needs. In this work, we explore how combining these techniques with large language models (LLMs) enables agile and automated generation of user stories from mockups. To this end, we present a case study that analyzes the ability of LLMs to extract user stories from high-fidelity mockups, both with and without the inclusion of a glossary of the Language Extended Lexicon (LEL) in the prompts. Our results demonstrate that incorporating the LEL significantly enhances the accuracy and suitability of the generated user stories. This approach represents a step forward in the integration of AI into requirements engineering, with the potential to improve communication between users and developers.
Diego Firmenich, Leandro Antonelli, Bruno Pazos +2
Departamento de Informática, Facultad de Ingeniería, Universidad Nacional de la Patagonia, Argentina · LIFIA, Facultad de Informática, Universidad Nacional de La Plata, Argentina · CAETI, Facultad de Tecnología Informática - Universidad Abierta Interamericana +3
As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause costly rework. However, existing benchmarks typically evaluate implementation from complete specifications or underspecified requests, overlooking pre-coding requirement clarification. Prior clarification studies mainly focus on function-level tasks. It therefore remains unclear how well LLMs can transform vague intent into clear requirements through interaction and how this capability affects subsequent code generation. We introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. It contains 341 tasks spanning game, miniapp, and web development, each with a vague user intent, a human-validated requirement document, and code evaluation artifacts. We employ LLM-based user simulators with diverse personas derived from real-world conversations. We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality. Results show that stronger clarification effectiveness does not necessarily translate into higher document quality, and performance varies substantially across user personas. We also find that coding-agent harnesses do not further improve requirement document quality compared with their base LLMs. These findings offer insights for building LLM-based end-to-end software development systems.
Mingyu Chen, Zhiqing Zhong, Manyi Wang +3
The Chinese University of Hong Kong, Shenzhen, Guangdong, China
User feedback is crucial for the evolution of mobile apps. However, research suggests that users tend to submit uninformative, vague, or destructive feedback. Unlike recent AI4SE approaches that focus on generating code and other development artifacts, our work aims at empowering users to submit better and more constructive UI feedback with concrete suggestions on how to improve the app. We propose LikeThis!, a GenAI-based approach that takes a user comment with the corresponding screenshot to immediately generate multiple improvement alternatives, from which the user can easily choose their preferred option. To evaluate LikeThis!, we first conducted a model benchmarking study based on a public dataset of carefully critiqued UI designs. The results show that GPT-Image-1 significantly outperformed three other state-of-the-art image generation models in improving the designs to address UI issues while keeping the fidelity and without introducing new issues. An intermediate step in LikeThis! is to generate a solution specification before sketching the design as a key to achieving effective improvement. Second, we conducted a user study with 10 production apps, where 15 users used LikeThis! to submit their feedback on encountered issues. Later, the developers of the apps assessed the understandability and actionability of the feedback with and without generated improvements. The results show that our approach helps generate better feedback from both user and developer perspectives, paving the way for AI-assisted user-developer collaboration.