Automating the Detection of Requirement Dependencies Using Large Language Models
Organizations: University of Ottawa, Ottawa, Canada. · École de technologie supérieure, Montreal, Canada. · Research Ireland Lero Centre for Software Research, University of Limerick, Ireland. · General Motors, Detroit, Michigan, United States.
Abstract
Requirements are inherently interconnected through various types of dependencies. Identifying these dependencies is essential, as they underpin critical decisions and influence a range of activities throughout software development. However, this task is challenging, particularly in modern software systems, given the high volume of complex, coupled requirements. These challenges are further exacerbated by the ambiguity of Natural Language (NL) requirements and their constant change. Consequently, requirement dependency detection is often overlooked or performed manually. Large Language Models (LLMs) exhibit strong capabilities in NL processing, presenting a promising avenue for requirement-related tasks. While they have shown to enhance various requirements engineering tasks, their effectiveness in identifying requirement dependencies remains unexplored. In this paper, we introduce LEREDD, an LLM-based approach for automated detection of requirement dependencies that leverages Retrieval-Augmented Generation (RAG) and In-Context Learning (ICL). It is designed to identify diverse dependency types directly from NL requirements. We empirically evaluate LEREDD against two state-of-the-art baselines. The results show that LEREDD provides highly accurate classification of dependent and non-dependent requirements, achieving an accuracy of 0.93, and an F1 score of 0.84, with the latter averaging 0.96 for non-dependent cases. LEREDD outperforms zero-shot LLMs and baselines, particularly in detecting fine-grained dependency types, where it yields average relative gains of 94.87% and 105.41% in F1 scores for the Requires dependency over the baselines. We also provide an annotated dataset of requirement dependencies encompassing 813 requirement pairs across three distinct systems to support reproducibility and future research.
Figures & tables
| Dependency Type | Definition | Example |
|---|---|---|
| Requires | if the fulfillment of one requirement is a prerequisite to the fulfillment of the other requirement. | “If a car in an oncoming lane is within 500 feet of the ADB system, or a car in the same lane is within 250 feet, the subsystem shall decrease light intensity by 50%.” requires “The subsystem shall check for cars and road elements every 10 milliseconds and determine whether the subsystem shall decrease light intensity.” |
| Implements | if one is a higher-level requirement (e.g., a system or subsystem level requirement) that is fulfilled by the other lower-level requirement (e.g., a subsystem or component level requirement). | “The system shall verify communication between subsystems.” implements “The subsystem shall have checks to mitigate the chances of a cybersecurity attack.” |
| Conflicts | if the fulfillment of one requirement restricts the fulfillment of the other requirement. | “The system shall not irritate the driver with more than 3 updates or alerts in 10 minutes for the system.” conflicts with “The system shall alert the driver if any malfunctions or errors have been detected within 5 seconds.” |
| Details | if both requirements describe the same action under the same condition, and one requirement provides additional details specifically regarding the shared action. | “The system shall respond to the surroundings around it in less than 2.5 seconds by dimming the light intensity 50%.” details “The system shall gradually change the light intensity to ensure a smooth operation for the driver by decreasing the intensity of the light by no greater than 50% in one update.” |
| Is similar | if one requirement replicates partially or totally the content of the other requirement, resulting in redundancy. | “The system shall authenticate updates.” is similar to “The system shall verify the source of the update.” |
| Dependency Type | ADB | TJA | APA | Total | |
|---|---|---|---|---|---|
| Pilot study | Evaluation | ||||
| # Requirements | 100 | 313 | 200 | 200 | 813 |
| Conflicts | 1 | 10 | - | 4 | 15 |
| Details | 2 | 13 | 2 | 1 | 18 |
| Implements | 6 | 18 | 10 | 3 | 37 |
| Is similar | - | 3 | 1 | 3 | 7 |
| LLM | Dependency | D1 | D2 | D3 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Type | Acc | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | |
| GPT | No Dep | 0.78 0.01 | 0.89 0.01 | 0.88 0.00 | 0.88 0.00 | 0.82 0.00 | 0.96 0.00 | 0.88 0.00 | 0.92 0.00 | 0.66 0.01 | 0.93 0.00 | 0.71 0.02 | 0.80 0.01 |
| Requires | 0.36 0.01 | 0.44 0.03 | 0.40 0.02 | 0.39 0.04 | 0.61 0.06 | 0.47 0.05 | 0.46 0.02 | 0.62 0.04 | 0.53 0.01 | ||||
| Implements | 0.57 0.08 | 0.64 0.07 | 0.60 0.08 | 0.33 0.05 | 0.40 0.00 | 0.36 0.03 | 0.07 0.01 | 0.56 0.16 | 0.13 0.03 | ||||
| Conflicts | 0.47 0.05 | 0.22 0.00 | 0.30 0.01 | N/A | N/A | N/A | 0.67 0.47 | 0.17 0.12 | 0.27 0.19 | ||||
| Details | 0.25 0.00 | 0.20 0.00 | 0.22 0.00 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 | 0.00 0.00 | ||||
| Examples | BGE | SBERT | ||||||
|---|---|---|---|---|---|---|---|---|
| Cosine | Euclidean | Cosine | Euclidean | |||||
| Avg | Max | Avg | Max | Avg | Max | Avg | Max | |
| 1 | 0.84 0.01 | 0.83 0.02 | 0.81 0.01 | 0.83 0.00 | 0.80 0.01 | 0.81 0.01 | 0.82 0.02 | 0.81 0.02 |
| 2 | 0.88 0.01 | 0.87 0.01 | 0.85 0.01 | 0.88 0.01 | 0.87 0.01 | 0.91 0.01 | 0.88 0.02 | 0.88 0.01 |
| 3 | 0.91 0.01 | 0.89 0.01 | 0.88 0.02 | 0.89 0.00 | 0.89 0.00 | 0.92 0.01 | 0.90 0.00 | 0.90 0.02 |
| 4 | 0.90 0.00 | 0.87 0.00 | 0.91 0.00 | 0.90 0.02 | 0.91 0.00 | 0.91 0.00 | 0.89 0.01 | 0.88 0.01 |
| Metric | 2 Chunks | 6 Chunks | 10 Chunks | Entire Doc | |||
|---|---|---|---|---|---|---|---|
| 500c | 1000c | 500c | 1000c | 500c | 1000c | ||
| Acc | 0.96 0.00 | 0.96 0.01 | 0.97 0.00 | 0.97 0.00 | 0.96 0.01 | 0.97 0.00 | 0.96 0.01 |
| P | 0.96 0.00 | 0.97 0.00 | 0.96 0.00 | 0.97 0.00 | 0.96 0.00 | 0.97 0.00 | 0.96 0.00 |
| R | 0.96 0.00 | 0.96 0.01 | 0.97 0.00 | 0.98 0.00 | 0.95 0.01 | 0.97 0.00 | 0.95 0.01 |
| F1 | 0.96 0.00 | 0.96 0.00 | 0.97 0.00 | 0.97 0.00 | 0.96 0.00 | 0.97 0.00 | 0.96 0.00 |
| Dataset | Dependency | Zero-shot | Few-shot | Few-shot & RAG | Supp | |||||||||
| Type | Acc | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | ||
| D1 | No Dep | 0.75 0.00 | 0.87 0.00 | 0.86 0.00 | 0.86 0.00 | 0.91 0.00 | 0.93 0.00 | 0.97 0.00 | 0.95 0.00 | 0.91 0.00 | 0.93 0.00 | 0.97 0.00 | 0.95 0.00 | 229 |
| Requires | 0.34 0.02 | 0.40 0.02 | 0.37 0.01 | 0.91 0.00 | 0.73 0.01 | 0.81 0.01 | 0.90 0.01 | 0.73 0.01 | 0.81 0.00 | 40 | ||||
| Implements | 0.60 0.03 | 0.70 0.05 | 0.64 0.01 | 0.73 0.02 | 0.80 0.03 | 0.76 0.02 | 0.75 0.01 | 0.87 0.03 | 0.80 0.01 | 18 | ||||
| Details | 0.62 0.09 | 0.41 0.10 | 0.49 0.08 | 1.00 0.00 | 0.79 0.04 | 0.89 0.02 | 0.94 0.05 | 0.74 0.04 | 0.83 0.04 | 10 | ||||
| Conflicts | 0.53 0.05 | 0.30 0.00 | 0.38 0.01 | 0.86 0.00 | 0.60 0.00 | 0.71 0.00 | 0.94 0.08 | 0.50 0.00 | 0.65 0.02 | 13 | ||||
| Dataset | Confidence score | Zero-shot | Few-shot | Few-shot & RAG |
|---|---|---|---|---|
| D1 | 5 | 59% | 97% | 95% |
| 4 | 36% | 3% | 4% | |
| D2 | 5 | 63% | 98% | 96% |
| 4 | 36% | 2% | 4% | |
| D3 | 5 | 35% | 98% | 97% |
| 4 | 58% | 2% | 2% |
| System | Dependency | TF-IDF & LSA | fine-tuned BERT | LEREDD | Supp | |||||||||
| Type | Acc | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | ||
| D1 | No Dep | 0.68 | 0.88 | 0.73 | 0.80 | 0.87 | 0.93 | 0.93 | 0.93 | 0.88 0.01 | 0.92 0.01 | 0.94 0.02 | 0.93 0.01 | 48 |
| Requires | 0.19 | 0.38 | 0.25 | 0.86 | 0.75 | 0.80 | 0.83 0.02 | 0.62 0.10 | 0.71 0.08 | 8 | ||||
| Implements | N/A | N/A | N/A | 0.50 | 0.67 | 0.57 | 0.70 0.07 | 1.00 0.00 | 0.82 0.05 | 3 | ||||
| Details | N/A | N/A | N/A | 0.67 | 0.67 | 0.67 | 1.00 0.00 | 0.67 0.00 | 0.80 0.00 | 3 | ||||
| Conflicts | N/A | N/A | N/A | 0.50 | 0.50 | 0.50 | 0.44 0.08 | 0.50 0.00 | 0.47 0.05 | 2 | ||||
| Training | Testing | Dependency | TF-IDF & LSA | fine-tuned BERT | LEREDD | LEREDD with RAG | Supp | ||||||||||||
| Type | Acc | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | |||
| D2 | D1 | No Dep | 0.70 | 0.86 | 0.76 | 0.81 | 0.67 | 0.84 | 0.79 | 0.82 | 0.84 0.01 | 0.86 0.00 | 0.98 0.00 | 0.92 0.00 | 0.86 0.00 | 0.87 0.01 | 0.98 0.00 | 0.92 0.00 | 229 |
| Requires | 0.18 | 0.30 | 0.23 | 0.16 | 0.28 | 0.20 | 0.68 0.11 | 0.21 0.03 | 0.32 0.05 | 0.83 0.06 | 0.27 0.01 | 0.40 0.01 | 40 | ||||||
| Implements | N/A | N/A | N/A | 0.00 | 0.00 | 0.00 | 0.64 0.04 | 0.48 0.03 | 0.55 0.03 | 0.72 0.03 | 0.67 0.05 | 0.69 0.03 | 18 | ||||||
| Overall | 0.76 | 0.69 | 0.72 | 0.69 | 0.67 | 0.68 | 0.82 0.02 | 0.84 0.01 | 0.81 0.01 | 0.85 0.01 | 0.86 0.01 | 0.83 0.01 | 287 | ||||||
| D3 | D1 | No Dep | 0.70 | 0.86 | 0.76 | 0.81 | 0.35 | 0.88 | 0.28 | 0.42 | 0.89 0.00 | 0.91 0.00 | 0.97 0.00 | 0.94 0.00 | 0.90 0.00 | 0.91 0.00 | 0.98 0.00 | 0.94 0.00 | 229 |