Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale
Organizations: School of Computer Science, Wuhan University, China · School of Computing Technologies, RMIT University, Australia · School of Computer Science, Central China Normal University, China
Abstract
As the complexity of Software Engineering (SE) tasks continues to escalate, Multi-Agent Systems (MASs) have emerged as a focal point of research and practice due to their autonomy and scalability. Furthermore, through leveraging the reasoning and planning capabilities of Large Language Models (LLMs), the application of LLM-based MASs in the field of SE is garnering increasing attention. However, there is no dedicated study that systematically explores the design of LLM-based MASs, including the Quality Attributes (QAs) on which designers mainly focus, the design patterns used by designers, and the rationale guiding the design of LLM-based MASs for SE tasks. To this end, we conducted a study to identify the QAs that LLM-based MASs for SE tasks focus on, the design patterns used in the MASs, and the design rationale for the MASs. We collected 106 papers on LLM-based MASs for SE tasks as the source. Our study shows that: (1) Code Generation is the most common SE task solved by LLM-based MASs among ten identified SE tasks, (2) Functional Suitability is the QA on which designers of LLM-based MASs pay the most attention, (3) Role-Based Cooperation is the design pattern most frequently employed among 16 patterns used to construct LLM-based MASs, and (4) Improving the Quality of Generated Code is the most common rationale behind the design of LLM-based MASs. Building on the study results, we presented the implications for the design of LLM-based MASs to support SE tasks.
Figures & tables
| # | Our Study | Review ( He et al., 2025 ) | Review ( Liu et al., 2026 ) | Review ( Wang et al., 2025b ) |
|---|---|---|---|---|
| Study Type | Empirical study | Systematic review and case studies | Comprehensive survey | Literature survey |
| agent architecture | LLM-based MASs for SE | LLM-based MASs for SE | LLM-based agents for SE | LLM-based agents for SE |
| Main Scope | Quality-driven design patterns and rationale of LLM-based MASs | LLM-based MAS applications across SDLC stages and research agenda | Design and use of LLM-based agents across software engineering tasks | Perception, memory, and action of LLM-based agents |
| SE Tasks | Taxonomy of SE tasks addressed by LLM-based MASs | SDLC-based taxonomy | SDLC-based taxonomy | SE tasks are used as examples to illustrate agent capabilities |
| Quality Attributes | Taxonomy of quality attributes based on ISO/IEC 25010:2023 | Coverage of robustness, reliability, scalability, efficiency, maintainability, and privacy | Coverage of robustness, security, efficiency, and reusability | Coverage of efficiency, compatibility, and system integrity |
| Design Patterns | Taxonomy of design patterns used in LLM-based MASs | Agent roles, process models, and collaboration | Agent component, role, and collaboration | A component-level agent framework based on perception, memory, and action |
| Review ( He et al., 2025 ) | Review ( Liu et al., 2026 ) | Review ( Wang et al., 2025b ) | arXiv | |
|---|---|---|---|---|
| Start date | November, 2022 | No start date limit | 2020 | No start date limit |
| Cutoff date | November 14, 2024 | November 11, 2024 | June 17, 2024 | November 14, 2024 |
| # | Data Item | Description | Cardinality | RQ |
|---|---|---|---|---|
| D1 | SE Task | The SE task addressed by the LLM-based MAS proposed in a paper. | Single | RQ1 |
| D2 | Quality Attribute | The quality attribute(s) considered in the design of the LLM-based MAS proposed in a paper. | Multiple | RQ2 |
| D3 | Design Pattern | The design pattern(s) employed in the design of the LLM-based MAS proposed in a paper. | Multiple | RQ3 |
| D4 | Design Rationale | The design rationale that supports the design of the LLM-based MAS proposed in a paper. | Multiple | RQ4 |
| # | Inclusion Rule | Exclusion Rule | Boundary Case |
|---|---|---|---|
| D1 | Include the primary SE task that the LLM-based MAS is explicitly designed to address. | Do not treat auxiliary activities that merely support the primary SE task as separate SE tasks. | When multiple closely related tasks jointly constitute a higher-level SE task, code the corresponding composite task. For example, Fault Localization and Program Repair can jointly be coded as End-to-End Software Maintenance. |
| D2 | Include a QA when the paper provides explicit evidence that the corresponding quality concern is considered in the design of the LLM-based MAS. | Exclude general claims of improvement when the reported evidence is insufficient to identify a specific QA. | When the same design decision provides evidence for multiple QAs, all supported QAs can be recorded. |
| D3 | Include a design pattern when the study describes a recurring structural, interaction, coordination, or behavioral solution instantiated in the MAS design. | Exclude isolated implementation details that do not constitute a reusable design solution for organizing or coordinating the MAS. | Similar implementations are coded separately when they exhibit substantively different agent structures or interaction mechanisms. |
| D4 | Include design rationale when the study provides evidence explaining the motivation or intended benefit underlying a design decision of the LLM-based MAS. | Exclude descriptions that only state what the system does or how it is structured without indicating why the corresponding design is adopted. | When the paper provides evidence of multiple distinct motivations for the same design decision, record each motivation as design rationale for that paper. |
| Data Item | Initial Scheme | Final Scheme | Evolution |
|---|---|---|---|
| SE Task | 8 categories | 10 categories | New task categories were identified through more precise characterization of the SE tasks. |
| Quality Attribute | 9 categories, 21 subcategories | 8 categories, 19 subcategories | Categories and subcategories considered unsuitable were excluded during refinement. |
| Design Pattern | 13 categories | 16 categories | Additional recurring design patterns were identified during iterative analysis. |
| Design Rationale | 10 categories | 8 categories | Conceptually similar design rationale were merged during refinement. |
| SE Tasks | Example | Count(%) | Studies |
|---|---|---|---|
| Code Generation | To enable LLMs to handle these real-world repo-level code generation, we present CODEAGENT, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. [S12] | 49 (46.2%) | [S1] , [S7] , [S9] , [S11] , [S12] , [S15] , [S20] , [S23] , [S24] , [S27] , [S29] , [S33] , [S34] , [S36] , [S37] , [S38] , [S40] , [S41] , [S42] , [S45] , [S46] , [S51] , [S52] , [S53] , [S54] , [S58] , [S61] , [S62] , [S63] , [S64] , [S67] , [S68] , [S71] , [S73] , [S74] , [S76] , [S78] , [S79] , [S82] , [S83] , [S84] , [S86] , [S87] , [S92] , [S94] , [S96] , [S98] , [S100] , [S105] |
| End-to-End Software Development | MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. [S81] | 11 (10.4%) | [S5] , [S13] , [S21] , [S57] , [S65] , [S81] , [S93] , [S97] , [S99] , [S101] , [S106] |
| Fault Localization | To address the limitation, this paper presents AGENTFL, a multi-agent system based on ChatGPT for automated fault localization. [S4] | 10 (9.4%) | [S4] , [S16] , [S19] , [S25] , [S26] , [S35] , [S49] , [S75] , [S77] , [S103] |
| End-to-End Software Maintenance | In this paper, we introduce MarsCode Agent , a novel framework that leverages LLMs to automatically identify and repair bugs in software code. MarsCode Agent combines the power of LLMs with advanced code analysis techniques to accurately localize faults and generate patches. [S31] | 9 (8.5%) | [S2] , [S3] , [S28] , [S31] , [S32] , [S39] , [S59] , [S60] , [S104] |
| Program Repair | In this paper, we propose an automated approach for solving GitHub issues to autonomously achieve program improvement. [S8] | 8 (7.5%) | [S8] , [S14] , [S17] , [S50] , [S69] , [S70] , [S89] , [S90] |
| Software Testing | The proposed system mainly consists of three LLM-based agents responsible for action planning, state checking and parameter selecting, respectively, and two additional modules for state sensing and case rewriting. [S43] | 7 (6.6%) | [S43] , [S47] , [S66] , [S80] , [S85] , [S91] , [S102] |
| Quality Attribute | Sub_Quality Attribute | Studies |
|---|---|---|
| Functional Suitability | Functional Correctness | [S1] , [S2] , [S3] , [S4] , [S5] , [S7] , [S8] , [S9] , [S11] , [S12] , [S13] , [S14] , [S15] , [S16] , [S17] , [S19] , [S20] , [S22] , [S23] , [S24] , [S25] , [S26] , [S27] , [S28] , [S29] , [S30] , [S31] , [S32] , [S33] , [S34] , [S36] , [S38] , [S39] , [S40] , [S41] , [S42] , [S43] , [S44] , [S46] , [S47] , [S48] , [S49] , [S50] , [S51] , [S52] , [S53] , [S54] , [S55] , [S56] , [S57] , [S58] , [S59] , [S61] , [S62] , [S63] , [S64] , [S65] , [S66] , [S67] , [S68] , [S69] , [S70] , [S71] , [S72] , [S73] , [S74] , [S75] , [S76] , [S77] , [S78] , [S79] , [S80] , [S81] , [S82] , [S83] , [S84] , [S85] , [S86] , [S87] , [S88] , [S89] , [S90] , [S91] , [S92] , [S93] , [S94] , [S95] , [S96] , [S97] , [S98] , [S99] , [S100] , [S101] , [S102] , [S103] , [S104] , [S105] |
| Functional Completeness | [S6] , [S12] , [S18] , [S21] , [S24] , [S29] , [S30] , [S34] , [S44] , [S47] , [S54] , [S55] , [S62] , [S64] , [S73] , [S76] , [S79] , [S80] , [S83] , [S84] , [S85] , [S91] , [S93] , [S94] , [S97] , [S99] | |
| Functional Appropriateness | [S22] , [S34] , [S36] , [S41] , [S49] , [S55] , [S67] , [S69] , [S73] , [S93] , [S94] , [S106] | |
| Performance Efficiency | Resource Utilization | [S2] , [S4] , [S6] , [S8] , [S18] , [S19] , [S20] , [S25] , [S28] , [S35] , [S36] , [S37] , [S39] , [S40] , [S45] , [S46] , [S47] , [S48] , [S49] , [S58] , [S63] , [S64] , [S68] , [S81] , [S87] , [S88] , [S94] , [S104] , [S105] |
| Time Behavior | [S2] , [S3] , [S4] , [S8] , [S10] , [S13] , [S21] , [S22] , [S27] , [S39] , [S45] , [S46] , [S47] , [S49] , [S55] , [S56] , [S58] , [S60] , [S65] , [S73] , [S80] , [S81] , [S82] , [S87] , [S90] , [S91] , [S94] , [S104] , [S106] | |
| Capacity | [S18] , [S26] , [S38] , [S45] , [S48] , [S57] , [S59] , [S85] , [S88] , [S89] , [S94] |
| Design Pattern | Example | Count(%) | Studies |
|---|---|---|---|
| Role-Based Cooperation | Specifically, INTERVENOR employs two LLM-based agents to play different roles in code repair. [S78] | 49 (46.2%) | [S2] , [S4] , [S5] , [S7] , [S11] , [S13] , [S14] , [S16] , [S18] , [S19] , [S21] , [S26] , [S28] , [S29] , [S30] , [S31] , [S32] , [S35] , [S36] , [S37] , [S38] , [S42] , [S44] , [S46] , [S47] , [S49] , [S51] , [S55] , [S56] , [S57] , [S58] , [S59] , [S60] , [S61] , [S63] , [S64] , [S67] , [S69] , [S71] , [S78] , [S81] , [S86] , [S93] , [S94] , [S95] , [S96] , [S97] , [S99] , [S103] |
| Self-Reflection | …, in order to reduce hallucinations and inefficient planning, we apply a self-reflection mechanism. [S43] | 36 (34.0%) | [S1] , [S2] , [S3] , [S11] , [S19] , [S22] , [S24] , [S27] , [S31] , [S36] , [S37] , [S38] , [S39] , [S42] , [S43] , [S45] , [S47] , [S49] , [S53] , [S57] , [S58] , [S60] , [S62] , [S63] , [S72] , [S73] , [S74] , [S75] , [S79] , [S81] , [S87] , [S90] , [S91] , [S92] , [S100] , [S102] |
| Tool-Agent Registry | Across five tasks with Python and WikiSearch API as tools, Lemur-70B-Chat outperforms both Llama-2-70B-Chat and CodeLlama-34B-INST by large margins. [S79] | 16 (15.1%) | [S2] , [S8] , [S9] , [S12] , [S25] , [S27] , [S75] , [S76] , [S79] , [S80] , [S82] , [S85] , [S88] , [S94] , [S101] , [S104] |
| Cross-Reflection | Meanwhile, this Stderr information is provided to the questioner, who will generate a natural language description based on the Stderr. This natural language description is also appended to the dialogue messages. Next, both the natural language description and the Stderr are provided as new questions to the programmer, who will continue to modify the code. [S7] | 14 (13.2%) | [S7] , [S13] , [S26] , [S41] , [S50] , [S61] , [S65] , [S67] , [S69] , [S71] , [S74] , [S78] , [S98] , [S105] |
| Retrieval-Augmented Generation (RAG) | Furthermore, we draw inspiration from the RAG (Retrieval-Augmented Generation) approach, where we match similar content from a memory pool as additional information. [S36] | 12 (11.3%) | [S8] , [S23] , [S36] , [S45] , [S51] , [S54] , [S55] , [S64] , [S89] , [S94] , [S101] , [S104] |
| Incremental Model Querying | These patches might be from running a single SWE agent multiple times or running multiple SWE agents. [S17] | 10 (9.4%) | [S4] , [S15] , [S17] , [S98] , [S100] , [S102] , [S103] , [S104] , [S105] , [S106] |
| Design Rationale | Example | Count(%) | Studies |
|---|---|---|---|
| Improving the Quality of Generated Code | We instructed each agent to iterate with the others at least three times to refine and update the code based on feedback from the previous agents, ensuring a thorough and collaborative development process. [S13] | 51 (48.1%) | [S1] , [S5] , [S6] , [S7] , [S11] , [S12] , [S13] , [S24] , [S27] , [S29] , [S31] , [S33] , [S36] , [S40] , [S41] , [S42] , [S45] , [S46] , [S47] , [S49] , [S50] , [S51] , [S52] , [S54] , [S57] , [S58] , [S59] , [S60] , [S61] , [S62] , [S64] , [S65] , [S66] , [S70] , [S73] , [S79] , [S81] , [S83] , [S90] , [S91] , [S92] , [S93] , [S95] , [S96] , [S97] , [S98] , [S99] , [S100] , [S101] , [S103] , [S105] |
| Simulating Human Processes of Solving SE Tasks | Specifically, iAudit is inspired by the observation that expert human auditors first perceive what could be wrong and then perform a detailed analysis of the code to identify the cause. [S16] | 31 (29.2%) | [S2] , [S5] , [S7] , [S10] , [S11] , [S12] , [S14] , [S16] , [S19] , [S26] , [S28] , [S29] , [S30] , [S35] , [S42] , [S44] , [S53] , [S55] , [S58] , [S59] , [S64] , [S67] , [S71] , [S72] , [S78] , [S81] , [S86] , [S87] , [S95] , [S97] , [S99] |
| Optimizing Software Resource Management | We posit that our approach will considerably augment the field of software quality assurance, rendering it more efficient, precise, and cost-effective. [S88] | 29 (27.4%) | [S2] , [S4] , [S6] , [S8] , [S14] , [S18] , [S19] , [S20] , [S25] , [S28] , [S32] , [S34] , [S35] , [S36] , [S37] , [S40] , [S45] , [S46] , [S47] , [S48] , [S49] , [S58] , [S63] , [S68] , [S81] , [S87] , [S88] , [S104] , [S105] |
| Improving the Efficiency of Generating Software Artifacts | Moreover, our analysis of agent interactions within AGENTVERSE reveals the emergence of specific collaborative behaviors, contributing to heightened group efficiency. [S62] | 25 (23.6%) | [S2] , [S3] , [S4] , [S9] , [S13] , [S17] , [S19] , [S21] , [S27] , [S39] , [S40] , [S45] , [S49] , [S61] , [S62] , [S63] , [S64] , [S73] , [S75] , [S87] , [S88] , [S90] , [S91] , [S104] , [S106] |
| Reducing the Difficulty of Task Resolution | Specifically, by decomposing the localization process into several steps and achieving FL through the collaboration of multiple LLM-driven agents (i.e., intelligent entities that can perceive the environment, make decisions, and perform actions), the advantages we expect are twofold. [S4] | 25 (23.6%) | [S1] , [S2] , [S4] , [S13] , [S14] , [S15] , [S20] , [S22] , [S32] , [S43] , [S53] , [S56] , [S62] , [S64] , [S69] , [S72] , [S74] , [S84] , [S86] , [S93] , [S100] , [S101] , [S102] , [S104] , [S106] |
| Improving the Adaptability of Agent Systems | Our proposal for dynamic process generation aims to accommodate this variability, enabling a wider array of diverse instances to emerge and guide the development process accordingly. [S57] | 20 (18.9%) | [S1] , [S2] , [S9] , [S12] , [S31] , [S33] , [S36] , [S37] , [S38] , [S47] , [S52] , [S53] , [S57] , [S75] , [S76] , [S89] , [S91] , [S94] , [S100] , [S101] |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| ID | Author, Publication Title, and Venue |
|---|---|
| [S1] | Sarah Fakhoury, Markus Kuppe, Shuvendu K. Lahiri, Tahina Ramananandro, Nikhil Swamy. 3DGen: AI-Assisted Generation of Provably Correct Binary Format Parsers . arXiv preprint arXiv:2404.10362 (2024). |
| [S2] | Cheryl Lee, Chunqiu Steven Xia, Longji Yang, Jen-tse Huang, Zhouruixin Zhu, Lingming Zhang, Michael R. Lyu. FixAgent: Hierarchical Multi-Agent Framework for Unified Software Debugging . arXiv preprint arXiv:2404.17153 (2024). |
| [S3] | Lyuye Zhang, Kaixuan Li, Kairan Sun, Daoyuan Wu, Ye Liu, Haoye Tian, Yang Liu. ACFIX: Guiding LLMs with Mined Common RBAC Practices for Context-Aware Repair of Access Control Vulnerabilities in Smart Contracts . arXiv preprint arXiv:2403.06838 (2024). |
| [S4] | Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, Xiaoguang Mao. AGENTFL: Scaling LLM-based Fault Localization to Project-Level Context . arXiv preprint arXiv:2403.16362 (2024). |
| [S5] | Minh Huynh Nguyen, Thang Phan Chau, Phong X. Nguyen, Nghi D. Q. Bui. AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology . arXiv preprint arXiv:2406.11912 (2024). |
| [S6] | Zeeshan Rasheed, Malik Abdul Sami, Muhammad Waseem, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen, Kari Systä, Pekka Abrahamsson. AI-powered Code Review with LLMs: Early Results . arXiv preprint arXiv:2404.18496 (2024). |
Multi-Agent LLM Systems
Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.
See all 1,108 papers