CIAO - Code In Architecture Out - Automated Software Architecture Documentation with Large Language Models
Authors: Marco De Luca, Tiziano Santilli, Domenico Amalfitano, Anna Rita Fasolino, Patrizio Pelliccione
Organizations: University of Naples Federico II Naples, Italy · University of Southern Denmark Odense, Denmark · Gran Sasso Science Institute L’Aquila, Italy
Software architecture documentation is essential for system comprehension, yet it is often unavailable or incomplete. While recent LLM-based techniques can generate documentation from code, they typically address local artifacts rather than producing coherent, system-level architectural descriptions. This paper presents a structured process for automatically generating system-level architectural documentation directly from GitHub repositories using Large Language Models. The process, called CIAO (Code In Architecture Out), defines an LLM-based workflow that takes a repository as input and produces system-level architectural documentation following a template derived from ISO/IEC/IEEE 42010, SEI Views & Beyond, and the C4 model. The resulting documentation can be directly added to the target repository. We evaluated the process through a study with 22 developers, each reviewing the documentation generated for a repository they had contributed to. The evaluation shows that developers generally perceive the produced documentation as valuable, comprehensible, and broadly accurate with respect to the source code, while also highlighting limitations in diagram quality, high-level context modeling, and deployment views. We also assessed the operational cost of the process, finding that generating a complete architectural document requires only a few minutes and is inexpensive to run. Overall, the results indicate that a structured, standards-oriented approach can effectively guide LLMs in producing system-level architectural documentation that is both usable and cost-effective.
Figures & tables
Fig. 1: Overview of the CIAO automated documentation workflow.
Repo
Domain
Lang. (LOC %)
Files
LOC
Group-Key-Phemapn [ 58 ]
IoT
C (17.0%), C++ (83.0%)
9
1061
pyALS-RF-tmr [ 59 ]
Machine Learning
C (0.9%), C++ (0.3%), CMake (1.6%), JSON (0.7%), Python (95.8%), Shell (0.7%)
85
9270
sa-bsn [ 60 ]
Healthcare
C (12.6%), C++ (75.5%), CMake (4.1%), Python (4.5%), Shell (0.5%), XML (2.9%)
175
9877
ocean-lib [ 61 ]
Machine Learning
Python (80%), YAML (20%)
33
1208
robethichor [ 62 ]
Robotic Systems
CMake (1.7%), Dockerfile (1.6%), JSON (3.7%), Python (88.1%), XML (4.9%)
28
697
Openjob [ 63 ]
Web Application
Dockerfile (0.6%), Java (65.3%), XML (32.3%), YAML (1.9%)
146
5386
TABLE I: Overview of analyzed repositories with languages and code size.
ID
Question
Type
Part 1 – Perceived value (RQ1)
Q1
The LLM-generated architectural documentation provides valuable architectural insights about the system.
L
Q2
The LLM-generated documentation is useful for understanding and maintaining the system’s structure and dependencies.
L
Q3
The LLM-generated documentation is valuable enough that I would consider using it in my own project.
L
Q4
Which sections or features of the LLM-generated documentation were most or least valuable to you?
O
Part 2 – Comprehensibility (RQ2)
TABLE II: Overview of the questionnaire structure, question types, and mapping to questionnaire parts. Legend: L = 5-point Likert scale; O = Open-ended question.
Fig. 2: Distribution of participants’ Likert-scale ratings for RQ1
Fig. 3: Distribution of participants’ Likert-scale ratings for RQ2
Fig. 4: Distribution of participants’ Likert-scale ratings for RQ3 - General Part
Fig. 5: Distribution of participants’ Likert-scale ratings for RQ3 - Section Part
Category
Occurrences
Diagram errors
32
Deployment issues
14
Improvement suggestions
13
Missing information
8
Inconsistencies
6
Redundancy
1
TABLE III: Limitation categories from open-ended responses.