A Retrieval-Augmented Language Assistant for Unmanned Aircraft Safety Assessment and Regulatory Compliance
Organizations: School of Engineering Zurich University of Applied Sciences ZHAW Winterthur, Switzerland
Abstract
This paper presents the design and validation of a retrieval-based assistant that supports safety assessment, certification activities, and regulatory compliance for unmanned aircraft systems. The work is motivated by the growing complexity of drone operations and the increasing effort required by applicants and aviation authorities to apply established assessment frameworks, including the Specific Operations Risk Assessment and the Pre-defined Risk Assessment, in a consistent and efficient manner. The proposed approach uses a controlled text-based architecture that relies exclusively on authoritative regulatory sources. To enable traceable and auditable outputs, the assistant grounds each response in retrieved passages and enforces citation-driven generation. System-level controls address common failure modes of generative models, including fabricated statements, unsupported inferences, and unclear provenance, by separating evidence storage from language generation and by adopting conservative behavior when supporting documentation is insufficient. The assistant is intentionally limited to decision support; it does not replace expert judgment and it does not make autonomous determinations. Instead, it accelerates context-specific information retrieval and synthesis to improve document preparation and review while preserving human responsibility for critical conclusions. The architecture is implemented using established open-source components, and key choices in retrieval strategy, interaction constraints, and response policies are evaluated for suitability in safety-sensitive regulatory environments. The paper provides technical and operational guidance for integrating retrieval-based assistants into aviation oversight workflows while maintaining accountability, traceability, and regulatory compliance.
Figures & tables
| Item | Description |
|---|---|
| Use Case ID | UC-CHAT-01 |
| Use Case Name | Ask SORA or drone operations question via chatbot |
| Primary Actor | User, operator, safety engineer, student |
| Supporting Actors or Systems | Web GUI or notebook client; RAG system, vector search and reranker; LLMChatbot; QueryGenerator (optional) |
| Goal | Get a correct, explainable answer grounded in SORA related documents, with references or snippets where possible |
| Trigger | User submits a natural language question, for example ground risk buffer, OSOs, or mitigations |
| Item | Description |
|---|---|
| Use Case ID | UC-CLS-01 |
| Use Case Name | Classify initial operation and return indicator answers |
| Primary Actor | User, operator, analyst, student |
| Supporting Actors or Systems | Notebook or Web GUI client; RAG system, vector search and reranker; LLM based indicator assistant (LLMIndicatorAssistant) |
| Goal | Produce consistent initial indicators, for example regulatory pathway and initial ground or air risk orientation, grounded in SORA documentation |
| Trigger | User provides an InitialOperationInput and selects one or more indicators to evaluate |
| Variant | Hit@1 | Hit@3 | Hit@5 | Hit@10 | MRR |
|---|---|---|---|---|---|
| Direct match | 100.0% | 100.0% | 100.0% | 100.0% | 1.000 |
| Synonym paraphrase | 75.0% | 75.0% | 100.0% | 100.0% | 0.800 |
| Reworked question | 75.0% | 75.0% | 75.0% | 75.0% | 0.750 |
| Overall | 83.3% | 83.3% | 91.7% | 91.7% | 0.850 |
| Variant | Grounded | Unsupported | Incomplete | Chunk used |
|---|---|---|---|---|
| Direct match | 50.0% | 0.0% | 0.0% | 50.0% |
| Synonym paraphrase | 25.0% | 25.0% | 0.0% | 50.0% |
| Reworked question | 50.0% | 25.0% | 0.0% | 25.0% |
| Indicator | Value consistency [%] | Explanation similarity [%] |
|---|---|---|
| Likely regulatory pathway | 83.3 | 59.3 |
| Initial ground risk orientation | 86.7 | 43.9 |
| Initial air risk orientation | 96.7 | 60.2 |
| Expected assessment depth | 100.0 | 66.9 |
| Overall | 91.7 | 57.6 |
| Indicator | Accuracy [%] |
|---|---|
| Likely regulatory pathway | 66.6 |
| Initial ground risk orientation | 80.3 |
| Initial air risk orientation | 80.3 |
| Expected assessment depth | 100.0 |
| Overall | 81.8 |