Offline Language-Based Support System for Tactical Decision-Making in Air Combat Operations
Organizations: Decision Support Systems Subdivision Institute for Advanced Studies São José dos Campos, São Paulo, Brazil, 12228-001
Abstract
An offline language-based support tool is presented to assist pilot training and tactical decision-making in air combat operations. The system, named MATE (Mission Assistant for Training and Evaluation), applies retrieval-augmented generation (RAG) to answer natural language questions using content extracted from doctrinal and tactical manuals. Built entirely with open-source components, it performs text preprocessing, embedding generation, semantic search, and language generation locally, ensuring full offline functionality. The tool was evaluated by three Brazilian Air Force fighter pilots, who rated the answers produced with three open-weight language models (Llama 3.1 8B, Qwen 2.5 14B, and Mistral NeMo 12B) to five tactical questions derived from a publicly available air combat manual. Average ratings ranged from 4.00 to 4.40 on a five-point scale, with Mistral NeMo 12B obtaining the highest score, and the pilots rated the response time of the tool as fully adequate compared to manual search. The relevance of the retrieved content was identified as the main point for improvement. Potential applications are discussed in onboard mission systems, flight simulators, and training environments.
Figures & tables
| Question | Llama 3.1 8B | Qwen 2.5 14B | Mistral NeMo 12B |
|---|---|---|---|
| Q1 – Advantage of flying as a section | 4.67 0.58 | 4.00 1.00 | 4.67 0.58 |
| Q2 – Advantages of the Fighting Wing | 4.67 0.58 | 4.33 0.58 | 4.67 0.58 |
| Q3 – Weaknesses of the Fighting Wing | 3.00 0.00 | 3.00 0.00 | 3.00 0.00 |
| Q4 – Innovation of the Double Attack | 3.33 0.58 | 4.00 1.00 | 4.67 0.58 |
| Q5 – Loose Deuce vs. Double Attack | 5.00 0.00 | 4.67 0.58 | 5.00 0.00 |
| Average over questions | 4.13 | 4.00 | 4.40 |
| Evaluation question | Rating |
|---|---|
| Relevance of retrieval : did the information used in the answer correspond to the correct content from the reference material? | 3.67 0.58 |
| Conceptual and terminological accuracy : did the model use correct tactical terms consistent with operational doctrine? | 4.33 1.15 |
| Integration of retrieved context : did the model properly incorporate the retrieved text into its explanation? | 3.67 0.58 |
| Clarity and coherence : was the answer clear, logical, and well structured? | 4.33 0.58 |
| Response time : was the generation time appropriate compared to manually locating the information in the reference text? | 5.00 0.00 |
AI-Assisted Decision Making
AI-assisted decision making studies how computational systems can inform human choices while accounting for context, uncertainty, and consequences.
See all 144 papers