Toward a Decision-Assurance Layer for AI-Assisted Flight Planning in Air Traffic Management
Organizations: George Mason University · Department of Cyber Security Engineering George Mason University Fairfax, VA, USA · Tecnológico de Monterrey · DECEA: Department of Airspace Control
Abstract
Generative AI is increasingly being used informally in Air Traffic Management (ATM) for tasks such as flight plan generation, trajectory interpretation, and constraint checking. Although these tools can reduce workload and accelerate planning, their non-deterministic outputs create safety and operational risks in human-in-the-loop settings. This paper proposes the AI Trust and Assurance Layer (ATAL), a model-agnostic decision assurance architecture that evaluates whether AI-generated flight-planning outputs are sufficiently reliable for operational use. ATAL combines semantic stability under prompt variation, operational consistency of structured outputs, and normative constraint validation against domain rules, and maps these signals to a Decision Readiness Level (DRL) for human operators. An ATM-inspired experimental study shows how unsafe, inconsistent, or misleading outputs can be identified before influencing flight-plan validation or execution. Although demonstrated in aviation, the framework is also transferable to other safety-critical decision-support domains that require human oversight under regulatory constraints.
Figures & tables
| Category | Capability | Limitation (Gap) | Existing Approaches | ATAL Contribution |
|---|---|---|---|---|
| Behavioral Evaluation | Linguistic quality and fluency | Focuses on surface-level correctness without ensuring semantic or structural reliability | Perplexity, GLUE, HELM | Not primary focus; used as auxiliary signal |
| Factual Consistency | Hallucination detection | Detects factual errors but lacks structural validation and cross-field consistency | TruthfulQA, factuality benchmarks | Partial detection via behavioral inconsistency analysis |
| Constraint Enforcement | Domain and rule validation | Limited to syntactic or schema-level validation; lacks semantic and contextual enforcement | Guardrails, schema validation | Deterministic and contextual constraint validation (NCV) |
| Robustness | Stability under perturbations | No systematic evaluation of output invariance under controlled input variations | Rarely addressed explicitly | Semantic Stability Score (SSS) for perturbation-based robustness |
| Adversarial Resilience | Detection of adversarial manipulation | Isolated detection without integration into reliability assessment | Adversarial prompt filters | Integrated adversarial signal detection (IEI) |
| Operational Monitoring | Longitudinal degradation tracking | Monitoring exists but disconnected from decision-making and reliability scoring | Drift detection systems | Structured longitudinal monitoring (TRL layer) |
| VQS | SSS | Interpretation |
|---|---|---|
| High | High | Model is robust and reliable |
| High | Low | Model is unstable under valid variations |
| Low | Low | Invalid experiment (bad perturbations) |
| Low | High | Model is robust even under noisy input |
| Scenario | Type | Description | Operational Significance |
|---|---|---|---|
| S01 | Valid baseline | Complete and consistent flight plan advisory generated by the model | Represents ideal conditions where the advisory can be directly executed without additional validation |
| S02 | Alternate required | Advisory omits or inconsistently handles alternate aerodrome requirements | Reflects real-world contingency planning gaps that may compromise safety under degraded conditions |
| S03 | Context-sensitive | Advisory must respect contextual constraints (e.g., airspace or restrictions) | Captures dependence on external operational conditions not explicitly encoded in the prompt |
| S04 | Incomplete input | Critical fields (e.g., departure, destination) are missing | Represents unusable outputs that must be rejected before any operational consideration |
| S05 | Conflicting input | Advisory contains contradictory information (e.g., inconsistent route or parameters) | Models ambiguity or internal inconsistency that can lead to unsafe execution |
| S06 | Unsafe but consistent | Structurally valid plan that violates safety constraints (e.g., restricted route) | Demonstrates risk of high-confidence but unsafe model outputs |
| Metric | High Cases | Medium Cases | Low Cases |
|---|---|---|---|
| SSS | S01, S07, S08 | S03 | S12 |
| OCS | S01, S06, S11 | S09 | S05 |
| NCV | S01, S07 | S02, S09 | S04, S05, S12 |