cs.AIAug 19, 2026
SaveAir Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation
Organizations: George Washington University, Washington, DC 20052, USA · California Institute of Technology, Pasadena, CA 91125, USA · University of California, Berkeley, CA 94720, USA
Abstract
Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco "Bay Tour" route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.
Figures & tables
Figure 1: Overall architecture. (1) Generation: pilot transmissions transcribed from an experimental Bay Area tour flight are issued one turn at a time to the language model, which is conditioned on a system prompt and on the communication history accumulated so far; each pseudo-ATC response is appended back into that history. (2) Teacher-forced generation: the same scenario is replayed with the controller side of the history replaced by the ground truth transmissions. The example contrasts the two regimes at turn 24. Conditioned on its own replies (left), the model has fallen into an Acknowledgment loop and answers “Roger,” missing the required radar-service termination and Hayward handoff. Given the true history (right), the same turn is answered with the correct facility, the correct “keep the code” instruction, and the correct frequency 120.2. (3) Evaluation: every turn is scored by automatic similarity metrics, by an LLM-as-judge, and—on a stratified subsample—by a human expert.
| ROUGE-L | Semantic cosine | |||||||
|---|---|---|---|---|---|---|---|---|
| Prompt | no ICL, self | no ICL, GT | ICL, self | ICL, GT | no ICL, self | no ICL, GT | ICL, self | ICL, GT |
| C1 | 0.216 | 0.209 | 0.244 | 0.244 | 0.464 | 0.449 | 0.497 | 0.490 |
| C2 | 0.217 | 0.214 | 0.243 | 0.230 | 0.482 | 0.472 | 0.492 | 0.477 |
| C3 | 0.219 | 0.212 | 0.243 | 0.224 | 0.487 | 0.470 | 0.493 | 0.467 |
| C4 | 0.209 | 0.220 | 0.216 | 0.238 | 0.430 | 0.445 | 0.447 | 0.470 |
| C5 | 0.167 | 0.218 | 0.159 | 0.224 | 0.302 | 0.383 | 0.298 | 0.396 |
Table 1: Per-prompt similarity to the C0 ground truth, by condition (ICL history), averaged over the 9 models and 3 attempts. Best per column in bold.
Figure 2: Prompt (ICL history) interaction, pooled over the nine models (mean 95% CI across models). Color encodes the dialogue history the model conditions on, line style encodes whether a worked in-context example was supplied. Under the light prompts C1–C3 the two history regimes are indistinguishable; under the over-specified C5 the self-history condition collapses and ground-truth grounding recovers it.
Figure 3: Cumulative running-mean similarity over the 36-turn conversation (ICL on, pooled over models). Under C1–C3 the self- and ground-truth-history trajectories track each other; under C5 self-history drifts steadily downward as early errors propagate through the dialogue history, while ground-truth injection holds the trajectory flat.
| ROUGE-L | Semantic cosine | |||||||
| Model | no ICL, self | no ICL, GT | ICL, self | ICL, GT | no ICL, self | no ICL, GT | ICL, self | ICL, GT |
| Open-source | ||||||||
| Qwen2.5-7B | 0.197 | 0.215 | 0.212 | 0.234 | 0.413 | 0.458 | 0.443 | 0.483 |
| Llama-3.1-8B | 0.193 | 0.181 | 0.208 | 0.203 | 0.437 | 0.403 | 0.452 | 0.418 |
| Gemma-2-9B | 0.188 | 0.218 | 0.216 | 0.248 | 0.405 | 0.441 | 0.438 | 0.487 |
| Qwen2.5-14B | 0.203 | 0.200 | 0.215 | 0.195 | 0.481 | 0.464 | 0.483 | 0.456 |
Table 2: Per-model similarity to the C0 ground truth, by condition (ICL history), averaged over prompts C1–C5 and 3 attempts. Best per column in bold.
Figure 4: LLM-judge score per rubric dimension, pooled over the nine models. Color encodes the dialogue history the model conditions on; hatching marks the in-context learning condition. Models score highly on callsign handling and conciseness but poorly on operational correctness.
Figure 5: Per-dimension scores from the LLM judge (a) and the human expert (b) on the same 180 annotated outputs, broken out by condition (color = dialogue history, hatching = in-context example). Both raters produce the same cross-dimension profile, peaking on callsign handling and conciseness and dipping on operational correctness, but the human sits about one point higher on every dimension. Because both panels use the identical 180-sample subset, the vertical offset between them is the genuine human–judge leniency gap.
Figure 6: Model-level agreement between each automatic similarity metric and the LLM judge. Each point is one model, plotted as its mean metric score against its mean judge rating, with a fitted line and 95% confidence band. ROUGE-L and TF-IDF track expert judgment closely; BLEU-1 and sentence-embedding similarity do not. The open/closed legend in (a) applies to all four panels.
Figure 7: Mean overall rating assigned by the LLM judge and by the human expert, split by the judge’s own score band (mean 95% CI). The human is systematically more lenient, and the gap grows as output quality falls.
| Measure | Wins | Statistic | Effect | ||
|---|---|---|---|---|---|
| In-context example vs. none (paired, , ) | |||||
| ROUGE-L (pooled) | 8/9 | 0.008 | |||
| Normalized BLEU (pooled) | 8/9 | 0.008 | |||
| LLM-judge overall | 5/9 | ||||
| Ground-truth vs. self dialogue history (paired, , ) | |||||
| ROUGE-L (pooled) | 5/9 | ||||
Table 3: Hypothesis tests for the three experimental contrasts, computed on per-model means across the nine models. Paired contrasts use an exact Wilcoxon signed-rank test ( ); the closed-versus-open contrast uses a one-sided Mann–Whitney test. is reported in the direction named in each group heading, wins counts models favoring that direction, and the last column is the effect size ( for the paired contrasts, Cohen’s for the unpaired one). Bold indicates .
| Model | Phraseology | Correctness | Callsign | Conciseness | Safety | Halluc.-free | Overall |
|---|---|---|---|---|---|---|---|
| Open-source | |||||||
| Qwen2.5-7B | 2.46 | 1.75 | 4.36 | 3.97 | 2.63 | 2.68 | 1.99 |
| Llama-3.1-8B | 2.40 | 1.74 | 3.65 | 3.88 | 2.51 | 2.22 | 1.89 |
| Gemma-2-9B | 2.80 | 1.94 | 4.14 | 4.34 | 2.80 | 2.37 | 2.12 |
| Qwen2.5-14B | 2.35 | 1.70 | 4.44 | 3.17 | 2.47 | 2.00 | 1.84 |
| Qwen2.5-32B | 2.81 | 2.00 | 4.29 | 3.83 | 2.83 | 2.33 | 2.20 |
Table 4: LLM-judge scores (1–5) per model, averaged over conditions and the top-25 judged outputs per cell. Best per column in bold.
Figure 8: Per-model similarity by history mode (ICL on), grouped into open- and closed-source models. Bars are self history (hatched // ) versus ground-truth history (hatched \ ); panels are the four similarity metrics.
Figure 9: Per-prompt similarity by history mode (ICL on), across prompts C1–C5. Self history (indigo) collapses at C5 on every metric while ground-truth history (coral) remains flat, the per-prompt counterpart of Figure 8 .
| ROUGE-L | TF-IDF | BLEU-1 | Semantic | LLM judge | |
|---|---|---|---|---|---|
| TF-IDF | 0.73 | ||||
| BLEU-1 | 0.58 | 0.36 | |||
| Semantic | 0.32 | 0.47 | 0.30 | ||
| LLM judge | 0.53 | 0.49 | 0.02 | 0.10 | |
| Human | 0.34 | 0.38 | -0.05 | 0.09 | 0.67 |
Table 5: Spearman rank-correlation matrix over all six evaluation measures, computed on the 180 human-annotated outputs so that every measure is available for every observation. The two reference-overlap metrics (ROUGE-L, TF-IDF) are the only automatic measures that track either human-facing judgement.
| Comparison | Level | Pearson | Spearman | |
|---|---|---|---|---|
| Automatic metric vs. LLM-judge overall (sample ) | ||||
| ROUGE-L | sample | 0.51 | 0.52 | – |
| ROUGE-L | model | 0.90 | 0.93 | – |
| TF-IDF | sample | 0.50 | 0.48 | – |
| TF-IDF | model | 0.91 | 0.90 | – |
| BLEU-1 | sample | 0.12 | 0.06 | – |
Table 6: Agreement between the three evaluation layers. Sample-level statistics treat each scored output as one observation; model-level statistics use per-model means ( ). is quadratic-weighted Cohen’s kappa, defined only when both measures share the 1–5 ordinal scale.
| Dimension | Judge mean | Human mean | Spearman | ||
|---|---|---|---|---|---|
| Phraseology | 3.24 | 4.30 | +1.06 | 0.45 | 0.36 |
| Correctness | 2.89 | 3.79 | +0.90 | 0.63 | 0.52 |
| Callsign | 4.43 | 4.86 | +0.43 | 0.35 | 0.43 |
| Conciseness | 4.34 | 4.51 | +0.17 | 0.41 | 0.59 |
| Safety | 3.63 | 3.91 | +0.28 | 0.53 | 0.59 |
| Halluc.-free | 3.61 | 3.96 | +0.34 | 0.48 | 0.46 |
Table 7: Per-dimension and per-stratum agreement between the LLM judge and the human annotator ( ). is the human mean minus the judge mean; positive values mean the human is more lenient. Within-stratum correlations are attenuated by range restriction and is uninformative there, so it is reported only for the pooled sample.
Figure 10: Agreement between the evaluation layers. (a) Mean automatic similarity as a function of the LLM-judge overall rating (sample level, ): ROUGE-L and TF-IDF rise monotonically with judge rating, BLEU-1 responds only at rating 5, and embedding similarity is nearly flat. (b) Per-model mean judge rating against mean human rating on the 180 annotated outputs (color = mean ROUGE-L); every model lies above the identity line, so the human is the more lenient rater for all nine.
Figure 11: Distribution of overall scores per model, as diverging Likert bars. Relative to the judge (a), the human distribution (b) shifts markedly toward 4 and 5, the same leniency offset quantified in Table 7 . Note the differing sample sizes: the judge panel is over the full judged subset, the human panel over the 180-sample validation set.
Figure 12: Serving cost and stability per model. Open models run locally on a GPU; closed models are API calls. (a) Median per-turn latency with interquartile-range whiskers, spanning 0.26 s (Llama-3.1-8B) to 1.74 s (Claude-Sonnet-4.6). (b) Standard deviation across the three attempts; bars are the mean over the four metrics, dots the individual per-metric values ( throughout).