Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation
Authors: Mahyar Ghazanfari, Matthias Casanova, Jordan Kam, Alex Zongo, Peng Wei, Torsten Darrell, Alexandre Bayen
Organizations: George Washington University, Washington, DC 20052, USA · California Institute of Technology, Pasadena, CA 91125, USA · University of California, Berkeley, CA 94720, USA
Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco "Bay Tour" route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.
Figures & tables
Figure 1: Overall architecture. (1) Generation: pilot transmissions transcribed from an experimental Bay Area tour flight are issued one turn at a time to the language model, which is conditioned on a system prompt and on the communication history accumulated so far; each pseudo-ATC response is appended back into that history. (2) Teacher-forced generation: the same scenario is replayed with the controller side of the history replaced by the ground truth transmissions. The example contrasts the two regimes at turn 24. Conditioned on its own replies (left), the model has fallen into an Acknowledgment loop and answers “Roger,” missing the required radar-service termination and Hayward handoff. Given the true history (right), the same turn is answered with the correct facility, the correct “keep the code” instruction, and the correct frequency 120.2. (3) Evaluation: every turn is scored by automatic similarity metrics, by an LLM-as-judge, and—on a stratified subsample—by a human expert.
ROUGE-L
Semantic cosine
Prompt
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
C1
0.216
0.209
0.244
0.244
0.464
0.449
0.497
0.490
C2
0.217
0.214
0.243
0.230
0.482
0.472
0.492
0.477
C3
0.219
0.212
0.243
0.224
0.487
0.470
0.493
0.467
C4
0.209
0.220
0.216
0.238
0.430
0.445
0.447
0.470
C5
0.167
0.218
0.159
0.224
0.302
0.383
0.298
0.396
Table 1: Per-prompt similarity to the C0 ground truth, by condition (ICL × history), averaged over the 9 models and 3 attempts. Best per column in bold.
Figure 2: Prompt × (ICL × history) interaction, pooled over the nine models (mean ± 95% CI across models). Color encodes the dialogue history the model conditions on, line style encodes whether a worked in-context example was supplied. Under the light prompts C1–C3 the two history regimes are indistinguishable; under the over-specified C5 the self-history condition collapses and ground-truth grounding recovers it.
Figure 3: Cumulative running-mean similarity over the 36-turn conversation (ICL on, pooled over models). Under C1–C3 the self- and ground-truth-history trajectories track each other; under C5 self-history drifts steadily downward as early errors propagate through the dialogue history, while ground-truth injection holds the trajectory flat.
ROUGE-L
Semantic cosine
Model
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
Open-source
Qwen2.5-7B
0.197
0.215
0.212
0.234
0.413
0.458
0.443
0.483
Llama-3.1-8B
0.193
0.181
0.208
0.203
0.437
0.403
0.452
0.418
Gemma-2-9B
0.188
0.218
0.216
0.248
0.405
0.441
0.438
0.487
Qwen2.5-14B
0.203
0.200
0.215
0.195
0.481
0.464
0.483
0.456
Table 2: Per-model similarity to the C0 ground truth, by condition (ICL × history), averaged over prompts C1–C5 and 3 attempts. Best per column in bold.
Figure 4: LLM-judge score per rubric dimension, pooled over the nine models. Color encodes the dialogue history the model conditions on; hatching marks the in-context learning condition. Models score highly on callsign handling and conciseness but poorly on operational correctness.
Figure 5: Per-dimension scores from the LLM judge (a) and the human expert (b) on the same 180 annotated outputs, broken out by condition (color = dialogue history, hatching = in-context example). Both raters produce the same cross-dimension profile, peaking on callsign handling and conciseness and dipping on operational correctness, but the human sits about one point higher on every dimension. Because both panels use the identical 180-sample subset, the vertical offset between them is the genuine human–judge leniency gap.
Figure 6: Model-level agreement between each automatic similarity metric and the LLM judge. Each point is one model, plotted as its mean metric score against its mean judge rating, with a fitted line and 95% confidence band. ROUGE-L and TF-IDF track expert judgment closely; BLEU-1 and sentence-embedding similarity do not. The open/closed legend in (a) applies to all four panels.
Figure 7: Mean overall rating assigned by the LLM judge and by the human expert, split by the judge’s own score band (mean ± 95% CI). The human is systematically more lenient, and the gap grows as output quality falls.
Measure
Δ
Wins
Statistic
p
Effect
In-context example vs. none (paired, W , n=9 )
ROUGE-L (pooled)
0.016
8/9
1.0
0.008
1.46
Normalized BLEU (pooled)
0.009
8/9
1.0
0.008
1.47
LLM-judge overall
0.036
5/9
13.0
0.547
0.25
Ground-truth vs. self dialogue history (paired, W , n=9 )
ROUGE-L (pooled)
0.010
5/9
14.0
0.359
0.49
Table 3: Hypothesis tests for the three experimental contrasts, computed on per-model means across the nine models. Paired contrasts use an exact Wilcoxon signed-rank test ( W ); the closed-versus-open contrast uses a one-sided Mann–Whitney U test. Δ is reported in the direction named in each group heading, wins counts models favoring that direction, and the last column is the effect size ( dz for the paired contrasts, Cohen’s d for the unpaired one). Bold indicates p<0.05 .
Model
Phraseology
Correctness
Callsign
Conciseness
Safety
Halluc.-free
Overall
Open-source
Qwen2.5-7B
2.46
1.75
4.36
3.97
2.63
2.68
1.99
Llama-3.1-8B
2.40
1.74
3.65
3.88
2.51
2.22
1.89
Gemma-2-9B
2.80
1.94
4.14
4.34
2.80
2.37
2.12
Qwen2.5-14B
2.35
1.70
4.44
3.17
2.47
2.00
1.84
Qwen2.5-32B
2.81
2.00
4.29
3.83
2.83
2.33
2.20
Table 4: LLM-judge scores (1–5) per model, averaged over conditions and the top-25 judged outputs per cell. Best per column in bold.
Figure 8: Per-model similarity by history mode (ICL on), grouped into open- and closed-source models. Bars are self history (hatched // ) versus ground-truth history (hatched \ ); panels are the four similarity metrics.
Figure 9: Per-prompt similarity by history mode (ICL on), across prompts C1–C5. Self history (indigo) collapses at C5 on every metric while ground-truth history (coral) remains flat, the per-prompt counterpart of Figure 8 .
ROUGE-L
TF-IDF
BLEU-1
Semantic
LLM judge
TF-IDF
0.73
BLEU-1
0.58
0.36
Semantic
0.32
0.47
0.30
LLM judge
0.53
0.49
0.02
0.10
Human
0.34
0.38
-0.05
0.09
0.67
Table 5: Spearman rank-correlation matrix over all six evaluation measures, computed on the 180 human-annotated outputs so that every measure is available for every observation. The two reference-overlap metrics (ROUGE-L, TF-IDF) are the only automatic measures that track either human-facing judgement.
Comparison
Level
Pearson r
Spearman ρ
κw
Automatic metric vs. LLM-judge overall (sample n=4,500 )
ROUGE-L
sample
0.51
0.52
–
ROUGE-L
model
0.90
0.93
–
TF-IDF
sample
0.50
0.48
–
TF-IDF
model
0.91
0.90
–
BLEU-1
sample
0.12
0.06
–
Table 6: Agreement between the three evaluation layers. Sample-level statistics treat each scored output as one observation; model-level statistics use per-model means ( n=9 ). κw is quadratic-weighted Cohen’s kappa, defined only when both measures share the 1–5 ordinal scale.
Dimension
Judge mean
Human mean
Δ
Spearman ρ
κw
Phraseology
3.24
4.30
+1.06
0.45
0.36
Correctness
2.89
3.79
+0.90
0.63
0.52
Callsign
4.43
4.86
+0.43
0.35
0.43
Conciseness
4.34
4.51
+0.17
0.41
0.59
Safety
3.63
3.91
+0.28
0.53
0.59
Halluc.-free
3.61
3.96
+0.34
0.48
0.46
Table 7: Per-dimension and per-stratum agreement between the LLM judge and the human annotator ( n=180 ). Δ is the human mean minus the judge mean; positive values mean the human is more lenient. Within-stratum correlations are attenuated by range restriction and κw is uninformative there, so it is reported only for the pooled sample.
Figure 10: Agreement between the evaluation layers. (a) Mean automatic similarity as a function of the LLM-judge overall rating (sample level, n=4,500 ): ROUGE-L and TF-IDF rise monotonically with judge rating, BLEU-1 responds only at rating 5, and embedding similarity is nearly flat. (b) Per-model mean judge rating against mean human rating on the 180 annotated outputs (color = mean ROUGE-L); every model lies above the identity line, so the human is the more lenient rater for all nine.
Figure 11: Distribution of overall scores per model, as diverging Likert bars. Relative to the judge (a), the human distribution (b) shifts markedly toward 4 and 5, the same leniency offset quantified in Table 7 . Note the differing sample sizes: the judge panel is over the full judged subset, the human panel over the 180-sample validation set.
Figure 12: Serving cost and stability per model. Open models run locally on a GPU; closed models are API calls. (a) Median per-turn latency with interquartile-range whiskers, spanning 0.26 s (Llama-3.1-8B) to 1.74 s (Claude-Sonnet-4.6). (b) Standard deviation across the three attempts; bars are the mean over the four metrics, dots the individual per-metric values ( ≤0.04 throughout).
Air Traffic Control (ATC) is a safety-critical domain in which incorrect interpretation of instructions may lead to severe operational consequences. While large language models (LLMs) demonstrate strong general performance, their reliability in operational ATC environments remains unclear. Existing evaluation approaches, largely based on aggregate metrics such as F1 or macro accuracy, treat all errors uniformly and fail to account for the asymmetric consequences of high-risk semantic mistakes (e.g., incorrect runway identifiers or movement constraints). To address this gap, we propose a safety-oriented, consequence-aware evaluation framework tailored to ATC operations. Our results reveal that while current LLMs achieve reasonable aggregate accuracy, their operational reliability is severely limited. Evaluated on clean transcripts, the peak Risk Score reaches only 0.69, with most models scoring below 0.6 despite high macro-F1 performance. Further analysis shows that errors concentrate in high-impact entities despite relatively stable action-type classification, indicating structural grounding deficiencies. These findings highlight the necessity of consequence-aware evaluation protocols for the responsible deployment of AI-assisted ATC systems.
Yujing Chang, Yash Guleria, Duc-Thinh Pham +4
1ATMRI, Nanyang Technological University (NTU), Singapore · School of Management, Indian Institute of Technology Mandi, India · Centre of AI Research, VinUniversity, Vietnam
Aviation knowledge question answering cannot directly assess the operational effectiveness and safety compliance of large language models throughout aircraft emergency procedures. We introduce AeroCopilotBench and its executable cockpit environment, ACOE, which define state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks (POHs). The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating task completion, safety compliance, execution discipline, and repeatability. Across repeated evaluations of 12 models, the highest safety-gated success rate is 72.6%. Most failed episodes achieve all critical goals but do not satisfy all remaining terminal goals. Across repeated runs, models still omit steps that they execute in other runs of the same task. Analysis of failed trajectories further reveals that some reasons for actions that conflict with the aircraft's POH recur across models. These results expose shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.
Yuchen Yuan, Zhenghuang Wu, Yuangan Li +2
School of Aeronautic Science and Engineering, Beihang University
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
Alex Brooker, Tim Hughes
Airside Labs, London, United Kingdom · Mahino Research, Christchurch, New Zealand