Can Large Language Models Explain Flight Safety Events? A Prior-Guided Semantic LLM-based Approach
Authors: Lu Xu, Xu Li, Linjiang Zheng, Fan Li, Riquan Zhang, Jiaxing Shang
Organizations: College of Computer Science, Chongqing University, Chongqing, China · School of Statistics and Data Science, Shanghai University of International Business and Economics, Shanghai, China
Improving flight safety with flight data requires not only accurate detection of risk events, but more importantly, clear interpretation of their underlying causes at the level of pilot control behavior. Existing explainable AI techniques, such as feature importance maps, often require considerable domain knowledge to translate them into operationally meaningful explanations. Large Language Models (LLMs), which excel at language reasoning, bring a promising solution to this issue. However, applying LLMs in this domain presents key challenges such as modal inconsistency, limited classification ability, scarcity of task-specific data for fine-tuning, and lack of domain knowledge. To overcome these challenges, we propose FlightLLM, a prior-guided semantic LLM-based approach for interpretable flight safety analysis. Specifically, we first perform feature engineering to address modal inconsistency, combining statistical descriptors with physically meaningful flight indicators. This representation is further processed by a Semantic Discretization module, which converts abstract numerical patterns into qualitative descriptions that are more compatible with language reasoning. In addition, since LLMs are not inherently strong classifiers, CatBoost is incorporated as a statistical expert, and its prediction results are injected into the prompt as prior guidance. A contrastive few-shot learning strategy is further adopted to compensate for limited data. Finally, we design structured prompts to embed aviation-specific knowledge into the inference process. Using hard landing, a representative risk event with complex causal mechanisms, as an anchor point, we evaluate FlightLLM on a dataset of 704 real-world A320 flight samples. Experimental results show that the proposed approach achieves competitive classification performance while generating direct and reasonable explanations for event causes.
Figures & tables
Fig. 1: The framework of FlightLLM consists of five modules: Data Preprocessing and Feature Engineering, Semantic Discretization, Statistical Expert Hinting, Dynamic Context Retrieval and Prompt Construction and LLM Invocation. Statistical Expert denotes CatBoost model.
Category
Feature Symbol
Description
Part A: Automated Statistical Features ( Fstat )
Vertical Speed
Var(ΔIVV0.41.0)
The variance of vertical speed in the 40th to 100th percentile range.
Var(∣ΔIVV∣)
The absolute variance of the vertical speed change within the full percentile range.
Var(ΔIVVmax−min)
The variance between the maximum and minimum changes in vertical speed.
Freal12(IVV)
The real frequency energy of the vertical speed in the higher frequency band.
Altitude
Q0.1(RADIOLH)
The 10th percentile value of the left radio height.
TABLE I: Detailed Description of the Extracted Hybrid Feature Space
Fig. 2: High-Dimensional Feature Profile Match
Fig. 3: The Design of the Prior-Guided Prompt Template.
Prompt Module
Content / Instruction Summary
1. System Persona
“You are an expert in aviation flight data. Your task is to determine whether a landing is normal or hard based on strict evidence.”
2. Context Injection
Feature Dictionary: Inject semantic definitions mapping abstract statistical variables to specific aircraft states.
Auxiliary Report ( H ): Inject the statistical prior distribution generated by the CatBoost expert model.
Few-Shot Examples ( E ): Provide historical normal and hard landing prototypes to construct a contrastive learning context.
3. Input Formulation
Test Sample Features ( Vtest ): The sequence of semantic tokens representing the current flight parameters.
4. Task Constraints
1. Rely strictly on provided observable features; avoid speculation or hallucination.
TABLE II: The Design of the Prior-Guided Prompt Template
Parameter
Description
Frequency(Hz)
Parameter
Description
Frequency(Hz)
ALT_QNH
Altitude
1
ROLL
Roll angel
2
ALT_STD
Standard altitude corrected
1
ROLL_CPT
ROLL_CPT
8
RADIO_LH
Left radio height
4
ROLL_FO
Deputy captain roll control
8
RADIO_RH
Right radio height
4
HEAD_MAG
Magnetic heading direction
1
LDGL
Left landing gear state
4
WIN_DIR
Wind direction
1
LDGR
Right landing gear state
4
WIN_SPD
Wind speed
1
TABLE III: The parameters from QAR data
Model
Accuracy
Precision
Recall
F1
LSTM
56.03
45.95
60.71
52.31
SVM
57.72
46.82
54.87
50.53
RF
56.74
50.88
46.77
48.74
KNN
54.61
34.62
16.07
21.95
CNN
77.30
67.14
83.93
74.60
IMTCN
65.96
55.88
67.86
61.29
TABLE IV: Classification results of different methods
Fig. 4: The parameters’ curves of the analyzed case and corresponding curves of group.
Fig. 5: The detailed textual output of LLM.
Variant
Accuracy
Precision
Recall
F1
Variant A(w/o Semanticization)
76.60
67.16
80.36
73.17
Variant B(w/o Expert Hint)
75.18
75.61
55.36
63.92
Variant C(w/o Context Retrieval)
73.76
63.01
82.14
71.32
FlightLLM
81.56
85.71
64.29
73.41
TABLE V: The result of ablation experiment
Fig. 6: Feature Attention Heatmap of 3 language models.
As Large Language Models (LLMs) advance toward embodied AI agents operating in physical environments, a fundamental question emerges: can models trained on text corpora reliably reason about complex physics while adhering to safety constraints? We address this through PilotBench, a benchmark evaluating LLMs on safety-critical flight trajectory and attitude prediction. Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction through comparative analysis of LLMs and traditional forecasters. We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance. Comparative evaluation across 41 models uncovers a Precision-Controllability Dichotomy: traditional forecasters achieve superior MAE of 7.01 but lack semantic reasoning capabilities, while LLMs gain controllability with 86--89% instruction-following at the cost of 11--14 MAE precision. Phase-stratified analysis further exposes a Dynamic Complexity Gap-LLM performance degrades sharply in high-workload phases such as Climb and Approach, suggesting brittle implicit physics models. These empirical discoveries motivate hybrid architectures combining LLMs' symbolic reasoning with specialized forecasters' numerical precision. PilotBench provides a rigorous foundation for advancing embodied AI in safety-constrained domains.
Yalun Wu, Haotian Liu, Zhoujun Li +1
School of Computing, National University of Singapore, China · School of Informatics, Xiamen University, China · CESS, Beihang University, China
The integration of Large Language Models (LLMs) into aviation safety decision-making represents a significant technological advancement, yet their standalone application poses critical risks due to inherent limitations such as factual inaccuracies, hallucination, and lack of verifiability. These challenges undermine the reliability required for safety-critical environments where errors can have catastrophic consequences. To address these challenges, this paper proposes a novel, end-to-end framework that synergistically combines LLMs and Knowledge Graphs (KGs) to enhance the trustworthiness of safety analytics. The framework introduces a dual-phase pipeline: it first employs LLMs to automate the construction and dynamic updating of an Aviation Safety Knowledge Graph (ASKG) from multimodal sources. It then leverages this curated KG within a Retrieval-Augmented Generation (RAG) architecture to ground, validate, and explain LLM-generated responses. The implemented system demonstrates improved accuracy and traceability over LLM-only approaches, effectively supporting complex querying and mitigating hallucination. Results confirm the framework's capability to deliver context-aware, verifiable safety insights, addressing the stringent reliability requirements of the aviation industry. Future work will focus on enhancing relationship extraction and integrating hybrid retrieval mechanisms.
Evaluating large language models (LLMs) in safety-critical, physics-governed environments requires more than accuracy-based metrics, because predictions that are numerically close to the ground truth can still violate operational constraints, combine fields in physically inconsistent ways, or fail to produce usable structured outputs. Existing evaluation protocols do not measure these failure modes reliably. We propose FLY-EVAL++, an evidence-driven evaluation protocol that combines deterministic verification of protocol compliance, physical feasibility, and safety constraints with fixed rubric-guided aggregation into interpretable multi-dimensional scores. We instantiate FLY-EVAL++ for Flight Trajectory and Attitude Prediction (FTAP) by extending the PilotBench setting with history-conditioned and multi-step prediction tasks. Across 66 LLMs, safety compliance is the most discriminative dimension of model behavior: models with comparable predictive performance differ by more than 28 points in safety score, and we observe recurrent failures including safety violations under physically plausible predictions and instability in multi-step rollouts. These results show that evaluation in safety-critical domains should measure constraint satisfaction and structured validity explicitly rather than rely on accuracy-centric reporting alone.
Yalun Wu, Junfeng Fang, Jiawei Wang +6
NExT++ Lab, School of Computing, National University of Singapore · 3Xiamen University · University of Manchester +4