Decomposed Relation Extraction from Aviation Accident Narratives: Evaluating Transformer-Based Safety Knowledge Discovery under Relational Sparsity

Aziida Nanyonga; Mustafa Tahsin Yilmaz; Saad Ezzini; Ugur Turhan; Graham Wild

cs.CLOct 19, 2026

Decomposed Relation Extraction from Aviation Accident Narratives: Evaluating Transformer-Based Safety Knowledge Discovery under Relational Sparsity

Authors: Aziida Nanyonga, Mustafa Tahsin Yilmaz, Saad Ezzini, Ugur Turhan, Graham Wild

Organizations: SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum and Minerals (KFUPM), Dhahran 31261, Saudi Arabia · Department of Industrial Engineering, Faculty of Engineering, King Abdulaziz University, Jeddah 21589, Saudi Arabia · Department of Information and Computer Science, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia · Capability Systems Centre, University of New South Wales, Canberra, ACT 2600, Australia

Abstract

Relation extraction from aviation accident narratives is challenging because genuine safety relations are sparse among possible concept pairs. This study evaluates a decomposed framework separating relation classification from concept identification. A benchmark of 300 National Transportation Safety Board (NTSB) narratives containing 2,524 unique concept-text nodes was expanded into 21,038 ordered candidate pairs, with only 248 (1.18%) representing RESULTS_IN or CONTRIBUTES_TO relations. A TF-IDF Linear Support Vector Machine (SVM) and weighted RoBERTa classifier were developed using an independent graph-annotated resource with hard-negative sampling and frozen before sealed evaluation. Development Macro-F1 scores of 0.4738 (SVM) and 0.4809 (RoBERTa) declined to 0.2741 and 0.2420 under exhaustive evaluation, with Matthews correlation coefficients of 0.0128 and 0.0197. False positives dominated: the SVM recovered 96 reference relations but generated 7,018 false positives, while RoBERTa recovered 125 but generated 9,824. Post-test diagnostics revealed overlapping RoBERTa confidence distributions and false positives even within valid relation-category directions. Complementary end-to-end SmolLM2 evaluation identified missed and unsupported relations alongside malformed outputs. These findings demonstrate that known concept endpoints do not guarantee reliable relation extraction and that sampled-negative development performance may poorly reflect exhaustive evaluation. Controlling false positives and managing candidate-space sparsity remain essential for aviation safety knowledge extraction.

Figures & tables

Figure 1. Experimental framework for aviation safety relation extraction. The decomposed evaluation uses reference safety concepts to construct exhaustive ordered concept pairs for relation classification, whereas the end-to-end generative evaluation requires SmolLM2-1.7B-Instruct to generate concepts and relations directly from the narrative. Both settings use the same 300-narrative benchmark and reference relation set but represent different task formulations.

Model	Accuracy	Macro-F1	Weighted F1	MCC	NO_REL. F1	RESULTS_IN F1	CONTRIB. F1
TF–IDF + Linear SVM	0.5460	0.4738	0.5605	0.1941	0.6445	0.5000	0.2769
Weighted RoBERTa	0.5000	0.4809	0.5143	0.2447	0.5464	0.5060	0.3902

Table 1. Performance of the relation classifiers on the frozen development validation set. Macro-F1 was the predefined primary model-selection criterion. Best values for each metric are shown in bold.

Model	Accuracy	Macro-F1	Weighted F1	MCC	NO_REL. F1	RESULTS_IN F1	CONTRIB. F1
TF–IDF + Linear SVM	0.6582	0.2741	0.7844	0.0128	0.7935	0.0268	0.0021
Weighted RoBERTa	0.5262	0.2420	0.6802	0.0197	0.6880	0.0331	0.0049

Table 2. Performance of the frozen relation classifiers on the sealed exhaustive gold-endpoint candidate set. The benchmark contained 21,038 ordered candidate pairs, of which 248 (1.18%) represented positive relations. Bold values indicate the higher value between the two models for each metric and do not represent post-test model selection.

Model	TP	FP	FN	TN	Precision	Recall	F1
TF–IDF + Linear SVM	96	7018	152	13772	0.0135	0.3871	0.0261
Weighted RoBERTa	125	9824	123	10966	0.0126	0.5040	0.0245

Table 3. Binary relation-detection results on the sealed exhaustive candidate set. The two positive relation classes were collapsed into a single relation-positive class for this diagnostic analysis.

Figure 2. Relation-prediction proliferation under exhaustive candidate generation. (a) Relation-positive outputs: Number of reference relations compared with the total number of relation-positive predictions produced by the frozen TF–IDF + Linear SVM and weighted RoBERTa classifiers. (b) Composition of predicted relations: Composition of each model’s relation-positive predictions as true positives and false positives. The sealed benchmark contained 21,038 ordered candidate pairs, of which only 248 (1.18%) were positive reference relations.

Figure 3. Performance transition from sampled development validation to exhaustive sealed evaluation. Macro-F1 and Matthews correlation coefficient (MCC) are shown for the TF–IDF + Linear SVM and weighted RoBERTa classifiers. Development evaluation used the sampled hard-negative distribution (2:1 negative-to-positive ratio), whereas sealed evaluation used exhaustive ordered candidate pairs, of which 98.82% were NO_RELATION . Both models exhibited substantially lower Macro-F1 and MCC under the naturally sparse exhaustive candidate distribution.

Outcome	$n$	Mean	Median	Q1	Q3	Minimum	Maximum
True positive	125	0.6787	0.7002	0.6234	0.7348	0.5168	0.7928
False positive	9824	0.6555	0.6602	0.5973	0.7153	0.5149	0.8001
False negative	123	0.3671	0.3504	0.3073	0.4221	0.2501	0.5220
True negative	10966	0.3800	0.3727	0.3184	0.4416	0.2186	0.5578

Table 4. Distribution of RoBERTa relation probabilities according to binary sealed-test prediction outcome. Relation probability is defined as

p_{\mathrm{rel}}=p(\texttt{RESULTS\_IN})+p(\texttt{CONTRIBUTES\_TO})

Figure 4. Distribution of frozen RoBERTa relation probabilities on the sealed benchmark. Distributions are shown for true-positive (TP), false-positive (FP), false-negative (FN), and true-negative (TN) candidate pairs. Relation probability was calculated as $p(\texttt{RESULTS\_IN})+p(\texttt{CONTRIBUTES\_TO})$ . The dashed horizontal line at 0.5 provides a visual reference for the binary relation/no-relation interpretation of the frozen three-class outputs and was not selected through post-test threshold optimization.

Source $\rightarrow$ Target	Candidates	Ref. Pos.	RoBERTa TP	RoBERTa FP	SVM FP
Event $\rightarrow$ Event	3570	151	84	1783	1268
Event $\rightarrow$ Phase of Flight	1859	39	34	975	610
Phase of Flight $\rightarrow$ Event	1859	4	4	943	778
Consequence $\rightarrow$ Event	1503	9	5	938	573
Hazard $\rightarrow$ Event	2090	52	33	909	608
Event $\rightarrow$ Hazard	2090	5	2	885	760

Table 5. Selected source–target concept-category directions from the sealed-test diagnostic analysis. TP and FP denote true-positive and false-positive relation predictions, respectively. Counts are category-membership based; because concept nodes may belong to multiple categories, rows are not mutually exclusive and should not be summed to recover the 21,038 unique candidate pairs.

Prompting Condition	TP	FP	FN	Precision	Recall	F1
Zero-shot	0	129	248	0.0000	0.0000	0.0000
One-shot	15	266	233	0.0534	0.0605	0.0567
Few-shot	9	734	239	0.0121	0.0363	0.0182
Retrieval-grounded	13	638	235	0.0200	0.0524	0.0289

Table 6. End-to-end SmolLM2-1.7B-Instruct relation-extraction results on the 300-narrative benchmark. Each prompting condition was evaluated against the same 248 reference relations. TP, FP, and FN denote relation-level true positives, false positives, and false negatives, respectively.

Abbreviation	Definition
AI	Artificial Intelligence
BERT	Bidirectional Encoder Representations from Transformers
CRF	Conditional Random Field
FP	False Positive
FN	False Negative
KG	Knowledge Graph

Table 11

Relation Extraction

Relation extraction identifies and classifies relationships between entities or events in text and multimodal data.

See all 82 papers

Semantic Causal-Factor Inference from Aviation Incident Narratives Using A Variational Autoencoder with Cosine-Similarity-Based Reconstruction
Oct 3, 2026AZIIDA NANYONGA1, HASSAN WASSWA, UGUR TURHAN +2Aviation Safety Variational Autoencoders
SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum and Minerals (KFUPM), Dhahran 31261, Saudi Arabia · School of Systems and Computing, University of New South Wales, Canberra, ACT 2600, Australia · School of Science, University of New South Wales, Canberra, ACT 2612, Australia +1
ConstructCIE: A Dataset for Extracting Causal Information from Construction Accident Narratives
Aug 6, 2026Hung Nguyen, Jaehoon Lee, Namgyun Kim +1Document Information Extraction Causal Attribution
Department of Computer Science, Texas A&M University · Department of Construction Science, Texas A&M University
A Hybrid Framework for Explainable Aviation Risk: Causal Chain Extraction from Safety Reports with Physics-Based Failure Quantification and Interventional Analysis
Oct 26, 2026Yutian Pang, Yifang Tian, Jiacheng Xie +4Aviation Safety Causal Inference
Department of Aerospace Engineering and Engineering Mechanics, The University of Texas at Austin, Austin, TX 78712, USA. · Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 3G4, Canada. · Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA. +2

Oct 3, 2026J/K move · Enter open · S save

Semantic Causal-Factor Inference from Aviation Incident Narratives Using A Variational Autoencoder with Cosine-Similarity-Based Reconstruction

AZIIDA NANYONGA1, HASSAN WASSWA, UGUR TURHAN, KEITH FRANCIS JOINER+1

SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum and Minerals (KFUPM), Dhahran 31261, Saudi Arabia · School of Systems and Computing, University of New South Wales, Canberra, ACT 2600, Australia · School of Science, University of New South Wales, Canberra, ACT 2612, Australia+1

Aviation Safety Variational Autoencoders

Open paper AskPDF arXiv

Aviation accident and incident investigations generate extensive unstructured textual information containing evidence relevant to the causes and contributing factors of safety occurrences. Automatically extracting such information is challenging because causal evidence may be distributed across long and complex investigation narratives. This study proposes a semantic causal-factor inference framework combining natural language processing with a variational autoencoder (VAE) to learn the relationship between aviation investigation narratives and expert-reported probable causes. Investigation narratives and their corresponding probable causes are transformed into numerical representations, after which the encoder maps narrative representations to a probabilistic latent space. The decoder estimates representations of the corresponding probable causes and is trained using an objective that combines Kullback-Leibler divergence with cosine-similarity-based semantic reconstruction. The framework was evaluated using 20,919 finalized U.S. National Transportation Safety Board investigation reports from 2005 to 2020. On the held-out test set, the predicted and expert-reported probable-cause representations achieved a mean cosine similarity of 0.786 (SD = 0.120). The predicted representations also yielded interpretable terms associated with causal information in the reports. The results demonstrate the potential of probabilistic latent representation learning for AI-assisted extraction of causal information from aviation safety narratives while retaining expert investigation as the basis for formal causal determination.

Abstract

Relation Extraction

Explore similar work

Semantic Causal-Factor Inference from Aviation Incident Narratives Using A Variational Autoencoder with Cosine-Similarity-Based Reconstruction

ConstructCIE: A Dataset for Extracting Causal Information from Construction Accident Narratives

A Hybrid Framework for Explainable Aviation Risk: Causal Chain Extraction from Safety Reports with Physics-Based Failure Quantification and Interventional Analysis