Authors: Anas Shrinah, Chris Bennett, Kerstin Eder
Organizations: Department of Software Engineering, Applied Science Private University, Amman, Jordan · Department of Computer Science, University of Bristol, Bristol, United Kingdom
We investigate a symmetry-aware surrogate-assisted testing approach for UAV autopilots. The proposed approach exploits geometric symmetries between flight missions by trans- forming obstacle configurations into a shared canonical coordi- nate frame. This enables a neural-network surrogate trained on a single reference mission to be reused across arbitrary missions, reducing the amount of high-fidelity simulation data required for training. The surrogate model is trained to predict the minimum clearance distance between a UAV and surrounding obstacles and is used to guide test generation towards safety-critical scenarios. We evaluate this approach using the Aerialist UAV test bench and benchmark its performance against techniques from the 2024 to 2026 UAV Testing Competitions. Results show that the approach achieves an average critical-test-case hit rate of 42% across the evaluated missions, compared with approximately 10% for the constrained-random baseline used to generate the training data. This approach also achieves high input diversity and competitive benchmark performance. However, our findings reveal limitations in transferring surrogate predictions across missions with differing flight-path characteristics, highlighting the boundaries of geometric canonicalization. To support reproducibility and future research, we provide the trained surrogate model and a dataset comprising more than 2,500 high-fidelity simulations.
Figures & tables
Fig. 1 : A segment of an arbitrary mission path is transformed to the canonical mission representation. The segment-placement zone is chosen in the arbitrary mission frame such that when transformed between representations, it always remains inside the designated obstacle-placement area.
Fig. 2 : The architecture of the surrogate model.
Fig. 3 : The flight missions, path segments, and segment-placement zones for transformation to the canonical mission.
Mission
RMSE
CS2
1.35m
CS4
0.71m
CS5
1.46m
Validation Set
0.50m
TABLE I : Surrogate Model Prediction Error
Benchmark
Simulation
Flying
Number of Test Cases
Hit Rate
Year
Budget
Mission
Timed Out
Non-Critical
Critical
(%)
2024
200
CS2
22
116
62
31
200
CS4
1
111
88
44
200
CS5
15
106
79
40
2025/2026
100
CS2
17
43
40
40
100
CS4
5
36
59
59
TABLE II : Test case outcomes and hit rates across benchmark years and flight missions.
CS2
CS4
CS5
Overall
Tool
Tests (critical)
Score (rank)
Tests (critical)
Score (rank)
Tests (critical)
Score (rank)
Score sum (rank sum)
Our approach
62 (19)
18.27 (1)
88 (18)
20.32 (1)
78 (16)
19.21 (1)
57.80 (3)
WOGAN-UAV
61 (14)
12.55 (2)
81 (4)
2.35 (4)
39 (12)
8.40 (2)
23.30 (8)
CAMBA
36 (7)
11.84 (3)
33 (0)
0.00 (7)
11 (1)
3.16 (3)
15.00 (13)
TUMB
69 (2)
0.12 (8)
135 (16)
7.12 (3)
114 (7)
2.73 (4)
9.97 (15)
DeepHyperion
2 (2)
1.22 (6)
10 (8)
7.74 (2)
22 (4)
1.08 (7)
10.04 (15)
TABLE III : Benchmark results for the 2024 evaluation. Each entry reports the number of test cases (critical test cases) or the score (score rank).
CS2
CS4
CS5
Overall
Tool
Tests (critical)
Failure (rank)
Diversity (rank)
Tests (critical)
Failure (rank)
Diversity (rank)
Tests (critical)
Failure (rank)
Diversity (rank)
Rank value
Our approach
40 (20)
11.90 (1)
0.94 (2)
59 (19)
9.14 (2)
0.85 (4)
47 (19)
10.89 (1)
0.93 (1)
4.75
Pseudo-Random
31 (7)
6.54 (3)
0.86 (5)
66 (12)
9.30 (1)
0.87 (3)
17 (5)
2.71 (3)
0.82 (5)
8.5
Evolv-1
57 (12)
9.02 (2)
0.88 (4)
22 (2)
1.08 (6)
0.82 (5)
37 (6)
2.93 (2)
0.85 (4)
10.75
TGen-UQ
5 (1)
0.50 (5)
1.00 (1)
6 (1)
1.44 (5)
0.94 (1)
6 (0)
0.00 (6)
0.89 (2)
13.00
PALM
58 (1)
0.14 (7)
0.57 (6)
82 (20)
3.58 (3)
0.42 (6)
76 (19)
0.80 (4)
0.46 (6)
15.00
TABLE IV : Benchmark results for the 2025 evaluation. Each entry reports test cases (critical test cases) or a failure/diversity score (rank).
CS2
CS4
CS5
Overall
Tool
Tests (critical)
Failure (rank)
Din (rank)
Dout (rank)
Tests (critical)
Failure (rank)
Din (rank)
Dout (rank)
Tests (critical)
Failure (rank)
Din (rank)
Dout (rank)
Rank value
Intelligent-UAV-Gen
66 (20)
19.23 (1)
10.74 (2)
4.13 (2)
16 (4)
3.12 (2)
6.89 (3)
3.07 (2)
77 (19)
25.33 (1)
8.64 (2)
3.11 (3)
5.50
Our approach
40 (20)
11.90 (3)
39.11 (1)
2.72 (3)
59 (19)
9.14 (1)
16.85 (1)
1.37 (4)
47 (19)
10.89 (2)
32.76 (1)
2.41 (4)
6.50
TLQD
100 (7)
6.66 (4)
9.79 (3)
23.31 (1)
100 (4)
1.11 (4)
16.74 (2)
14.44 (1)
100 (10)
9.13 (3)
4.74 (3)
39.62 (1)
8.25
Surrealist
80 (17)
18.25 (2)
2.80 (4)
1.00 (4)
13 (4)
2.44 (3)
2.68 (4)
2.67 (3)
79 (0)
0.00 (4)
3.00 (4)
4.26 (2)
9.75
TABLE V : Benchmark results for the 2026 evaluation. Each entry reports test cases (critical test cases) or a failure/diversity score (rank).
School of Computing, Macquarie University, Sydney, NSW, Australia. · Department of Computer Science, University of California, Santa Barbara, CA, USA. · Anitron
School of Engineering and Computing, Christopher Newport University · Department of Mathematics and Statistics, Texas Tech University · Department of Computer Science, Texas A&M University-Corpus Christi