Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection

Yuyi Zhi; Min Du; Bill Shi; Guodong Yuan; Junheng Liang; Peifan Liao; Xin Guan

cs.CVOct 10, 2026

Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection

Authors: Yuyi Zhi, Min Du, Bill Shi, Guodong Yuan, Junheng Liang, Peifan Liao, Xin Guan

Organizations: Yuyi Zhi is with Zhejiang University. · Min Du, Bill Shi, Guodong Yuan, and Xin Guan are with Avant Robotics. · Junheng Liang is with University of Science and Technology Beijing. · Peifan Liao is with Tsinghua University.

Abstract

Synthetic UAV imagery can expand pre-deployment test evidence when task-specific Real data are scarce, but visual plausibility or semantic label alignment alone does not establish validity for model selection. We study when Synthetic evaluation preserves the model-selection conclusions obtained from Real UAV data through two complementary cases under fixed evaluation protocols. In a requirement-specific road-debris benchmark, four language-conditioned perception systems obtain lower absolute F1 on Synthetic data, yet the point-estimate ordering and top-ranked model are preserved; bootstrap analysis yields no supported pairwise reversal, and size-matched resampling preserves the top-model decision. In contrast, a closed-set stress test on a separately materialized business corpus, using four VisDrone-compatible detectors over eight empirically supported classes, changes the point-estimate top-ranked detector and reverses three of six pairwise point-estimate preference directions. A complementary audit of five UAV business scenarios further reveals substantial Real-Synthetic mismatch in class composition, object density, and target scale, including systematic under-representation of small targets. These results show that Synthetic UAV benchmark trust is conditional: semantic coverage is insufficient, and validity must be established empirically for the task, operating regime, and model panel of interest.

Figures & tables

Fig. 1: Overview of the Real–Synthetic validation study. The construction example and pictured SUTs refer to road debris; the VisDrone-overlap test uses a separately materialized corpus and a distinct detector panel. The two cases yield different model-selection outcomes, while the five-scenario audit documents distributional discrepancies in the business corpus.

Fig. 2: Road-debris Synthetic test-set construction pipeline. A Real UAV source frame and task instruction condition image generation, followed by multimodal screening, human review, manual bounding-box annotation, independent cross-review, and provenance recording before admission to the final evaluation set.

Panel of Figure 3 — Fig. 3: Pairwise Real–Synthetic model-selection fidelity. Black dots denote point-estimate performance margins and horizontal bars denote 95% bootstrap intervals. Preserved, reversed, and unresolved outcomes are determined by the interval positions relative to zero.

Model	System type	Task interface
Gemini 3.5 Flash	General MLLM	Road-debris instruction
Qwen3-VL-8B	General MLLM	Road-debris instruction
LocateAnything-3B	V-L grounding	road debris
Rex-Omni	Detection MLLM	road debris

TABLE I: Road-debris SUT panel and fixed task interfaces.

Fig. 4: Representative Real and Synthetic examples from the five business-scenario families. The examples illustrate semantic scenario coverage in the separately materialized business corpus but do not establish Real–Synthetic evaluation validity.

Model	P	R	F1	$\Delta F_{1}$
Gemini	0.928/0.734	0.619/0.610	0.742/0.667	-0.076
Qwen3-VL	0.628/0.466	0.477/0.421	0.542/0.443	-0.100
LocateAnything	0.843/0.440	0.374/0.358	0.518/0.395	-0.124
Rex-Omni	0.541/0.188	0.410/0.431	0.467/0.262	-0.205

TABLE II: Road-debris performance at IoU

=0.5

(Real/Synthetic).

Model	TP/img	FP/img	Zero %	Matched IoU
Gemini	3.39/3.36	0.26/1.22	1.3/1.0	0.814/0.816
Qwen3-VL	2.62/2.32	1.55/2.66	7.9/7.0	0.771/0.782
LocateAnything	2.05/1.97	0.38/2.51	28.9/35.0	0.792/0.773
Rex-Omni	2.25/2.37	1.91/10.24	2.6/0.0	0.745/0.755

TABLE III: Road-debris diagnostics (Real/Synthetic). TP and FP are per image; IoU is mean IoU over matched TPs.

Model	AP	AP50	AP75	AR@300	Rank
Faster R-CNN	20.45/26.37	38.88/39.43	19.34/26.92	36.71/51.43	1/4
RF-DETR	19.13/39.12	35.94/51.78	17.92/41.90	33.96/71.55	2/1
YOLO11m	18.99/31.37	33.18/41.01	19.21/33.59	29.51/46.78	3/2
YOLOv8m	18.25/31.23	31.63/40.18	18.45/32.33	28.22/48.40	4/3

TABLE IV: VisDrone-overlap performance (Real/Synthetic, 0–100). Rank is by AP; maxDets =300.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 236 papers

Synthetic-to-Real ViT-Based Pose Estimation of a Noncooperative UAV
Oct 3, 2026Krishnanujam Srinivas, Hanish Acharla, Brij Agrawal +1Object Pose Estimation Synthetic-to-Real Domain Adaptation
Spacecraft Research and Design Center, Department of Mechanical and Aerospace Engineering, Naval Postgraduate School, Monterey.
Beyond Clear Skies: Synthetic Seasonal and Weather Variations for Real-World Drone Detection
Aug 17, 2026Tamara R. Lenhard, Andreas Weinmann, Tobias KochAerial Object Detection Synthetic Data Augmentation
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR), Sankt Augustin, Germany · ACIDA Lab, Technical University of Applied Sciences Würzburg-Schweinfurt, Schweinfurt, Germany
Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance
Jun 18, 2026Myeongseok Nam, Donghun Yeo, Seungwook KimDistribution Matching Object Detection
GenGenAI, South Korea

Oct 3, 2026J/K move · Enter open · S save

Synthetic-to-Real ViT-Based Pose Estimation of a Noncooperative UAV

Krishnanujam Srinivas, Hanish Acharla, Brij Agrawal, Leonardo Herrera

Spacecraft Research and Design Center, Department of Mechanical and Aerospace Engineering, Naval Postgraduate School, Monterey.

Object Pose Estimation Synthetic-to-Real Domain Adaptation Vision Transformer

Open paper AskPDF arXiv

Remote pose estimation of noncooperative Unmanned Aerial Vehicles (UAVs) from imagery is critical, as they cannot be influenced or instrumented in advance. Deep-learning-based approaches offer a promising solution; however, their development is constrained by the cost and difficulty of acquiring large-scale real-world datasets with accurate pose labels. Synthetic imagery provides an alternative, but models trained on synthetic data must overcome the synthetic-to-real domain gap to generalize to real-world imagery. This work investigates the inherent synthetic-to-real generalization capability of a Vision Transformer (ViT)-based model for monocular UAV pose estimation. The proposed approach employs a self-supervised DINOv2 backbone and is trained exclusively on labeled synthetic imagery while being evaluated on labeled real-world imagery. Pose ambiguity-aware strategies are incorporated during training and inference to address ambiguities arising from the projection of a three-dimensional target onto a two-dimensional image plane and from target symmetries. An

α

β

filter is further integrated during inference to improve pose estimations. To assess the model under operational requirements, it is evaluated in terms of Mean Angular Error (MAE) and inference time, both before and after filtering, using a real-world dataset containing 77,077 labeled UAV images. Before filtering, the model achieves an MAE of

19.18^{\circ}

and an inference time of

13.25

ms, whereas after filtering, these values are

8.74^{\circ}

and

13.42

ms, respectively.

Abstract

Synthetic Data Evaluation

Explore similar work

Synthetic-to-Real ViT-Based Pose Estimation of a Noncooperative UAV

Beyond Clear Skies: Synthetic Seasonal and Weather Variations for Real-World Drone Detection

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance