MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency

Markus Gross; Andreas Greiner; Taehyoung Kim; Sivasubiramaniam Subbiah; Tomaž Cotič; Sai Bharadwaj Matha; Conrad Christoph; Oussema Dhaouadi

cs.ROOct 7, 2026

MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency

Authors: Markus Gross, Andreas Greiner, Taehyoung Kim, Sivasubiramaniam Subbiah, Tomaž Cotič, Sai Bharadwaj Matha, Conrad Christoph, Oussema Dhaouadi, +7 more

Organizations: Autonomous Aerial Systems, Fraunhofer Institute IVI · Computer Vision Group, Technical University of Munich · Munich Center for Machine Learning (MCML) · Computer Vision for Digital Twins, University of Cambridge · Institute of Innovative Mobility, Univ. of Applied Sciences Ingolstadt

Abstract

We introduce MultiFly, a real-world, low-altitude UAV dataset for semantic perception across RGB, thermal, LiDAR, and radar modalities. MultiFly provides 17,272 synchronized samples from four suburban scenes with frame-wise annotations for 15 semantic classes, together with calibration and GNSS-RTK/IMU measurements. To avoid costly and inconsistent modality-specific annotation, we propagate labels from only 115 manually annotated RGB images through shared geometric representations to all four modalities. This approach generates semantic labels for 17,157 additional RGB images, 17,272 thermal images, 840M LiDAR points, and 3.4M radar points. Transferred annotations achieve 89.93% average agreement with held-out manual annotations, and 90.94% average semantic consistency across all six modality pairs. We further establish semantic segmentation benchmarks for all four modalities, revealing distinct architectural behavior for dense LiDAR and sparse radar data. Taken together, MultiFly provides a scalable foundation for multimodal aerial perception and, to the best of our knowledge, the first public real-world low-altitude aerial benchmark that combines consistent frame-wise semantic annotations for RGB, thermal, LiDAR, and radar. Data at https://github.com/markus-42/multifly.

Figures & tables

Dataset	Perception Modalities				Auxiliary Signals		Sample Count	Environments	Altitude [m]	Semantic Classes	Cross-Modal Semantic Consistency	Manual Correction Not Required
Dataset	RGB	Thermal	LiDAR	Radar	IMU	GNSS	Sample Count	Environments	Altitude [m]	Semantic Classes	Cross-Modal Semantic Consistency	Manual Correction Not Required
IndraEye [ 1 ]	✓	✓	-	-	-	-	$5{,}612$	urban	$7$ – $30$	$13$	-	-
MVUAV [ 2 ]	✓	✓	-	-	-	-	$2{,}183$	urban	$5$ – $20$	$36$	-	-
CART [ 3 ]	✓	✓	-	-	-	-	$2{,}282$	rural	$40$	$11$	-	-
Kust4K [ 4 ]	✓	✓	-	-	-	-	$4{,}024$	urban	-	$8$	-	-
SegFly [ 5 ]	✓	✓	-	-	-	-	$15{,}007$	urban, industrial, rural	$30$ , $40$ , $50$	$15$	✓	✓

TABLE I : Comparison of real-world multimodal UAV datasets with frame-wise semantic labels (see Section II ).

Sensor	Model	Count	Frequency	Data Type	Field of View			Resolution
Sensor	Model	Count	Frequency	Data Type	Range	Azimuth	Elevation	Range	Azimuth	Elevation
RGB Camera and Lens	FLIR Blackfly S GigE BFS-PGE-31S4C-C with Tamron M112FM06	$1$	$10\text{\,}\mathrm{H}\mathrm{z}$	$8\text{-}\mathrm{bit}$ Bayer image	-	$62\text{\,}\mathrm{\SIUnitSymbolDegree}$	$48\text{\,}\mathrm{\SIUnitSymbolDegree}$	-	$2048\text{\,}\mathrm{px}$	$1536\text{\,}\mathrm{px}$
Thermal Camera	IRCAM NIK III microbolometer camera	$1$	$20\text{\,}\mathrm{H}\mathrm{z}$ raw $10\text{\,}\mathrm{H}\mathrm{z}$ selected	$16\text{-}\mathrm{bit}$ monochrome thermal image	-	$57.5\text{\,}\mathrm{\SIUnitSymbolDegree}$	$47.39\text{\,}\mathrm{\SIUnitSymbolDegree}$	-	$1280\text{\,}\mathrm{px}$	$1024\text{\,}\mathrm{px}$
LiDAR	Ouster OS1-128	$1$	$10\text{\,}\mathrm{H}\mathrm{z}$	x, y, z, intensity, ambient, reflectivity	$90\text{\,}\mathrm{m}$ recommended $200\text{\,}\mathrm{m}$ maximum	$360\text{\,}\mathrm{\SIUnitSymbolDegree}$ $90\text{\,}\mathrm{\SIUnitSymbolDegree}$ selected	$45\text{\,}\mathrm{\SIUnitSymbolDegree}$	-	$2048$ cols per scan	$128$ channels
Radar	Continental ARS 548 RDI	$2$	$10\text{\,}\mathrm{H}\mathrm{z}$	x, y, z, velocity, RCS, range, azimuth, elevation	$301\text{\,}\mathrm{m}$ $1514\text{\,}\mathrm{m}$ extended	$100\text{\,}\mathrm{\SIUnitSymbolDegree}$ $120\text{\,}\mathrm{\SIUnitSymbolDegree}$ processed	$8\text{\,}\mathrm{\SIUnitSymbolDegree}$ $28\text{\,}\mathrm{\SIUnitSymbolDegree}$ below $100\text{\,}\mathrm{m}$	$0.22\text{\,}\mathrm{m}$	$1.2\text{\,}\mathrm{\SIUnitSymbolDegree}$ near boresight $1.68\text{\,}\mathrm{\SIUnitSymbolDegree}$ at $\pm 45\text{\,}\mathrm{\SIUnitSymbolDegree}$	$2.3\text{\,}\mathrm{\SIUnitSymbolDegree}$
GNSS-RTK / INS	OxTS xRED with dual GNSS receivers	$1$	$100\text{\,}\mathrm{H}\mathrm{z}$	time, latitude, longitude, altitude, linear velocity, angular velocity, yaw, pitch, roll, status	-	-	-	-	-	-

TABLE II : Sensor specifications of the MultiFly sensor suite. For more details, see Section III .

Fig. 2 : MultiFly sensor rig (see Section III-A ).

Fig. 3 : Registration results (see Section III-D ).

	MultiFly Scenes and Altitudes								Total
	#1		#2		#3		#4
Statistic	$30\text{\,}\mathrm{m}$	$50\text{\,}\mathrm{m}$	$30\text{\,}\mathrm{m}$	$50\text{\,}\mathrm{m}$	$30\text{\,}\mathrm{m}$	$50\text{\,}\mathrm{m}$	$30\text{\,}\mathrm{m}$	$50\text{\,}\mathrm{m}$
Synchronized samples	2,209	2,903	1,987	2,166	1,766	2,214	2,097	1,930	17,272
IMU/GNSS-RTK pose	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	—
Covered area [ ${\mathrm{m}}^{2}$ ]	17,400		11,100		13,000		15,200		56,700

TABLE III : MultiFly dataset statistics (see Section V-B ).

Fig. 4 : MultiFly sample with semantic classes (Sec. V-B ).

Altitude	RGB	Thermal	LiDAR	Radar	Average
$30\text{\,}\mathrm{m}$	$91.78\text{\,}\%$	$90.51\text{\,}\%$	$91.80\text{\,}\%$	$85.92\text{\,}\%$	$90.00\text{\,}\%$
$50\text{\,}\mathrm{m}$	$92.47\text{\,}\%$	$90.11\text{\,}\%$	$91.11\text{\,}\%$	$85.76\text{\,}\%$	$89.86\text{\,}\%$
Average	$92.12\text{\,}\%$	$90.31\text{\,}\%$	$91.45\text{\,}\%$	$85.84\text{\,}\%$	$89.93\text{\,}\%$

TABLE IV : Modal-wise label transfer evaluation (Sec. VI ).

Modality Pair	MultiFly Scene				Average
Modality Pair	#1	#2	#3	#4	Average
2D–2D
RGB–Thermal	$90.82\text{\,}\%$	$89.83\text{\,}\%$	$91.97\text{\,}\%$	$91.47\text{\,}\%$	$91.02\text{\,}\%$
2D–3D
RGB–LiDAR	$91.09\text{\,}\%$	$91.41\text{\,}\%$	$90.96\text{\,}\%$	$92.79\text{\,}\%$	$91.56\text{\,}\%$
RGB–Radar	$91.89\text{\,}\%$	$89.14\text{\,}\%$	$85.84\text{\,}\%$	$88.07\text{\,}\%$	$88.73\text{\,}\%$

TABLE V : Cross-modal semantic consistency across all modalities and scenes of MultiFly (see Section VI-B ).

Fig. 5 : Cross-modal semantic consistency (Sec. VI-B ).

	UPerNet [ 26 ]	SegFormer [ 27 ]	Firefly [ 5 ]
RGB
Mean accuracy	$53.17\%$	$\underline{55.62\%}$	$\mathbf{58.21\%}$
Mean intersection over union	$39.96\%$	$\underline{41.77\%}$	$\mathbf{44.05\%}$
Thermal
Mean accuracy	$44.22\%$	$\underline{47.81\%}$	$\mathbf{53.42\%}$
Mean intersection over union	$32.08\%$	$\underline{34.71\%}$	$\mathbf{40.24\%}$

TABLE VI : RGB and Thermal experiments ( Section VII-B ).

Fig. 6 : RGB and Thermal experiments (see Section VII-B ).

	SpUNet [ 28 ]	LitePT [ 29 ]	PTv3 [ 30 ]
LiDAR
Mean accuracy	$49.49\%$	$\underline{49.91\%}$	$\mathbf{51.76\%}$
Mean intersection over union	$34.72\%$	$\underline{35.62\%}$	$\mathbf{37.09\%}$
Radar
Mean accuracy	$\mathbf{25.95\%}$	$21.34\%$	$\underline{22.68\%}$
Mean intersection over union	$\mathbf{14.50\%}$	$12.09\%$	$\underline{12.90\%}$

TABLE VII : LiDAR and Radar experiments ( Section VII-C ).

Fig. 7 : LiDAR and Radar experiments (see Section VII-C ).

Explore similar work

SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale
Mar 18, 2026Markus Gross, Sai Bharadhwaj Matha, Rui Song +4Remote Sensing Image Segmentation Cross-Modal Image Registration
Fraunhofer Institute IVI · TU Munich · MCML +2
GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception
Aug 28, 2026Jingpu Yang, Debin Tang, Yilin Sun +4Cross-Modal Alignment Multimodal Pretraining
Beihang University, Beijing, China. · Zhongguancun Academy, Beijing, China. · Northeastern University, Shenyang, China.
PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation
Apr 17, 2026Shuyan Ke, Yifan Mei, Changli Wu +4Image Segmentation Remote Sensing Image Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute

Mar 18, 2026, cs.CVJ/K move · Enter open · S save

SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale

Markus Gross, Sai Bharadhwaj Matha, Rui Song, Viswanathan Muthuveerappan+3

Fraunhofer Institute IVI · TU Munich · MCML+2

Remote Sensing Image Segmentation Cross-Modal Image Registration Semantic Segmentation

Open paper AskPDF arXiv

Semantic segmentation for uncrewed aerial vehicles (UAVs) is fundamental for aerial scene understanding, yet existing RGB and RGB-T datasets remain limited in scale, diversity, and annotation efficiency due to the high cost of manual labeling and the difficulties of accurate RGB-T alignment on off-the-shelf UAVs. To address these challenges, we propose a scalable geometry-driven 2D-3D-2D paradigm that leverages multi-view redundancy in high-overlap aerial imagery to automatically propagate labels from a small subset of manually annotated RGB images to both RGB and thermal modalities within a unified framework. By lifting less than 3% of RGB images into a semantic 3D point cloud and rendering it into all views, our approach enables dense pseudo ground-truth generation across large image collections, automatically producing 97% of RGB labels and 100% of thermal labels while achieving 91% and 88% annotation accuracy without any 2D manual refinement. We further extend this 2D-3D-2D paradigm to cross-modal image registration, using 3D geometry as an intermediate alignment space to obtain fully automatic, strong pixel-level RGB-T alignment with 87% registration accuracy and no hardware-level synchronization. Applying our framework to existing geo-referenced aerial imagery, we construct SegFly, a large-scale benchmark with over 20,000 high-resolution RGB images and more than 15,000 geometrically aligned RGB-T pairs spanning diverse urban, industrial, and rural environments across multiple altitudes and seasons. On SegFly, we establish the Firefly baseline for RGB and thermal semantic segmentation and show that both conventional architectures and vision foundation models benefit substantially from SegFly supervision, highlighting the potential of geometry-driven 2D-3D-2D pipelines for scalable multi-modal aerial scene understanding. Data and Code available at https://github.com/markus-42/SegFly.

Abstract

Explore similar work

SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale

GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception

PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation