Real-Time Monocular Scene Analysis for UAV in Outdoor Environments

Yara AlaaEldin

Figure 1

Figure 2

Figure 1.1 : Our objective of designing a joint architecture for estimating both depth and semantic segmentation maps given as input video frames and camera pose data.

Board	GPU?	Memory	Storage	Power
Arduino Uno [ arduino_uno ]	No	32KB	-	5V
Arduino Portenta [ arduino_port ]	No	2MB	16MB	5V
Rapberry Pi [ raspberry ]	No	8GB	8GB	15W
Jetson Nano [ jetson ]	Yes	4GB	16GB	5W-10W
Jetson TX2 [ jetson ]	Yes	4/8GB	16/32GB	7.5W-15W
Jetson Xavier NX [ jetson ]	Yes	8/16GB	16GB	10W-20W

Table 1.1 : Specifications of commonly used micro-controllers and single board computers in robotics

Figure 1.2 : Sample images captured in simulation (a) and real-world (b) outdoor environments. It can be observed the differences in color, texture, lighting, and semantic detail between the two domains.

Figure 1.3 : Illustration of the camera moving from one frame

a

to the next

b

Figure 2.1 : Output depth map using SfM method [ meshroom ] resulting in holes and unknown regions in black

Figure 2.2 : Sample images from different aerial datasets that contain depth and semantic segmentation annotation

Figure 2.3 : Sample images from different aerial datasets that contain only depth annotation

Figure 2.4 : Sample images from different aerial datasets that contain only semantic segmentation annotation

Figure 2.5 : Sample images from different marine datasets that are annotated with either depth or semantic segmentation

Figure 2.6 : Sample images from different non-annotated marine datasets

Figure 3.1 : The architecture of M4Depth. It is fed by two consecutive frames and the camera motion. Each convolution layer is followed by a ReLU activation unit. The depth map is shown in the viridis scale. Details of the modules are given in Figure 3.2

Figure 3.2 : An illustration of the modules included in M4Depth. The preprocessing unit does not contain any learnable parameters. Feature vectors are subdivided into K sub-vectors using the split layer for subsequent parallel processing. The value of K at each level L is provided. For the encoder, the number of filters y at each level L is provided.

Figure 3.3 : Our M4Semantic Architecture. It is composed of an encoder and a decoder module with a pyramidal structure. Each level of the decoder is composed of a preprocessing unit and a semantic refiner.

Figure 3.4 : An illustration of the modules in our M4Semantic architecture.

N_{c}

is the number of semantic classes

Figure 3.5 : Our proposed joint architecture Co-SemDepth. It predicts both depth and semantic segmentation maps. It is composed of a shared encoder and two decoders. The encoder and the depth decoder are the same presented in [ m4depth ] . The semantic decoder makes use of the encoded feature maps to give an estimate of the semantic segmentation map. The depth and semantic maps get scaled up as they go forward through the successive levels of the decoders. The number of shown levels here is 3, while in our experiments, we use 5 levels. Depth map is shown in gray scale

Figure 3.6 : Sample images and annotations from the MidAir dataset

Figure 3.7 : Sample images and annotations from the AeroScapes dataset

Figure 3.8 : Loss curves showing the difference in the loss range of values for the two tasks during training. After multiplying the semantic loss (b) by a weighting factor of 0.15 (c), its range of values became comparable to the depth loss (a).

Architecture	Output	Params(M)	Inf. Time (ms/f)	Semantic	Depth
Architecture	Output	Params(M)	Inf. Time (ms/f)	mIoU $\uparrow$	RMSE $\downarrow$	RelErr $\downarrow$	$\delta 1$ $\uparrow$	$\delta 2$ $\uparrow$	$\delta 3$ $\uparrow$
M4Depth	D	3.06	44.9	-	6.99	0.109	92.0%	95.4%	97.0%
M4Semantic	S	2.61	9.8	76.80 %	-	-	-	-	-
Co-SemDepth	D+S	5.2	49.6	75.44%	6.70	0.096	92.3 %	95.7 %	97.2 %

Table 3.1 : Evaluation of our joint vs single architectures for depth estimation and semantic segmentation on the MidAir dataset.

Method	optimizer	lr	sched wt decay	max iter	epochs	batch
FCN	Adam	$1\times 10^{-4}$	-	-	60	4
ERFNet	Adam	$5\times 10^{-4}$	$1\times 10^{-4}$	-	60	6
SegFormer-B0	Adam	$6\times 10^{-5}$	$1\times 10^{-2}$	80K	-	4
RefineNet	Adam	$1\times 10^{-4}$	$1\times 10^{-4}$	-	60	8
TaskPrompter	Adam	$1\times 10^{-5}$	$1\times 10^{-6}$	80K	-	3

Table 3.2 : Training parameters used for benchmarking the baseline methods on MidAir.

Method	Output	Params(M)	Inf. Time (ms/f) $\downarrow$	Semantic	Depth
Method	Output	Params(M)	Inf. Time (ms/f) $\downarrow$	mIoU $\uparrow$	RMSE $\downarrow$	RelErr $\downarrow$	$\delta 1$ $\uparrow$	$\delta 2$ $\uparrow$	$\delta 3$ $\uparrow$
MonoDepth2 ∗	D	14.8	23.9	-	12.35	0.394	61.0%	75.1%	83.3%
ST-CLSTM ∗	D	15.04	35.3	-	13.69	0.404	75.1%	86.5%	91.1%
ManyDepth ∗	D	46.3	82.9	-	10.92	0.203	72.3%	87.6%	93.3%
PWCDC-Net ∗	D	9.4	25.8	-	8.35	0.095	88.7%	93.8%	96.2%
DepthAnythingV2 @	D	24.8	75.2	-	33.37	0.640	12.3%	25.6%	39.7%

Table 3.3 : Benchmarking Co-SemDepth on MidAir against other state-of-the-art methods in both depth estimation (D) and semantic segmentation (S). The top part reports single depth estimation methods, the middle part for single segmentation methods, and the bottom part for joint methods. ∗ means the depth metrics values were reported in [ m4depth ] . @ means zero-shot prediction.

Method	Sky	Water	Trees	Land	Boulders	Road	Others	mIoU
FCN(VGG16) [ fcn ]	88.56%	83.12%	75.50%	82.30%	29.97%	88.04%	54.60%	72.93%
FCN(MobileNetv2) [ fcn ]	87.82%	82.42%	73.42%	81.28%	26.57%	84.64%	46.09%	69.82%
ERFNet [ romera2017erfnet ]	91.50%	87.64%	82.48%	85.10%	40.63%	90.90%	63.70%	77.42%
SegFormer-B0 [ xie2021segformer ]	90.54%	88.58%	79.70%	83.33%	30.13%	92.19%	61.20%	75.10%
RefineNet [ nekrasov2019real ]	89.70%	81.60%	79.70%	82.20%	30.15%	91.36%	54.10%	72.69%
TaskPrompter [ taskprompter ]	90.60%	87.10%	79.10%	85.30%	41.00%	95.30%	83.20%	80.23 %

Table 3.4 : Per-Class IoU Evaluation of Co-SemDepth architecture and other baseline methods on MidAir.

Figure 3.9 : Qualitative evaluation of the semantic map predictions of ERFNet, FCN MobileNet, TaskPrompter, and Co-SemDepth respectively, on sample images from MidAir dataset.

Figure 3.10 : Qualitative evaluation of the zero-shot generalization performance of MidAir-trained Co-SemDepth on sample images from TartanAir (top and 2nd row) and WildUAV (3rd and bottom rows).

RMSE $\downarrow$	AbsRelErr $\downarrow$	$\delta 1$ $\uparrow$	$\delta 2$ $\uparrow$	$\delta 3$ $\uparrow$
14.9	0.303	58.7%	80.7%	91.5%

Table 3.5 : Depth estimation results on WildUAV obtained using Co-SemDepth tained on MidAir+TopAir

Sky	Water	Trees	Land	Rocks	Road	Others	mIoU
-	0.9%	32.8%	34.8%	-	14.9%	0%	16.7%

Table 3.6 : Semantic segmentation results on WildUAV obtained using Co-SemDepth tained on MidAir+TopAir

Figure 3.11 : Visualization of the predicted semantic maps using M4Semantic on sample images from the Aeroscapes test data.

Method	Params(M)	P/S	Open-Source	mIoU $\uparrow$
FCN-8S [ zheng2023deep ]	14.7	P	No	43.12%
FCN-16S [ zheng2023deep ]	14.7	P	No	44.52%
FCN-32S [ zheng2023deep ]	14.7	P	No	45.51%
FCN-ImageNet-4S [ aeroscapes ]	14.7	P	No	48.96%
FCN-Cityscapes [ aeroscapes ]	14.7	P	No	49.55%
FCN-ADE20K [ aeroscapes ]	14.7	P	No	51.62%

Table 3.7 : Comparison of our M4Semantic architecture with other semantic segmentation methods benchmarked on Aeroscapes. P means pretrained on other datasets and S means trained from scratch

Architecture	Inf. Time (ms/f) $\downarrow$	mIoU $\uparrow$
M4Semantic (4level)	9	75.1%
M4Semantic (5level)	9.8	76.8 %
M4Semantic (6level)	10.9	74.9%
Original-{DINL}	9.5	75.6%
Original-{Normalize}	9.7	74.1%
Original+{SNCV}	17.7	71.9%

Table 3.8 : Evaluation of our M4Semantic architecture on MidAir with the addition (+) or ablation (-) of different modules. The top part evaluates choosing a different number of levels. The bottom part was performed on M4Semantic (5level).

Setting	Value
SimMode	ComputerVision
ImageType	0, 3, 5
RecordOnMove	True
RecordInterval	0.05 (Sec)
Width	384
Height	384

Table 4.1 : Settings adjusted of AirSim during data acquisition

Figure 4.1 : Sample images from the collected TopAir dataset showing the variety of environments used and the variety of altitudes

Figure 4.2 : Applying a median filter on the collected depth and semantic segmentation maps of TopAir to smooth the appearance and remove unnecessary details

Figure 4.3 : Curating the semantic segmentation maps of the collected TopAir data by manually painting the Water region after applying a median filter

Figure 4.4 : TopAir per-pixel semantic segmentation classes distribution

Figure 4.5 : TopAir per-pixel depth values distribution

Figure 4.6 : An illustration of the convention of the camera and world reference frames used in AirSim while collecting TopAir data

Figure 4.7 : Illustration of the spatial-channel multi-task prompt learning framework used in each layer of the transformer in TaskPrompter. Task generic representations are extracted from the image patch tokens, while task-specific representations are extracted from the relations between spatial task prompts, channel task prompts, and image patch tokens. The cross-task interactions are extracted from the spatial task prompts. Taken from [ j8 ] , courtesy of the authors, in accordance with the policy of ICLR

Figure 4.8 : Histograms of pixel-wise depth distribution in each dataset. Each pin on the x-axis indicates all depth values less than or equal to the pin value. It can be marked how the range of depth values in WildUAV is more relevant to the depth values of the synthetic datasets used for training, while this does not hold true for DroneScapes.

Figure 4.9 : Histograms of pixel-wise class distribution in each dataset after mapping to our common set of segmentation classes

Train Dataset	optimizer	lr	weight decay	epochs	batch	Train size	Valid size	resolution
MidAir+TopAir	Adam	$10^{-4}$	-	60	3	16868	5381	$384\times 384$
SkyScenes+SynDrone	Adam	$10^{-4}$	-	60	3	12972	4138	$480\times 320$

Table 4.3 : Training parameters used for training Co-SemDepth on MidAir+TopAir and SkyScenes+SynDrone

Train Dataset	optimizer	lr	weight decay	iterations	batch	train size	valid size	resolution
MidAir+TopAir	Adam	$10^{-5}$	$10^{-6}$	80k	3	16868	5381	$384\times 384$
SkyScenes+SynDrone	Adam	$10^{-5}$	$10^{-6}$	80k	3	12972	4138	$480\times 320$

Table 4.4 : Training parameters used for training TaskPrompter on MidAir+TopAir and SkyScenes+SynDrone

Figure 4.10 : Plots of the train and validation RMSE and mIoU values of the models during training on MidAir+TopAir and SkyScenes+SynDrone. While TaskPrompter tends to overfit on the training data due to its high capacity, Co-SemDepth’s performance is almost equal on the train and validation data (no overfitting). For all the models, the checkpoint that produced the best validation results was selected.

Test Data	Train Data	Method	Params	RMSE $\downarrow$	AbsRelErr $\downarrow$	$\delta<1.25$ $\uparrow$	$\delta<1.25^{2}$ $\uparrow$	$\delta<1.25^{3}$ $\uparrow$
WildUAV	MidAir+TopAir	Co-SemDepth	5.2M	14.9	0.303	58.7 %	80.7 %	91.5 %
	MidAir+TopAir	TaskPrompter	126M	21.26	0.459	18.5%	37.5%	54.5%
	SkyScenes+SynDrone	Co-SemDepth	5.2M	18.9	0.388	33.78%	58.6%	73.4%
	SkyScenes+SynDrone	TaskPrompter	126M	23	0.48	10.5%	25.2%	46.2%

Table 4.5 : Evaluation of depth estimation performance of joint architectures on WildUAV real dataset. Best results are highlighted.

Figure 4.11 : Visualization of the estimated depth maps of sample input images from WildUAV using Co-SemDepth and TaskPrompter (displayed in relative scale where red is the highest distance and blue is the lowest). The output of Co-SemDepth is closer to the ground truth than TaskPrompter, and training on MidAir+TopAir (Mid+Top) is better than SkyScenes+SynDrone (Sky+Syn).

Test Data	Method	Train Data	Finetuning Data	RMSE $\downarrow$	AbsRelErr $\downarrow$	$\delta<1.25$ $\uparrow$	$\delta<1.25^{2}$ $\uparrow$	$\delta<1.25^{3}$ $\uparrow$
WildUAV	TaskPrompter	MidAir+TopAir	-	21.26	0.459	18.49%	37.46%	54.54%
	TaskPrompter	MidAir+TopAir	Dronescapes	15.87	0.32	31.42 %	62.39 %	84.27 %
	TaskPrompter	SkyScenes+SynDrone	-	23	0.48	10.5%	25.2%	46.2%
	TaskPrompter	SkyScenes+SynDrone	Dronescapes	18.05	0.354	29.4%	53.86%	73.14%
DroneScapes	TaskPrompter	MidAir+TopAir	-	96.77	0.78	0.1%	0.6%	2.5%
	TaskPrompter	MidAir+TopAir	Dronescapes	76.12	0.485	8.12%	34.53%	55.15%

Table 4.6 : Evaluation of depth estimation performance of TaskPrompter on WildUAV and DroneScape real datasets without and with adding real data to the training.

Figure 4.12 : Qualitative visualization of the estimated depth maps using TaskPrompter of sample input images from WildUAV before and after adding DroneScapes to the training (displayed in relative scale where red is the highest distance and blue is the lowest). Adding real data to the training enhanced the results.

Figure 4.13 : Visualization of the estimated depth maps (displayed in relative scale where red is the highest distance and blue is the lowest) of sample input images from FSI using TaskPrompter. The model trained on MidAir+TopAir produced the best output. Finetuning on DroneScapes enhanced the results of the model trained on SkyScenes+SynDrone

Figure 4.14 : Visualization of the estimated depth maps (displayed in relative scale where red is the highest distance and blue is the lowest) of sample input images from ICG using TaskPrompter. The output of the model trained on MidAir+TopAir is generally better than the others. Finetuning on DroneScapes did not enhance the results except in the last row (notice the umbrellas).

Test Data	Train Data	Method	Params	Sky	Water	Trees	Land	Vehicle	Rocks	Road	Building	Others
WildUAV	MidAir+TopAir	Co-SemDepth	5.2M	-	0.93%	32.8%	34.76%	0%	-	14.9%	-	0%
	MidAir+TopAir	TaskPrompter	126M	-	61.66%	55.36 %	67.63 %	1.65%	-	53.5 %	-	3.1%
	SkyScenes+SynDrone	Co-SemDepth	5.2M	-	17.63%	4.2%	14.1%	1.2%	-	1.8%	-	0.04%
	SkyScenes+SynDrone	TaskPrompter	126M	-	62.1 %	51%	62.05%	46.2 %	-	45.9%	-	5.9 %
FSI	MidAir+TopAir	Co-SemDepth	5.2M	-	10.1%	40.6%	34.67%	0%	-	9.14%	1.43%	0.01%
	MidAir+TopAir	TaskPrompter	126M	-	49.57 %	67.36 %	63.7%	3.32%	-	34.1%	59.83 %	1.73%

Table 4.7 : Per-Class IoU segmentation evaluation of the joint architectures on a variety of real data. Best results for each dataset are highlighted.

Figure 4.15 : Qualitative visualization of the estimated depth maps of sample input images from FSI using different training modalities and architectures. It can be seen that using TaskPrompter trained on MidAir+TopAir gives the closest results to the ground truth. TaskPrompter trained on SkyScenes+SynDrone is the best in the detection of cars (top row).

Figure 4.16 : Qualitative visualization of the estimated segmentation maps of sample input images from ICG using different training modalities. Best performing method is TaskPrompter trained on MidAir+TopAir except in the detection of cars (third row) where TaskPrompter trained on SkyScenes+SynDrone is better.

Test Data	Train Data	Finetuning Data	Method	Sky	Water	Trees	Land	Vehicle	Rocks	Road	Building	Others
WildUAV	MidAir+TopAir	-	TaskPrompter	-	61.66%	55.36%	67.63%	1.65%	-	53.5%	-	3.1%
	MidAir+TopAir	Dronescapes	TaskPrompter	-	59.36%	56.88%	77.54%	7.62%	-	39.12%	-	2.6%
	SkyScenes+SynDrone	-	TaskPrompter	-	62.1%	51%	62.05%	46.2%	-	45.9%	-	5.9%
	SkyScenes+SynDrone	Dronescapes	TaskPrompter	-	63.9%	47.77%	55.66%	22.89%	-	45.26%	-	5.5%
RuralScapes	MidAir+TopAir	-	TaskPrompter	92.7%	9.13%	40.14%	42.12%	0.2%	-	23.1%	31.45%	6.8%
	MidAir+TopAir	Dronescapes	TaskPrompter	89.61%	14.07%	46.8%	47.65%	2.1%	-	33.6%	46.93%	7.8%

Table 4.8 : Per-Class IoU segmentation evaluation of joint architectures on WildUAV and RuralScapes without and with adding real data to the training.

Figure 4.17 : Qualitative visualization of the estimated semantic segmentation maps of sample input images from RuralScapes using TaskPrompter before and after adding real data (DroneScapes) to the training

Figure 4.18 : 3D semantic map reconstruction using the predicted depth and segmentation maps using TaskPrompter (trained on MidAir

+

TopAir) on a sample image from the FSI dataset. Depth is truncated at 200m.

Figure 4.19 : 3D semantic map reconstruction using the predicted depth and segmentation maps using TaskPrompter (trained on SkyScenes

+

SynDrone

+

DroneScapes) on a sample image from DroneScapes test data. Depth is truncated at 200m

Figure 5.1 : Overview of the Cycle-GAN method for image style transfer [ cyclegan ] . The process is composed of two mapping functions

G

and

F

(a). The loss used for training of the networks is the addition of two losses: forward cycle-consistency loss (b), and backward cycle-consistency loss (c).

Figure 5.2 : a simple illustration of the forward and reverse diffusion processes

Figure 5.3 : Overview of InST method for image style transfer using Stable Diffusion Models (SDM) [ inst ] . CLIP [ clip ] is used to give image embedding of the style image, then it is inserted as a caption conditioning in the standard form of SDM.

Figure 5.4 : Sample output of style transfer of images from SynDrone using Cycle-GAN

Figure 5.5 : Sample output of style transfer of images from TopAir using Cycle-GAN

Figure 5.6 : Sample output of style transfer of images from SynDrone using Diffusion Model InST. It can be noted that the output images do not hold exactly the same semantic segmentation layout.

Figure 5.7 : Sample output of style transfer of images from TopAir using Diffusion Model InST. It can be noted that the output images do not hold exactly the same semantic segmentation layout.

Figure 6.1 : Sample images from the collected marine data in the Tropical Island environment

Figure 6.2 : Plots of the metrics evaluated on the validation data of MidSea while training Co-SemDepth

Method	Semantic Metrics	Depth Metrics
	mIoU $\uparrow$	RMSE $\downarrow$	AbsRelErr $\downarrow$	$\delta<1.25$ $\uparrow$	$\delta<1.25^{2}$ $\uparrow$	$\delta<1.25^{3}$ $\uparrow$
Base CoSemDepth	83.38%	4.24	0.092	81.7%	95.5%	98.5%
CoSemDepth + custom augm.	91.21%	3.59	0.081	88.0%	97.3%	98.9%
Finetuned MoCo	75.0%	9.03	0.265	67.3%	79.1%	87.4%
ViT+DINO encoder	79.11%	9.42	0.221	65.0%	75.2%	83.4%

Table 6.1 : Comparison of different methods on the collected Synthetic dataset MidSea. Base CosemDepth includes the default augmentations, while the "custom augm." version includes more aggressive color jittering and z-axis rotation.

Figure 6.3 : Sample prediction of depth and segmentation maps on MidSea. Depth is shown in a linear scale, and distance values greater than 255 meters are clamped to 255; that’s why the far island disappears in the depth map. Semantic segmentation color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Figure 6.4 : Sample semantic predictions on SMD using Co-SemDepth trained on the synthetic MidSea. Semantic segmentation color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Figure 6.5 : Sample semantic predictions on foggy images from SMD using Co-SemDepth trained on the synthetic MidSea. Semantic segmentation color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Figure 6.6 : Sample zero-shot predictions on images from MIT dataset using Co-SemDepth trained on the synthetic MidSea. Depth is displayed in a linear scale and clamped at 255 meters. Semantic segmentation color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Figure 6.7 : Sample prediction on MIT dataset after finetuning Co-SemDepth on MaSTr dataset. Color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Figure 6.8 : Sample predictions on MIT dataset using Co-SemDepth after adding custom augmentations. Depth is viewed in metric scale. Segmentation color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Figure 6.9 : Sample predictions on the MIT dataset using Co-SemDepth after finetuning the weights obtained from MoCo pre-training. Depth is viewed in metric scale. Segmentation color index is: Blue

=

Sea, Cyan

=

Sky, Yellow

=

Ground/Buildings/Vegetation, Magenta

=

Other

Real-Time Monocular Scene Analysis for UAV in Outdoor Environments

Abstract

Figures & tables

Appendix

Explore similar work

AerialMetric: Benchmarking and Adapting UAV Monocular Metric Depth Estimation in the Real World

Synthetic-to-Real ViT-Based Pose Estimation of a Noncooperative UAV

DAPM: UAV Monocular Depth Estimation from Any Height, Pitch, Roll and FOV