DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems

Yasaswini Chebolu

cs.CVMar 17, 2026

DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems

Authors: Yasaswini Chebolu

Organizations: Department of Computer Science & Engineering (AI & ML), Gayatri Vidya Parishad College of Engineering for Women, Visakhapatnam, India

Abstract

Reliable terrain perception is a fundamental requirement for autonomous navigation in unstructured, off-road environments. Desert landscapes present unique challenges due to low chromatic contrast between terrain categories, extreme lighting variability, and sparse vegetation that defy the assumptions of standard road-scene segmentation models. We present DesertFormer, a semantic segmentation pipeline for off-road desert terrain analysis based on SegFormer B2 with a hierarchical Mix Transformer (MiT-B2) backbone. The system classifies terrain into ten ecologically meaningful categories -- Trees, Lush Bushes, Dry Grass, Dry Bushes, Ground Clutter, Flowers, Logs, Rocks, Landscape, and Sky -- enabling safety-aware path planning for ground robots and autonomous vehicles. Trained on a purpose-built dataset of 4,176 annotated off-road images at 512x512 resolution, DesertFormer achieves a mean Intersection-over-Union (mIoU) of 64.4% and pixel accuracy of 86.1%, representing a +24.2% absolute improvement over a DeepLabV3 MobileNetV2 baseline (41.0% mIoU). We further contribute a systematic failure analysis identifying the primary confusion patterns -- Ground Clutter to Landscape and Dry Grass to Landscape -- and propose class-weighted training and copy-paste augmentation for rare terrain categories. Code, checkpoints, and an interactive inference dashboard are released at https://github.com/Yasaswini-ch/Vision-based-Desert-Terrain-Segmentation-using-SegFormer.

Figures & tables

Fig. 1: DesertFormer pipeline overview. Dataset images (

512{\times}512

) pass through preprocessing and augmentation before entering the SegFormer B2 encoder (MiT-B2). Four hierarchical encoder stages produce multi-scale feature maps F1–F4 at strides

\{4,8,16,32\}

, which the lightweight MLP decoder fuses into a 256-channel representation via linear projection, upsampling, and concatenation. Final per-pixel logits are supervised with combined CE + Dice loss with class weights (left annotation). At inference, TTA (H-flip + multi-scale ensemble, right annotation) further improves accuracy. The predicted mask is mapped to a three-tier navigation safety costmap for downstream path planning.

Fig. 2: Dataset visualisation: four representative samples. Each row shows (left to right): original RGB image $|$ ground-truth segmentation mask $|$ model prediction . Colour coding follows the class palette in Figure 3 . The diversity of terrain types—from sparse scrubland and rocky outcrops to dense vegetation—highlights the breadth of the annotation effort.

Model	Backbone	mIoU	PA	$\Delta$ mIoU
DeepLabV3	MobileNetV2	41.0%	76.5%	—
DesertFormer	MiT-B2	64.4%	86.1%	+24.2%

TABLE I: Model comparison on the validation set.

Class	IoU	Pixel %
Sky	98.2%	37.84%
Trees	85.7%	4.07%
Dry Grass	70.2%	19.31%
Lush Bushes	68.4%	6.01%
Landscape	63.1%	23.72%
Flowers	62.1%	2.44%

TABLE II: Per-class segmentation performance on the validation set, sorted by IoU (descending). “Pixel %” is the fraction of validation pixels.

Fig. 3: Per-class IoU bar chart. Bar colours match the class segmentation palette. The dashed red line marks overall mean IoU (64.4%). Sky and Trees are the best-predicted classes; Ground Clutter and Dry Bushes are the most challenging.

Hardware	Precision	FPS	Latency (ms)
GPU (RTX 3060)	FP32	22	45
GPU (RTX 3060)	FP16	38	26
CPU (Intel Core i7)	FP32	1.3	769
Edge (Jetson Nano)	INT8	$\sim$ 4	$\sim$ 250

TABLE III: Inference speed on 512

\times

512 images.

Panel of Figure 7 — Fig. 4: Training dynamics of DesertFormer. Loss converges steadily over 40 epochs with no sign of overfitting. mIoU improves rapidly in the first 10 epochs and plateaus near epoch 30, consistent with the cosine annealing schedule.

Fig. 5: Row-normalised confusion matrix on the validation set. Diagonal entries represent per-class recall. The off-diagonal hotspots at (Ground Clutter, Landscape) and (Dry Grass, Landscape) reveal the primary spectral confusion caused by similar sandy/earthy colours under desert lighting.

Panel of Figure 9 — Fig. 6: Qualitative results on eight randomly selected test images. Each panel shows (left to right): input RGB $|$ model prediction $|$ 50%-alpha overlay . Class colours follow the palette in Figure 3 . Vegetation (Trees, Dry Grass) and Sky are predicted with high spatial precision; the dominant failure mode is boundary ambiguity between Landscape, Dry Grass, and Ground Clutter in the mid-image transition zone.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data
May 26, 2026Marcus G Müller, Wout Boerdijk, Maximilian Durner +5Mobile Robotics Synthetic Data Generation for Robotics
Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Weßling, Germany · Federal Institute of Technology Zurich (ETH Zurich), Zurich, Switzerland · Robotics and AI Institute (RAI), Zurich, Switzerland
Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs
Apr 6, 2026Abdelmoamen Nasser, Yousef Baba'a, Murad Mebrahtu +3Off-Road Robot Navigation VLM Reasoning
Robotics, Khalifa University, Abu Dhabi, United Arab Emirates · Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates · Computer and Information Engineering, Khalifa University, Abu Dhabi, United Arab Emirates
A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation
Mar 30, 2026Seongkyu Choi Jhonghyun AnSemantic Segmentation Multi-Scale Feature Fusion

May 26, 2026J/K move · Enter open · S save

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data

Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato+4

Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Weßling, Germany · Federal Institute of Technology Zurich (ETH Zurich), Zurich, Switzerland · Robotics and AI Institute (RAI), Zurich, Switzerland

Mobile Robotics Synthetic Data Generation for Robotics Semantic Segmentation

Open paper AskPDF arXiv

Terrain understanding is fundamental for mobile robots operating in unstructured outdoor environments. Existing vision-based traversability estimation methods rely on robot-specific annotations or semantic class mappings, limiting transferability across platforms and requiring costly re-annotation when robot capabilities change, while standard semantic segmentation methods only focus on specific predefined classes, which do not capture the variety of terrains. In this work, we propose a transformer-based architecture that jointly performs class-specific semantic segmentation and class-agnostic terrain segmentation within a unified network, called Trinity. Terrain regions are segmented based solely on visual appearance, without predefined semantic labels or robot-dependent traversability scores. This formulation enables the learning of robot-agnostic visual terrain priors that can be combined with robot-specific experience for downstream tasks such as traversability estimation, visual odometry, and mission planning. To enable large-scale training with diverse terrain appearances, we extend the OAISYS simulator and introduce RUGDSynth, a synthetic dataset inspired by RUGD with class-agnostic terrain samples. Furthermore, we present the EXTerra Dataset, providing real-world images annotated with both class-specific and class-agnostic terrain labels. Experiments demonstrate the feasibility of the proposed task and the effectiveness of our joint segmentation approach in complex outdoor environments. Code and datasets will be released with this publication (after review).

Abstract

Semantic Segmentation

Explore similar work

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data

Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs

A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation