ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments

Pragat Wagle; Zheng Chen; Lantao Liu

cs.CVMar 30, 2026

ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments

Authors: Pragat Wagle, Zheng Chen, Lantao Liu

Organizations: Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN 47408, USA

Abstract

Robust scene understanding is essential for intelligent vehicles operating in natural, unstructured environments. While semantic segmentation datasets for structured urban driving are abundant, the datasets for extremely unstructured wild environments remain scarce due to the difficulty and cost of generating pixel-accurate annotations. These limitations hinder the development of perception systems needed for intelligent ground vehicles tasked with forestry automation, agricultural robotics, disaster response, and all-terrain mobility. To address this gap, we present ForestSim, a high-fidelity synthetic dataset designed for training and evaluating semantic segmentation models for intelligent vehicles in forested off-road and no-road environments. ForestSim contains 2094 photorealistic images across 25 diverse environments, covering multiple seasons, terrain types, and foliage densities. Using Unreal Engine environments integrated with Microsoft AirSim, we generate consistent, pixel-accurate labels across 20 classes relevant to autonomous navigation. We benchmark ForestSim using state-of-the-art architectures and report strong performance despite the inherent challenges of unstructured scenes. ForestSim provides a scalable and accessible foundation for perception research supporting the next generation of intelligent off-road vehicles. The dataset and code are publicly available: Dataset: https://vailforestsim.github.io Code: https://github.com/pragatwagle/ForestSim

Figures & tables

Fig. 2: Example RGB images of seasonal environments. These pictures demonstrate the unstructured, off-road, and forested characteristics of complex environments.

Fig. 3: More dense environments, an example is seen on the left, required manual control. On the right, data was able to be collected programmatically with no manual control.

Fig. 4: Examples of segmentation images captured directly from AirSim are on the left. These images were processed by manually determining the object each RGB value corresponds with and using this mapping to generate the ground truth pixelwise labels on the right.

Fig. 5: Numbers of total pixels per class in the dataset in descending order.

Fig. 6: Examples of ground truth annotations from the ForestSim dataset. The first row is the photorealistic RGB image collected from the environment, and the second row is the corresponding semantic segmentation. Please note that these are the pixel-wise, true semantic segmentation images after consolidation and labeling.

Fig. 7: The original image, the ground truth, and the predicted image annotation for models 1 to 7.

Fig. 8: The original image, the ground truth, and the predicted image annotation for models 8 to 13.

Model	Method	IOU $\downarrow$	Pix. Acc. $\downarrow$	M. Pix. Acc. $\downarrow$
m1	resnet50v1c + ResNetV1c + PSPHead	61.64	89.85	72.14
m2	resnet50v1c + ResNetV1c + ASPPHead	61.87	89.91	72.81
m3	resnet101v1c + ResNetV1c + ASPPHead	62.81	89.86	73.13
m4	resnet50v1c + ResNetV1c + DepthwiseSeparableASPPHead	59.16	89.31	72.93
m5	resnet101-v1c + ResNetV1c + DepthwiseSeparableASPPHead	59.22	88.32	69.56
m6	mit-b0 + MixVisionTransformer + SegformerHead	61.82	90.52	71.12

TABLE I: Prediction results from architectures used beginning with the model number, the pretrained model, encoder, and decoder.

Autonomous Driving Datasets

Autonomous driving datasets collect and organize driving-scene data to train and evaluate perception, prediction, planning, and safety systems.

See all 141 papers

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data
May 26, 2026Marcus G Müller, Wout Boerdijk, Maximilian Durner +5Mobile Robotics Synthetic Data Generation for Robotics
Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Weßling, Germany · Federal Institute of Technology Zurich (ETH Zurich), Zurich, Switzerland · Robotics and AI Institute (RAI), Zurich, Switzerland
DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems
Mar 17, 2026Yasaswini CheboluSemantic Segmentation Off-Road Robot Navigation
Department of Computer Science & Engineering (AI & ML), Gayatri Vidya Parishad College of Engineering for Women, Visakhapatnam, India
Leveraging Image Generators to Address Training Data Scarcity: The Gen4Regen Dataset for Forest Regeneration Mapping
May 7, 2026Gabriel Jeanson, David-Alexandre Duclos, William Larrivée-Hardy +5Synthetic Data Augmentation Remote Sensing Image Segmentation
1Northern Robotics Laboratory, Université Laval, Québec, QC, G1V 0A6, Canada · 2Département d’informatique et de génie logiciel, Université Laval, Québec, QC, G1V 0A6, Canada

May 26, 2026J/K move · Enter open · S save

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data

Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato+4

Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Weßling, Germany · Federal Institute of Technology Zurich (ETH Zurich), Zurich, Switzerland · Robotics and AI Institute (RAI), Zurich, Switzerland

Mobile Robotics Synthetic Data Generation for Robotics Semantic Segmentation

Open paper AskPDF arXiv

Terrain understanding is fundamental for mobile robots operating in unstructured outdoor environments. Existing vision-based traversability estimation methods rely on robot-specific annotations or semantic class mappings, limiting transferability across platforms and requiring costly re-annotation when robot capabilities change, while standard semantic segmentation methods only focus on specific predefined classes, which do not capture the variety of terrains. In this work, we propose a transformer-based architecture that jointly performs class-specific semantic segmentation and class-agnostic terrain segmentation within a unified network, called Trinity. Terrain regions are segmented based solely on visual appearance, without predefined semantic labels or robot-dependent traversability scores. This formulation enables the learning of robot-agnostic visual terrain priors that can be combined with robot-specific experience for downstream tasks such as traversability estimation, visual odometry, and mission planning. To enable large-scale training with diverse terrain appearances, we extend the OAISYS simulator and introduce RUGDSynth, a synthetic dataset inspired by RUGD with class-agnostic terrain samples. Furthermore, we present the EXTerra Dataset, providing real-world images annotated with both class-specific and class-agnostic terrain labels. Experiments demonstrate the feasibility of the proposed task and the effectiveness of our joint segmentation approach in complex outdoor environments. Code and datasets will be released with this publication (after review).

Abstract

Autonomous Driving Datasets

Explore similar work

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data

DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems

Leveraging Image Generators to Address Training Data Scarcity: The Gen4Regen Dataset for Forest Regeneration Mapping