A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation

Seongkyu Choi Jhonghyun An

cs.CVMar 30, 2026

A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation

Authors: Seongkyu Choi Jhonghyun An

Abstract

Off-road semantic segmentation is fundamentally challenged by irregular terrain, vegetation clutter, and inherent annotation ambiguity. Unlike urban scenes with crisp object boundaries, off-road environments exhibit strong class-level similarity among terrain categories, resulting in thick and uncertain transition regions that degrade boundary coherence and destabilize training. Rare or thin structures, such as narrow traversable gaps or isolated obstacles, further receive sparse and unreliable supervision and are easily overwhelmed by dominant background textures. Existing decoder designs either rely on low-scale bottlenecks that oversmooth fine structural details, or repeatedly fuse high-detail features, which tends to amplify annotation noise and incur substantial computational cost. We present a cross-scale decoder that explicitly addresses these challenges through three complementary mechanisms. First, a global--local token refinement module consolidates semantic context on a compact bottleneck lattice, guided by boundary-aware regularization to remain robust under ambiguous supervision. Second, a gated detail bridge selectively injects fine-scale structural cues only once through cross-scale attention, preserving boundary and texture information while avoiding noise accumulation. Third, an uncertainty-guided class-aware point refinement selectively updates the least reliable pixels, improving rare and ambiguous structures with minimal computational overhead. The resulting framework achieves noise-robust and boundary-preserving segmentation tailored to off-road environments, recovering fine structural details while maintaining deployment-friendly efficiency. Experimental results on standard off-road benchmarks demonstrate consistent improvements over prior approaches without resorting to heavy dense feature fusion.

Figures & tables

Fig. 1: Off-road scenes exhibit diverse challenges, including irregular and uneven terrain, ambiguous boundaries between similar surfaces, strong light reflections, and severe occlusion by vegetation.

Fig. 2: Overall framework of our CSTR. Multi-scale backbone features are compressed into a compact bottleneck token $T_{0}$ and refined by Global–Local Token Refinement (GLTR). Boundary-Guided Correction extracts fine-scale structural cues, which are selectively consulted via a Gated Cross-Scale Interaction (GCS) to recover thin or ambiguous structures efficiently.

Fig. 3: Structure of the Boundary-Guided Correction module. Boundary-sensitive cues are extracted through an edge path, while smoothed contextual cues are obtained via a grid path. These fine-scale structural cues are later consulted through gated cross-scale interaction without dense fusion.

Dataset	Methods (IoU)	Smooth	Rough	Bumpy	Forbidden	Obstacle	Background	mIoU $\uparrow$	aAcc $\uparrow$
RUGD	PSPNet [ 13 ]	48.62	88.92	69.45	29.07	87.98	78.29	67.06	92.85
	DeepLabv3+ [ 14 ]	5.86	84.99	50.40	25.04	87.50	81.47	55.88	91.51
	DANet [ 25 ]	2.26	81.47	8.69	15.00	82.54	74.86	44.14	88.81
	OCRNet [ 18 ]	66.29	89.47	76.15	59.14	88.77	79.17	76.50	93.46
	PSANet [ 36 ]	34.92	87.70	35.64	8.66	86.95	78.97	55.47	92.13
	BiSeNetv2 [ 28 ]	24.27	89.99	89.99	83.31	90.93	75.29	75.10	93.40

TABLE I: Comparison on RUGD and RELLIS-3D . We report per-group IoU (%), mean IoU (

\mathrm{mIoU}\!\uparrow

), and average accuracy (

\mathrm{aAcc}\!\uparrow

). Asterisks (*) denote transformer-based methods. Best per dataset in bold , second best underlined .

Terrain Group	RUGD Labels	RELLIS-3D Labels
Smooth	Concrete, asphalt	Concrete, asphalt
Rough	Gravel, grass, dirt, sand	Dirt, grass
Bumpy	Rock, rock bed	Mud, rubble
Forbidden	Water, bushes, tall vegetation	Water, bush
Obstacles	Trees, poles, logs, etc.	Tree, pole, vehicle, etc.
Background	Void, sky, signs	Void, sky

TABLE II: Unified grouping of fine-grained RUGD and RELLIS-3D labels into a 6-class terrain hierarchy based on surface texture, navigability, and semantic characteristics.

Fig. 4: Qualitative comparison on RUGD highlighting rare and thin structures. Compared to GA-Nav, CSTR better preserves small objects and narrow regions while maintaining coherent terrain boundaries.

Fig. 5: Qualitative comparison on RELLIS-3D under visually ambiguous terrain transitions. CSTR produces more coherent region interiors and smoother class boundaries around obstacles and vegetation compared to GA-Nav.

Gate	Inputs to Gate	mIoU $\uparrow$	bIoU $\uparrow$	F1 $\uparrow$	aAcc $\uparrow$
1-way	CA	89.72	32.22	48.87	95.93
2-way	CA+ $T_{0}$	89.71	32.12	48.78	95.83
2-way	CA+TB	89.86	32.62	49.14	95.95
3-way	CA+TB+ $T_{0}$	89.97	32.75	49.18	95.98

TABLE III: Ablation of gated fusion configurations on RUGD, analyzing the effect of different gating inputs on semantic accuracy and boundary-sensitive performance.

Variant	mIoU $\uparrow$	bIoU $\uparrow$	F1 $\uparrow$	aAcc $\uparrow$
Baseline	88.32	28.53	46.97	95.19
+ GLTR	88.66	29.78	47.81	95.44
+ BGC	88.80	29.75	47.80	95.51
+ GCS (w/o point-wise)	88.86	30.03	48.05	95.52
+ GCS (w/ point-wise)	89.97	32.75	49.18	95.98

TABLE IV: Incremental ablation results on RUGD demonstrating the contribution of each decoder component. We report region-level metrics (

\mathrm{mIoU}

\mathrm{aAcc}

) together with boundary- and structure-sensitive metrics (

\mathrm{bIoU}

F_{1}

Fig. 6: Quantitative robustness evaluation on RUGD with noisy ground truth. CSTR exhibits smaller performance degradation than GA-Nav under increasing label noise.

Model	Noise Level	mIoU $\uparrow$	bIoU $\uparrow$	$F1_{\text{boundary}}\uparrow$	aAcc $\uparrow$
GA-Nav	clean	89.08	39.66	93.65	95.66
	$r=1$	87.66	39.34	93.28	95.20
	$r=3$	88.36	39.24	93.69	95.43
	$r=5$	88.05	39.84	93.51	95.60
Ours	clean	89.97	45.94	94.64	95.98
	$r=1$	89.18	43.55	94.19	95.73

TABLE V: Qualitative comparison on RUGD with noisy ground truth. CSTR produces more coherent and structurally consistent predictions than GA-Nav under label noise.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

May 28, 2026

How to Relieve Distribution Shifts in Semantic Segmentation for Off-Road Environments

Semantic segmentation is crucial for autonomous navigation in off-road environments, enabling precise classification of surroundings to identify traversable regions. However, distinctive factors inherent to off-road conditions, such as source-target domain discrepancies and sensor corruption from rough terrain, can result in distribution shifts that alter the data differently from the trained conditions. This often leads to inaccurate semantic label predictions and subsequent failures in navigation tasks. To address this, we propose ST-Seg, a novel framework that expands the source distribution through style expansion (SE) and texture regularization (TR). Unlike prior methods that implicitly apply generalization within a fixed source distribution, ST-Seg offers an intuitive approach for distribution shift. Specifically, SE broadens domain coverage by generating diverse realistic styles, augmenting the limited style information of the source domain. TR stabilizes local texture representation affected by style-augmented learning through a deep texture manifold. Experiments across various distribution-shifted target domains demonstrate the effectiveness of ST-Seg, with substantial improvements over existing methods. These results highlight the robustness of ST-Seg, enhancing the real-world applicability of semantic segmentation for off-road navigation.

Ji-Hoon Hwang, Daeyoung Kim, Hyung-Suk Yoon +2

Department of Electrical and Communication Engineering, Seoul National University, Korea

Distribution Shift Robustness Domain Generalization Semantic Segmentation

Ask PDF

Mar 17, 2026

DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems

Reliable terrain perception is a fundamental requirement for autonomous navigation in unstructured, off-road environments. Desert landscapes present unique challenges due to low chromatic contrast between terrain categories, extreme lighting variability, and sparse vegetation that defy the assumptions of standard road-scene segmentation models. We present DesertFormer, a semantic segmentation pipeline for off-road desert terrain analysis based on SegFormer B2 with a hierarchical Mix Transformer (MiT-B2) backbone. The system classifies terrain into ten ecologically meaningful categories -- Trees, Lush Bushes, Dry Grass, Dry Bushes, Ground Clutter, Flowers, Logs, Rocks, Landscape, and Sky -- enabling safety-aware path planning for ground robots and autonomous vehicles. Trained on a purpose-built dataset of 4,176 annotated off-road images at 512x512 resolution, DesertFormer achieves a mean Intersection-over-Union (mIoU) of 64.4% and pixel accuracy of 86.1%, representing a +24.2% absolute improvement over a DeepLabV3 MobileNetV2 baseline (41.0% mIoU). We further contribute a systematic failure analysis identifying the primary confusion patterns -- Ground Clutter to Landscape and Dry Grass to Landscape -- and propose class-weighted training and copy-paste augmentation for rare terrain categories. Code, checkpoints, and an interactive inference dashboard are released at https://github.com/Yasaswini-ch/Vision-based-Desert-Terrain-Segmentation-using-SegFormer.

Yasaswini Chebolu

Department of Computer Science & Engineering (AI & ML), Gayatri Vidya Parishad College of Engineering for Women, Visakhapatnam, India

Semantic Segmentation Off-Road Robot Navigation

Ask PDF

Jun 17, 2026

Technical Report for ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Leveraging DINOv3 for Robust Outdoor Scene Understanding in Field Robotics

The GOOSE 2D Fine-Grained Semantic Segmentation Challenge at the ICRA 2026 Workshop on Field Robotics evaluates dense semantic segmentation of off-road imagery over a fine-grained taxonomy of 64 classes and 11 evaluated non-void coarse categories. We present the first-place solution to this challenge. Our solution comprises two complementary improvements: (a) a network-level design that combines a self-supervised DINOv3 ViT-L/16 backbone, a ViT-Adapter, and a Mask2Former mask-classification decoder, together with a coarse-category auxiliary loss on the global [CLS] token; and (b) an inference-time aggregation strategy based on multi-scale and horizontal-flip test-time augmentation and an ensemble of the top three checkpoints selected using Codabench scores. Our method achieves an official composite score of 76.57%, consisting of 69.32% fine-class mIoU and 83.81% category-level mIoU, and ranks first on the final phase leaderboard: www.codabench.org/competitions/14257/#/results-tab.

Jaeil Park, Hyobin Choi, Sangjin Lee +2

Multimodal Intelligence and Perception Laboratory (MIP), Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu, Republic of Korea. · Laboratory for Information and Decision Systems (LIDS), Massachusetts Institute of Technology (MIT), Cambridge, MA, USA.

Robotics Robotic Perception Fine-Grained Semantic Segmentation Semantic Segmentation

Ask PDF

Abstract

Semantic Segmentation

Explore similar work

How to Relieve Distribution Shifts in Semantic Segmentation for Off-Road Environments

DesertFormer: Transformer-Based Semantic Segmentation for Off-Road Desert Terrain Classification in Autonomous Navigation Systems

Technical Report for ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Leveraging DINOv3 for Robust Outdoor Scene Understanding in Field Robotics