Organizations: Department of Electronics and Computer Engineering, Institute of Engineering, Thapathali Campus, Tribhuvan University, Kathmandu, 44600, Bagmati Province, Nepal
Parcel-scale land-use data is missing for most rapidly urbanizing Global South cities, as current segmentation architectures rely on multispectral imagery and planned layouts. We introduce a hand-annotated six-class land-use corpus covering 202 km2 of Kathmandu Valley across 408 fine-resolution RGB scenes. The dataset is uniquely challenging: RGB spectral similarity is high, class imbalance approaches 100:1, and key structural features like rivers measure just two to three pixels wide at a standard hierarchical decoder's prediction stride. Evaluating five architectures reveals that standard random data splitting conceals massive spatial leakage. Switching to a strict sequence-block split drops performance by up to 0.095 mIoU. This penalty exceeds the performance gap between the architectures themselves. We modified the SegFormer baseline by integrating a half-resolution residual detail path, boundary-reliability reweighting, a topology-aware loss, and a presence-balanced sampler. Adding only 1.9% more parameters, the enhanced architecture improves mean IoU across every test fold, with gains concentrated in rare, thin-structure classes. However, both models achieve identical results on the external LoveDA benchmark. Finally, augmenting with diffusion-generated pairs at matched compute degraded accuracy across all architectures, especially harming rare classes it aimed to improve. This demonstrates that blindly incorporating synthetic data can severely damage performance, highlighting a critical need to rigorously evaluate and filter generated samples before training.
Figures & tables
Figure 1: The corpus. (a) Approximate extent of the 202.4 km 2 covered by 408 annotated tiles. Tile footprints were not preserved, so a georeferenced tile index is unavailable. (b) Three representative tiles from the dense urban core, river corridor and cultivated fringe, shown as images, annotations and overlays.
Control
True (m)
Measured (px)
GSD
Error (px)
Penalty area width
40.32
134.50
0.2998
− 0.82
Penalty area depth
16.50
55.20
0.2989
− 0.49
Goal area width
18.32
59.85
0.3061
+ 0.89
Goal area depth
5.50
17.81
0.3087
+ 0.42
Length-weighted
—
—
0.3016
—
Half-range
—
—
± 0.0049
—
Table 1: Ground sampling distance estimated from four fixed dimension controls. Error shows the deviation in pixels along each control’s baseline.
Figure 2: Recovering the acquisition scale. (a) Football pitch rotated upright with four regulation fixed controls marked. (b) Ground sampling distance estimated from these controls with marker size reflecting baseline length.
Class
Share
Area
Tiles
Worst
Width
px @
(%)
(km 2 )
JM
(m)
H/4
Residential
56.02
113.41
360
0.204
—
—
Agricultural
18.10
36.64
191
0.051
—
—
Unused land
12.97
26.25
225
0.051
—
—
Forest
10.39
21.02
132
0.476
—
—
Road
1.57
3.18
214
0.290
11.54
2.10
Table 2: Per-class statistics across 408 scenes at 512-pixel resolution. “Worst JM” indicates the lowest Jeffries–Matusita separability from another class on a 0–2 scale. “px @ H/4 ” gives the median width at the hierarchical decoder’s prediction stride.
Figure 3: Properties of the corpus that make it hard. (a) Share of annotated pixels by class, logarithmic axis. (b) Number of tiles containing each class, total 408. (c) Jeffries–Matusita separability for every class pair, from per-class Gaussians in RGB.
Figure 4: Two partitioning regimes for the 408-scene export sequence. (a) Sequence-block fold 1 showing dropped two-tile buffers. (b) Random split of the same scenes. (c, d) Residential and river proportions in each block fold’s validation set.
Figure 5: The study pipeline. Left: segmentation benchmark comparing five architectures under one protocol across two partitioning regimes, with thirty fold-runs. Right: generative pipeline covering unconditional mask diffusion, mask-conditioned image synthesis and compute-matched downstream evaluation.
Figure 6: Per-pixel reliability weights from Eq. ( 1 ) for a crop containing a road and large residential block. (a) Image. (b) Annotation with boundary set B marked in white. (c) Distance transform of the complement of B . (d) Resulting weight field, floored at 0.25.
Figure 7: The proposed model. (a) Forward pass with the unchanged SegFormer-B0 backbone (grey) and three added components (blue). The detail path adds a gated six-channel residual to the upsampled backbone logits. (b) Training objective using reliability weights in cross-entropy and a hard mask in the Lovász-softmax term.
Figure 8: The two-stage generator. Stage 1 generates six-class label maps at 128 × 128 through diffusion. Stage 2 uses a frozen Stable Diffusion 1.5 backbone with low-rank adaptation and a control branch conditioned on class channels instead of a colour palette. Stage 3 chains the two and filters to the real class pixel marginal.
Figure 9: The label-representation ablation for the mask generator. (a) Kullback–Leibler divergence between generated and real class share marginals on a logarithmic axis. (b) Connected components per thin class relative to real masks, with the dashed line marking parity.
Figure 10: Why binary label encoding fails for thin classes. In the three channel code, river lies midway between residential and forest. This is the corpus’s most common class boundary where a single bit transition incorrectly produces river labels.
Architecture
Sequence-block split
Random split
Gap
SD ratio
Fold 0
Fold 1
Fold 2
Mean (SD)
Fold 0
Fold 1
Fold 2
Mean (SD)
U-Net (R18)
0.3213
0.4516
0.4228
0.3986 (0.0684)
0.4533
0.4465
0.4484
0.4494 (0.0035)
+ 0.0508
19.5 ×
UNetFormer (R18)
0.2795
0.5069
0.4736
0.4200 (0.1228)
0.5263
0.5007
0.4905
0.5058 (0.0184)
+ 0.0858
6.7 ×
SegFormer-B0
0.2877
0.4657
0.4351
0.3962 (0.0952)
0.4948
0.4736
0.4611
0.4765 (0.0170)
+ 0.0803
5.6 ×
SegNeXt-T
0.3345
0.5701
0.5122
0.4723 (0.1228)
0.5646
0.5498
0.5450
0.5531 (0.0102)
+ 0.0809
12.0 ×
Proposed
0.3403
0.5582
0.5061
0.4682 (0.1138)
0.5525
0.5650
0.5717
0.5631 (0.0097)
+ 0.0949
11.7 ×
Table 3: Per-fold mIoU for five architectures under both partitioning regimes. Block-split SD measures variation across three regions, not noise. Gap is random minus block.
Figure 11: Effect of the partitioning protocol. (a) mIoU for both regimes, with open markers showing three folds, filled markers showing regime means and connecting lines showing the gaps. (b) Ratio of between-fold standard deviations under block and random splits.
Figure 12: Performance difference between the proposed model and its SegFormer-B0 backbone across six fold-runs. Both share the same encoder, initialisation and training schedule, making this the only architectural comparison unaffected by capacity or pretraining differences.
Class
Block
Random
Penalty
Residential
0.7093–0.7872
0.7969–0.8354
+ 0.0669
Agricultural
0.4757–0.5311
0.4614–0.5324
+ 0.0144
Forest
0.5694–0.5935
0.6027–0.6690
+ 0.0593
Road
0.2461–0.3918
0.2886–0.4526
+ 0.0573
Unused land
0.1899–0.2428
0.2893–0.3978
+ 0.1301
River
0.0868–0.3305
0.1546–0.5155
+ 0.1433
Table 4: Per-class IoU averaged over the three folds in each regime, shown as the range across five architectures. “Penalty” is the mean random minus block across the five.
Figure 13: Per-class results by architecture. (a) IoU averaged across the three sequence-block folds. (b) Spatial penalty for the same cells, computed as random minus block.
Figure 14: Qualitative comparison across four held-out scenes from block fold 1. Columns show the four scenes, with class colours indicated in the legend. Rows show the input images, reference annotations, predictions from SegFormer-B0 and the proposed model under the base-tile split, and the final two rows compare both models under the file-level split described in Section 6.7 .
Figure 15: Per-class IoU on the LoveDA validation data after 20,000 training steps. Mean performance is similar, but the proposed model improves water, building, agriculture and road while performing worse on barren, forest and background.
Figure 16: Synthetic pool quality at three set sizes. (a) Layout fidelity, with the dashed line marking the frozen scorer’s 0.4646 ceiling on real images. (b) Nearest-neighbour distance to training samples, relative to held-out real tiles (dashed).
Figure 17: Denoising error selects the wrong image-stage checkpoint. Validation error increases even as layout fidelity improves from 0.24 to 0.42.
Figure 18: Three synthetic pairs with different fidelity levels. Each shows a generated mask, its corresponding synthetic image and the frozen scorer’s prediction against the original mask. The rows show the error causing the decline in downstream performance.
Architecture
Mean IoU
River IoU
R0
R1
Δ
95 % CI
R0
R1
Δ
95 % CI
U-Net (R18)
0.4572
0.4515
− 0.0057
[ − 0.0130, + 0.0021]
0.1587
0.1181
− 0.0406
[ − 0.0520, − 0.0291]
UNetFormer (R18)
0.5055
0.4897
− 0.0159
[ − 0.0292, − 0.0022]
0.3290
0.2281
− 0.1008
[ − 0.1225, − 0.0778]
SegFormer-B0
0.4646
0.4575
− 0.0071
[ − 0.0180, + 0.0037]
0.1463
0.1138
− 0.0324
[ − 0.0457, − 0.0209]
SegNeXt-T
0.5675
0.5311
− 0.0364
[ − 0.0496, − 0.0214]
0.4535
0.3463
− 0.1072
[ − 0.1310, − 0.0839]
Proposed
0.5608
0.5560
− 0.0047
[ − 0.0170, + 0.0078]
0.4761
0.4723
− 0.0039
[ − 0.0416, + 0.0351]
Table 5: Downstream experiment on block fold 1 with seed 42 and 136 real validation scenes. Both arms use 6000 optimizer steps. R0 contains 804 real pairs, while R1 adds 804 synthetic pairs. Intervals use 2000 paired bootstrap resamples of validation scenes, sampled jointly across both arms.
Figure 19: Effect of mixing 804 synthetic image–label pairs with 804 real pairs over 6000 steps on block fold 1. Changes in (a) mean, (b) river and (c) road IoU with 95 % paired-bootstrap intervals. Filled markers indicate intervals excluding zero.
Class
Mean Δ
CI excl. 0
Layout fidelity
Contam. (%)
Synthetic
Real
Ratio
River
− 0.0570
4 − / 0 +
0.0567
0.1339
0.424
47.2
Road
− 0.0310
3 − / 0 +
0.2187
0.2991
0.731
39.1
Residential
− 0.0101
3 − / 0 +
0.8143
0.8006
1.017
0 9.3
Forest
− 0.0058
0 − / 0 +
0.6757
0.7472
0.904
16.2
Agricultural
+ 0.0059
0 − / 1 +
0.5436
0.5626
0.966
22.8
Table 6: Per-class IoU changes after adding synthetic pairs and the factors linked to them. “Synthetic” and “Real” give the frozen scorer’s IoU on generated and real images. Their ratio is synthetic divided by real IoU. Contamination is estimated as 0.5(1−synthetic IoU) .
Figure 20: Per-class IoU changes after adding synthetic data across architectures. Asterisks mark 95 % paired-bootstrap intervals excluding zero. River and road, left of the rule, are the predefined targets of the generative pipeline.
Figure 21: Layout fidelity and downstream performance. (a) mIoU change against layout fidelity relative to the frozen scorer’s ceiling on real images. (b) mIoU change against absolute layout fidelity on synthetic images.
Figure 22: Parameter count versus mIoU across sequence-block folds. The arrow shows the proposed model’s additional 0.078 M parameters, 1.9 % of the SegFormer-B0 backbone.
Figure 23: Effect of file-level splitting on mIoU for block fold 1, compared with the base-tile split. Arrows show measured variant leakage. The two leaked results are not directly comparable.
Figure 24: Three separate causes of rare-class failure, with the measurement that identifies each and the intervention it implies.
Configuration
mIoU
Δ mIoU
Δ River
Δ Road
Δ Unused
Full model
0.5582
—
—
—
—
− detail path
0.5449
− 0.0133
− 0.0392
− 0.0295
− 0.0180
− sampler
0.5509
− 0.0073
− 0.0030
− 0.0075
− 0.0262
− soft-clDice
0.5522
− 0.0060
− 0.0162
− 0.0159
− 0.0004
− reweighting
0.5564
− 0.0018
− 0.0077
+ 0.0013
+ 0.0024
Sum of the four
—
− 0.0284
—
—
—
Table 7: Leave-one-out ablation on block fold 1 with seed 42. Deltas are calculated as variant minus full model, with negative values indicating a useful component. Per-class changes are shown for the three targeted classes.
Figure 25: Leave-one-out ablation on block fold 1. (a) mIoU for the full model, each variants with one component disabled, and the bare backbone. (b) Per-class differences. The shaded band is the 0.005 below which nothing is defensible at a single seed.
Figure 26: Effect of synthetic data on mIoU and topology. (a) Changes in unweighted beside frequency-weighted IoU. (b) Change in centreline Dice for the two thin classes. The right hand axis gives the number of images entering each river mean in the two arms.
Driven by rapid advances in large-scale generative models, synthetic data has emerged as a promising solution for visual understanding. While modern diffusion models achieve remarkable photorealistic image synthesis, their potential in complex visual segmentation tasks remains underexplored. In this work, we conduct a systematic analysis of synthetic images from state-of-the-art diffusion models to uncover the factors governing their utility. In particular, synthetic images characterized by dense scene composition and fine instance fidelity demonstrate distinctive benefits, yielding significantly more discriminative spatial representations. Building on these insights, we propose SENSE, a unified framework that leverages flexible and scalable synthetic data to substantially enhance segmentation performance. Notably, SENSE is model-agnostic, compatible with diverse architectures (e.g., DPT and Mask2Former), and scales effectively across models with varying parameter capacities. Extensive experiments on Cityscapes, COCO, and ADE20K validate the effectiveness and generalization capability of our approach. Code is available at https://github.com/zhang0jhon/SENSE.
Jinjin Zhang, Xiefan Guo, Yizhou Jin +2
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China
Semantic segmentation models struggle with data sparsity and rare or visually diverse regions, e.g., dense regions or small objects in aerial or autonomous mobility data. While synthetic augmentation is an appealing solution, directly generating new labeled data risks misalignment of labels and generated pixels. Existing solutions to this problem often rely on external models, or employ coarse heuristics such as indiscriminately augmenting all foreground objects or entire backgrounds, which wastes capacity on uninformative pixels. To address this, we propose an uncertainty-guided synthetic context augmentation strategy that strictly preserves label validity and efficiently maximizes pixel informativeness per synthetic sample - no external guardrails required. Using a baseline segmenter's predictive entropy, we identify uncertain semantic regions and inpaint only the complementary visual context. When fine-tuning the segmenter on this synthetic data, we compute the loss only over the original pixels, excluding inpainted regions. This focuses learning on the unmodified, uncertain regions while presenting them in novel contexts. We demonstrate substantial mIoU gains on Cityscapes, UAVID, and BDD100K with the largest gains on rare and difficult classes such as buses, trains, or (from the aerial perspective) cars. Our results demonstrate that uncertainty-guided context augmentation is a highly effective lever to improve segmentation performance on complex datasets, with code provided at https://github.com/XITASO/Preserve-the-Hard-Regenerate-the-Rest.
Nikolai Röhrich, Julian Gleißner, Ahmed H. A. Ibrahim +2
1XITASO GmbH · 2Zuse School of Excellence in Reliable AI · 3Technische Hochschule Ingolstadt +1
Deep learning models benefit from increasing data diversity and volume, motivating synthetic data augmentation to improve existing datasets. However, existing evaluation metrics for synthetic data typically calculate latent feature similarity, which is difficult to interpret and does not always correlate with the contribution to downstream tasks. We propose a vision-language grounded framework for interpretable synthetic data augmentation and evaluation in remote sensing. Our approach combines generative models, semantic segmentation and image captioning with vision and language models. Based on this framework, we introduce ARAS400k: A large-scale Remote sensing dataset Augmented with Synthetic data for segmentation and captioning, containing 100k real images and 300k synthetic images, each paired with segmentation maps and descriptions. ARAS400k enables the automated evaluation of synthetic data by analyzing semantic composition, minimizing caption redundancy, and verifying cross-modal consistency between visual structures and language descriptions. Experimental results indicate that while models trained exclusively on synthetic data reach competitive performance levels, those trained with augmented data (a combination of real and synthetic images) consistently outperform real-data baselines. Consequently, this work establishes a scalable benchmark for remote sensing tasks, specifically in semantic segmentation and image captioning. The dataset is available at zenodo.org/records/18890661 and the code base at github.com/caglarmert/ARAS400k.