We introduce DORMS (Depth-Ordered Region Merging for Segmentation), a purely algorithmic approach to image segmentation from monocular depth maps, requiring no neural networks or learned parameters in the segmentation process. An off-the-shelf monocular depth estimator provides the input depth maps. Our method constructs depth-ordered merge trees through a deterministic union-find sweep, combining component area, depth persistence, and spatial contact constraints to guide region merging. A threshold search determines the segmentation granularity according to a specified target region count, while global depth detrending improves the separation of local geometric structures. Extensive experiments on COCO Panoptic, Cityscapes, and ADE20K demonstrate consistent improvements over classical segmentation baselines and competitive performance against zero-shot neural segmentation models. Notably, our method outperforms all evaluated zero-shot models on Cityscapes, while remaining competitive with several approaches on COCO and ADE20K. These results highlight the potential of purely algorithmic geometric reasoning for image segmentation without learned segmentation models.
Figures & tables
Figure 1: Overview of the proposed depth-based segmentation pipeline on a real image. An off-the-shelf monocular depth estimator provides the input depth map, which is globally detrended before depth-ordered region merging. Our algorithm repeatedly evaluates merges using component area, depth persistence, and gated accumulated spatial contact, and selects a threshold according to a target region count. The final panel shows our actual class-agnostic segmentation prediction overlaid on the RGB image, not the ground truth.
Figure 2: Depth-ordered pixel activation and component evolution on a COCO image. The left panel shows the input RGB image; the remaining panels show the active regions after processing 10%, 25%, 50%, 75%, 90%, and 100% of pixels in descending detrended-depth order. Dark pixels have not yet been activated; colors distinguish components within each snapshot. Components can form and merge as the sweep progresses, so their number need not change monotonically.
Dataset / Method
Without
With
Δ
COCO / DORMS
0.2037
0.2332
+0.0295
Cityscapes / DORMS (stereo)
0.0612
0.1045
+0.0433
ADE20K / DORMS
0.1753
0.2206
+0.0453
COCO / Felzenszwalb
0.1496
0.1516
+0.0020
COCO / SLIC
0.1712
0.1626
-0.0086
Table 3: Effect of global depth detrending on Hungarian-matched mIoU. COCO and ADE20K results use their full validation sets.
Figure 3: Effect of global depth detrending across COCO planarity quartiles, ordered by increasing plane-fit R2 (1,250 images per quartile).
Figure 4: Qualitative segmentation results from COCO Panoptic, Cityscapes, and ADE20K (top three rows), followed by a crowded ADE20K failure example (bottom row). From left to right: RGB image, estimated monocular depth, detrended depth, our predicted segmentation overlaid on RGB, and ground-truth regions overlaid on RGB. The bottom example illustrates under-segmentation in a scene containing many annotated regions.
Disparity preprocessing
H-mIoU
Covering
Original disparity
0.0612
0.2963
+ Depth detrending
0.1045 ( k=20 )
0.4751 ( k=7 )
+ Disparity preprocessing
0.1163 ( k=25 )
0.4994 ( k=7 )
Table 4: Cityscapes stereo-disparity ablation on the full 500-image validation set.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
k
m
g
COCO Panoptic
6
16
0.2%
Cityscapes
30
3
10%
ADE20K
8
8
2%
Appendix
Table 5: Dataset-specific configurations of the proposed method. The area gate g is expressed as a percentage of the image area.
k
4
5
6
7
8
10
H-mIoU
0.20114
0.20273
0.20357
0.20102
0.19624
0.18822
Appendix
Table 6: Target region count selection on the held-out COCO train2017 subset, using naturally reweighted Hungarian-matched mIoU.
Dataset
Before
After
Removed
Below k
COCO
6.054
5.927
0.127
371 (7.4%)
Cityscapes
30.034
25.842
4.192
460 (92.0%)
ADE20K
8.021
7.886
0.135
175 (8.8%)
Appendix
Table 7: Effect of small-region cleanup on the three evaluation datasets. Region counts are averaged over images. The final column reports the number and percentage of images with fewer than k regions after cleanup.
Dataset
DA-V2 checkpoint
Parameters
COCO Panoptic
Small
24.8M
Cityscapes
Large
335.3M
ADE20K
Small
24.8M
Appendix
Table 8: Pretrained depth estimation models used in the main experiments. Model sizes refer to approximate parameter counts.
Depth source
Original
Detrended
Δ
Stereo disparity
0.0648
0.1023
+0.0375
Marigold
0.0807
0.1195
+0.0388
Depth Anything V1
0.0884
0.1625
+0.0741
ZoeDepth
0.0922
0.1307
+0.0385
Depth Anything V2
0.1110
0.1919
+0.0809
Appendix
Table 9: Depth-source ablation on 50 stratified Cityscapes images. We report the highest mean H-mIoU over the evaluated segmentation configurations for each depth source, with and without global depth detrending. The improvement ( Δ ) compares the best configurations within each condition, which may use different segmentation hyperparameters.
Dataset
Split
Images
GT regions/image
COCO Panoptic
val2017
5,000
11.35
Cityscapes
Fine val
500
29.34
ADE20K
Val
2,000
15.21
Appendix
Table 10: Evaluation datasets and ground-truth segmentation statistics. The average number of segments is computed per image.
Method
Input
Configuration
Setting
#R
H-mIoU
Covering
DiffCut
RGB
SSD-1B backbone
Zero-shot
6.31
0.3245
0.6668
SAM2
RGB
16 points
Zero-shot
10.11
0.2783
0.6002
SAM2
RGB
32 points
Zero-shot
14.59
0.2648
0.6107
CLIPSeg
RGB
133-category vocabulary
Zero-shot
13.31
0.2471
0.6684
DORMS
DA-V2 Small
k=6,m=16,g=0.2%
Depth-only
5.93
0.2332
0.5119
SLIC
Depth (raw)
n=6
Classical
5.16
0.1712
0.3823
Appendix
Table 11: Extended class-agnostic segmentation comparison on COCO Panoptic val2017 (5,000 images). Classical configurations correspond to their H-mIoU-selected settings. Mask2Former uses a COCO-trained Swin-Tiny checkpoint as a fully supervised reference. Bold indicates the highest H-mIoU among non-supervised methods. The fixed-grid control is reported separately from the segmentation methods.
Method
Input
Configuration
Setting
#R
H-mIoU
Covering
DORMS
DA-V2 Large
k=30,m=3,g=10%
Depth-only
25.84
0.2086
0.5866
CLIPSeg
RGB
19-category vocabulary
Zero-shot
9.24
0.1506
0.6713
SAM2
RGB
32 points
Zero-shot
41.35
0.1464
0.4356
SAM2
RGB
16 points
Zero-shot
22.77
0.1365
0.4063
DiffCut
RGB
SSD-1B backbone
Zero-shot
5.54
0.1309
0.6834
SAM
RGB
16 points
Zero-shot
26.53
0.1234
0.4536
Appendix
Table 12: Extended class-agnostic segmentation comparison on Cityscapes fine validation. All methods use 500 images except DiffCut, which is evaluated on 499 images following one inference failure. The depth-based classical rows use sensor-derived stereo disparity, whereas DORMS uses DA-V2 Large monocular depth. Mask2Former uses a Cityscapes-trained Swin-Tiny checkpoint as a fully supervised reference. Bold indicates the highest H-mIoU among non-supervised methods.
Method
Input
Configuration
Setting
#R
H-mIoU
Covering
DiffCut
RGB
SSD-1B backbone
Zero-shot
6.20
0.2841
0.6329
SAM2
RGB
32 points
Zero-shot
11.92
0.2696
0.5474
SAM2
RGB
16 points
Zero-shot
8.66
0.2615
0.5357
DORMS
DA-V2 Small
k=8,m=8,g=2%
Depth-only
7.89
0.2206
0.5126
CLIPSeg
RGB
150-category vocabulary
Zero-shot
17.25
0.2126
0.6488
SAM
RGB
16 points
Zero-shot
26.95
0.1946
0.5558
Appendix
Table 13: Extended class-agnostic segmentation comparison on ADE20K validation (2,000 images). All methods are evaluated using the same constructed panoptic annotations. Classical configurations correspond to their H-mIoU-selected settings. Mask2Former uses an ADE20K-trained Swin-Large panoptic checkpoint as a fully supervised reference; its backbone is larger than the Swin-Tiny checkpoint used for COCO and Cityscapes. Bold indicates the highest H-mIoU among non-supervised methods.
Quartile
R2 range
Without
With
Δ H-mIoU
95% CI
p -value
Improved images
Q1
[0.0003, 0.3640]
0.2127
0.2284
+0.0158
[0.0117, 0.0198]
1.08×10−13
761 / 1250
Q2
[0.3641, 0.5883]
0.1954
0.2341
+0.0387
[0.0344, 0.0431]
6.35×10−61
915 / 1250
Q3
[0.5884, 0.7538]
0.1974
0.2428
+0.0454
[0.0408, 0.0502]
2.96×10−70
929 / 1250
Q4
[0.7539, 0.9939]
0.2092
0.2274
+0.0182
[0.0123, 0.0241]
1.26×10−9
802 / 1250
Appendix
Table 14: Effect of global depth detrending across COCO Panoptic planarity quartiles. Each quartile contains 1,250 images. Improvements are calculated from paired image-level H-mIoU differences, with 95% confidence intervals obtained using 10,000 paired bootstrap resamples. Statistical significance is assessed using two-sided paired t -tests. Displayed values are rounded independently.
Method
Without
With
Δ
Felzenszwalb (depth)
0.1496
0.1516
+0.0020
SLIC (depth)
0.1712
0.1626
-0.0086
Appendix
Table 15: Effect of depth detrending on classical segmentation baselines on COCO Panoptic val2017. Each input variant uses its H-mIoU-selected configuration. Differences are computed from the displayed rounded scores.
Preprocessing variant
H-mIoU
Covering
Fill split
0.1184
0.4287
Log transform
0.1170
0.5100
Fill far
0.1152
0.3875
Median ( 3×3 )
0.1143
0.4107
Bilateral filtering
0.1142
0.3957
Half resolution
0.1131
0.4007
Appendix
Table 16: Initial screening of disparity preprocessing variants on 20 Cityscapes images at k=30 . Results are preliminary and are used only to select configurations for subsequent experiments.
Disparity preprocessing + detrending
Detrending only
k
H-mIoU
Covering
#R
H-mIoU
Covering
#R
4
0.0688
0.4708
4.0
–
–
–
5
0.0772
0.4887
4.9
–
–
–
7
0.0892
0.4994
6.9
0.0858
0.4751
7.0
10
0.1002
0.4969
9.9
–
–
–
15
0.1112
0.4913
14.4
0.1014
0.4343
14.6
Appendix
Table 17: Full-validation Cityscapes stereo-disparity results across target region counts. The proposed disparity preprocessing uses fill_split+log+median3+half followed by global detrending. The control uses nearest-filled disparity with global detrending only. Region counts are measured after cleanup. A dash indicates a target count not evaluated for the control.
Method
CPU (s)
GPU (s)
CPU images × repeats
Proposed segmentation stage
Ours (full τ search)
1.1230
–
8×3
Ours (fixed τ )
0.0328
–
8×3
Classical methods
Felzenszwalb (depth)
0.2719
–
8×3
SLIC (depth)
0.0330
–
8×3
Appendix
Table 18: Single-image inference latency on COCO Panoptic, in seconds. CPU and GPU measurements use the same hardware but separate execution configurations. GPU measurements use eight images and three repetitions per image; the final column specifies the CPU measurement sample count and repetitions. Dashes indicate unavailable or unmeasured configurations. The fixed- τ variant does not perform per-image region-count targeting. A dash indicates that the corresponding device configuration was not benchmarked; DiffCut was evaluated on GPU only.
Method
COCO
ADE20K
Cityscapes
Ours (segmentation)
231
233
465
Ours (complete pipeline)
1128
1311
3210
Felzenszwalb
231
257
811
SLIC
157
162
321
Appendix
Table 19: Peak CPU process memory (MB) across image resolutions. The measurements use one image from each dataset in a dedicated process. The complete pipeline includes depth estimation and segmentation in the same CPU process.
Method
Params (M)
Alloc.
Reserv.
Process VRAM
Host RSS
Depth Anything V2 Small
24.8
219
272
732
1407
Depth Anything V2 Large
335.3
1500
1758
2218
2246
GroupViT
55.7
278
326
786
1693
Mask2Former
47.4
375
510
972
1773
SAM (16 points)
93.7
2426
3880
4342
1698
SAM (32 points)
93.7
2426
3880
4342
1708
Appendix
Table 20: Detailed resource usage of neural models during GPU inference on COCO. Parameter counts are in millions (M), and all memory measurements are in MB. Alloc. and Reserv. denote peak CUDA-allocated and peak CUDA-reserved memory. Process VRAM is the GPU-process memory reported after inference, including the CUDA context. Host RSS is peak CPU process memory during GPU execution and is not equivalent to the CPU-only inference measurements in Table 2 .
COCO (held-out)
Cityscapes (val)
k
H-mIoU
k
Veto off
Veto on
4
0.2241
10
0.1543
0.1550
5
0.2329
20
0.1952
0.1996
6
0.2357
25
0.1987
0.2067
7
0.2300
30
0.1945
0.2086
8
0.2252
35
0.1844
0.2061
Appendix
Table 21: Sensitivity to target region count k and the gated neck-width constraint, measured using Hungarian-matched mIoU. COCO results are from the 500-image held-out train2017 subset with natural-distribution reweighting, using the adopted neck-width configuration. Cityscapes results are from the full 500-image validation set, with the neck-width constraint disabled ( m=1 ) or enabled ( m=3 ).
Figure 5: Additional qualitative results, arranged by row: (1) COCO 000000145781 , (2) COCO 000000070048 , (3) Cityscapes lindau_000038_000019 , (4) Cityscapes lindau_000018_000019 , and (5) ADE20K ADE_val_00000783 . Predictions and ground truth are shown as colored overlays on RGB images. These examples include successful foreground separation, partial merging, and a case where the predicted region granularity exceeds that of the ground-truth annotation.
Figure 6: Evolution of connected components during one depth-ordered sweep on COCO image 000000435081 . Each panel shows the active components at the indicated fraction of pixels processed in descending detrended-depth order; dark areas are not yet activated. Colors distinguish components within a snapshot. The last panel shows the partition before cleanup; for this particular image, cleanup does not change the partition, leaving six regions.
Figure 7: Threshold-dependent accepted-merge forest for ADE20K image ADE_val_00001000 . (a) Component lifetimes and approved merges during the sweep. (b) The same trace with markers for rejected evaluations under the gated spatial-contact constraint and area–persistence score. Rejected evaluations do not permanently prohibit a merge; the corresponding components can be reconsidered upon later contact. Because this is a threshold-dependent forest with multiple surviving roots, the plot should not be interpreted as a conventional threshold-independent merge tree.
School of Artificial Intelligence, Shanghai Jiao Tong University · 2The Hong Kong University of Science and Technology (Guangzhou) · 3King Abdullah University of Science and Technology