Granularity-Aware Transfer for Tree Instance Segmentation in Synthetic and Real Forests
Organizations: Robotics Research Lab, University of Kaiserslautern-Landau, Kaiserslautern, Germany · University of Kaiserslautern-Landau, Kaiserslautern, Germany
Abstract
We address the challenge of synthetic-to-real transfer in forestry perception where real data have only coarse Tree labels while synthetic data provide fine-grained trunk/crown annotations. We introduce MGTD, a mixed-granularity dataset with 53k synthetic and 3.6k real images, and a four-stage protocol isolating domain shift and granularity mismatch. Our core contribution is granularity-aware distillation, which transfers structural priors from fine-grained synthetic teachers to a coarse-label student via logit-space merging and mask unification. Experiments show consistent mask AP gains, especially for small/distant trees, establishing a testbed for Sim-Real transfer under label granularity constraints.
Figures & tables
| Model | Type | Backbone | [email protected]:0.95 (BBox) | [email protected] (BBox) | [email protected]:0.95 (Segm) |
|---|---|---|---|---|---|
| Mask-RCNN | Tree trunk | X101 | 0.732 | 0.904 | 0.686 |
| Mask-RCNN | Tree trunk | Swin-T | 0.788 | 0.936 | 0.753 |
| Mask-RCNN | Whole tree | X101 | 0.859 | 0.947 | 0.670 |
| Mask-RCNN | Whole tree | Swin-T | 0.880 | 0.948 | 0.688 |
| Model | Type | Task | Sim Val | Sim Real | Abs. Gap (Rel. Drop) |
|---|---|---|---|---|---|
| Mask-RCNN | Tree trunk | BBox | 0.788 | 0.225 | 0.563 (71.4%) |
| Mask-RCNN | Tree trunk | Segm | 0.753 | 0.147 | 0.606 (80.5%) |
| Mask-RCNN | Whole tree | BBox | 0.880 | 0.066 | 0.814 (92.5%) |
| Mask-RCNN | Whole tree | Segm | 0.688 | 0.039 | 0.649 (94.3%) |
| Model | Type | Backbone | [email protected]:0.95 (BBox) | [email protected] (BBox) | [email protected]:0.95 (Segm) |
|---|---|---|---|---|---|
| Mask R-CNN | Tree | R50 | 0.487 | 0.817 | 0.396 |
| Mask R-CNN | Tree | X101 | 0.461 | 0.810 | 0.363 |
| Mask R-CNN | Tree | Swin-T | 0.500 | 0.792 | 0.420 |
| Model (Backbone) | Training Regime | mAP 50:95 (Segm) | Improvement |
| Mask R-CNN (ResNet-50) | Phase 3 (Real-only) | 0.396 | – |
| Mask R-CNN (ResNeXt-101) | Phase 3 (Real-only) | 0.363 | – |
| Mask R-CNN (Swin-T) | Phase 3 (Real-only) | 0.420 | – |
| Mask R-CNN (ResNet-50) | Phase 4 (Distillation) | 0.429 | +8.3% vs. same backbone |
| Mask R-CNN (ResNet-50) | Phase 4 (Distillation) | 0.429 | Outperforms ResNeXt-101 (0.363) |
| Outperforms Swin-T (0.420) |
| Subset | Train (I/O/A) | Val (I/O/A) | Test (I/O/A) | Total (I/O) |
|---|---|---|---|---|
| 1 / Tree Trunk | 10036 / 111068 / 11.07 | 2867 / 31832 / 11.10 | 1434 / 15632 / 10.90 | 14337 / 158532 |
| 1 / Whole Tree | 8921 / 29282 / 3.28 | 2549 / 8338 / 3.27 | 1274 / 4184 / 3.28 | 12744 / 41704 |
| 2 / Tree Trunk | 9712 / 101677 / 10.47 | 2775 / 29301 / 10.56 | 1388 / 14474 / 10.43 | 13875 / 145452 |
| 2 / Whole Tree | 9400 / 47411 / 5.04 | 2686 / 13714 / 5.11 | 1343 / 6811 / 5.07 | 13429 / 67936 |
| 3 / Tree Trunk | 7447 / 84651 / 11.37 | 2128 / 23852 / 11.21 | 1064 / 12314 / 11.57 | 10639 / 120817 |
| 3 / Whole Tree | 7337 / 45580 / 6.21 | 2096 / 12606 / 6.01 | 1049 / 6486 / 6.18 | 10482 / 64672 |
| Subset | Train (I/O/A) | Val (I/O/A) | Test (I/O/A) | Total (I/O) |
|---|---|---|---|---|
| Part 1 | 1400 / 14065 / 10.05 | 400 / 4064 / 10.16 | 200 / 1974 / 9.87 | 2000 / 20103 |
| Part 2 | 1120 / 9156 / 8.18 | 320 / 2663 / 8.32 | 160 / 1285 / 8.03 | 1600 / 13104 |
| Parameter Group | Setting |
|---|---|
| Image size | 1024 1024 (fixed), multi_scale = false |
| Batching | Batch size = 1 (auto-batch), single GPU; no gradient accumulation |
| Optimizer | SGD (momentum = 0.937, weight decay = 5e–4) |
| Learning rate schedule | Linear LR; base LR = 0.01; final LR fraction = 0.01 final LR = ; Warmup: 3 epochs (momentum = 0.8, bias_lr = 0.0) |
| Augmentations | Mosaic: on (prob = 1.0), disabled in last 10 epochs; MixUp: off; HSV jitter: h = 0.015, s = 0.7, v = 0.4; RandomAffine: translate = 0.1, scale = 0.5; Flips: fliplr = 0.5, flipud = 0.0; Random Erasing: prob = 0.4; CutMix / Copy-Paste: off; AutoAugment: RandAugment |
| Losses | Box: 7.5 (CIoU style); Class: 0.5 (BCE); DFL: 1.5 (distributional regression); Label smoothing: none |
| Parameter Group | Setting |
|---|---|
| Image size | 640 640 (fixed, multi_scale: false) |
| Batching | Batch size = 16, single/multi-GPU depending on device, no gradient accumulation |
| Optimizer | SGD (momentum: 0.937, weight decay: 0.0005) |
| Learning rate schedule | Linear LR (cos_lr: false), base LR = 0.01, final LR fraction = 0.01 ends at ; warmup (epochs = 3.0, momentum = 0.8, bias_lr = 0.1) |
| Augmentations | Same as simulated trees: Mosaic on (prob = 1.0), off in last 10 epochs (close_mosaic = 10); MixUp off; HSV jitter (h = 0.015, s = 0.7, v = 0.4); RandomAffine (degrees = 0.0, translate = 0.1, scale = 0.5, shear = 0.0, perspective = 0.0); Flips (flipud = 0.0, fliplr = 0.5); CutMix off; Random Erasing on (prob = 0.4); Copy-Paste off; AutoAugment: RandAugment |
| Losses | Box loss: 7.5 (CIoU-style); Class loss: 0.5 (BCE, no focal); DFL: 1.5 (distributional regression on); Label smoothing: 0.0 |
| Parameter Group | Setting |
|---|---|
| Backbone & Init | RegNet (return_idx = [1,2,3]); initialized with ImageNet-1K weights (torchvision) |
| Epochs | 130 total |
| Optimizer | AdamW; LR = 1e-4 (all modules); weight decay = 1e-4 (bias/norm excluded); gradient clipping = 0.1 |
| LR Schedule | MultiStepLR (milestones = [1000], → no effective decay); no warmup |
| Queries & Matcher | 300 object queries; Hungarian costs — cls: 2, L1: 5, GIoU: 2; focal matcher |
| Losses | Varifocal classification loss ; Box: L1 (w=5) + GIoU (w=2); Denoising with 100 queries |
| Model | Backbone | Optimizer | LR | Weight Decay | Schedule / Augmentations |
|---|---|---|---|---|---|
| Mask R-CNN | ResNeXt-101-64x4d + FPN | SGD ( ) | 0.02 | 0.0001 | Step LR + linear warmup ( ), Resize + Flip + Norm + Pad |
| Swin-T + FPN | AdamW | 1e-4 | 0.05 | Step LR + linear warmup ( ), Resize + Flip + Norm + Pad | |
| Cascade Mask R-CNN | ResNeXt-101-64x4d + FPN | SGD ( ) | 0.02 | 0.0001 | Step LR + linear warmup ( ), Resize + Flip + Norm + Pad |
| Swin-T + FPN | AdamW | 1e-4 | 0.05 | Step LR + linear warmup ( ), Resize + Flip + Norm + Pad | |
| Mask2Former | R50_lsj_50e | AdamW | 1e-4 | 0.05 | Step LR + linear warmup ( ), Resize + Flip + Norm + Pad |
| R101_lsj_50e / tuned | AdamW | 1e-4 | 0.05 | Step LR + linear warmup ( ), Resize + Flip + Norm + Pad |
| Hyperparameter | Value |
|---|---|
| Teacher architecture | Mask R-CNN (Swin-T, MMDetection). |
| Student architecture | Mask R-CNN (ResNet-50 + FPN). |
| Training epochs | 100 |
| Batch size (total) | 48 |
| Labeled batch size | 16 |
| Data loader workers | 8 |
| Type | Model | Backbone | Epochs | mAP@[.5:.95] (Box) | mAP@50 (Box) | mAP@75 (Box) | APs (Box) | APm (Box) | APl (Box) | mAR@100 | mAR@300 | mAR@1000 | ARs | ARm | ARl |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Tree Trunk | Mask-RCNN | X101 | 24 | 0.732 | 0.904 | 0.858 | 0.694 | 0.769 | 0.837 | 0.812 | 0.812 | 0.812 | 0.778 | 0.840 | 0.916 |
| Swin-T | 24 | 0.788 | 0.936 | 0.908 | 0.772 | 0.801 | 0.868 | 0.847 | 0.847 | 0.847 | 0.822 | 0.865 | 0.939 | ||
| Cascade Mask-RCNN | X101 | 24 | 0.762 | 0.905 | 0.872 | 0.728 | 0.801 | 0.877 | 0.829 | 0.829 | 0.829 | 0.797 | 0.858 | 0.937 | |
| Swin-T | 24 | 0.764 | 0.912 | 0.877 | 0.729 | 0.803 | 0.874 | 0.837 | 0.837 | 0.837 | 0.798 | 0.844 | 0.942 | ||
| Whole Tree | Mask-RCNN | X101 | 24 | 0.859 | 0.947 | 0.930 | 0.717 | 0.880 | 0.856 | 0.905 | 0.905 | 0.905 | 0.764 | 0.895 | 0.909 |
| Swin-T | 24 | 0.880 | 0.948 | 0.936 | 0.798 | 0.903 | 0.874 | 0.923 | 0.923 | 0.923 | 0.823 | 0.916 | 0.926 |
| Model | Type | Epochs | [email protected]:0.95 | [email protected] | [email protected] | APs | APm | API | mAR@100 | AR@300 | AR@1000 | ARI |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Bounding Box Metrics | ||||||||||||
| Mask-RCNN (Swin-T) | Tree trunk | 24 | 0.225 | 0.445 | 0.205 | -1.000 | -1.000 | 0.225 | 0.364 | 0.364 | 0.364 | 0.364 |
| Mask-RCNN (Swin-T) | Whole tree | 24 | 0.066 | 0.191 | 0.038 | -1.000 | -1.000 | 0.066 | 0.178 | 0.178 | 0.178 | 0.178 |
| Segmentation Metrics | ||||||||||||
| Mask-RCNN (Swin-T) | Tree trunk | 24 | 0.147 | 0.300 | 0.127 | -1.000 | -1.000 | 0.147 | 0.249 | 0.249 | 0.249 | 0.249 |
| Mask-RCNN (Swin-T) | Whole tree | 24 | 0.039 | 0.104 | 0.024 | -1.000 | -1.000 | 0.039 | 0.131 | 0.131 | 0.131 | 0.131 |
| Task | Model | Backbone | Epochs | mAP@[.5:.95] | mAP@50 | AP50 | AP75 | APs | APm | APl | AR@[.5:.95] |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Bbox | Mask-RCNN | Swin-T | 24 | 0.500 | 0.792 | 0.792 | 0.544 | - | - | 0.500 | 0.617 |
| X-101 | 24 | 0.461 | 0.810 | 0.810 | 0.467 | - | - | 0.461 | 0.592 | ||
| Cascade RCNN | X-101 | 24 | 0.571 | 0.834 | 0.834 | 0.629 | - | - | 0.571 | 0.684 | |
| Swin-T | 24 | 0.571 | 0.842 | 0.842 | 0.629 | - | - | 0.571 | 0.685 | ||
| Mask2Former | R50 | 24 | 0.265 | 0.623 | 0.623 | 0.186 | - | - | 0.265 | 0.510 | |
| R101 | 24 | 0.280 | 0.662 | 0.662 | 0.194 | - | - | 0.280 | 0.523 |
| Metric | Tree Trunk + Student | Whole Tree + Student | Key Insight |
|---|---|---|---|
| Final IoU | 0.512 | 0.630 | Whole tree supervision leads to better final overlap . |
| Peak IoU | 0.689 (Epoch 97) | 0.667 (Epoch 90) | Trunk supervision enables higher best-ever overlap. |
| Final mAP@[.5:.95] | 0.138 | 0.326 | Whole tree supervision yields better final detection quality . |
| Peak mAP@[.5:.95] | 0.427 (Epoch 88) | 0.381 (Epoch 90) | Trunk supervision achieves superior peak detection precision . |
| Stability (Post-Epoch 50) | High | Low | Trunk teacher provides more consistent learning signals . |
| Task | Model | Epoch | train/box_loss | train/cls_loss | train/dfl_loss | Precision | Recall | mAP@50 | mAP@50:95 | val/box_loss | val/cls_loss | val/dfl_loss | lr/pg0 | lr/pg1 | lr/pg2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Tree Trunk | YOLOv8n | 100 | 0.64198 | 0.51119 | 0.85470 | 0.82755 | 0.87099 | 0.92217 | 0.77399 | 0.64272 | 0.53417 | 0.85107 | 0.000199 | 0.000199 | 0.000199 |
| YOLOv8s | 100 | 0.55581 | 0.42323 | 0.83554 | 0.83738 | 0.87747 | 0.92875 | 0.79685 | 0.59484 | 0.49071 | 0.84336 | 0.000199 | 0.000199 | 0.000199 | |
| YOLOv8m | 100 | 0.52055 | 0.39357 | 0.83282 | 0.83614 | 0.88321 | 0.93024 | 0.80409 | 0.57936 | 0.47443 | 0.84955 | 0.000199 | 0.000199 | 0.000199 | |
| YOLOv8l | 100 | 0.50205 | 0.36515 | 0.82760 | 0.84004 | 0.88467 | 0.93074 | 0.80960 | 0.56565 | 0.46984 | 0.85554 | 0.000199 | 0.000199 | 0.000199 | |
| YOLOv11n | 100 | 0.62017 | 0.50105 | 0.84677 | 0.83320 | 0.87276 | 0.92494 | 0.78447 | 0.61998 | 0.52759 | 0.84075 | 0.000199 | 0.000199 | 0.000199 | |
| YOLOv11s | 100 | 0.54407 | 0.42742 | 0.83043 | 0.83856 | 0.88160 | 0.93020 | 0.80234 | 0.58274 | 0.48856 | 0.83403 | 0.000199 | 0.000199 | 0.000199 |
| Type (Tree trunk / Whole tree) | Model | [email protected]:0.95 | [email protected] | AP50 | AP75 | APs | APm | API | mAR@100 |
|---|---|---|---|---|---|---|---|---|---|
| Whole tree | RT-DETR + RegNet | 0.920 | 0.962 | 0.962 | 0.957 | 0.817 | 0.938 | 0.917 | 0.974 |
| Tree trunk | RT-DETR + RegNet | 0.825 | 0.947 | 0.947 | 0.928 | 0.797 | 0.861 | 0.899 | 0.973 |
| Model | Type | Task | Simulated accuracy (val set) | Sim→Real acc. | Absolute Gap | Relative Drop |
|---|---|---|---|---|---|---|
| yolo11m | Tree trunk | BBox | 0.93341 | 0.40 | 0.53 | 56.8% |
| yolov8m | Whole tree | BBox | 0.724 | 0.4817 | 0.24 | 33% |
| Task | Model | [email protected]:0.95 | [email protected] | AP50 | AP75 | APs | APm | API | Pretrained on |
|---|---|---|---|---|---|---|---|---|---|
| Object detection | RT-DETR + RegNet | 0.232 | 0.438 | 0.438 | 0.220 | -1 | -1 | 0.232 | Tree trunk sim |
| Object detection | RT-DETR + RegNet | 0.062 | 0.182 | 0.182 | 0.036 | -1 | -1 | 0.062 | Whole tree sim |
| Task | Model | Epoch | train/box_loss | train/cls_loss | train/dfl_loss | Precision | Recall | mAP@50 | mAP@50:95 | val/box_loss | val/cls_loss | val/dfl_loss | lr |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Object Detection | YOLOv8n | 50 | 0.9830 | 0.8459 | 1.0656 | 0.821 | 0.805 | 0.879 | 0.616 | 0.928 | 0.837 | 1.058 | 9.92e-05 |
| YOLOv8s | 50 | 0.8869 | 0.7426 | 1.0244 | 0.816 | 0.826 | 0.886 | 0.637 | 0.928 | 0.772 | 1.058 | 9.92e-05 | |
| YOLOv8m | 50 | 0.8173 | 0.6936 | 1.0125 | 0.821 | 0.825 | 0.886 | 0.649 | 0.898 | 0.720 | 1.120 | 9.92e-05 | |
| YOLOv8l | 50 | 0.8129 | 0.6565 | 1.0225 | 0.828 | 0.823 | 0.890 | 0.656 | 0.882 | 0.705 | 1.126 | 9.92e-05 | |
| YOLOv8x | 50 | 0.7900 | 0.6260 | 1.0095 | 0.818 | 0.835 | 0.892 | 0.660 | 0.883 | 0.690 | 1.140 | 9.92e-05 | |
| YOLOv11s | 50 | 0.9979 | 0.8583 | 1.0650 | 0.787 | 0.807 | 0.863 | 0.590 | 0.961 | 0.814 | 1.102 | 5.96e-05 |
| Task | Model | [email protected]:0.95 | [email protected] | AP50 | AP75 | APs | APm | API |
|---|---|---|---|---|---|---|---|---|
| Object detection | RT-DETR + RegNet | 34.7 | 68.6 | 68.6 | 30.3 | -1 | -1 | 34.7 |
Instance Segmentation
Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.
See all 159 papers