Unsupervised Domain Adaptation for Adverse Weather Semantic Segmentation (UDA-ASS) aims to transfer semantic knowledge from labeled normal-weather images to unlabeled adverse environments. Existing approaches implicitly assume that restoration and segmentation provide mutually beneficial guidance. However, under severe degradation and without target-domain supervision, the validity of cross-task optimization directions becomes fundamentally unidentifiable, leading to hallucination-driven error propagation. In this work, we propose a novel Unsupervised Restoration-Segmentation Collaborative Learning Framework (Ultra), which reframes cross-task interaction as direction selection under uncertainty and causal effect estimation, enabling reliable collaboration through candidate direction generation and intervention-based filtering. In detail, we propose CTDN and CMIL. The former exploits complementary visual structures and semantic information to generate candidate optimization directions and performs cooperative direction selection between restoration and segmentation. The latter reformulates cross-task information transfer from correlation-based propagation into causal effect assessment, suppressing hallucination propagation. Extensive experiments on three widely used UDA-ASS benchmarks demonstrate state-of-the-art segmentation performance. Beyond segmentation, our framework achieves better unsupervised restoration results than existing UDA-ASS restoration methods and generalizes to unsupervised restoration and object detection collaboration tasks. Code and models will be available at https://github.com/Wang-Shiqin/Ultra.
Figures & tables
Figure 1: Challenges in unsupervised restoration-segmentation collaboration: (a) Ambiguous cross-task optimization directions due to the lack of reliable supervision in heterogeneous task spaces; (b) Self-reinforcing hallucination loops caused by iterative propagation of unreliable cross-task biases.
Figure 2: Overview of the proposed Unsupervised Restoration-Segmentation Collaborative Learning Framework (Ultra) . Given mixed images from cross-domain sampling, Ultra jointly optimizes image restoration and semantic segmentation through bidirectional cross-task guidance. Specifically, segmentation features Si are transformed into semantic priors PiS by the SGRA, which performs conditional modulation on restoration features to provide potential restoration optimization directions. The modulated restoration features Ri+ are further converted into restoration priors PiR through the RGSA, providing potential segmentation optimization directions via conditional modulation. UNB selects the optimal cross-task optimization directions during training. To mitigate the Self-reinforcing Hallucination Loop in unsupervised cross-task collaboration, CMIL estimates the causal effects of semantic and restoration interventions and only propagates beneficial interventions.
Method
Backbone
ACDC test
ACDC val
DeepLabV2 [ 6 ]
DeepLabV2
38.0
34.6
Refign [ 4 ]
DeepLabV2
48.0
55.8
CMA [ 3 ]
DeepLabV2
50.4
48.8
VBLC [ 14 ]
DeepLabV2
47.8
46.0
Ultra (Ours)
DeepLabV2
58.6
56.7
DAFormer [ 9 ]
DAFormer
55.3
55.3
Table 1: Comparison of the state-of-the-art in Cityscapes → ACDC domain adaptation on the ACDC test and ACDC val sets. The evaluation metric is mIoU, where a higher value indicates better performance. The best results are presented in bold .
Figure 3: Qualitative comparison of our method and existing UDA-ASS approaches with DeepLabV2, DAFormer, and HRDA backbones on the ACDC val set. Under the same backbone configurations, our method generates predictions that better match the ground-truth annotations compared with previous approaches ( e.g. , Refign, CMA, VBLC, and ACSegFormer). The HRDA-based model achieves the best segmentation quality, further demonstrating the effectiveness of the proposed framework.
Figure 4: Qualitative comparison with existing UDA-ASS approaches using DAFormer and HRDA backbones on Dark Zurich val (top two rows) and Nighttime Driving test (bottom two rows).
Method
Backbone
Dark Zurich-val
Nighttime Driving
mIoU
mIoU
DAFormer [ 9 ]
DAFormer
37.1
54.0
InforMS [ 26 ]
DAFormer
45.1
56.0
Ultra (Ours)
DAFormer
45.8
56.8
HRDA [ 10 ]
HRDA
42.1
54.1
InforMS [ 26 ]
HRDA
52.5
58.5
Table 2: Comparison with state-of-the-art methods on the Dark Zurich-val set and Nighttime Driving test set when performing Cityscapes → Dark Zurich domain adaptation.
Method
DeepLabV2
DAFormer
LOE ↓
LOE ↓
VBLC [ 14 ]
37.90
37.90
Ultra (Ours)
30.26
29.98
Table 3: Comparison of image restoration performance between our method and existing UDA-ASS restoration methods on the ACDC val set. The no-reference LOE metric is adopted, where lower values indicate better performance.
Figure 5: Qualitative comparison of Ultra (ours) and VBLC on ACDC val. Our method produces more natural restored images that are closer to the coarse-aligned target-domain reference images provided by ACDC [ 20 ] .
Cross-Task Direction Negotiation (CTDN)
Causal Mutual Intervention Learning (CMIL)
ACDC val
w/o Unsupervised Nash Bargaining (UNB)
w/ UNB
65.0 (+0.0)
✓
65.2 (+0.2)
✓
65.4 (+0.4)
✓
✓
65.5 (+0.5)
✓
✓
65.6 (+0.6)
Table 4: Ablation Study on several model variants of our method on the ACDC val (Backbone: DAFormer).
Method
mAP ↑
mAP50 ↑
mAP75 ↑
mAPs ↑
mAPm ↑
2PCNet [ 11 ]
16.6
30.8
14.9
6.0
24.8
2PCNet + Ultra (Ours)
18.9 (+2.3)
34.2 (+3.4)
19.3 (+4.4)
7.7 (+1.7)
27.5 (+2.7)
Instance-Warp [ 42 ]
17.9
32.1
18.2
6.8
26.7
Instance-Warp + Ultra (Ours)
18.8 (+0.9)
33.4 (+1.3)
18.6 (+0.4)
6.9 (+0.1)
28.3 (+1.6)
Table 5: Comparison of UDA object detection performance on BDD100K Clear → ACDC Object Detection on the ACDC object detection validation set. The better result within each baseline pair is shown in bold .