cs.CVAug 17, 2026

Ultra: Unsupervised Cross-Task Optimization for Reliable Restoration Segmentation Collaboration under Adverse Weather

Authors: Shiqin Wang, Zhiqian Li, Haoyuan Du, Junming Chen, Jiayuan Li, Tianrun Xu, Haoyang Chen

Organizations: Wuhan University · Beijing Institute of Technology · Zhongguancun Academy · Tsinghua University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Unsupervised Domain Adaptation for Adverse Weather Semantic Segmentation (UDA-ASS) aims to transfer semantic knowledge from labeled normal-weather images to unlabeled adverse environments. Existing approaches implicitly assume that restoration and segmentation provide mutually beneficial guidance. However, under severe degradation and without target-domain supervision, the validity of cross-task optimization directions becomes fundamentally unidentifiable, leading to hallucination-driven error propagation. In this work, we propose a novel Unsupervised Restoration-Segmentation Collaborative Learning Framework (Ultra), which reframes cross-task interaction as direction selection under uncertainty and causal effect estimation, enabling reliable collaboration through candidate direction generation and intervention-based filtering. In detail, we propose CTDN and CMIL. The former exploits complementary visual structures and semantic information to generate candidate optimization directions and performs cooperative direction selection between restoration and segmentation. The latter reformulates cross-task information transfer from correlation-based propagation into causal effect assessment, suppressing hallucination propagation. Extensive experiments on three widely used UDA-ASS benchmarks demonstrate state-of-the-art segmentation performance. Beyond segmentation, our framework achieves better unsupervised restoration results than existing UDA-ASS restoration methods and generalizes to unsupervised restoration and object detection collaboration tasks. Code and models will be available at https://github.com/Wang-Shiqin/Ultra.

Figures & tables

Figure 1: Challenges in unsupervised restoration-segmentation collaboration: (a) Ambiguous cross-task optimization directions due to the lack of reliable supervision in heterogeneous task spaces; (b) Self-reinforcing hallucination loops caused by iterative propagation of unreliable cross-task biases.
Figure 2: Overview of the proposed Unsupervised Restoration-Segmentation Collaborative Learning Framework (Ultra) . Given mixed images from cross-domain sampling, Ultra jointly optimizes image restoration and semantic segmentation through bidirectional cross-task guidance. Specifically, segmentation features SiS_{i} are transformed into semantic priors PiSP_{i}^{S} by the SGRA, which performs conditional modulation on restoration features to provide potential restoration optimization directions. The modulated restoration features Ri+R_{i}^{+} are further converted into restoration priors PiRP_{i}^{R} through the RGSA, providing potential segmentation optimization directions via conditional modulation. UNB selects the optimal cross-task optimization directions during training. To mitigate the Self-reinforcing Hallucination Loop in unsupervised cross-task collaboration, CMIL estimates the causal effects of semantic and restoration interventions and only propagates beneficial interventions.
MethodBackboneACDC testACDC val
DeepLabV2 [ 6 ]DeepLabV238.034.6
Refign [ 4 ]DeepLabV248.055.8
CMA [ 3 ]DeepLabV250.448.8
VBLC [ 14 ]DeepLabV247.846.0
Ultra (Ours)DeepLabV258.656.7
DAFormer [ 9 ]DAFormer55.355.3
Table 1: Comparison of the state-of-the-art in Cityscapes →\rightarrow ACDC domain adaptation on the ACDC test and ACDC val sets. The evaluation metric is mIoU, where a higher value indicates better performance. The best results are presented in bold .
Figure 3: Qualitative comparison of our method and existing UDA-ASS approaches with DeepLabV2, DAFormer, and HRDA backbones on the ACDC val set. Under the same backbone configurations, our method generates predictions that better match the ground-truth annotations compared with previous approaches ( e.g. , Refign, CMA, VBLC, and ACSegFormer). The HRDA-based model achieves the best segmentation quality, further demonstrating the effectiveness of the proposed framework.
Figure 4: Qualitative comparison with existing UDA-ASS approaches using DAFormer and HRDA backbones on Dark Zurich val (top two rows) and Nighttime Driving test (bottom two rows).
MethodBackboneDark Zurich-valNighttime Driving
mIoUmIoU
DAFormer [ 9 ]DAFormer37.154.0
InforMS [ 26 ]DAFormer45.156.0
Ultra (Ours)DAFormer45.856.8
HRDA [ 10 ]HRDA42.154.1
InforMS [ 26 ]HRDA52.558.5
Table 2: Comparison with state-of-the-art methods on the Dark Zurich-val set and Nighttime Driving test set when performing Cityscapes →\rightarrow Dark Zurich domain adaptation.
MethodDeepLabV2DAFormer
LOE ↓\downarrowLOE ↓\downarrow
VBLC [ 14 ]37.9037.90
Ultra (Ours)30.2629.98
Table 3: Comparison of image restoration performance between our method and existing UDA-ASS restoration methods on the ACDC val set. The no-reference LOE metric is adopted, where lower values indicate better performance.
Figure 5: Qualitative comparison of Ultra (ours) and VBLC on ACDC val. Our method produces more natural restored images that are closer to the coarse-aligned target-domain reference images provided by ACDC [ 20 ] .
Cross-Task Direction Negotiation (CTDN)Causal Mutual Intervention Learning (CMIL)ACDC val
w/o Unsupervised Nash Bargaining (UNB)w/ UNB
   65.0 (+0.0)
✓  65.2 (+0.2)
 ✓ 65.4 (+0.4)
✓ ✓65.5 (+0.5)
 ✓✓65.6 (+0.6)
Table 4: Ablation Study on several model variants of our method on the ACDC val (Backbone: DAFormer).
MethodmAP ↑\uparrowmAP50 ↑\uparrowmAP75 ↑\uparrowmAPs ↑\uparrowmAPm ↑\uparrow
2PCNet [ 11 ]16.630.814.96.024.8
2PCNet + Ultra (Ours)18.9 (+2.3)34.2 (+3.4)19.3 (+4.4)7.7 (+1.7)27.5 (+2.7)
Instance-Warp [ 42 ]17.932.118.26.826.7
Instance-Warp + Ultra (Ours)18.8 (+0.9)33.4 (+1.3)18.6 (+0.4)6.9 (+0.1)28.3 (+1.6)
Table 5: Comparison of UDA object detection performance on BDD100K Clear →\rightarrow ACDC Object Detection on the ACDC object detection validation set. The better result within each baseline pair is shown in bold .
More in this topic

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 183 papers

Explore similar work

CardsList
Sep 23, 2026

ICM: Intra-class Mixing for Domain Adaptation in Adverse Weather

Unsupervised domain adaptation (UDA) for semantic segmentation remains challenging under adverse weather conditions because severe appearance changes enlarge the domain gap and degrade the reliability of pseudo labels in the target domain. To address this problem, we propose an Intra-Class Mixing Consistency (ICM) framework that enforces prediction consistency between an intra-class mixed image and its original counterpart. Unlike previous mixing-based consistency methods that combine regions across different images or domains and may introduce unrealistic semantic inconsistencies, ICM performs mixing within the same image and semantic class, preserving realistic semantic layout for consistency regularization. With ICM, we establish a new state-of-the-art performance for clear-to-adverse-weather unsupervised domain adaptation (UDA) in semantic segmentation. On the Cityscapes →\rightarrow ACDC benchmark, our method achieves 75.7% mIoU, outperforming the previous state of the art by +1.9 pp, demonstrating its effectiveness in mitigating class confusion under challenging environmental conditions. The code is provided in the supplementary material.

Boying Li, Chang Liu, Britta Ayano Wilde +4

Unsupervised Domain AdaptationDomain AdaptationConsistency RegularizationSemantic SegmentationUnsupervised Image Segmentation
AskPDF
Oct 1, 2026

Learning from Failure: Leveraging Unreliable Predictions in Semi-Supervised Real-World Adverse Weather Removal

Adverse weather image restoration aims to recover images degraded by rain, haze, snow, and other weather-induced artifacts, thereby improving the robustness of outdoor vision systems. Existing unified restoration models exhibit limited generalization to real-world scenes due to their reliance on synthetic supervision and insufficient semantic constraints. In this paper, we propose a novel student--teacher semi-supervised framework that addresses both challenges. Specifically, we introduce an unreliable database that preserves failed teacher predictions as informative negative samples for contrastive learning, while a reliable database stores high-quality teacher predictions as positive samples. By jointly exploiting reliable pseudo-ground truths and unreliable teacher outputs, the proposed framework learns to enhance desirable restoration characteristics while avoiding common failures. We further propose a phase spectrum-based semantic constraint that replaces computationally expensive text-based supervision with an efficient and naturally aligned semantic prior. An adaptive phase consistency loss is also designed to dynamically balance supervision between the degraded input and teacher pseudo-ground truths according to degradation severity. Extensive experiments on real-world benchmarks demonstrate that the proposed method consistently outperforms existing state-of-the-art approaches in restoration quality and perceptual fidelity while exhibiting stronger generalization to real-world adverse weather conditions.

Cap Dang Xuan Kiet, Tat-Jen Cham

College of Computing and Data Science, Nanyang Technological University, Singapore

Unsupervised Image RestorationAll-in-One Image RestorationImage Restoration
AskPDF
May 5, 2026

Dual-Foundation Models for Unsupervised Domain Adaptation

Semantic segmentation provides pixel-level scene understanding essential for autonomous driving and fine-grained perception tasks. However, training segmentation models requires costly, labor-intensive annotations on real-world datasets. Unsupervised Domain Adaptation (UDA) addresses this by training models on labeled synthetic data and adapting them to unlabeled real images. While conceptually simple, adaptation is challenging due to the domain gap, i.e., differences in visual appearance and scene structure between synthetic and real data. Prior approaches bridge this gap through pixel-level mixing or feature-level contrastive learning. Yet, these techniques suffer from two major limitations: (1) reliance on high-confidence pseudo-labels restricts learning to a subset of the target domain, and (2) prototype-based contrastive methods initialize class prototypes from source-trained models, yielding biased and unstable anchors during adaptation. To address these issues, we propose a dual-foundation UDA framework that leverages two complementary foundation models. First, we employ the Segment Anything Model (SAM) with superpixel-guided prompting to enable learning from a broader range of target pixels beyond high-confidence predictions. Second, we incorporate DINOv3 to construct stable, domain-invariant class prototypes through its robust representation learning. Our method achieves consistent improvements of +1.3% and +1.4% mIoU over strong UDA baselines on GTA-to-Cityscapes and SYNTHIA-to-Cityscapes, respectively.

Yerin Cheon, Aruna Balasubramanian, Francois Rameau

Stony Brook University, Stony Brook, NY, USA · SUNY Korea, The State University of New York, Korea

Synthetic-to-Real Domain AdaptationUnsupervised Domain AdaptationSemantic Segmentation
AskPDF