cs.CVNov 19, 2026

Attention Drift Regularization with Adaptive Initialization for Domain-Robust Visual Recognition

Authors: Shivangi Rai, Vinod K Kurmi

Organizations: Indian Institute of Science Education and Research Bhopal, India

Abstract

Source-free domain adaptation (SFDA) adapts a pretrained source model to unlabeled target domains without source data access, yet two critical failure modes remain unaddressed. First, attention drift: source-learned attention maps shift toward spurious or background features under distribution shift, destabilizing feature representations. Second, domain-gap-agnostic initialization: applying a single fixed source model uniformly across all target domains ignores domain gap severity, causing premature overfitting to noisy pseudo-labels under large shifts. We propose DGA-CAFC to address both failures. The Counterfactual Attention-based Feature Consistency (CAFC) module enforces feature stability by minimizing the discrepancy between representations computed under learned attention and those under counterfactual-inspired normalized perturbations, thereby promoting domain-robust features independent of spurious attention shifts. The Domain-Gap Aware Source Model Selection (DGASS) module estimates domain gap severity via target prediction entropy and selects the source model variant whose confidence level best matches the estimated gap, reducing negative transfer and stabilizing pseudo-label quality. Crucially, CAFC transfers beyond SFDA: applied as a lightweight regularizer to standard domain generalization objectives without target data. Extensive experiments across SFDA and DG benchmarks demonstrate that our method consistently improves over recent baselines, demonstrating the effectiveness and generality of the proposed framework. Project page is available at https://visdomlab.github.io/DGA_CAFC

Figures & tables

Explore similar work

May 4, 2026

ProS2T: Progressive Fusion with Structure-Shape Transformations for Source-Free Domain Adaptation from Scratch

Most Vision-Language (ViL)-guided works for source-free domain adaptation (SFDA) still rely on source-trained models. In those methods, we observe that different source models yield close final performance on the same target domain, suggesting the source initialization may has limited impact. Motivated by this, we propose ViL-guided Fully-Source-Free Domain Adaptation (VF-SFDA) setting, which trains a target model from scratch using only unlabeled target data and a pretrained ViL model. Reliable ViL supervision is therefore critical, however, resizing and cropping during pre-processing can distort object shape or remove important structural cues, leading to inaccurate predictions. To address this, we propose Progressive Fusion with Structure-Shape Transformations (ProS2T). Its S2 module constructs complementary views: Structure Priority retains more image content, while Shape Priority largely preserves object proportions. Guided by our analysis of adaptation dynamics, ProS2T progressively fuses each view's ViL prediction with the same target model prediction to form two candidates, then selects the more confident one to generate hard pseudo-labels to supervise the target model, whose predictions guide ViL prompt updates. Experiments on Office-Home, VisDA-C, and DomainNet-126 demonstrate that ProS2T achieves competitive or state-of-the-art performance compared to existing SFDA methods, despite operating under a stricter setting. Moreover, S2T improves ViL zero-shot classification and existing ViL-guided SFDA methods as a plug-and-play module.
Apr 20, 2026

Source-Free Domain Adaptation with Vision-Language Prior

Source-Free Domain Adaptation (SFDA) seeks to adapt a source model, which is pre-trained on a supervised source domain, for a target domain, with only access to unlabeled target training data. Relying on pseudo labeling and/or auxiliary supervision, conventional methods are inevitably error-prone. To mitigate this limitation, in this work we for the first time explore the potentials of off-the-shelf vision-language (ViL) multimodal models (e.g., CLIP) with rich whilst heterogeneous knowledge. We find that directly applying the ViL model to the target domain in a zero-shot fashion is unsatisfactory, as it is not specialized for this particular task but largely generic. To make it task-specific, we propose a novel DIFO++ approach. Specifically, DIFO++ alternates between two steps during adaptation: (i) Customizing the ViL model by maximizing the mutual information with the target model in a prompt learning manner, (ii) Distilling the knowledge of this customized ViL model to the target model, centering on gap region reduction. During progressive knowledge adaptation, we first identify and focus on the gap region, where enclosed features are entangled and class-ambiguous, as it often captures richer task-specific semantics. Reliable pseudo-labels are then generated by fusing predictions from the target and ViL models, supported by a memory mechanism. Finally, gap region reduction is guided by category attention and predictive consistency for semantic alignment, complemented by referenced entropy minimization to suppress uncertainty. Extensive experiments show that DIFO++ significantly outperforms the state-of-the-art alternatives. Our code and data are available at https://github.com/tntek/DIFO-Plus.
Aug 5, 2026

COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptation

Source-free domain adaptation (SFDA) adapts a source-trained model to an unlabeled target domain without source data, a practical setting under privacy or storage constraints. Yet its self-generated supervision can reinforce source bias under substantial domain shifts. Pretrained vision-language models (VLMs) offer complementary semantic knowledge, but the relative reliability of the source model and VLM varies across target samples. Existing cross-model guidance does not explicitly account for this variation and may overwrite valid source-derived evidence under conflict, a failure we term source-derived evidence forgetting. We formulate VLM-guided SFDA as a sample-wise reliability-allocation problem and propose Consensus-Driven Shift Modulation (COSMO). COSMO replaces expert-to-expert guidance with co-adaptation through an anchored shared consensus. It first forms a sample-specific initial consensus that favors the more concentrated prediction. During adaptation, COSMO re-aggregates both branches' evolving evidence and regulates how far the resulting consensus moves from its initial anchor based on consensus uncertainty and training progress. This keeps the shared supervision anchored yet adaptive. Across four benchmarks, COSMO achieves state-of-the-art performance under matched VLM backbones. Further analyses indicate that it better balances the retention of valid source-derived evidence with the absorption of complementary VLM evidence.