cs.CVNov 19, 2026

Attention Drift Regularization with Adaptive Initialization for Domain-Robust Visual Recognition

Authors: Shivangi Rai, Vinod K Kurmi

Organizations: Indian Institute of Science Education and Research Bhopal, India

Abstract

Source-free domain adaptation (SFDA) adapts a pretrained source model to unlabeled target domains without source data access, yet two critical failure modes remain unaddressed. First, attention drift: source-learned attention maps shift toward spurious or background features under distribution shift, destabilizing feature representations. Second, domain-gap-agnostic initialization: applying a single fixed source model uniformly across all target domains ignores domain gap severity, causing premature overfitting to noisy pseudo-labels under large shifts. We propose DGA-CAFC to address both failures. The Counterfactual Attention-based Feature Consistency (CAFC) module enforces feature stability by minimizing the discrepancy between representations computed under learned attention and those under counterfactual-inspired normalized perturbations, thereby promoting domain-robust features independent of spurious attention shifts. The Domain-Gap Aware Source Model Selection (DGASS) module estimates domain gap severity via target prediction entropy and selects the source model variant whose confidence level best matches the estimated gap, reducing negative transfer and stabilizing pseudo-label quality. Crucially, CAFC transfers beyond SFDA: applied as a lightweight regularizer to standard domain generalization objectives without target data. Extensive experiments across SFDA and DG benchmarks demonstrate that our method consistently improves over recent baselines, demonstrating the effectiveness and generality of the proposed framework. Project page is available at https://visdomlab.github.io/DGA_CAFC

Figures & tables

Explore similar work

CardsList
  1. ProS2T: Progressive Fusion with Structure-Shape Transformations for Source-Free Domain Adaptation from Scratch

    May 4, 2026Zhou Bingtao, Xiang Mian, Chen Bingcai +1Source-Free Domain AdaptationVLM Adaptation

  2. Source-Free Domain Adaptation with Vision-Language Prior

    Apr 20, 2026Song Tang, Yunxiang Bai, Wenxin Su +3Source-Free Domain AdaptationCross-Modal Knowledge Distillation

  3. COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptation

    Aug 5, 2026Bo Li, Junjie Peng, Xiaohua Xie +1Source-Free Domain AdaptationVLM Adaptation