Revisiting Cross-Attention Mechanisms: Leveraging Beneficial Noise for Domain-Adaptive Learning

Zelin Zang; Yehui Yang; Fei Wang; Liangyu Li; Baigui Sun

cs.CVMar 18, 2026

Revisiting Cross-Attention Mechanisms: Leveraging Beneficial Noise for Domain-Adaptive Learning

Authors: Zelin Zang, Yehui Yang, Fei Wang, Liangyu Li, Baigui Sun

Organizations: School of Engineering, Westlake University, Hangzhou, China · Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences (HKISI-CAS), Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, China ·

Abstract

Unsupervised Domain Adaptation (UDA) seeks to transfer knowledge from a labeled source domain to an unlabeled target domain but often suffers from severe domain and scale gaps that degrade performance. Existing cross-attention-based transformers can align features across domains, yet they struggle to preserve content semantics under large appearance and scale variations. To explicitly address these challenges, we introduce the concept of beneficial noise, which regularizes cross-attention by injecting controlled perturbations, encouraging the model to ignore style distractions and focus on content. We propose the Domain-Adaptive Cross-Scale Matching (DACSM) framework, which consists of a Domain-Adaptive Transformer (DAT) for disentangling domain-shared content from domain-specific style, and a Cross-Scale Matching (CSM) module that adaptively aligns features across multiple resolutions. DAT incorporates beneficial noise into cross-attention, enabling progressive domain translation with enhanced robustness, yielding content-consistent and style-invariant representations. Meanwhile, CSM ensures semantic consistency under scale changes. Extensive experiments on VisDA-2017, Office-Home, and DomainNet demonstrate that DACSM achieves state-of-the-art performance, with up to +2.3% improvement over CDTrans on VisDA-2017. Notably, DACSM achieves a +5.9% gain on the challenging "truck" class of VisDA, evidencing the strength of beneficial noise in handling scale discrepancies. These results highlight the effectiveness of combining domain translation, beneficial-noise-enhanced attention, and scale-aware alignment for robust cross-domain representation learning.

Figures & tables

Fig. 1: Style/appearance gap and scale gap are two frequently observed challenges in UDA. Two modules, Domain Adaptive Transformer and Cross Scale Matching, are carefully designed for the specific issues. (a), the two objects share the same structure but have different appearances. Additionally, identical semantic categories may manifest different scales in different datasets, which can also cause confusion for the model. (b) The scale gap is frequently observed in many datasets but has not received much attention. The objects may appear at different scales.

Fig. 2: Transformer blocks with self and cross attentions. One attentional module consists of three input sources: the input feature

\bm{z}

, the query

\bm{Q}

, and the key and value pairs

\bm{K}/\bm{V}

. In self attention,

\bm{Q}

and

\bm{K}/\bm{V}

come from the same domain. In cross attention,

\bm{Q}

and

\bm{K}/\bm{V}

may come from different domains.

Fig. 3: DACSM Framework. Domain-Adaptive Cross-Scale Matching (DACSM) integrates the Domain-Adaptive Transformer (DAT) and the Cross-Scale Matching (CSM) modules into a single end-to-end architecture. Note that the Tokenizer (Patch Embedding) is applied only once at the input stage; subsequent blocks operate on feature tokens. The DAT employs self- and cross-attention to disentangle domain-shared content (queries) from domain-specific style (keys/values), ensuring query-consistent and content-biased feature learning. The residual connection adds the Source Query stream to the attention output, ensuring dimension matching. Beneficial noise is injected into cross-attention to suppress spurious style correlations and enhance robustness. The CSM module addresses scale discrepancies by rescaling source images with multiple factors and aligning target features through a sub-center classifier. Each sub-center learns scale-specific class features in the source domain, while the target features are adaptively matched to the most compatible scale.

Panel of Figure 4 — Fig. 4: (Top) Illustration of a style swap operation. A 2D convolution extracts $3\times 3$ patches with stride 1 and computes the normalized cross-correlations. There are $n_{c}=9$ spatial locations and $n_{s}=4$ feature channels immediately before and after the channel-wise argmax operation. A 2D transposed convolution reconstructs the full activations by placing each best-matching style patch at the corresponding spatial location. (Bottom) Style transfer results of StyleSwap and optimization-based methods. Both figures are borrowed from StyleSwap [ 35 ] .

Method	Ar 2 Cl	Ar 2 Pr	Ar 2 Re	Cl 2 Ar	Cl 2 Pr	Cl 2 Re	Pr 2 Ar	Pr 2 Cl	Pr 2 Re	Re 2 Ar	Re 2 Cl	Re 2 Pr	avg
JAN [ 18 ]	45.9	61.2	68.9	50.4	59.7	61.0	45.8	43.4	70.3	63.9	52.4	76.8	58.3
MCD [ 19 ]	48.9	68.3	74.6	61.3	67.6	68.8	57.0	47.1	75.1	69.1	52.2	79.6	67.8
CDAN [ 40 ]	50.7	70.6	76.0	57.6	70.0	70.0	57.4	50.9	77.3	70.9	56.7	81.6	65.8
CGDM-B [ 21 ]	67.1	83.9	85.4	77.2	83.3	83.7	74.6	64.7	85.6	79.3	69.5	87.7	78.5
SHOT-B [ 23 ]	67.1	83.5	85.5	76.6	83.4	83.7	76.3	65.3	85.3	80.4	66.7	83.4	78.1
DOT-B [ 25 ]	69.0	85.6	87.0	80.0	85.2	86.4	78.2	65.4	87.9	79.7	67.3	89.3	80.1

TABLE I: Comparison with state-of-the-art methods on Office-Home. The A 2 B in each cell represents the adaptation from domain A to domain B. For transformer-based methods, the suffix ’-B’ is added to the method name. The ’avg’ means the average accuracy of all the target domains. The best performance is marked as bold . The number in the bracket represents the improvement over the best baseline.

Method	plane	bcycl	bus	car	horse	knife	mcycl	person	plant	sktbrd	train	truck	avg
ALDA [ 42 ]	93.8	74.1	82.4	69.4	90.6	87.2	89.0	67.6	93.4	76.1	87.7	22.2	77.8
CGDM-B [ 21 ]	96.0	87.0	87.0	87.0	92.0	98.0	92.0	78.0	96.0	48.0	89.0	39.0	82.3
SHOT-B [ 23 ]	98.0	90.0	86.0	73.0	97.0	99.0	94.0	55.0	95.0	87.0	93.0	63.0	85.9
DOT-B [ 25 ]	99.0	93.0	89.0	79.0	98.0	96.0	93.0	80.0	98.0	96.0	94.0	69.0	90.3
CDTrans-B [ 13 ]	98.0	86.0	87.0	83.0	98.0	97.0	96.0	84.0	98.0	84.0	95.0	55.0	88.4
CSTrans-B [ 30 ]	97.0	90.9	90.3	62.5	95.2	94.6	91.1	49.8	97.0	93.6	82.8	60.3	83.8

TABLE II: Comparison with state-of-the-art methods on VisDA-2017. The ’plane’ to ’truck’ in each cell represents the adaptation from domain ’synthetic’ to domain ’real’. For transformer-based methods, the suffix ’-B’ is added to the method name. The ’avg’ means the average accuracy of all the target domains. The best performance is marked as bold . Suboptimal results are underlined . The number in the bracket represents the improvement over the best baseline.

CDAN	clp	inf	pnt	qdr	rel	skt	avg	CGDM	clp	inf	pnt	qdr	rel	skt	avg
clp	-	13.5	28.3	9.3	43.8	30.2	25.0	clp	-	16.9	35.3	10.8	53.5	36.9	30.7
inf	18.9	-	21.4	1.9	36.3	21.3	20.0	inf	27.8	-	28.2	4.4	48.2	22.5	26.2
pnt	29.6	14.4	-	4.1	45.2	27.4	24.2	pnt	37.7	14.5	-	4.6	59.4	33.5	30.0
qdr	11.8	1.2	4.0	-	9.4	9.5	7.2	qdr	14.9	1.5	6.2	-	10.9	10.2	8.7
rel	36.4	18.3	40.9	3.4	-	24.6	24.7	rel	49.4	20.8	47.2	4.8	-	38.2	32.0
skt	38.2	14.7	33.9	7.0	36.6	-	26.1	skt	50.1	16.5	43.7	11.1	55.6	-	35.4

TABLE III: Comfusion matrax comparison with state-of-the-art methods on DomainNet. Each cell represents the adaptation from the source domain to the target domain. Columns are source domains, and rows are target domains. The best performance is marked as bold . For transformer-based methods, the suffix ‘-B’ is added to the method name. The ‘avg’ means the average accuracy of all the target domains. The number in the bracket represents the improvement over the best baseline.

Method	plane	bcycl	bus	car	horse	knife	mcycl	person	plant	sktbrd	train	truck	Average
CDTrans-B	97.7	86.4	86.9	83.3	97.8	97.2	95.9	84.1	97.9	83.5	94.6	55.3	88.4
+only CSM	98.2	91.3	89.5	84.1	98.3	97.6	95.9	84.2	97.7	89.4	95.6	59.7	90.1 (+1.7)
+only DAT	98.3	91.2	89.3	84.3	98.4	97.5	95.9	84.1	98.0	89.4	95.6	60.7	90.2 (+1.8)
+DAT+CSM (DACSM)	98.7	92.1	90.1	85.6	98.4	97.5	96.2	84.3	98.0	90.6	95.6	61.2	90.7 (+2.3)

TABLE IV: Ablation studies. Explore the enhancements and effects of the proposed innovations on VisDA-2017. The best performance is marked as bold . The number in the bracket represents the improvement over the best baseline.

Fig. 5: Scatter visualization of CDTrans and DACSM : The t-SNE visualization showcases the effects of domain alignment on the VisDA-2017 dataset. It is evident that CDTrans struggles to align the ’red’ cluster, resulting in noisy boundaries. In contrast, this figure highlights the capability of DACSM in aligning features from the source and target domains more effectively than CDTrans.

Fig. 6: Hyper parameter analysis. Average accuracy of different number of scales in CSM module on VisDa-2017.

	Train (100 epoch)	Test (14k sample)
CDtrans	90m35s	21s
DACSM (Ours)	100m16s(+21s)	21s

TABLE V: The computational cost of DACSM on VisDA & DomainNet. DACSM does have a slightly higher computational cost than baseline methods. (m: minutes; s: seconds)

Fig. 7: Cross attention construction in DAT. (a) is a picture of source domain, (b) is the reconstructed graph by using $\bm{f}^{s\rightarrow t}$ obtained from CDTrans, (c) is the reconstructed graph by using $\bm{f}^{s\rightarrow t}$ obtained from DACSM, (d) is the reconstructed graph by using $\bm{f}^{t\rightarrow s}$ obtained from DACSM, (e) is a picture of target domain. This means that the reconstructed graph by $\bm{f}^{s\rightarrow t}$ learns the content of source domain and also learns the style information on the target domain.

Method	VisDA	DomainNet	$A$ -distance
Base DAT	79.2	43.7	1.92
DAT+Noise	81.0	45.3	1.75
DAT+CSM	82.1	46.0	1.71
Full DACSM (Ours)	83.5	47.8	1.60

TABLE VI: Comparison of module configurations on VisDA-2017 and DomainNet, demonstrating the complementary effects of noise injection and cross-scale matching.

Method	$\mathcal{L}_{cls}^{s}$	$\mathcal{L}_{dst}$	$\mathcal{L}_{cls}^{t}$	$\mathcal{L}_{cls}^{s\rightarrow t}$	$\mathcal{L}_{s}$	Average
ViT-B	✓	-	-	-	-	73.4
CDTrans-B	✓	✓	✓	-	-	88.4 (+15.0)
DACSM	✓	✓	✓	-	-	89.6 (+1.2)
	✓	✓	✓	✓	-	89.9 (+0.3)
	✓	✓	✓	✓	✓	90.7 (+0.8)

TABLE VII: Comparison among different losses on VisDA-2017, cls and dst imply the classification and the distillation loss. cls

\rightarrow

t and cls

\rightarrow

\rightarrow

s imply the classification loss and the classification loss from source to target. s implies the style loss. The ✓indicates that the loss is used and the - indicates that the loss is not used. The best results are in bold. The numbers in the brackets are the improvements over the Baseline.

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 183 papers

A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning
Jul 3, 2026Sushant Dagaji Desale, Rahul Mishra, Ashutosh Kumar SinhaAdversarial Training Unsupervised Domain Adaptation
Indian Institute of Technology Patna
Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation
Oct 11, 2026Yuling Jiao, Lican Kang, Jerry Zhijian Yang +2Unsupervised Domain Adaptation Representation Learning
Return of Frustratingly Easy Unsupervised Video Domain Adaptation
May 19, 2026Pengfei Wei, Yiqun Sun, Zhiqiang Xu +2Unsupervised Domain Adaptation Video Action Recognition
1Magellan Technology Research Institute (MTRI), Japan · 2Mohamed bin Zayed University of Artificial Intelligence, UAE · 3Nanyang Technological University, Singapore.

Jul 3, 2026J/K move · Enter open · S save

A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning

Sushant Dagaji Desale, Rahul Mishra, Ashutosh Kumar Sinha

Indian Institute of Technology Patna

Adversarial Training Unsupervised Domain Adaptation Adversarial Robustness Vision Foundation Model Adaptation Pseudo-Labeling

Open paper AskPDF arXiv

Adversarial robustness in Unsupervised Domain Adaptation (UDA) remains a significant challenge due to noisy pseudo labels and inherent distributional shifts between the clean source and adversarially perturbed target domains. Existing approaches often fail to achieve an optimal trade-off between robustness and accuracy, as pseudo-labels generated by domain-adapted models tend to introduce classification errors under adversarial attacks. In this work, we propose SFT+RL, a two-stage robust UDA framework that integrates Supervised Fine Tuning (SFT) and Reinforcement Learning (RL) on top of CLIP's pre-trained visual encoder. In the SFT stage, we adversarially fine-tune a linear classifier using PGD-based perturbations over the labelled source domain while partially unfreezing CLIP's projection layer. It allows adaptation to adversarial noise while preserving CLIP's rich semantic priors. We introduce a confidence-guided pseudo-labeling strategy in the RL stage to annotate unlabeled target samples progressively. Pseudo labels are filtered using a decaying confidence threshold to balance quality and coverage, and the model is trained on a composite dataset formed by combining clean source samples with high-confidence target samples. Adversarial training is applied to mixed batches of clean and adversarial examples to enhance cross-domain robustness. Comprehensive evaluations on three benchmark datasets OfficeHome~\cite{tomm-ude}, PACS~\cite{pacs}, and VisDA~\cite{visda} demonstrate the effectiveness of our approach. Notably, SFT+RL achieves average improvements of 10.2% in clean accuracy and 15.8% in adversarial robustness across all three datasets, outperforming existing state-of-the-art methods.

Abstract

Unsupervised Domain Adaptation

Explore similar work

A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning

Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation

Return of Frustratingly Easy Unsupervised Video Domain Adaptation