Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models

Wentao Yue; Qingyu Mao; Tianyou Lai; Ahmed M. Abdelmoniem; Qilei Li

cs.CVOct 1, 2026

Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models

Authors: Wentao Yue, Qingyu Mao, Tianyou Lai, Ahmed M. Abdelmoniem, Qilei Li

Abstract

Federated parameter-efficient fine-tuning enables distributed clients to adapt pretrained vision-language models without sharing raw data or updating the full backbone. Its effectiveness, however, is limited by domain heterogeneity across clients. Existing personalized methods separate globally shared knowledge from client-specific style, but they largely treat each domain as a class-agnostic transformation. We show that this abstraction is insufficient: the cross-domain displacement associated with a fixed domain varies across semantic classes, and only a subset of these class-domain residuals damages the image-text decision margin. We therefore propose Margin-Oriented Semantic-Appearance Interaction Correction (MOSAIC), which first constructs a decision-aware harmfulness score that measures whether a training-derived class-domain residual favors a competing text prototype over the true class. It then models fine-grained class-domain interactions with a low-rank residual adapter whose class factors and residual basis are globally shared while domain factors remain client-private. An image-conditioned gate further controls candidate-wise correction, and harmful-pair-aware reweighting prioritizes decision-relevant residuals during local optimization. Extensive experiments on Office31, OfficeHome, and DomainNet100 demonstrate that MOSAIC consistently improves macro-client top-1 accuracy across all evaluated domain-shift and joint domain-label-shift settings.

Figures & tables

Figure 1: Motivation of MOSAIC. Existing personalized federated VLM tuning usually treats each domain as a class-agnostic style, while the same domain can shift different semantic classes differently and only some class-domain residuals harm the decision margin.

Figure 2: Class-conditioned domain variation on DomainNet100. Rows show dog and chair, while columns cover all six rendering domains. Cross-domain changes differ between the two semantic classes, indicating that domain heterogeneity cannot be represented faithfully by a single class-agnostic shift.

Figure 3: Feature-space evidence on DomainNet100. Left: class-specific shifts into sketch deviate from one shared shift. Right: only residuals that reduce the true-versus-competitor margin receive positive harmfulness scores. The harmful margin score is defined in Eq. 8 .

Figure 4: Overview of MOSAIC. Stage 1 identifies harmful class-domain pairs; Stage 2 factorizes and gates candidate-wise residual correction. Only shared parameters are aggregated.

Setting ①: one domain for one client
Method	Office31				OfficeHome					DomainNet100
Method	A	W	D	Avg.	A	C	P	R	Avg.	C	I	P	Q	R	S	Avg.
Zero-shot CLIP [ICML2021]	81.14	72.45	74.05	75.88	84.30	66.28	89.06	89.66	82.33	71.93	53.30	65.73	13.57	83.49	66.46	59.08
PromptFL [TMC2023]	88.90	87.55	94.30	90.25	86.94	75.76	94.32	93.59	87.65	86.55	70.29	79.89	34.31	91.54	79.97	73.76
PromptFL+Prox [TMC2023; MLSys2020]	89.22	89.80	93.04	90.68	86.16	76.28	94.25	93.59	87.57	87.47	71.25	82.15	32.63	91.79	81.20	74.41
FedOTP [CVPR2024]	85.73	94.69	94.94	91.79	79.71	76.24	92.18	87.10	83.81	86.73	69.80	82.05	50.37	90.75	82.69	77.06

Table 1: Top-1 accuracy (%) under domain shift. Setting ① uses one client per domain and Setting ② uses two. Brackets show publication venues. Pale red, yellow, and green mark the first-, second-, and third-best results; bold and underline additionally identify the top two.

Method	Office31			OfficeHome			DomainNet100
Method	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$
Zero-shot CLIP [ICML2021]	75.36	75.01	75.49	82.27	82.39	82.30	59.07	59.24	59.11
PromptFL [TMC2023]	89.01	90.44	88.40	87.35	87.23	87.01	73.14	73.88	74.23
PromptFL+Prox [TMC2023; MLSys2020]	89.27	89.66	88.11	87.46	87.32	87.36	73.41	73.66	74.07
FedOTP [CVPR2024]	90.78	90.54	88.75	84.81	85.66	84.28	77.52	77.70	76.94
FedPGP [ICML2024]	91.78	90.88	91.68	89.49	89.63	88.78	80.72	82.46	82.12

Table 2: Macro-client top-1 accuracy (%) under joint domain and label shifts in Setting ②. Brackets show publication venues. Pale red, yellow, and green mark the first-, second-, and third-best results; bold and underline additionally identify the top two.

Figure 5: Component ablation on Office31 under Setting ①. Each panel uses its own vertical scale to make the domain-wise differences visible. A5 denotes the complete MOSAIC configuration.

Figure 6: Component ablation on OfficeHome under Setting ①. The complete configuration (A5) gives the highest macro-client average.

Axis	Value	Final	Best
Rank $r$	8	95.85	96.44
	16	95.67	96.52
	32	96.43	96.86
Scale $\alpha$	0.03	95.97	96.69
	0.05	96.50	96.80
	0.10	95.67	96.52

Table 3: Parameter sensitivity on Office31 Setting ①. Bold and underline mark the best and second-best results.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Rank $r$	Scale $\alpha$	Harmful weight $\gamma$	Quantile $q$	Regularization $\lambda$
Office31	32	0.10	1.5	20%	$10^{-4}$
OfficeHome	16	0.05	1.5	20%	$10^{-4}$
DomainNet100	32	0.10	1.5	20%	$10^{-4}$
Candidate values	$\{8,16,32\}$	$\{0.03,0.05,0.10\}$	$\{1.0,1.5,2.0\}$	$\{10,20,25,50\}\%$	$\{10^{-5},10^{-4},5{\times}10^{-4}\}$

Appendix

Table 4: Per-dataset residual-module hyperparameters. The final rows give the configurations used in the reported experiments. The candidate row lists values considered across the completed staged searches; it is not an exhaustive Cartesian-product sweep on every dataset.

Method	Adapted state	Shared mechanism	Private mechanism	Correction granularity
Zero-shot CLIP	None	Pretrained image–text space	None	None
PromptFL	Text prompt	Federated prompt averaging	None	Task/global prompt
PromptFL+Prox	Text prompt	Prompt averaging with proximal control	None	Task/global prompt
FedOTP	Text prompts	Global prompt	Local prompt with optimal transport	Client/prompt
FedPGP	Text prompts	CLIP-guided global knowledge	Low-rank personalized prompt	Client/prompt
FedDDA	Text and visual adapters	Global prompt and shared adapter	Local prompt and specific adapter	Client/domain

Appendix

Table 5: Mechanistic scope of the comparison methods. “Private” denotes a client-retained adaptation state.

Dataset	$(C,r)$	Shared params	Round trip	Statistics
			(KiB)	(KiB)
Office31	$(31,32)$	33,791	264.0	62.1
OfficeHome	$(65,16)$	17,489	136.6	130.3
DomainNet100	$(100,32)$	36,068	281.8	200.4

Appendix

Table 6: Residual-module storage and communication overhead. Communication assumes FP32 and reports one client upload plus one server download per round. Statistics are a one-time per-domain upper bound when all classes are observed.

Panel of Figure 7 — Figure 7: Additional qualitative evidence of class-conditioned domain drift. Classes form rows and domains form columns. The same domain changes different semantic classes through different visual cues, rather than through one class-agnostic offset.

Dataset	$\rho_{\mathrm{CLIP}}$	$\rho_{\mathrm{FedDDA}}$	Intersection/size	Jaccard
Office31	0.656	0.724	9/18	0.333
OfficeHome	0.908	0.852	27/52	0.351
DomainNet100	0.930	0.897	69/120	0.404

Appendix

Table 7: Robustness to the diagnostic representation.

\rho_{\mathrm{CLIP}}

and

\rho_{\mathrm{FedDDA}}

are Spearman correlations between

h

and held-out

\Delta m

in frozen-CLIP and round-49 FedDDA spaces. Set overlap compares domain-balanced top-20% harmful sets; both sets have the size shown after the slash.

Figure 8: Pair-level harmfulness versus held-out counterfactual margin gain. Each point is one class-domain pair; the residual is computed on training data and evaluated on held-out test images. Spearman rank correlations are shown in each panel.

Figure 9: Domain-balanced harmfulness-quintile analysis. Values are unweighted pair means. Higher harmfulness generally corresponds to a more favorable response to oracle residual removal, while low-score pairs are frequently damaged by indiscriminate correction.

Figure 10: Representative Q5 class-domain pairs. Columns correspond to Office31, OfficeHome, and DomainNet100; the top and bottom rows show success and failure cases, respectively. Pair identities and metrics are given in the surrounding text. Displayed images are representative examples, not individually scored predictions.

Figure 11: Domain-level accuracy profiles of all comparison methods under Setting 1, using the values reported in the main tables. From left to right, the three panels correspond to Office31, OfficeHome, and DomainNet100; domain abbreviations follow the main paper. MOSAIC and FedDDA are emphasized with solid and dash-dotted contours, respectively.

Dataset	Pairs	$\rho(h,\Delta m)$	$\rho(\\|r\\|,\Delta m)$	$\rho(h,e)$
Office31	93	0.724	$-0.275$	$-0.144$
OfficeHome	260	0.852	0.207	0.364
DomainNet100	600	0.897	0.145	0.619

Appendix

Table 8: Pair-level Spearman correlations.

h

is the harmfulness score,

\|r\|

is residual norm,

e

is held-out error, and

\Delta m

is counterfactual margin gain.

		Setting 1: one client/domain			Setting 2: two clients/domain
Dataset	Domain	FedDDA	MOSAIC	$\Delta$	FedDDA	MOSAIC	$\Delta$
Office31	Amazon	89.32	90.57	+1.25	88.39	90.00	+1.61
	Webcam	97.14	100.00	+2.86	95.06	98.10	+3.04
	DSLR	98.23	100.00	+1.77	98.10	96.90	$-1.20$
	Average	94.90	96.86	+1.96	93.85	95.00	+1.15
OfficeHome	Art	87.07	88.60	+1.53	86.69	88.25	+1.56

Appendix

Table 9: Complete domain-level FedDDA and MOSAIC top-1 accuracy (%) under the two client-partition protocols in the main paper. Average is the macro-client accuracy; because every domain contains the same number of clients, it is also the unweighted mean of the domain-level accuracies.

\Delta

is MOSAIC minus FedDDA.

Class	Acc.	Class	Acc.
back_pack	96.0	mouse	100.0
bike	100.0	mug	95.8
bike_helmet	100.0	paper_notebook	89.3
bookcase	91.3	pen	96.2
bottle	69.2	phone	95.8
calculator	100.0	printer	92.3

Appendix

Table 10: Class-level MOSAIC accuracy (%) on Office31 at the saved terminal checkpoint.

Class	Acc.	Class	Acc.	Class	Acc.
Alarm_Clock	100.0	Flowers	100.0	Postit_Notes	83.9
Backpack	93.0	Folder	83.3	Printer	98.1
Batteries	91.5	Fork	83.8	Push_Pin	90.9
Bed	98.0	Glasses	100.0	Radio	88.9
Bike	96.8	Hammer	85.7	Refrigerator	91.3
Bottle	88.2	Helmet	100.0	Ruler	89.5

Appendix

Table 11: Class-level MOSAIC accuracy (%) on OfficeHome at the saved terminal checkpoint.

Class	Acc.	Class	Acc.	Class	Acc.	Class	Acc.
aircraft_carrier	70.7	bed	80.3	cactus	87.6	coffee_cup	77.2
airplane	89.3	bee	91.4	cake	59.6	compass	79.3
alarm_clock	73.0	belt	61.6	calculator	87.6	computer	81.4
ambulance	85.2	bench	72.4	calendar	74.6	cookie	80.0
angel	90.2	bicycle	94.2	camel	88.8	cooler	64.1
animal_migration	90.4	binoculars	81.8	camera	92.2	couch	73.5

Appendix

Table 12: Class-level MOSAIC accuracy (%) on DomainNet100 at the saved terminal checkpoint.

Explore similar work

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
May 27, 2026Yuting Ma, Lechao Cheng, Xiaohua XuFederated Learning Vision-Language Model Adaptation
School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.
FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models
May 27, 2026Xucong Wang, Pengkun Wang, Zhe Zhao +3Multi-Label Classification Fedavg
University of Science and Technology of China (USTC) · Suzhou Institute for Advanced Research, USTC · City University of Hong Kong
VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models
Oct 15, 2025Dominick Reilly, Manish Kumar Govind, Le Xue +1Vision-Language Model Adaptation Domain Adaptation
University of North Carolina at Charlotte · Elorian AI

May 27, 2026, cs.CVJ/K move · Enter open · S save

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

Yuting Ma, Lechao Cheng, Xiaohua Xu

School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.

Federated Learning Vision-Language Model Adaptation Fedavg Model Fine-Tuning Efficient Task Adaptation Reinforcement Fine-Tuning

Open paper AskPDF arXiv

Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation under insufficient local data while preserving generalization. However, these methods emphasize fully local optimization with simple parameter aggregation,which can amplify inter-client optimization inconsistency and intra-client over-specialization under heterogeneous and full-data FL settings, making it difficult to balance global task adaptation and generalization. To address these challenges, we propose FedDTL, a novel federated VLM framework that decouples the image encoder and text encoder across clients and the server. Through decoupled encoder training with server-client modality alignment, FedDTL promotes coherent global semantic update and reduces inter-client optimization inconsistency, improving global task adaptation.To further mitigate intra-client over-specialization,we introduce a two-stage local fine-tuning, where a supervised fine-tuning stage enables rapid and reliable warm-start, followed by a reinforcement learning stage that enhances generalization. Extensive experiments on multiple benchmarks, including label skew and feature shift, demonstrate that FedDTL achieves an effective balance between global task adaptation and generalization under various FL data distributions in both few-shot and full-data regimes.

Abstract

Explore similar work

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models