Model merging promises to construct a single multi-task model from independently fine-tuned task models without accessing the original task data. This makes it attractive when task data cannot be centralized, but released task models may still leak private fine-tuning data. Differential privacy (DP) provides a principled mechanism for limiting such leakage, yet its effect on model merging remains poorly understood. In this paper, we study the geometry of differentially private model merging and identify two geometric obstacles that make private task models difficult to merge: local sharpness, which makes task losses sensitive to the parameter displacement induced by merging, and reference drift, which measures the displacement of private task models from the shared pretrained initialization and amplifies cross-task interference. Based on these observations, we propose DP-Merging, a geometry-aware framework that improves the mergeability of differentially private task models. DP-Merging uses a DP-compatible sharpness-aware objective to guide each private task model toward flatter loss regions, and a reference-based alignment regularizer to keep task models close to the shared pretrained initialization. We derive a merge-gap upper bound showing that reducing local curvature and reference drift tightens the bound on the loss increase induced by merging. Experiments on vision and language tasks across multiple privacy budgets show that DP-Merging consistently improves private merged-model performance while preserving the privacy guarantees of the underlying DP fine-tuning procedures.
Figures & tables
Figure 1: Geometric illustration of model merging under DP. Task vectors are fine-tuned from w0 and merged in parameter space. (a) Without DP, flat and compatible solutions merge effectively. (b) Naive DP can yield sharper solutions with weaker reference alignment, leading to high-loss merging. (c) DP-Merging improves flatness and reference alignment, yielding lower-loss merging under DP.
Figure 2: Interpolation loss landscapes from wt to wmerge . We evaluate test loss along w(γ)=(1−γ)wt+γwmerge . DP-Merging consistently reduces the sharp loss barriers induced by standard DP, yielding smoother paths across vision and language tasks.
Figure 3: Empirical validation on the eight vision tasks using CLIP ViT-B/32.
1:
Input: pretrained model w0 ; private datasets {Dt}t=1T ; iterations K ; learning rate η ; perturbation radius ρt ; alignment strength λ ; clipping threshold C ; noise multiplier σ ; sampling rate q
2:
Output: merged model wmerge
3:
for each task t=1,…,T do
4:
Initialize task model wt,0←w0
5:
for k=0,…,K−1 do
6:
Sample minibatch Bt,k⊂Dt with sampling rate q
7:
Compute private gradient g~t,k←g~(wt,k;Bt,k) using Eq. ( 4 )
Algorithm 1 DP-Merging
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
DP + Individual
59.2
67.7
86.1
89.7
87.5
88.6
89.3
69.4
79.7
DP + Weight Averaging
45.3
43.4
51.9
52.7
43.2
33.8
67.1
30.9
46.0
DP-Merging + Weight Averaging
48.2
49.5
56.3
57.5
43.2
37.4
69.0
36.3
49.7_{{\color[rgb]{1,0,0}+3.7}}
DP + RegMean
45.8
44.9
58.3
58.6
59.1
48.3
73.7
32.0
52.6
DP-Merging + RegMean
49.4
48.6
63.1
62.4
63.6
53.1
76.8
38.2
56.9_{{\color[rgb]{1,0,0}+4.3}}
DP + Task Arithmetic
45.2
44.9
56.7
58.9
60.2
59.7
77.3
30.4
54.2
Table 1: Multi-task accuracy (%) on the 8-task vision benchmark with ViT-B/32, ε=4 .
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
DP + Individual
64.8
74.2
93.1
92.9
92.2
95.4
96.7
75.6
85.5
DP + Weight Averaging
50.3
49.4
57.4
57.7
50.2
39.8
72.5
35.1
51.6
DP-Merging + Weight Averaging
54.4
54.4
62.1
62.2
55.4
45.4
80.1
44.2
57.3_{{\color[rgb]{1,0,0}+5.7}}
DP + RegMean
50.3
49.8
61.0
64.6
63.2
53.4
79.7
41.6
57.9
DP-Merging + RegMean
53.3
54.8
65.9
70.6
69.2
55.0
82.7
49.6
62.7_{{\color[rgb]{1,0,0}+4.8}}
DP + Task Arithmetic
50.2
50.9
61.7
73.5
75.4
54.7
82.3
45.4
61.8
Table 2: Multi-task accuracy (%) on the 8-task vision benchmark with ViT-L/14, ε=4 .
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
DP + Individual
60.5
79.1
81.0
89.2
82.3
77.1
93.8
82.1
80.6
DP + Weight Averaging
56.1
73.8
67.9
79.5
78.6
69.3
91.2
67.6
72.6
DP-Merging + Weight Averaging
58.2
76.6
69.2
79.3
81.6
69.9
91.8
68.2
73.9_{{\color[rgb]{1,0,0}+1.3}}
DP + RegMean
55.6
73.8
65.7
79.2
78.6
67.9
91.0
62.5
71.6
DP-Merging + RegMean
58.8
78.7
68.6
79.5
80.9
69.3
91.9
64.6
74.7_{{\color[rgb]{1,0,0}+3.1}}
DP + Task Arithmetic
56.1
75.5
67.2
78.9
81.5
69.9
91.2
61.7
71.8
Table 3: Multi-task performance of RoBERTa-Large models on GLUE benchmark, ε=4 .
Privacy Budget ε
Visual Tasks (ViT-B/32)
Language Tasks (RoBERTa-Large)
DP(AdamW)
DP-Merging
Gain
DP(AdamW)
DP-Merging
Gain
1.0
46.1
52.2
+6.1
64.6
68.4
+3.8
2.0
52.8
58.6
+5.8
68.3
70.6
+2.3
4.0
57.5
62.1
+4.6
74.3
76.5
+2.2
8.0
62.0
66.7
+4.7
76.3
77.8
+1.5
Table 4: Performance under different privacy budgets on visual and language tasks using TIES-Merging. Smaller ε indicates stronger privacy protection.
Variant
Weight Averaging
Task Arithmetic
TIES-Merging
PCB-Merging
DP-AdamW
46.0
54.2
57.5
54.5
DP-Merging
49.7
58.1
62.1
59.2
DP-Merging w/o Alignment
48.1
55.6
60.7
57.4
DP-Merging w/o Flatness
47.5
56.2
59.8
57.9
Table 5: Ablation study of different merging algorithms on the 8-task vision benchmark with ViT-B/32. “w/o Alignment” represents without the reference-anchored alignment component, and “w/o Flatness” represents without sharpness-aware fine-tuning.
Table 10
Training
Weight Averaging
Task Arithmetic
TIES-Merging
PCB-Merging
WUDI-Merging
DP(AdamW)
46.0
54.2
57.5
54.5
56.5
DP-Merging (Ours)
49.7
58.1
62.1
59.2
60.8
Table 8: Robustness to different merging operators on the 8-task vision benchmark with ViT-B/32.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
w0
Public pretrained initialization shared by all downstream tasks.
T
Number of downstream tasks.
t∈[T]
Task index.
Dt
Private training dataset of task t .
ℓ(w;x,y)
Sample-wise loss evaluated at model parameter w on example (x,y) .
Lt(w)
Empirical loss of task t on Dt .
Appendix
Table 9: Summary of notation.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
DP + Individual
59.8
68.2
88.1
86.9
86.2
89.4
90.7
70.6
80.0
DP + Weight Averaging
45.3
43.4
51.4
51.7
44.2
32.8
67.5
30.1
46.8
DP-Merging + Weight Averaging
49.4
48.4
57.1
56.2
49.4
39.4
75.1
38.6
52.1_{{\color[rgb]{1,0,0}+5.3}}
DP + RegMean
45.3
43.8
55.9
58.6
58.2
47.4
73.7
36.0
52.1
DP-Merging + RegMean
48.3
47.8
59.9
64.6
63.6
49.4
76.7
44.0
56.3_{{\color[rgb]{1,0,0}+4.2}}
DP + Task Arithmetic
45.2
44.9
56.7
68.4
70.5
49.7
77.3
40.4
57.9
Appendix
Table 10: Multi-task accuracy (%) on the 8-task vision benchmark with ViT-B/16, ε=4 .
Figure 4: Loss landscapes of ViT models that were first Naive DP fine-tuned under ε=4 and then merged using Task Arithmetic (ViT-B/32, ViT-B/16, ViT-L/14).
Figure 5: Loss landscapes of the merged ViT models (ViT-B/32, ViT-B/16, and ViT-L/14) under ε=4 , obtained using DP-Merging with Task Arithmetic.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
DP + Individual
59.2
77.7
80.1
88.7
81.5
76.7
92.7
79.4
79.8
DP + Weight Averaging
55.3
73.4
67.4
78.7
78.2
68.8
90.5
66.1
72.3
DP-Merging + Weight Averaging
58.2
76.1
69.0
79.1
81.3
69.7
91.6
68.2
73.0_{{\color[rgb]{1,0,0}+0.7}}
DP + RegMean
55.3
73.5
65.6
78.6
78.1
67.4
90.7
62.0
70.4
DP-Merging + RegMean
58.6
78.2
68.4
79.0
80.6
69.2
91.6
63.8
73.7_{{\color[rgb]{1,0,0}+3.3}}
DP + Task Arithmetic
55.2
74.9
66.7
78.9
80.2
69.7
90.3
61.4
70.8
Appendix
Table 11: Multi-task performance of RoBERTa-Base models on GLUE benchmark, ε=4 .
In machine learning, privacy requirements at inference or deployment time often evolve due to changing policies, regulations, or user preferences. In this work, we aim to construct a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post-processing techniques, namely random selection and linear combination, to generate final private models satisfying any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions on general problems, as well as on private mean estimation, where we precisely characterize the privacy/utility tradeoffs and compare the two mechanisms. Empirically, we demonstrate the effectiveness of our approaches and validate our analyses on several models and both synthetic and real-world datasets.
Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
Kuangpu Guo, Aijing Yu, Jian Liang +4
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3
Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods
Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2
Mila & DIRO, Universit´e de Montr´eal · University of Toronto & Vector Institute