Model merging promises to construct a single multi-task model from independently fine-tuned task models without accessing the original task data. This makes it attractive when task data cannot be centralized, but released task models may still leak private fine-tuning data. Differential privacy (DP) provides a principled mechanism for limiting such leakage, yet its effect on model merging remains poorly understood. In this paper, we study the geometry of differentially private model merging and identify two geometric obstacles that make private task models difficult to merge: local sharpness, which makes task losses sensitive to the parameter displacement induced by merging, and reference drift, which measures the displacement of private task models from the shared pretrained initialization and amplifies cross-task interference. Based on these observations, we propose DP-Merging, a geometry-aware framework that improves the mergeability of differentially private task models. DP-Merging uses a DP-compatible sharpness-aware objective to guide each private task model toward flatter loss regions, and a reference-based alignment regularizer to keep task models close to the shared pretrained initialization. We derive a merge-gap upper bound showing that reducing local curvature and reference drift tightens the bound on the loss increase induced by merging. Experiments on vision and language tasks across multiple privacy budgets show that DP-Merging consistently improves private merged-model performance while preserving the privacy guarantees of the underlying DP fine-tuning procedures.
Figures & tables
Figure 1: Geometric illustration of model merging under DP. Task vectors are fine-tuned from w0 and merged in parameter space. (a) Without DP, flat and compatible solutions merge effectively. (b) Naive DP can yield sharper solutions with weaker reference alignment, leading to high-loss merging. (c) DP-Merging improves flatness and reference alignment, yielding lower-loss merging under DP.
Figure 2: Interpolation loss landscapes from wt to wmerge . We evaluate test loss along w(γ)=(1−γ)wt+γwmerge . DP-Merging consistently reduces the sharp loss barriers induced by standard DP, yielding smoother paths across vision and language tasks.
Figure 3: Empirical validation on the eight vision tasks using CLIP ViT-B/32.
1:
Input: pretrained model w0 ; private datasets {Dt}t=1T ; iterations K ; learning rate η ; perturbation radius ρt ; alignment strength λ ; clipping threshold C ; noise multiplier σ ; sampling rate q
2:
Output: merged model wmerge
3:
for each task t=1,…,T do
4:
Initialize task model wt,0←w0
5:
for k=0,…,K−1 do
6:
Sample minibatch Bt,k⊂Dt with sampling rate q
7:
Compute private gradient g~t,k←g~(wt,k;Bt,k) using Eq. ( 4 )
Algorithm 1 DP-Merging
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
DP + Individual
59.2
67.7
86.1
89.7
87.5
88.6
89.3
69.4
79.7
DP + Weight Averaging
45.3
43.4
51.9
52.7
43.2
33.8
67.1
30.9
46.0
DP-Merging + Weight Averaging
48.2
49.5
56.3
57.5
43.2
37.4
69.0
36.3
49.7_{{\color[rgb]{1,0,0}+3.7}}
DP + RegMean
45.8
44.9
58.3
58.6
59.1
48.3
73.7
32.0
52.6
DP-Merging + RegMean
49.4
48.6
63.1
62.4
63.6
53.1
76.8
38.2
56.9_{{\color[rgb]{1,0,0}+4.3}}
DP + Task Arithmetic
45.2
44.9
56.7
58.9
60.2
59.7
77.3
30.4
54.2
Table 1: Multi-task accuracy (%) on the 8-task vision benchmark with ViT-B/32, ε=4 .
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
DP + Individual
64.8
74.2
93.1
92.9
92.2
95.4
96.7
75.6
85.5
DP + Weight Averaging
50.3
49.4
57.4
57.7
50.2
39.8
72.5
35.1
51.6
DP-Merging + Weight Averaging
54.4
54.4
62.1
62.2
55.4
45.4
80.1
44.2
57.3_{{\color[rgb]{1,0,0}+5.7}}
DP + RegMean
50.3
49.8
61.0
64.6
63.2
53.4
79.7
41.6
57.9
DP-Merging + RegMean
53.3
54.8
65.9
70.6
69.2
55.0
82.7
49.6
62.7_{{\color[rgb]{1,0,0}+4.8}}
DP + Task Arithmetic
50.2
50.9
61.7
73.5
75.4
54.7
82.3
45.4
61.8
Table 2: Multi-task accuracy (%) on the 8-task vision benchmark with ViT-L/14, ε=4 .
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
DP + Individual
60.5
79.1
81.0
89.2
82.3
77.1
93.8
82.1
80.6
DP + Weight Averaging
56.1
73.8
67.9
79.5
78.6
69.3
91.2
67.6
72.6
DP-Merging + Weight Averaging
58.2
76.6
69.2
79.3
81.6
69.9
91.8
68.2
73.9_{{\color[rgb]{1,0,0}+1.3}}
DP + RegMean
55.6
73.8
65.7
79.2
78.6
67.9
91.0
62.5
71.6
DP-Merging + RegMean
58.8
78.7
68.6
79.5
80.9
69.3
91.9
64.6
74.7_{{\color[rgb]{1,0,0}+3.1}}
DP + Task Arithmetic
56.1
75.5
67.2
78.9
81.5
69.9
91.2
61.7
71.8
Table 3: Multi-task performance of RoBERTa-Large models on GLUE benchmark, ε=4 .
Privacy Budget ε
Visual Tasks (ViT-B/32)
Language Tasks (RoBERTa-Large)
DP(AdamW)
DP-Merging
Gain
DP(AdamW)
DP-Merging
Gain
1.0
46.1
52.2
+6.1
64.6
68.4
+3.8
2.0
52.8
58.6
+5.8
68.3
70.6
+2.3
4.0
57.5
62.1
+4.6
74.3
76.5
+2.2
8.0
62.0
66.7
+4.7
76.3
77.8
+1.5
Table 4: Performance under different privacy budgets on visual and language tasks using TIES-Merging. Smaller ε indicates stronger privacy protection.
Variant
Weight Averaging
Task Arithmetic
TIES-Merging
PCB-Merging
DP-AdamW
46.0
54.2
57.5
54.5
DP-Merging
49.7
58.1
62.1
59.2
DP-Merging w/o Alignment
48.1
55.6
60.7
57.4
DP-Merging w/o Flatness
47.5
56.2
59.8
57.9
Table 5: Ablation study of different merging algorithms on the 8-task vision benchmark with ViT-B/32. “w/o Alignment” represents without the reference-anchored alignment component, and “w/o Flatness” represents without sharpness-aware fine-tuning.
Table 10
Training
Weight Averaging
Task Arithmetic
TIES-Merging
PCB-Merging
WUDI-Merging
DP(AdamW)
46.0
54.2
57.5
54.5
56.5
DP-Merging (Ours)
49.7
58.1
62.1
59.2
60.8
Table 8: Robustness to different merging operators on the 8-task vision benchmark with ViT-B/32.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
w0
Public pretrained initialization shared by all downstream tasks.
T
Number of downstream tasks.
t∈[T]
Task index.
Dt
Private training dataset of task t .
ℓ(w;x,y)
Sample-wise loss evaluated at model parameter w on example (x,y) .
Lt(w)
Empirical loss of task t on Dt .
Appendix
Table 9: Summary of notation.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
DP + Individual
59.8
68.2
88.1
86.9
86.2
89.4
90.7
70.6
80.0
DP + Weight Averaging
45.3
43.4
51.4
51.7
44.2
32.8
67.5
30.1
46.8
DP-Merging + Weight Averaging
49.4
48.4
57.1
56.2
49.4
39.4
75.1
38.6
52.1_{{\color[rgb]{1,0,0}+5.3}}
DP + RegMean
45.3
43.8
55.9
58.6
58.2
47.4
73.7
36.0
52.1
DP-Merging + RegMean
48.3
47.8
59.9
64.6
63.6
49.4
76.7
44.0
56.3_{{\color[rgb]{1,0,0}+4.2}}
DP + Task Arithmetic
45.2
44.9
56.7
68.4
70.5
49.7
77.3
40.4
57.9
Appendix
Table 10: Multi-task accuracy (%) on the 8-task vision benchmark with ViT-B/16, ε=4 .
Figure 4: Loss landscapes of ViT models that were first Naive DP fine-tuned under ε=4 and then merged using Task Arithmetic (ViT-B/32, ViT-B/16, ViT-L/14).
Figure 5: Loss landscapes of the merged ViT models (ViT-B/32, ViT-B/16, and ViT-L/14) under ε=4 , obtained using DP-Merging with Task Arithmetic.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
DP + Individual
59.2
77.7
80.1
88.7
81.5
76.7
92.7
79.4
79.8
DP + Weight Averaging
55.3
73.4
67.4
78.7
78.2
68.8
90.5
66.1
72.3
DP-Merging + Weight Averaging
58.2
76.1
69.0
79.1
81.3
69.7
91.6
68.2
73.0_{{\color[rgb]{1,0,0}+0.7}}
DP + RegMean
55.3
73.5
65.6
78.6
78.1
67.4
90.7
62.0
70.4
DP-Merging + RegMean
58.6
78.2
68.4
79.0
80.6
69.2
91.6
63.8
73.7_{{\color[rgb]{1,0,0}+3.3}}
DP + Task Arithmetic
55.2
74.9
66.7
78.9
80.2
69.7
90.3
61.4
70.8
Appendix
Table 11: Multi-task performance of RoBERTa-Base models on GLUE benchmark, ε=4 .
Dec 1, 2025·Kuangpu Guo, Aijing Yu, Jian Liang +4Model Merging
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3