Joint Null-Space Projection for Data-Free Model Merging
Abstract
Merging independently fine-tuned checkpoints into a single multi-task model requires no data, no labels, and no gradient steps. In practice the task vectors of different tasks interfere, and the standard remedies--averaging, magnitude pruning, sign election--operate entry by entry, ignoring that each delta induces a low-dimensional update on the hidden features it acts upon. We recast merging as a subspace interference problem. For every two-dimensional weight tensor we take the top-r right-singular subspace of each task's delta--the input directions along which that task moves the representation most--and remove from each task vector the component lying in the union of the other tasks' subspaces. The resulting Joint Null-Space Projection (JNSP) is closed-form and order-independent, needs no forward pass, and is defined entirely by the checkpoint set. On eight CLIP image-classification tasks JNSP merges ViT-B/16 experts to 89.45% average accuracy without using a single image, 11.88 points above the strongest parameter-wise training-free baseline (TIES-Merging, 77.57%), 14.22 above Task Arithmetic, and 0.98 above WUDI's data-free consensus merge (88.47%). The gain holds across three backbones on the independently trained TALL release (+10.8, +12.6, +9.1 over TIES for ViT-B/16, ViT-B/32, ViT-L/14) and replicates there at 89.20%; controls confirm the source--a single shared subspace instead of per-task ones loses 2.9 points, and a sequential variant collapses to 69.6%. On RoBERTa-large the same projection underperforms plain task arithmetic, which we report rather than hide. Code is available at https://github.com.
Figures & tables
| Method | MNIST | EuroSAT | GTSRB | SVHN | DTD | RESISC45 | Cars | SUN397 | Avg | Pub. |
|---|---|---|---|---|---|---|---|---|---|---|
| Task Arithmetic [ 23 ] | 97.15 | 81.81 | 71.08 | 83.02 | 52.55 | 77.51 | 70.42 | 68.29 | 75.23 | — |
| DARE [ 53 ] | 97.06 | 81.30 | 70.91 | 82.40 | 52.07 | 77.10 | 70.10 | 68.31 | 74.91 | — |
| TIES-Merging [ 50 ] | 98.71 | 82.78 | 71.86 | 90.35 | 57.98 | 81.10 | 71.09 | 66.70 | 77.57 | — |
| WUDI [ 6 ] | 99.44 | 98.00 | 96.55 | 95.30 | 73.62 | 90.13 | 81.58 | 73.13 | 88.47 | — |
| TSV-M † [ 17 ] | 99.27 | 96.67 | 93.96 | 93.93 | 72.61 | 89.16 | 80.67 | 71.49 | 87.22 | 89.01 |
| Iso-C † | 98.78 | 96.78 | 91.74 | 88.27 | 71.65 | 92.16 | 82.54 | 75.28 | 87.15 | n/a |
| Backbone | Method | MNIST | EuroSAT | GTSRB | SVHN | DTD | RESISC45 | Cars | SUN397 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| ViT-B/16 | Task Arithmetic | 97.10 | 84.89 | 72.04 | 80.45 | 52.87 | 76.95 | 71.44 | 68.30 | 75.51 |
| DARE | 97.08 | 84.70 | 72.43 | 80.20 | 52.77 | 76.43 | 71.16 | 68.09 | 75.36 | |
| TIES-Merging | 98.69 | 89.41 | 73.34 | 89.19 | 56.17 | 79.71 | 73.64 | 66.69 | 78.36 | |
| TSV-M † | 99.33 | 97.52 | 93.99 | 93.12 | 71.44 | 88.95 | 82.19 | 71.44 | 87.25 | |
| Iso-C † | 98.86 | 97.56 | 92.78 | 87.05 | 72.07 | 92.21 | 83.74 | 75.42 | 87.46 | |
| Iso-CTS † | 98.55 | 96.78 | 92.15 | 83.35 | 70.96 | 92.06 | 83.61 | 75.50 | 86.62 |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | own top-16 | captured | retained | |
|---|---|---|---|---|
| MNIST | 0.364 | 0.824 | 0.789 | 0.615 |
| EuroSAT | 0.366 | 0.736 | 0.835 | 0.550 |
| GTSRB | 0.288 | 0.761 | 0.828 | 0.560 |
| SVHN | 0.409 | 0.784 | 0.806 | 0.592 |
| DTD | 0.307 | 0.511 | 0.774 | 0.634 |
| RESISC45 | 0.359 | 0.639 | 0.790 | 0.613 |
| Dataset | Classes | Test | Domain |
|---|---|---|---|
| MNIST [ 27 ] | 10 | 10,000 | handwritten digits |
| EuroSAT [ 19 ] | 10 | 2,700 | satellite land use |
| GTSRB [ 41 ] | 43 | 12,630 | traffic signs |
| SVHN [ 33 ] | 10 | 26,032 | street numbers |
| DTD [ 8 ] | 47 | 1,880 | describable textures |
| RESISC45 [ 5 ] | 45 | 6,300 | aerial scenes |
| 0.1 | 0.2 | 0.3 | 0.5 | |
|---|---|---|---|---|
| Task Arithmetic | 69.53 | 75.23 | 74.87 | 60.23 |
| DARE | 69.53 | 74.91 | 74.27 | 58.78 |
| 1.0 | 1.5 | 2.0 | 3.0 | |
| TIES-Merging | 77.57 | 69.37 | 55.13 | 33.49 |
| MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | ||
|---|---|---|---|---|---|---|---|---|---|
| 0.1 | 91.6 | 75.8 | 56.3 | 71.7 | 48.6 | 74.8 | 69.2 | 68.2 | |
| 0.2 | 97.2 | 81.8 | 71.1 | 83.0 | 52.6 | 77.5 | 70.4 | 68.3 | |
| 0.3 | 98.2 | 79.1 | 75.5 | 88.2 | 52.3 | 75.6 | 67.4 | 62.5 | |
| 0.5 | 98.6 | 60.3 | 66.7 | 89.4 | 40.3 | 53.6 | 44.7 | 28.3 |
| MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | ||
|---|---|---|---|---|---|---|---|---|---|
| 0.1 | 91.6 | 75.8 | 56.5 | 71.5 | 48.7 | 74.7 | 69.3 | 68.2 | |
| 0.2 | 97.1 | 81.3 | 70.9 | 82.4 | 52.1 | 77.1 | 70.1 | 68.3 | |
| 0.3 | 98.2 | 78.0 | 75.3 | 87.7 | 51.6 | 74.9 | 66.5 | 61.9 | |
| 0.5 | 98.4 | 58.8 | 66.1 | 88.2 | 37.3 | 51.7 | 42.5 | 27.3 |
| MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | ||
|---|---|---|---|---|---|---|---|---|---|
| 0.1 | 65.0 | 62.4 | 46.6 | 59.4 | 47.0 | 70.0 | 67.0 | 66.6 | |
| 0.2 | 81.1 | 69.1 | 49.6 | 65.6 | 48.6 | 73.0 | 68.6 | 67.8 | |
| 0.3 | 89.5 | 73.4 | 52.8 | 70.9 | 49.4 | 75.7 | 70.0 | 69.0 | |
| 0.5 | 95.9 | 80.6 | 61.5 | 79.5 | 53.4 | 79.6 | 72.4 | 70.4 | |
| 0.7 | 97.9 | 83.5 | 69.5 | 85.8 | 56.3 | 81.6 | 73.1 | 70.3 | |
| 1.0 | 98.7 | 82.8 | 71.9 | 90.4 | 58.0 | 81.1 | 71.1 | 66.7 |
| Backbone | Peak | |||||
|---|---|---|---|---|---|---|
| ViT-B/16 | 88.03 | 89.30 | 89.45 | 89.22 | 87.08 | 89.45 |
| ViT-B/32 | 81.57 | 82.93 | 82.92 | 82.54 | 79.72 | 82.93 |
| ViT-L/14 | 89.99 | 90.68 | 90.87 | 90.83 | 90.13 | 90.87 |
| Variant | MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | Avg | |
|---|---|---|---|---|---|---|---|---|---|---|
| Joint | 8 | 99.31 | 97.26 | 96.41 | 94.41 | 74.41 | 90.56 | 80.53 | 71.38 | 88.03 |
| Joint | 12 | 99.27 | 98.07 | 96.63 | 93.46 | 77.18 | 92.49 | 83.45 | 73.86 | 89.30 |
| Joint | 16 | 99.07 | 98.44 | 96.26 | 92.16 | 77.23 | 93.22 | 84.25 | 74.97 | 89.45 |
| Joint | 20 | 98.96 | 98.33 | 95.33 | 90.61 | 76.38 | 93.43 | 85.12 | 75.58 | 89.22 |
| Joint | 32 | 98.46 | 96.93 | 89.91 | 84.77 | 73.56 | 92.37 | 84.91 | 75.76 | 87.08 |
| Shared | 8 | 98.63 | 96.63 | 94.75 | 92.18 | 70.80 | 86.51 | 74.05 | 66.43 | 85.00 |
| Captured by neighbours | Own top- | |||||||
|---|---|---|---|---|---|---|---|---|
| Task | ||||||||
| MNIST | 63.0 | 71.5 | 80.2 | 89.3 | 75.4 | 83.7 | 89.6 | 94.0 |
| EuroSAT | 53.6 | 64.4 | 75.9 | 87.6 | 62.8 | 72.8 | 81.6 | 89.0 |
| GTSRB | 53.8 | 63.4 | 74.1 | 86.0 | 66.1 | 76.7 | 85.5 | 91.8 |
| SVHN | 61.7 | 70.5 | 79.5 | 89.1 | 70.9 | 80.4 | 87.4 | 92.7 |
| DTD | 44.3 | 54.7 | 67.6 | 82.4 | 50.1 | 60.0 | 70.6 | 80.4 |
| MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | |
| MNIST | — | 33.1 | 48.5 | 59.9 | 32.0 | 31.9 | 32.2 | 30.7 |
| EuroSAT | 31.4 | — | 35.6 | 34.6 | 34.9 | 45.1 | 30.4 | 34.6 |
| GTSRB | 43.7 | 35.0 | — | 46.1 | 32.4 | 32.1 | 31.5 | 31.4 |
| SVHN | 55.4 | 35.9 | 49.2 | — | 31.9 | 31.4 | 29.9 | 30.0 |
| DTD | 26.9 | 30.9 | 28.1 | 27.2 | — | 33.1 | 29.5 | 33.4 |
| RESISC45 | 27.2 | 39.9 | 28.8 | 27.6 | 33.7 | — | 30.9 | 36.2 |
| Task | Position | Sequential | Joint | |
| MNIST | 1 | 99.59 | 99.31 | |
| EuroSAT | 2 | 93.81 | 97.26 | |
| GTSRB | 3 | 72.19 | 96.41 | |
| SVHN | 4 | 88.78 | 94.41 | |
| DTD | 5 | 50.69 | 74.41 | |
| RESISC45 | 6 | 64.78 | 90.56 |
| Variant | |||||
|---|---|---|---|---|---|
| Joint | 33.44 | 31.73 | 30.36 | 29.16 | 25.98 |
| Shared basis | 35.47 | 34.04 | 32.83 | 32.01 | 29.98 |
| Task | ||||
|---|---|---|---|---|
| MNIST | 79.8 | 75.3 | 71.8 | 68.8 |
| EuroSAT | 84.2 | 80.0 | 76.3 | 73.1 |
| GTSRB | 83.5 | 79.5 | 76.0 | 72.9 |
| SVHN | 79.6 | 75.1 | 71.6 | 68.5 |
| DTD | 87.8 | 84.4 | 81.3 | 78.5 |
| RESISC45 | 86.0 | 82.1 | 78.7 | 75.6 |
| Module type | ||||
|---|---|---|---|---|
| MLP output projection ( c_proj ) | 87.7 | 85.7 | 84.0 | 82.5 |
| Attention output projection ( out_proj ) | 88.1 | 84.0 | 80.2 | 76.8 |
| Attention input projections ( in_proj ) | 84.6 | 80.3 | 76.5 | 73.1 |
| MLP input projection ( c_fc ) | 85.0 | 80.7 | 76.9 | 73.5 |
| Embeddings and visual/text projections | 71.4 | 65.9 | 62.1 | 58.9 |
| Spread (max min) | 16.7 | 19.8 | 21.9 | 23.6 |
| MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | |
|---|---|---|---|---|---|---|---|---|
| MNIST | – | 0.077 | 0.109 | 0.214 | 0.074 | 0.074 | 0.072 | 0.069 |
| EuroSAT | 0.156 | – | 0.075 | 0.080 | 0.078 | 0.104 | 0.068 | 0.072 |
| GTSRB | 0.241 | 0.189 | – | 0.122 | 0.073 | 0.073 | 0.071 | 0.071 |
| SVHN | 0.338 | 0.180 | 0.278 | – | 0.071 | 0.072 | 0.067 | 0.068 |
| DTD | 0.134 | 0.177 | 0.142 | 0.138 | – | 0.082 | 0.073 | 0.078 |
| RESISC | 0.140 | 0.272 | 0.152 | 0.142 | 0.207 | – | 0.071 | 0.084 |
| Block | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | emb |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retained | 25.5 | 28.7 | 28.0 | 28.7 | 28.7 | 29.2 | 29.5 | 30.1 | 30.4 | 30.8 | 30.3 | 27.2 | 44.2 |
| Dataset | Ours only | Baselines only | Ratio | |
|---|---|---|---|---|
| MNIST | 51 | 14 | 10,000 | 3.6 |
| EuroSAT | 354 | 12 | 2,700 | 29.5 |
| GTSRB | 2,340 | 12 | 12,630 | 195.0 |
| SVHN | 765 | 138 | 26,032 | 5.5 |
| DTD | 349 | 24 | 1,880 | 14.5 |
| RESISC45 | 702 | 79 | 6,300 | 8.9 |
| Dataset | #cls | Mean | SD | Min | Med | / | ||
|---|---|---|---|---|---|---|---|---|
| MNIST | 10 | 99.07 | 0.81 | 97.0 | 99.3 | 98.9/99.5 | 1.000 | 0.000 |
| EuroSAT | 10 | 98.44 | 1.86 | 94.4 | 99.3 | 97.8/100.0 | 1.000 | 0.000 |
| GTSRB | 43 | 94.61 | 6.42 | 64.4 | 96.0 | 93.3/99.0 | 0.907 | 0.000 |
| SVHN | 10 | 91.90 | 2.05 | 89.3 | 91.2 | 90.4/92.9 | 0.800 | 0.000 |
| RESISC45 | 45 | 93.10 | 5.62 | 68.5 | 93.5 | 91.1/97.5 | 0.844 | 0.000 |
| Cars | 196 | 84.11 | 13.30 | 17.2 | 87.2 | 79.3/92.9 | 0.378 | 0.026 |
| Dataset | Pair 1 | Pair 2 | Pair 3 |
|---|---|---|---|
| MNIST | (4,9) 17 | (7,2) 15 | (7,9) 9 |
| EuroSAT | (0,5) 8 | (1,2) 5 | (6,5) 5 |
| GTSRB | (16,10) 18 | (40,42) 18 | (14,38) 14 |
| SVHN | (1,4) 106 | (4,1) 92 | (2,1) 82 |
| RESISC45 | (13,22) 27 | (27,7) 13 | (23,11) 12 |
| Cars | (83,82) 20 | (70,63) 20 | (12,21) 19 |
| Dataset | Acc. (%) | 95% interval | |
|---|---|---|---|
| MNIST | 99.07 | 10,000 | |
| EuroSAT | 98.44 | 2,700 | |
| GTSRB | 96.26 | 12,630 | |
| SVHN | 92.16 | 26,032 | |
| DTD | 77.23 | 1,880 | |
| RESISC45 | 93.22 | 6,300 |
| vs. Task Arithmetic | vs. TIES-Merging | |||||||
|---|---|---|---|---|---|---|---|---|
| Task | ||||||||
| MNIST | 209 | 17 | 58 | 22 | ||||
| EuroSAT | 463 | 14 | 436 | 13 | ||||
| GTSRB | 3205 | 24 | 3106 | 24 | ||||
| SVHN | 2567 | 187 | 863 | 391 | ||||
| DTD | 494 | 30 | 396 | 34 | ||||
| Method | MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | Avg |
|---|---|---|---|---|---|---|---|---|---|
| JNSP | 98.91 | 92.52 | 90.36 | 87.04 | 68.03 | 85.89 | 72.21 | 68.40 | 82.92 |
| TIES | 97.17 | 69.74 | 71.96 | 81.93 | 51.65 | 71.06 | 60.50 | 58.46 | 70.31 |
| TA | 90.00 | 66.74 | 63.44 | 69.20 | 49.26 | 69.05 | 60.41 | 62.39 | 66.31 |
| DARE | 90.06 | 65.56 | 61.63 | 68.37 | 48.72 | 68.92 | 59.94 | 62.08 | 65.66 |
| Method | MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | Avg |
|---|---|---|---|---|---|---|---|---|---|
| JNSP | 99.52 | 97.96 | 97.49 | 95.23 | 78.62 | 92.67 | 89.39 | 76.09 | 90.87 |
| TIES | 98.88 | 87.37 | 82.67 | 88.98 | 61.22 | 83.32 | 81.21 | 70.16 | 81.73 |
| TA | 97.74 | 82.48 | 79.41 | 81.91 | 60.64 | 79.89 | 79.91 | 70.38 | 79.05 |
| DARE | 97.69 | 82.15 | 79.26 | 82.09 | 60.32 | 79.71 | 79.64 | 70.13 | 78.87 |
| Method | MNIST | EurS | GTS | SVHN | DTD | RES | Cars | SUN | Avg |
|---|---|---|---|---|---|---|---|---|---|
| WUDI | 99.44 | 98.00 | 96.55 | 95.30 | 73.62 | 90.13 | 81.58 | 73.13 | 88.47 |
| FDA | 99.38 | 98.26 | 93.43 | 94.31 | 69.84 | 87.83 | 80.56 | 71.80 | 86.93 |
| JNSP DARE | 98.8 | 97.4 | 94.2 | 91.0 | 73.5 | 91.7 | 77.7 | 71.7 | 86.99 |
| JNSP TIES | 89.5 | 86.7 | 60.3 | 73.3 | 59.3 | 84.7 | 78.3 | 72.3 | 75.55 |
| WUDI-init FDA | 99.40 | 98.70 | 96.32 | 94.58 | 74.26 | 91.11 | 82.69 | 74.11 | 88.90 |
| JNSP | 99.07 | 98.44 | 96.26 | 92.16 | 77.23 | 93.22 | 84.25 | 74.97 | 89.45 |
| Backbone | JNSP init | distill (600) | CE (600) | distill (1800) |
|---|---|---|---|---|
| ViT-B/16 | 89.45 | 91.17 | 90.85 | 91.40 |
| ViT-B/32 | 82.91 | 87.95 | 87.77 | — |
| ViT-L/14 | 90.87 | 92.62 | 92.57 | — |
| Method | Data | Own | TALL | Pub. | TALL Pub. |
|---|---|---|---|---|---|
| TSV-M | – | 87.22 | 87.25 | 89.01 | |
| AlphaMerging | – | 87.22 ∗ | 85.40 ∗ | 89.9 | |
| Iso-C | – | 87.15 | 87.46 | — | — |
| Iso-CTS | – | 86.59 | 86.62 | — | — |
| JNSP | – | 89.45 | 89.20 | — | — |
| Task | Joint | Direct | Global | CR ( ) |
|---|---|---|---|---|
| CoLA | 6.6 | 47.7 | 53.8 | |
| SST-2 | 51.5 | 50.1 | 91.9 | 93.6 |
| MRPC | 22.3 | 59.0 | 75.8 | 75.7 |
| STS-B | 84.0 | 78.1 | ||
| QQP | 49.3 | 80.4 | 21.7 | 31.5 |
| MNLI | 38.6 | 44.3 | 49.0 | 70.4 |
| Check | Result |
|---|---|
| Recomputed global merge vs. archived file | |
| same, worst tensor | (abs.) |
| Aggregate retention, joint (target ) | |
| Wall-clock, 53 tensors, CPU (target s) | s |
| Permutation of the task list, three orders | , , |
| worst tensor ( token_embedding ) |