Rethinking Layer-wise Model Merging through Chain of Merges
Organizations: AImageLab, Department of Engineering, University of Modena and Reggio Emilia, Italy.
Abstract
Model merging has emerged as a simple yet effective method to fuse models without re-training. Existing techniques operate at the level of individual layers, thereby overlooking the inter-layer dependencies inherent in deep networks. We show that this simplification induces distributional mismatches in intermediate activations during merging, as changes applied to early layers fail to propagate to downstream ones. We identify these mismatches as a form of covariate shift that compounds across the network. To address this issue, we propose Chain of Merges (CoM), a novel procedure that sequentially merges weights across layers while accounting for inter-layer interactions. In particular, CoM mitigates covariate shift through a series of regression problems, where input activations are recomputed at each step to reflect the updated representations. Additionally, we introduce a dynamic weighting mechanism that prioritizes task-layer pairs most susceptible to performance degradation, improving robustness during merging. Experiments on standard benchmarks demonstrate that CoM achieves state-of-the-art performance across heterogeneous tasks.
Figures & tables
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Components | Architecture | ||||
| MCS | Norm | FC | ViT - B/32 | ViT - L/14 | Llama - 3 - 8B |
| ✗ | ✗ | ✗ | 72.77 | 80.69 | 97.31 |
| ✗ | ✓ | ✗ | 74.32 | 83.05 | 97.52 |
| ✓ | ✗ | ✗ | 83.45 | 84.26 | 99.71 |
| ✗ | ✗ | ✓ | 82.79 | 85.66 | 95.02 |
| ✗ | ✓ | ✓ | 84.02 | 88.27 | 97.52 |
| Calibration set | ViT-B/32 | ViT-L/14 | Llama-3-8B |
| Class-balanced, validation split (default, vision) | 92.74 0.14 | 91.35 0.02 | 99.57 0.15 |
| Random, validation split | 92.60 0.06 | 91.31 0.03 | 99.88 0.12 |
| Class-balanced, training split (default, Llama) | 92.53 0.05 | 91.30 0.03 | 99.76 0.12 |
| Random, training split | 92.39 0.06 | 91.25 0.01 | 99.69 0.10 |
| Noise | 90.64 0.13 | 85.52 0.14 | 99.72 0.11 |
| Noise | 82.05 0.13 | 81.61 0.23 | 99.70 0.12 |
| # of samples | 5 | 10 | 50 | 100 | 200 | 300 | 400 | 500 |
| ViT-B/32 | 76.94 | 81.95 | 88.74 | 91.02 | 92.24 | 92.57 | 92.67 | 92.74 |
| ViT-L/14 | 86.12 | 88.43 | 90.85 | 91.24 | 91.34 | 91.32 | 91.36 | 91.35 |
| Llama-3-8B | 97.17 | 97.59 | 98.97 | 99.43 | 99.71 | 99.48 | 99.66 | 99.76 |
| Arch. | TA | TIES | DARE TIES | Iso-C | TSV | Core ISO | RegMean | CoM |
| ViT-B/32 | 62.91 | 62.27 | 62.21 | 64.73 | 63.63 | 66.64 | 62.95 | 74.01 |
| ViT-L/14 | 72.14 | 73.25 | 71.97 | 77.52 | 73.18 | 77.53 | 75.24 | 81.59 |
| Merged | + Fine-tuning | |||||||
| Method | ||||||||
| ViT-B/32 | ||||||||
| Zero-shot | 60.6 | 60.6 | 60.6 | 60.6 | 63.8 | 73.4 | 78.4 | 92.5 |
| Task Arithmetic | 63.6 | 63.2 | 63.9 | 63.9 | 66.9 | 75.3 | 80.1 | 93.6 |
| TIES | 63.6 | 63.9 | 63.8 | 64.0 | 67.3 | 75.6 | 80.3 | 93.4 |
| DARE-TIES | 64.1 | 63.9 | 64.1 | 64.1 | 67.8 | 75.8 | 80.4 | 93.6 |
| ViT-B/32 | Cars | DTD | ESAT | GTSRB | MNIST | RESISC | SUN | SVHN | Average |
| TA | 81.97 | 73.99 | 49.01 | 41.93 | 53.45 | 71.57 | 97.47 | 42.08 | 63.93 |
| TIES | 81.57 | 72.90 | 49.20 | 37.87 | 57.40 | 70.05 | 96.64 | 46.87 | 64.06 |
| DARE TIES | 82.70 | 72.81 | 49.49 | 37.68 | 57.73 | 69.87 | 96.87 | 45.92 | 64.13 |
| Consensus TA | 82.72 | 73.08 | 48.75 | 40.79 | 60.51 | 71.46 | 96.95 | 47.78 | 65.26 |
| LiNeS | 81.59 | 74.91 | 50.84 | 39.88 | 52.81 | 71.07 | 97.99 | 39.27 | 63.54 |
| FisherAVG | 84.91 | 73.99 | 81.74 | 53.19 | 51.15 | 69.58 | 96.42 | 34.05 | 68.13 |
| ViT-L/14 | Cars | DTD | ESAT | GTSRB | MNIST | RESISC | SUN | SVHN | Average |
| TA | 80.07 | 79.73 | 65.51 | 60.01 | 82.27 | 79.52 | 86.68 | 64.95 | 74.84 |
| TIES | 79.54 | 78.21 | 64.27 | 61.18 | 84.16 | 79.48 | 87.36 | 70.19 | 75.55 |
| DARE TIES | 79.95 | 78.89 | 63.22 | 60.11 | 81.66 | 79.91 | 87.50 | 68.77 | 75.00 |
| Consensus TA | 81.88 | 81.32 | 68.60 | 64.55 | 85.04 | 81.56 | 86.90 | 63.72 | 76.69 |
| LiNeS | 79.70 | 79.73 | 66.04 | 57.88 | 81.86 | 78.24 | 86.48 | 64.10 | 74.25 |
| FisherAVG | 82.60 | 81.09 | 66.68 | 87.66 | 75.32 | 79.70 | 85.44 | 61.41 | 77.49 |
| Method | SNLI | MNLI | SICK | QNLI | RTE | SCITAIL | Average |
| TA | 93.66 | 95.11 | 88.14 | 68.66 | 99.19 | 96.83 | 90.27 |
| TIES | 95.07 | 96.32 | 83.46 | 70.70 | 100.00 | 96.35 | 90.32 |
| DARE TIES | 94.84 | 96.40 | 81.62 | 71.66 | 99.19 | 96.07 | 89.96 |
| Consensus TA | 94.26 | 94.63 | 88.78 | 66.43 | 101.61 | 96.93 | 90.44 |
| LiNeS | 92.87 | 95.35 | 89.14 | 73.24 | 97.58 | 96.64 | 90.80 |
| RegMean | 99.91 | 98.65 | 82.02 | 98.55 | 94.35 | 95.03 | 94.75 |
| Method | Abs. Acc. | Norm. Acc. |
| Zero-shot | 45.47 | 50.86 |
| Individual FT | 89.56 | 100.00 |
| Task Arithmetic | 77.64 | 86.85 |
| TIES | 77.31 | 86.39 |
| DARE-TIES | 77.13 | 86.29 |
| Consensus | 77.96 | 87.24 |
| Method | SNLI | MNLI | SICK | QNLI | RTE | SciTail | Avg |
| Zero-shot | 35.32 | 36.18 | 60.42 | 51.73 | 59.09 | 62.41 | 50.86 |
| Task Arithmetic | 94.72 | 94.27 | 79.70 | 69.54 | 92.73 | 90.15 | 86.85 |
| TIES | 98.62 | 99.02 | 68.05 | 75.02 | 89.09 | 88.51 | 86.39 |
| DARE-TIES | 96.17 | 95.83 | 74.60 | 68.09 | 93.03 | 90.04 | 86.29 |
| Consensus | 93.07 | 92.02 | 85.38 | 68.90 | 93.64 | 90.45 | 87.24 |
| LiNeS | 95.53 | 95.51 | 76.87 | 69.83 | 92.73 | 90.40 | 86.81 |
Model Merging
Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.
See all 214 papers