Model Merging

Model merging studies how to combine weights or adapters from separately trained models into one model, aiming to reuse complementary skills without storing an ensemble or repeating joint training. Common approaches average checkpoints or add task vectors, while sparsification, low-rank and spectral methods, geometric alignment, and calibration-based routing aim to reduce interference. The methods are used for multi-task and continual learning, domain adaptation, and combining language, vision, speech, and specialized models; evaluations often measure task accuracy, capability retention, worst-task performance, and inference or storage cost. The central tension is that updates that help one task can erase or distort another: success depends on shared initialization, representation compatibility, update scale, and data or prompt distribution, and some task combinations may not be recoverable through weight-space fusion alone.