Task Vector Merging

Task vector merging studies how to combine the weight changes made by separately fine-tuning a shared pretrained model, so one model can perform multiple tasks without pooling their training data. Common approaches include weighted averaging and sign-based methods such as TIES, sparsification, low-rank and SVD-based merging, and subspace alignment or input-dependent routing. Work spans vision and language benchmarks as well as continual learning, reinforcement learning, and multimodal models, with methods aiming to preserve useful updates while reducing interference. A central open tension is that task-vector geometry, update magnitudes, and task-specific structure can each affect outcomes, so a method that works across tasks and model settings—and reliably predicts harmful merges—remains elusive.