cs.LGMar 4, 2026

BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning

Authors: Yuhan Xie, Chen Lyu

Organizations: MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Model Merging (MM) has emerged as a scalable paradigm for multi-task learning (MTL), enabling multiple task-specific models to be integrated without revisiting the original training data. Despite recent progress, the reliability of MM under test-time distribution shift remains insufficiently understood. Most existing MM methods typically assume that test data are clean and distributionally aligned with both the training and auxiliary sources. However, this assumption rarely holds in practice, often resulting in biased predictions with degraded generalization. To address this issue, we present BD-Merging, a bias-aware unsupervised model merging framework that explicitly models uncertainty to achieve adaptive reliability under distribution shift. First, BD-Merging introduces a joint evidential head that learns uncertainty over a unified label space, capturing cross-task semantic dependencies in MM. Second, building upon this evidential foundation, we propose an Adjacency Discrepancy Score (ADS) that quantifies evidential alignment among neighboring samples. Third, guided by ADS, a discrepancy-aware contrastive learning mechanism refines the merged representation by aligning consistent samples and separating conflicting ones. Combined with general unsupervised learning, this process trains a debiased router that adaptively allocates task-specific or layer-specific weights on a per-sample basis, effectively mitigating the adverse effects of distribution shift. Extensive experiments across diverse tasks demonstrate that BD-Merging achieves superior effectiveness and robustness compared to state-of-the-art MM baselines.

Figures & tables

Figure 1: Averaged normalized accuracy under different distribution shift settings across multiple benchmarks.
Figure 2: The process of BD-Merging. BD-Merging includes three modules: (1) Modeling uncertainty using a Dirichlet-based evidential head; (2) Constructing an adjacency set and calculating the ADS to measure evidential alignment; and (3) Performing discrepancy-aware contrastive MM with a debiased router, guided by ADS. Stage (a)-(d) demonstrates the evolution of samples in the feature space through various stages.
MethodTypeLevelSUN397 [ 41 ]Cars [ 24 ]RESISC45 [ 7 ]EuroSAT [ 18 ]SVHN [ 29 ]GTSRB [ 35 ]MNIST [ 25 ]DTD [ 8 ]Avg Acc.
Task Arithmetic [ 20 ]Clean-55.3754.9866.1078.4080.2169.6597.3450.6469.09
CorruptedL1L_{1}53.07 ( ↓\downarrow 4.2)52.68 ( ↓\downarrow 4.2)63.90 ( ↓\downarrow 3.3)67.15 ( ↓\downarrow 14.3)71.33 ( ↓\downarrow 11.1)64.09 ( ↓\downarrow 8.0)92.04 ( ↓\downarrow 5.4)48.30 ( ↓\downarrow 4.6)64.07 ( ↓\downarrow 7.3)
L2L_{2}50.57 ( ↓\downarrow 8.7)49.24 ( ↓\downarrow 10.4)57.60 ( ↓\downarrow 12.9)63.35 ( ↓\downarrow 19.2)66.68 ( ↓\downarrow 16.9)59.09 ( ↓\downarrow 15.2)88.67 ( ↓\downarrow 8.9)47.23 ( ↓\downarrow 6.7)60.30 ( ↓\downarrow 12.7)
L3L_{3}47.35 ( ↓\downarrow 14.5)46.40 ( ↓\downarrow 15.6)53.50 ( ↓\downarrow 19.1)62.05 ( ↓\downarrow 20.9)64.81 ( ↓\downarrow 19.2)56.75 ( ↓\downarrow 18.5)85.71 ( ↓\downarrow 11.9)43.40 ( ↓\downarrow 14.3)57.50 ( ↓\downarrow 16.8)
Ties-Merging [ 43 ]Clean-64.5764.4274.6076.6581.2869.3896.5355.9672.92
CorruptedL1L_{1}62.75 ( ↓\downarrow 2.8)62.03 ( ↓\downarrow 3.7)72.00 ( ↓\downarrow 3.5)67.35 ( ↓\downarrow 12.1)72.71 ( ↓\downarrow 10.5)63.71 ( ↓\downarrow 8.2)91.37 ( ↓\downarrow 5.3)53.19 ( ↓\downarrow 4.9)68.14 ( ↓\downarrow 6.6)
Table 1: Performance comparison of MM methods under varying intensities of test-time bias. bold and Blue numbers indicate the best and second-best results, respectively.
Figure 3: Visualization of corrupted image instances.
MethodSeen TasksUnseen Tasks
EuroSAT [ 18 ]SVHN [ 29 ]GTSRB [ 35 ]MNIST [ 25 ]Avg Acc.SUN397 [ 41 ]Cars [ 24 ]RESISC45 [ 7 ]DTD [ 8 ]Avg Acc.
Pretrained [ 32 ]43.60 ±\pm 0.1232.62 ±\pm 0.4032.60 ±\pm 0.6748.26 ±\pm 0.5639.27 ±\pm 0.4462.46 ±\pm 0.2959.73 ±\pm 0.3059.60 ±\pm 0.2146.17 ±\pm 0.2356.99 ±\pm 0.26
Task Arithmetic [ 20 ]75.80 ±\pm 0.2886.69 ±\pm 0.1778.77 ±\pm 0.2297.80 ±\pm 0.5084.77 ±\pm 0.2956.41 ±\pm 0.2252.10 ±\pm 0.2250.60 ±\pm 0.4939.15 ±\pm 0.4449.57 ±\pm 0.44
Ties-Merging [ 43 ]70.85 ±\pm 0.2482.47 ±\pm 0.1470.43 ±\pm 0.4896.14 ±\pm 0.2179.97 ±\pm 0.2760.19 ±\pm 0.4257.07 ±\pm 0.1858.10 ±\pm 0.3342.77 ±\pm 0.3654.42 ±\pm 0.32
AdaMerging [ 47 ]89.85 ±\pm 0.5088.14 ±\pm 0.4887.31 ±\pm 0.4597.94 ±\pm 0.2590.79 ±\pm 0.3256.36 ±\pm 0.1153.46 ±\pm 0.1051.40 ±\pm 0.1738.09 ±\pm 0.3549.83 ±\pm 0.18
Twin-Merging [ 26 ]94.83 ±\pm 0.3187.10 ±\pm 0.2391.60 ±\pm 0.3998.72 ±\pm 0.4493.06 ±\pm 0.3459.72 ±\pm 0.4655.93 ±\pm 0.2955.63 ±\pm 0.4340.85 ±\pm 0.1553.03 ±\pm 0.33
Table 2: Performance of MM methods on seen vs. unseen tasks (4 each).
Figure 4: Evaluation of BD-Merging under test-time bias across various models.
Figure 5: Performance vs. time cost of MM methods.
Figure 6: Variation in debiased router’s average weights across unseen tasks.
ComponentCleanCorrupted
BD-Merging87.15 ±\pm 0.2178.78 ±\pm 0.37
w/o Sharp(⋅)\mathrm{Sharp}(\cdot)86.68 ±\pm 0.1277.94 ±\pm 0.15
w/o Div(⋅)\mathrm{Div}(\cdot)85.36 ±\pm 0.1376.28 ±\pm 0.24
w/o Conf(⋅)\mathrm{Conf}(\cdot)85.89 ±\pm 0.1276.95 ±\pm 0.25
w/o ADS84.48 ±\pm 0.2875.44 ±\pm 0.25
w/o Router78.31 ±\pm 0.3367.25 ±\pm 0.52
Table 3: Ablation study under clean and corrupted conditions.
More in this topic

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

Explore similar work

CardsList
Dec 1, 2025

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.

Kuangpu Guo, Aijing Yu, Jian Liang +4

University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3

Model Merging
AskPDF
May 13, 2026

ReForge: Refining Merged Models with Anchor-Regularized Regression

Model merging aims to combine multiple task-specific expert models into a single model without joint retraining, offering a practical alternative to multi-task learning when data access or computational budget is limited. Existing model merging methods rarely exploit strong merged models as priors for further improvement. To address this limitation, we propose ReForge, a bilevel optimization framework that formulates module-wise refinement as Bayesian linear regression with an anchor-centered prior. The inner level yields a closed-form MAP estimate from unlabeled calibration activations. The outer level uses Bayesian optimization to jointly select heterogeneous regularization strengths and assembly scales using held-out validation data. Furthermore, we develop a data-free variant of ReForge that replaces activation statistics with task-vector Grams, eliminating the need for calibration examples. Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, ReForge consistently outperforms all evaluated plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). On 20-task ViT-B/32, ReForge improves the strongest evaluated baseline, ISO-CTS, from 77.6% to 82.8% in the data-assisted setting and to 81.5% in the data-free setting. On eight-task ViT-L/14, the data-assisted variant achieves 95.1% mean accuracy, compared with 95.8% for the individual task experts. Our source code will be released soon.

Kaiyang Li, Shaobo Han, Qing Su +1

School of Computing, University of Connecticut, Storrs, CT 06269 · Optical Networking and Sensing, NEC Labs America, Princeton, NJ 08540

Model Merging
AskPDF
Mar 3, 2026

ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

Model merging aims to combine multiple task-specific expert models into a single model while preserving generalization across diverse tasks. However, interference among experts, especially when they are trained on different objectives, often leads to significant performance degradation. Despite recent progress, resolving this interference without data access, retraining, or architectural modification remains a fundamental challenge. This paper provides a theoretical analysis demonstrating that the input covariance of each task, which is a key factor for optimal merging, can be implicitly estimated from the parameter differences of its fine-tuned model, even in a fully data-free setting. Building on this insight, we introduce \acem, an Adaptive Covariance Estimation framework that effectively mitigates inter-task interference. Our approach features a principled, closed-form solution that contrasts with prior iterative or heuristic methods. Extensive experiments on both vision and language benchmarks demonstrate that \acem sets a new state-of-the-art among data-free methods. It consistently outperforms existing baselines; for example, \acem achieves an average absolute improvement of 4% over the previous methods across seven tasks on GPT-2. Owing to its efficient closed-form formulation, \acem delivers superior performance with a modest computational cost, providing a practical and theoretically grounded solution for model merging.

Bo Xu, Haotian Wu, Hehai Lin +4

The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China

Model MergingLanguage Model MergingCovariance Estimation
AskPDF