Selecting the best data mixture is critical for successful Supervised Fine-Tuning (SFT) of Multimodal Large Language Models. However, determining the optimal mixture weights across multiple domain-specific datasets remains a significant bottleneck due to the combinatorial search space and the high cost associated with even a single training run. This is the so-called Data Mixture Optimization (DMO) problem. On the other hand, model merging unifies domain-specific experts through parameter interpolation. This strategy is efficient, as it requires only a single training run per domain, yet it often leads to suboptimal models. In this work, we take the best of both worlds, studying model merging as an efficient strategy for estimating the performance of different data mixtures. We train domain-specific multimodal experts and evaluate their weighted parameter-space combinations to estimate the efficacy of corresponding data mixtures. We conduct extensive experiments on 14 multimodal benchmarks, and empirically demonstrate that the merged proxy models consistently select near-optimal mixtures. This decouples the search for optimal mixtures from the resource-intensive training process, thereby providing a scalable and efficient strategy for navigating the complex landscape of mixture weights. Code is publicly available at https://github.com/BerasiDavide/mLLMs_merging_4_DMO.
Figures & tables
General Multimodal Understanding ( General )
ALLaVA-Instruct (LAION) Chen et al. (2024a) , VQAv2 Goyal et al. (2017) , lnQA Pont-Tuset et al. (2020) , LVIS-Instruct-4v Wang et al. (2023) , Q-Align Wu et al. (2024) , GQA Hudson & Manning (2019) , VizWiz Gurari et al. (2018) , Visual7W Zhu et al. (2016)
Optical Character Recognition ( OCR )
SynthDog Modified Kim et al. (2022) , OCR-VQA Mishra et al. (2019) , DocVQA Mathew et al. (2021) , TextVQA Singh et al. (2019) , TextCaps Sidorov et al. (2020) , LLaVAr Zhang et al. (2023) , ST-VQA Biten et al. (2019) , Rendered-Text Wendler (2023) , InfoVQA Mathew et al. (2022)
Visual Perception & Counting ( Vis. Perc. )
Clevr Johnson et al. (2017) , TallyQA Acharya et al. (2019)
Charts Understanding ( Charts )
Table 1: SFT data collection comprising 23 datasets belonging to 4 categories: General Multimodal Understanding , Optical Character Recognition , Visual Perception & Counting , and Charts Understanding .
Figure 1: Correlation plots between downstream accuracies of mixture-trained models and our proposed merged proxy. Results are shown for Qwen2-VL-2B and Intern3.5-VL-2B models, fine-tuned on 2,3,4 -domains data mixtures. Each plot reports the Spearman’s rank correlation coefficient (R) of the average performance.
K=2 domains
K=3 domains
K=4 domains
Target Benchmarks
Uniform
Median
Selected
Best
Uniform
Median
Selected
Best
Uniform
Median
Selected
Best
Qwen2-VL-2B
GQA
57.8
57.8
58.80.0
58.8
57.8
57.5
58.70.0
58.7
56.9
57.4
58.40.0
58.4
OK-VQA
40.7
40.1
38.20.8
41.6
42.5
40.3
38.40.6
42.2
35.8
34.5
38.52.0
40.7
VQAv2
71.4
71.4
72.10.3
72.2
71.8
71.0
71.60.4
72.1
70.5
70.4
71.00.2
71.6
VizWiz
61.0
61.0
60.41.1
61.8
62.9
61.1
62.50.0
63.1
62.0
61.0
58.50.0
63.0
Table 2: Downstream accuracies of mixture-trained models. We compare the performance of the mixture selected by the merged proxy ( Selected ) against the Uniform mixture, Median performance among candidate mixtures, and the Best mixtures (the upper bound obtained via grid search). The Average Performance row corresponds to generalist scenarios, where the goal is to select a mixture that performs well overall. Other rows correspond to specialist cases, where the mixture is optimized for a downstream task. For Selected , we report the average accuracy and the standard deviation (subscript) over three runs.
Target Benchmarks
Model
Metric
GQA
OK-VQA
VQAv2
VizWiz
DocVQA
InfoVQA
TextVQA
OCRBench
CVB-Count
Pope
ChartQA
MME
VMCBench
MMStar
Average Performance
Qwen2-VL-7B
Uniform
60.9
51.2
76.5
63.8
93.3
72.1
81.6
79.8
61.7
88.0
81.3
79.8
78.3
52.0
72.9
Median
61.0
46.9
77.2
67.2
93.1
72.0
81.0
79.6
60.6
88.1
81.3
79.2
77.8
52.3
72.6
Selected
62.0
46.6
77.8
68.5
93.6
72.2
81.9
80.0
58.6
88.2
81.9
79.7
77.5
52.3
73.2
Best
62.0
51.9
78.0
69.5
93.6
73.2
82.5
81.0
64.0
88.6
82.0
80.5
78.5
53.7
73.2
Intern3.5-VL-8B
Uniform
60.5
36.5
76.2
64.5
91.6
71.3
78.7
78.5
67.0
89.2
84.7
81.8
79.0
58.8
72.7
Table 3: Downstream accuracy of different mixtures for larger models (Qwen2-VL-7B and Intern3.5-VL-8B). Naming follows table 2 .
Figure 2: Spearman rank correlation between regressor-predicted and ground-truth accuracies, as a function of the number of training runs used to fit the regressor. The merged proxy requires only the K domain-experts.
Figure 3: Loss functions in the neighbourhood of expert models, along 5 random directions. In a neighbourhood of their minimum, the loss functions remain convex.
Figure 4: Projections of mixture-finetuned checkpoints lie approximately along the line connecting the two experts, suggesting they can be approximated by linearly merging the experts’ parameters.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Composition of the domain-specific datasets used in our experiments.
Figure 6: Comparison of optimal data mixtures and data mixtures selected by the merged proxy.
Figure 7: Cross-data budget correlation plots. Results are shown for Qwen2-VL-2B and Intern3.5-VL-2B models for 4-domains mixtures.
N=10k samples
N=50k samples
N=100k samples
Target Benchmarks
Uniform
Median
Selected
Best
Uniform
Median
Selected
Best
Uniform
Median
Selected
Best
Qwen2-VL-2B
GQA
53.0
53.1
53.8
54.1
55.7
55.6
56.9
56.9
56.9
57.5
58.4
58.4
OK-VQA
33.0
31.0
35.7
35.7
33.7
27.1
38.7
38.7
35.8
34.6
39.0
40.7
VQAv2
68.6
68.2
69.2
69.9
69.7
69.5
69.4
70.7
70.5
70.5
70.9
71.6
VizWiz
49.9
51.6
55.3
55.3
59.0
58.8
57.6
62.6
62.0
61.0
58.5
63.0
Appendix
Table 4: Downstream accuracies of models fine-tuned on 4-domains data mixtures. We compare the performance of the mixture selected by our merged proxy ( Selected ) against the Uniform , Median and Best mixtures on an increasing number of data samples allocated to SFT ( N= 10k, 50k, 100k samples). Naming and formatting follow table 2 .
Target Benchmarks
Model
Metric
GQA
OK-VQA
VQAv2
VizWiz
DocVQA
InfoVQA
TextVQA
OCRBench
CVB-Count
Pope
ChartQA
MME
VMCBench
MMStar
Average Performance
Qwen2-VL-2B Full Fine-tuning
Uniform
60.39
42.14
69.78
52.40
87.22
56.65
76.11
73.10
54.19
86.74
73.00
64.19
70.20
42.16
64.88
Median
61.18
42.60
69.53
53.28
85.94
55.43
75.34
71.85
58.25
87.02
73.14
66.21
70.60
42.84
65.09
Selected
61.86
42.07
71.10
55.97
87.26
57.70
77.73
72.90
54.57
86.73
74.88
69.23
70.40
40.59
65.61
Best
62.41
46.02
71.62
59.22
88.47
58.83
78.82
73.80
66.50
88.46
74.88
69.87
72.70
46.37
66.67
Intern3.5-VL-2B Full Fine-tuning
Uniform
58.18
41.37
72.24
40.39
83.19
54.39
71.33
71.80
60.53
88.99
74.64
64.51
70.30
48.26
64.29
Appendix
Table 5: Downstream accuracy of different 4-domains mixtures for Qwen2-VL-2B and Intern3.5-VL-2B trained with full fine-tuning .
Figure 8: Spearman’s R correlation coefficient of accuracies predicted by a linear ridge regressor (left) and a LightGBM regressor (right), fitted on an increasing number of data points. Each data point comes from a finetuned model, while the merged proxy requires only the K expert models.
Figure 9: Loss functions in the neighbourhood of expert models, along 5 random directions. In a neighbourhood of their minimum, the loss functions remain convex.
Figure 10: Projections of mixture-finetuned checkpoints are aligned between the two experts, suggesting they can be approximated by linearly merging the experts’ parameters.
Target Benchmarks
Model
Metric
GQA
OK-VQA
VQAv2
VizWiz
DocVQA
InfoVQA
TextVQA
OCRBench
CVB-Count
Pope
ChartQA
MME
VMCBench
MMStar
Average Performance
Qwen2-VL-2B
Uniform
56.9
35.8
70.5
62.0
85.1
54.1
76.6
75.2
57.9
88.3
70.3
67.7
70.9
41.4
65.2
Median
57.4
34.5
70.4
61.0
84.1
54.6
75.8
74.8
56.8
88.2
70.1
66.9
71.5
41.2
64.8
Selected
58.4
39.0
70.9
58.5
86.6
56.1
76.4
75.4
56.5
88.5
71.6
66.9
70.8
41.7
65.3
Selected (reg.)
58.9
43.3
71.3
57.1
86.6
56.1
78.6
75.8
62.7
87.9
69.6
67.0
71.8
43.0
65.3
Best
58.4
40.7
71.6
63.0
86.6
58.4
77.5
76.0
61.3
88.6
71.7
67.8
72.6
42.7
65.5
Appendix
Table 6: Performance of mixtures selected by a regressor fitted on merged models performance. Naming follows table 2 .