Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
Organizations: University of Trento · Fondazione Bruno Kessler
Abstract
Selecting the best data mixture is critical for successful Supervised Fine-Tuning (SFT) of Multimodal Large Language Models. However, determining the optimal mixture weights across multiple domain-specific datasets remains a significant bottleneck due to the combinatorial search space and the high cost associated with even a single training run. This is the so-called Data Mixture Optimization (DMO) problem. On the other hand, model merging unifies domain-specific experts through parameter interpolation. This strategy is efficient, as it requires only a single training run per domain, yet it often leads to suboptimal models. In this work, we take the best of both worlds, studying model merging as an efficient strategy for estimating the performance of different data mixtures. We train domain-specific multimodal experts and evaluate their weighted parameter-space combinations to estimate the efficacy of corresponding data mixtures. We conduct extensive experiments on 14 multimodal benchmarks, and empirically demonstrate that the merged proxy models consistently select near-optimal mixtures. This decouples the search for optimal mixtures from the resource-intensive training process, thereby providing a scalable and efficient strategy for navigating the complex landscape of mixture weights. Code is publicly available at https://github.com/BerasiDavide/mLLMs_merging_4_DMO.
Figures & tables
| General Multimodal Understanding ( General ) |
| ALLaVA-Instruct (LAION) Chen et al. (2024a) , VQAv2 Goyal et al. (2017) , lnQA Pont-Tuset et al. (2020) , LVIS-Instruct-4v Wang et al. (2023) , Q-Align Wu et al. (2024) , GQA Hudson & Manning (2019) , VizWiz Gurari et al. (2018) , Visual7W Zhu et al. (2016) |
| Optical Character Recognition ( OCR ) |
| SynthDog Modified Kim et al. (2022) , OCR-VQA Mishra et al. (2019) , DocVQA Mathew et al. (2021) , TextVQA Singh et al. (2019) , TextCaps Sidorov et al. (2020) , LLaVAr Zhang et al. (2023) , ST-VQA Biten et al. (2019) , Rendered-Text Wendler (2023) , InfoVQA Mathew et al. (2022) |
| Visual Perception & Counting ( Vis. Perc. ) |
| Clevr Johnson et al. (2017) , TallyQA Acharya et al. (2019) |
| Charts Understanding ( Charts ) |
| domains | domains | domains | ||||||||||
| Target Benchmarks | Uniform | Median | Selected | Best | Uniform | Median | Selected | Best | Uniform | Median | Selected | Best |
| Qwen2-VL-2B | ||||||||||||
| GQA | 57.8 | 57.8 | 58.8 | 57.8 | 57.5 | 58.7 | 56.9 | 57.4 | 58.4 | |||
| OK-VQA | 40.7 | 40.1 | 41.6 | 42.5 | 40.3 | 42.2 | 35.8 | 34.5 | 40.7 | |||
| VQAv2 | 71.4 | 71.4 | 72.2 | 71.8 | 71.0 | 72.1 | 70.5 | 70.4 | 71.6 | |||
| VizWiz | 61.0 | 61.0 | 61.8 | 62.9 | 61.1 | 63.1 | 62.0 | 61.0 | 63.0 | |||
| Target Benchmarks | ||||||||||||||||
| Model | Metric | GQA | OK-VQA | VQAv2 | VizWiz | DocVQA | InfoVQA | TextVQA | OCRBench | CVB-Count | Pope | ChartQA | MME | VMCBench | MMStar | Average Performance |
| Qwen2-VL-7B | Uniform | |||||||||||||||
| Median | ||||||||||||||||
| Selected | ||||||||||||||||
| Best | ||||||||||||||||
| Intern3.5-VL-8B | Uniform | |||||||||||||||
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| samples | samples | samples | ||||||||||
| Target Benchmarks | Uniform | Median | Selected | Best | Uniform | Median | Selected | Best | Uniform | Median | Selected | Best |
| Qwen2-VL-2B | ||||||||||||
| GQA | 53.0 | 53.1 | 53.8 | 54.1 | 55.7 | 55.6 | 56.9 | 56.9 | 56.9 | 57.5 | 58.4 | 58.4 |
| OK-VQA | 33.0 | 31.0 | 35.7 | 35.7 | 33.7 | 27.1 | 38.7 | 38.7 | 35.8 | 34.6 | 39.0 | 40.7 |
| VQAv2 | 68.6 | 68.2 | 69.2 | 69.9 | 69.7 | 69.5 | 69.4 | 70.7 | 70.5 | 70.5 | 70.9 | 71.6 |
| VizWiz | 49.9 | 51.6 | 55.3 | 55.3 | 59.0 | 58.8 | 57.6 | 62.6 | 62.0 | 61.0 | 58.5 | 63.0 |
| Target Benchmarks | ||||||||||||||||
| Model | Metric | GQA | OK-VQA | VQAv2 | VizWiz | DocVQA | InfoVQA | TextVQA | OCRBench | CVB-Count | Pope | ChartQA | MME | VMCBench | MMStar | Average Performance |
| Qwen2-VL-2B Full Fine-tuning | Uniform | |||||||||||||||
| Median | ||||||||||||||||
| Selected | ||||||||||||||||
| Best | ||||||||||||||||
| Intern3.5-VL-2B Full Fine-tuning | Uniform | |||||||||||||||
| Target Benchmarks | ||||||||||||||||
| Model | Metric | GQA | OK-VQA | VQAv2 | VizWiz | DocVQA | InfoVQA | TextVQA | OCRBench | CVB-Count | Pope | ChartQA | MME | VMCBench | MMStar | Average Performance |
| Qwen2-VL-2B | Uniform | |||||||||||||||
| Median | ||||||||||||||||
| Selected | ||||||||||||||||
| Selected (reg.) | ||||||||||||||||
| Best | ||||||||||||||||
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,134 papers