Data Mixture Optimization

Ask about this topic

Overview

The field studies how the proportions, selection, and timing of training data from different domains or tasks affect models, especially in pretraining, fine-tuning, and multimodal training. Approaches range from scaling-law fits and designed proxy experiments to bilevel gradient methods, Bayesian search, and dynamic reweighting based on training losses or benchmark feedback. Researchers use these methods to improve target capabilities, balance domain coverage, and make better use of limited data and compute. A central open challenge is whether mixtures found with proxy models, datasets, or evaluation objectives transfer reliably to larger models and new settings, particularly when repetition and interactions between domains matter.

Latest papers

CardsList

40 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. LinMix: Data Mixing as Linear Domain Attribution

    Oct 27, 2026Hubert Baniecki, Przemyslaw Biecek, Martin PawelczykData Mixture Optimization

    University of Warsaw · Centre for Credible AI, Warsaw University of Technology · Faculty of Computer Science, University of Vienna

  2. Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

    Oct 15, 2026Yuanjian Xu, Jianing Hao, Tianze Sun +7LLM TrainingData Mixture Optimization

    The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology · OpenCSG +4

  3. CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives

    Oct 12, 2026Yuhan Wang, Rui Hu, Batian Di +5Semantic ClusteringInstruction-Tuning Data Selection

    Peking University · Renmin University of China · Zhejiang University +2

  4. Conditional Transfer from Controlled Pretraining Mixtures to Code

    Oct 8, 2026Ohad RubinSynthetic Data PretrainingData Mixture Optimization

  5. Differentially Private Mixing of Public Datasets Improves Private Learning

    Oct 5, 2026Yufei Chen, Tejumade Afonja, Anvith Thudi +1Privacy-Preserving MLDifferential Privacy

    University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

  6. When do data mixtures improve scaling laws? Insights from high-dimensional regression

    Sep 29, 2026Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1High-Dimensional Linear RegressionData Mixture Optimization

    Institute of Science and Technology Austria (ISTA). · Department of Applied and Computational Mathematics, Yale University. · Department of Statistics and Data Science, Yale University.

  7. CALIBUDGET: Calibration-Guided Source Allocation for Fixed-Budget Mixed-Reasoning Adaptation

    Sep 29, 2026Yupeng Chang, Yuan WuData Mixture OptimizationAdaptive Sampling

    School of Artificial Intelligence, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University

  8. Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

    Sep 8, 2026Yunpeng Xu, Kun ZhengLLM AlignmentDomain-Adaptive Pretraining

  9. LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering

    Aug 31, 2026Gopi Krishnan Rajbahadur, Amir M. Ebrahimi, Boyuan Chen +1Training Data CurationLLM Post-Training

    Queen’s University, Canada · Centre for Software Excellence, Huawei Technologies, Canada

  10. Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining

    Aug 24, 2026Yicheng Mao, Hongru DuData Mixture OptimizationLanguage Model Pretraining

    Department of Mathematics and Statistics, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada · Department of Systems and Information Engineering, University of Virginia, Charlottesville, VA, USA

  11. Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation

    Aug 16, 2026Ashima Sood, Bryan Gardiner, Joan CondellLLM Fine-TuningText Summarization

    School of Computing, Engineering and Intelligent Systems, Ulster University Londonderry, Northern Ireland, United Kingdom

  12. Small Models Scout Bottleneck Order for Large-Model Data Control

    Aug 14, 2026Seungmin Choi, Jiwon Sung, Muhammad Umer +4LLM TrainingEfficient Language Model Training

    Stanford University · Seoul National University

  13. Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting

    Jul 30, 2026Xiang Yuan, Kaiqing Lei, Zhenyu Jin +3Data Mixture OptimizationLLM Training

    School of Mathematics and Statistics, Xi’an Jiaotong University · Northwest University, Shaanxi, China

  14. DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

    Jul 27, 2026Jiahao Xie, Zhongbin Guo, Qianle Wang +4Training Data CurationData Mixture Optimization

    ByteDance · ByteDance Commercial AI Team

  15. DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning

    Jul 24, 2026He ZhangSupervised Fine-TuningLarge Language Model-Guided Optimization

    Tsinghua University

  16. Domain-Aware Scaling Laws Uncover Data Synergy

    Jul 13, 2026Kimia Hamidieh, Lester Mackey, David Alvarez-MelisLanguage Model PretrainingLanguage Model Scaling Laws

    MIT CSAIL · Microsoft Research · Harvard University

  17. HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

    Jul 2, 2026Ziyun Qiao, Yue Min, Ruining Chen +1Language Model PretrainingTraining Data Curation

    Wizard Quant, Beijing, China · University of Science and Technology of China, Hefei, China

  18. CausalMix: Data Mixture as Causal Inference for Language Model Training

    Jul 1, 2026Zinan Tang, Yukun Zhang, Shaomian Zheng +7Data Mixture OptimizationLLM Training

    Tsinghua University · Ant Group · Renmin University of China

  19. FastMix: Fast Data Mixture Optimization via Gradient Descent

    Jun 12, 2026Haoru Tan, Sitong Wu, Yanfeng Chen +5Language Model PretrainingData Mixture Optimization

    University of Hong Kong · 2Hunyuan LLM, Tencent · 3Chinese University of Hong Kong

  20. Explaining Data Mixing Scaling Laws

    Jun 6, 2026Rui Dai, Shuran ZhengData Mixture OptimizationNeural Scaling Laws

    Beijing Institute of Technology · IIIS, Tsinghua University

  21. TANDEM: Bi-Level Data Mixture Optimization with Twin Networks

    Jun 3, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Bilevel OptimizationData Mixture Optimization

    University of Oxford · Renmin University of China · University of Chinese Academy of Sciences

  22. Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them

    May 29, 2026Kevin Zhou, Lisa Alazraki, Kris Cao +1Language Model PretrainingData Mixture Optimization

    Imperial College London · Cohere

  23. Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training

    May 20, 2026Hongzhi Ruan, Pei Liu, Weiliang Ma +5Autonomous Driving DatasetsSynthetic Data Augmentation

    Li Auto · HKUST (GZ) · HKUST

  24. Scaling Laws for Mixture Pretraining Under Data Constraints

    May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Language Model PretrainingLanguage Model Scaling Laws

    Apple · ITU

  25. DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures

    May 11, 2026Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3Fine-TuningLLM Fine-Tuning

    Apple

  26. InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

    May 4, 2026Fengze Liu, Weidong Zhou, Binbin Liu +7Language Model PretrainingLanguage Model Scaling Laws

    1ByteDance · 2UC Santa Cruz

  27. GEM: Geometric Entropy Mixing for Optimal LLM Data Curation

    Apr 27, 2026Yue Min, Ziyun Qiao, Ruining Chen +1Language Model PretrainingTraining Data Curation

    Wizard Quant, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Peking University, Beijing, China +1

  28. Expectation Maximization (EM) Converges for General Agnostic Mixtures

    Apr 7, 2026Avishek GhoshExpectation-MaximizationData Mixture Optimization

    Department of Computer Science and Engineering, IIT Bombay, Mumbai, India

  29. MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtraining

    Apr 3, 2026Bingbing Wen, Sirajul Salekin, Feiyang Kang +4Data Mixture OptimizationMultimodal Large Language Models

    University of Washington · Work done during an internship at Apple · Apple +1

  30. OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

    Mar 30, 2026Haiyue Song, Masao UtiyamaContinued PretrainingData Mixture Optimization

    National Institute of Information and Communications Technology, Kyoto, Japan