Data Mixture Optimization

Follow this topic Ask about this topic

The field studies how the proportions, selection, and timing of training data from different domains or tasks affect models, especially in pretraining, fine-tuning, and multimodal training. Approaches range from scaling-law fits and designed proxy experiments to bilevel gradient methods, Bayesian search, and dynamic reweighting based on training losses or benchmark feedback. Researchers use these methods to improve target capabilities, balance domain coverage, and make better use of limited data and compute. A central open challenge is whether mixtures found with proxy models, datasets, or evaluation objectives transfer reliably to larger models and new settings, particularly when repetition and interactions between domains matter.

Latest papers

40 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
Mar 27, 2026Hao Liang, Zhengyang Zhao, Mingrui Chen +22Training Data Selection Data Mixture Optimization
Peking University · Institute for Advanced Algorithms Research, Shanghai
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
Mar 25, 2026Zhuo Chen, Yuxuan Miao, Supryadi +1Data Mixture Optimization Language Model Pretraining
The International Joint Institute of Tianjin University, Tianjin University, Tianjin 300072, China · TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China
mSFT: Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFT
Mar 23, 2026Woosung Koh, Jeyoung Jeon, Youngjin Song +4LLM Fine-Tuning Supervised Fine-Tuning
KAIST AI · Yonsei University · Samsung Advanced Institute of Technology
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
Mar 18, 2026Xuyang Cao, Qianying Liu, Chuan Xiao +7Language Model Scaling Laws Multilingual Language Models
NII LLMC · The University of Osaka · Nara Institute of Science and Technology +2
Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
Mar 17, 2026Ryo Kishino, Riku Shiomi, Hiroaki Yamagiwa +2Language Model Pretraining LLM Alignment
1Kyoto University · 2RIKEN
Towards Next-Generation LLM Training: From the Data-Centric Perspective
Mar 16, 2026Hao Liang, Zhengyang Zhao, Zhaoyang Han +8LLM Training Efficient Language Model Training
Peking University, Beijing, China · Hong Kong University of Science and Technology, Hong Kong SAR, China · Institute for Advanced Algorithms Research, Shanghai, China +2
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Mar 9, 2026Jingwei Li, Xinran Gu, Jingzhao ZhangData Mixture Optimization LLM Training
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute
Less is More: Convergence Benefits of Fewer Data Weight Updates over Longer Horizon
Feb 23, 2026Rudrajit Das, Neel Patel, Meisam Razaviyayn +1Data Mixture Optimization Bilevel Optimization
Google Research · EPFL · University of Southern California
Mixtures Closest to a Given Measure: A Semidefinite Programming Approach
Sep 26, 2025Srećko Đurašinović, Jean-Bernard Lasserre, Victor MagronSemidefinite Programming Data Mixture Optimization
College of Computing and Data Science, NTU, Singapore · LAAS-CNRS, Toulouse, France · Toulouse School of Economics +1
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Aug 16, 2025Haebin Shin, Lei Ji, Xiao Liu +4Multi-Armed Bandits Instruction Tuning
University of Michigan · Microsoft Research · KAIST AI +1

Mar 27, 2026J/K move · Enter open · S save

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

Hao Liang, Zhengyang Zhao, Mingrui Chen, Meiyi Qiang+21

Peking University · Institute for Advanced Algorithms Research, Shanghai

Training Data Selection Data Mixture Optimization LLM Training

Open paper AskPDF arXiv

Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.