Data Mixture Optimization

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 23

All topics

Differentially Private Mixing of Public Datasets Improves Private Learning
Oct 5, 2026Yufei Chen, Tejumade Afonja, Anvith Thudi +1Privacy-Preserving ML Differential Privacy
University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany
Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training
Sep 8, 2026Yunpeng Xu, Kun ZhengLLM Alignment Domain-Adaptive Pretraining
LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering
Aug 31, 2026Gopi Krishnan Rajbahadur, Amir M. Ebrahimi, Boyuan Chen +1Training Data Curation LLM Post-Training
Queen’s University, Canada · Centre for Software Excellence, Huawei Technologies, Canada
Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation
Aug 16, 2026Ashima Sood, Bryan Gardiner, Joan CondellLLM Fine-Tuning Text Summarization
School of Computing, Engineering and Intelligent Systems, Ulster University Londonderry, Northern Ireland, United Kingdom
Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting
Jul 30, 2026Xiang Yuan, Kaiqing Lei, Zhenyu Jin +3Data Mixture Optimization LLM Training
School of Mathematics and Statistics, Xi’an Jiaotong University · Northwest University, Shaanxi, China
DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes
Jul 27, 2026Jiahao Xie, Zhongbin Guo, Qianle Wang +4Training Data Curation Data Mixture Optimization
ByteDance · ByteDance Commercial AI Team
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
Jul 24, 2026He ZhangSupervised Fine-Tuning Large Language Model-Guided Optimization
Tsinghua University
Domain-Aware Scaling Laws Uncover Data Synergy
Jul 13, 2026Kimia Hamidieh, Lester Mackey, David Alvarez-MelisLanguage Model Pretraining Language Model Scaling Laws
MIT CSAIL · Microsoft Research · Harvard University
HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
Jul 2, 2026Ziyun Qiao, Yue Min, Ruining Chen +1Language Model Pretraining Training Data Curation
Wizard Quant, Beijing, China · University of Science and Technology of China, Hefei, China
CausalMix: Data Mixture as Causal Inference for Language Model Training
Jul 1, 2026Zinan Tang, Yukun Zhang, Shaomian Zheng +6Data Mixture Optimization LLM Training
Tsinghua University · Ant Group · Renmin University of China
FastMix: Fast Data Mixture Optimization via Gradient Descent
Jun 12, 2026Haoru Tan, Sitong Wu, Yanfeng Chen +5Language Model Pretraining Data Mixture Optimization
University of Hong Kong · 2Hunyuan LLM, Tencent · 3Chinese University of Hong Kong
Explaining Data Mixing Scaling Laws
Jun 6, 2026Rui Dai, Shuran ZhengData Mixture Optimization Neural Scaling Laws
Beijing Institute of Technology · IIIS, Tsinghua University
TANDEM: Bi-Level Data Mixture Optimization with Twin Networks
Jun 3, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Bilevel Optimization Data Mixture Optimization
University of Oxford · Renmin University of China · University of Chinese Academy of Sciences
Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
May 29, 2026Kevin Zhou, Lisa Alazraki, Kris Cao +1Language Model Pretraining Data Mixture Optimization
Imperial College London · Cohere
Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training
May 20, 2026Hongzhi Ruan, Pei Liu, Weiliang Ma +5Autonomous Driving Datasets Synthetic Data Augmentation
Li Auto · HKUST (GZ) · HKUST
Scaling Laws for Mixture Pretraining Under Data Constraints
May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Language Model Pretraining Language Model Scaling Laws
Apple · ITU
DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
May 11, 2026Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3Fine-Tuning LLM Fine-Tuning
Apple
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
May 4, 2026Fengze Liu, Weidong Zhou, Binbin Liu +7Language Model Pretraining Language Model Scaling Laws
1ByteDance · 2UC Santa Cruz
GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
Apr 27, 2026Yue Min, Ziyun Qiao, Ruining Chen +1Language Model Pretraining Training Data Curation
Wizard Quant, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Peking University, Beijing, China +1
DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
Mar 27, 2026Hao Liang, Zhengyang Zhao, Mingrui Chen +22Training Data Selection Data Mixture Optimization
Peking University · Institute for Advanced Algorithms Research, Shanghai
Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
Mar 17, 2026Ryo Kishino, Riku Shiomi, Hiroaki Yamagiwa +2Language Model Pretraining LLM Alignment
1Kyoto University · 2RIKEN
Mixtures Closest to a Given Measure: A Semidefinite Programming Approach
Sep 26, 2025Srećko Đurašinović, Jean-Bernard Lasserre, Victor MagronSemidefinite Programming Data Mixture Optimization
College of Computing and Data Science, NTU, Singapore · LAAS-CNRS, Toulouse, France · Toulouse School of Economics +1
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Aug 16, 2025Haebin Shin, Lei Ji, Xiao Liu +4Multi-Armed Bandits Instruction Tuning
University of Michigan · Microsoft Research · KAIST AI +1

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Differentially Private Mixing of Public Datasets Improves Private Learning

Yufei Chen, Tejumade Afonja, Anvith Thudi, Nicolas Papernot

University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

Privacy-Preserving ML Differential Privacy Data Mixture Optimization

Open paper AskPDF arXiv

Many machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility. However, the success of this depends on how relevant the selected public dataset is to the sensitive data. We introduce the first pipeline that privately learns the mixture of several public datasets to pretrain on for a given sensitive downstream task. Our key insight is that we can privately find the best mixture of multiple public datasets by privately learning a low-dimensional linear model. We tested our method on the NIH dataset for X-ray classification and the ENRON email dataset for language modeling. Applying our method to find tailored mixtures of X-ray datasets to pretrain on for diseases in the NIH ChestX-ray14 dataset, we improved macro AUC by up to 0.037 across privacy budgets compared to the baselines, with gains as large as +22.8% relative AUC on Cardiomegaly at

ε=1

. For DP training on the ENRON dataset, pre-training on our mixture of The Common Pile (a collection of public-domain text datasets) decreased test perplexity by 16% relative to the baseline mixtures.

Momentum

Related topics

Latest papers 23