Training Data Selection

Follow this topic Ask about this topic

Training data selection asks which examples, tokens, or data sources best help a model learn a target task, generalize, or retain existing capabilities. Methods include quality and difficulty filtering, diversity-aware coresets, gradient or influence scoring, active learning, and dynamic curricula or data mixtures. Researchers test these approaches in settings such as language-model pretraining and instruction tuning, vision, speech, and scientific learning, measuring both downstream performance and the cost of selecting and training on data. A central tension is that carefully chosen subsets can save resources or improve results, but selection may be expensive, depend on the model and data budget, or harm coverage and future learning; in some settings, random or full-data training remains competitive.

Latest papers

187 papers27 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation
Oct 26, 2026Yingjie Bai, Weiming ZhiData Selection Training Data Selection
School of Computer Science, The University of Sydney
Exact Unlearning After Data Curation
Oct 24, 2026Priyank Jayraj, Devanshu sahoo, Uday MittalMachine Unlearning Training Data Curation
Birla Institute of Technology and Science, Pilani Pilani, Rajasthan, India
Towards Faithful Data Selection: Statistical Limits and Scalable Algorithms
Oct 21, 2026Ahmed Mehdi Inane, Mahdi Haghifam, Gintare Karolina Dziugaite +1Training Data Selection Statistical Learning Theory
Université de Montréal, Mila · Toyota Technological Institute at Chicago · Google DeepMind +1
ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy
Oct 19, 2026Yuqiao Wen, Lili MouTraining Data Selection Data Selection
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair
Hidden Cartographer: Hidden State Guided Data Selection for Dual Budgeted Instruction Tuning
Oct 18, 2026Zixuan Qin, Qi Qin, Erbo Li +3Training Data Selection Instruction-Tuning Data Selection
Center for Applied Statistics, School of Statistics, Renmin University of China · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing
Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension
Oct 17, 2026Junlong Jia, Jiang Zhou, Xing Wu +3Training Data Selection Long-Context Language Modeling
Hunyuan Team, Tencent
Not All Unlabeled Data Are Equal: Unlabeled Data Selection for Semi-Supervised QSAR Modeling with Large Chemical Repositories
Oct 15, 2026Oliver Majer, Saso Dzeroski, Jurica LevaticTraining Data Selection Semi-Supervised Learning
Jožef Stefan Institute, Jamova cesta 39, Ljubljana, Slovenia · University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, Slovenia
Efficient Pairwise Difference Learning: A Comparative Study of Encoding Strategies and Training Pair Selection
Oct 15, 2026Bernard Georges, Mohamed Karim Belaid, Manuel RodriguesData-Efficient Learning Training Data Selection
University of Minho, ALGORITMI CENTER LASI, Braga, Portugal · IDIADA Fahrzeugtechnik GmbH, Munich, Germany
Mind the Tail: Data Selection Through Sparse Concepts
Oct 14, 2026Ziheng Zhang, Neha Hulkund, Colin Lee +7Training Data Selection Data Selection
University of Texas at Austin · The Ohio State University · Massachusetts Institute of Technology +2
EigDiff: Eigensample Diffusion - or how to significantly reduce training dataset size with marginal performance variation?
Oct 14, 2026Andrei Jarca, Vlad Hondru, Florinel-Alin Croitoru +1Diffusion Models Training Data Selection
University of Bucharest, Romania
Beyond Correctness: Compute-Responsive Data Selection for Reasoning Model Post-Training
Oct 14, 2026Liang He, Xiangyu Li, Yuxin Wang +4Training Data Selection Language Model Post-Training
Counterfactual Gate-Support Decomposition: Separating Selection from Redistribution in Training-Data Filtering
Oct 14, 2026Hyojung Han, Jongmin Kim, Seung-Hun JeonTraining Data Selection Counterfactual Learning
ThakiCloud
Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning
Oct 9, 2026Sida Li, Yuanxin Liu, Shicheng Li +4AI Coding Agents Training Data Selection
National Key Laboratory for Multimedia Information Processing, Peking University · Huawei Technologies Co., Ltd.
Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training
Oct 9, 2026Suorong Yang, Jiaxin Wang, Jiaming Su +4Efficient Language Model Reasoning Efficient Language Model Training
National University of Singapore · Nanjing University · Fudan University +3
DataSense-Bench: The First Step Toward an AI Scientist
Oct 8, 2026Yudi Zhang, Mingyu Cao, Lu Yin +2Training Data Selection AI Agent Benchmarks
Eindhoven University of Technology · Max Planck Institute for Intelligent Systems · University of Surrey +2
MAP4CS: A Multi-dimensional Data Pruning Framework for Efficient Code Retriever Fine-tuning
Oct 8, 2026Yuxuan Chen, Mingwei Liu, Guangsheng Ou +4Code Search Training Data Selection
School of Software Engineering, Sun Yat-sen University, China
CoTrace: Data Recipes for Training Terminal Agents with Harness-Model Co-Evolution
Oct 7, 2026Jixuan Chen, Jiaxin Zhang, Qinyuan Ye +11Terminal Agents LLM Agent Harnesses
University of California, San Diego · Salesforce AI Research · University of Washington
Dataset Pruning from First Principles: A Label-Free Linear Programming Approach
Oct 7, 2026Rodrigo Schuller, Francisco GanacimDataset Pruning Training Data Selection
Instituto de Matem´atica Pura e Aplicada (IMPA) Rio de Janeiro, Brazil
A Fine-Grained Analysis of the LoRA Fine-Tuning Landscape with Implications for Data Selection
Oct 5, 2026Bowen Zhang, Changrui Fang, Xinsong Ma +2Low-Rank Adaptation Training Data Selection
City University of Hong Kong · Shanghai University of Finance and Economics
LESSER: Post-Training Data Selection with Output-Layer Gradients
Oct 2, 2026Lyuxin David Zhang, Eric Wong, Surbhi Goel +1Training Data Selection Data Selection for Language Model Fine-Tuning
University of Pennsylvania · University of Texas at Austin
OptiSelect: How does the Optimizer Shape Data Curriculum?
Oct 2, 2026Simin Fan, Alireza Abdollahpoorrostam, Martin JaggiTraining Data Selection Language Model Pretraining
EPFL
Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)
Oct 1, 2026Zilin Du, Bowen Yang, Boyang Albert LiTraining Data Selection
College of Computing and Data Science Nanyang Technological University Singapore
Client and Training Data Selection for Computationally Efficient Synchronized Federated Learning
Sep 30, 2026Muzaffer Citir, Hiroki Nishikawa, Sangyoung ParkClient Selection in Federated Learning Non-IID Federated Learning
Smart Mobility Systems, Technical University of Berlin, Germany · Graduate School of Information Science and Technology, The University of Osaka, Japan
The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Sep 29, 2026Gonçalo Paulo, Louis Jaburi, Nora Belrose +2LLM Alignment LLM Fine-Tuning
EleutherAI
Beyond Prompt Count: How Data Shapes Transfer in On-Policy Distillation
Sep 29, 2026Jiaxuan Wang, Jiafei Lyu, Yuchen Cai +8Training Data Selection Language Model Distillation
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · LLM Department, Tencent +1
Loss-Guided Pretraining Data Selection for Time-Series Foundation Models
Sep 29, 2026Yike Li, Shaoxu Song, Jianmin WangTime Series Forecasting Training Data Selection
Tsinghua University
From internal representations to model improvement through prediction errors
Sep 28, 2026Yushi Nakaya, Kenichi Higuchi, Shuichi IshidaActive Learning Training Data Selection
Adansons Corp., 214 Homer Ave, Palo Alto, 94301, CA, USA. · Tohoku University, 6-6-11-803 Aza-Aoba Aramaki, Aoba-ku, Sendai, 980-8579, Miyagi, Japan.
When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation
Sep 28, 2026Minjae Park, Taesun Yeom, Jaeho LeeSelective Knowledge Distillation Training Data Selection
Pohang University of Science and Technology (POSTECH)
How code helps different tasks? A decompositional lens on LLM post-training
Sep 27, 2026Zheng Yu, Yiwei Li, Yishen Chen +4LLM Fine-Tuning Instruction-Tuning Data Selection
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +1
Selecting Diverse SFT Traces Improves Post-RL Generalization
Sep 27, 2026Dylan Zhang, Mingyuan Wu, Jinning LiDiversity-Aware Sampling Reasoning Diversity
University of Illinois Urbana-Champaign, work done at Google · Google

Oct 26, 2026J/K move · Enter open · S save

When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation

Yingjie Bai, Weiming Zhi

School of Computer Science, The University of Sydney

Data Selection Training Data Selection Data Valuation Statistical Learning Theory

Open paper AskPDF arXiv

Per-point data-valuation scores -- Data Shapley, influence, leverage -- are routinely used to select training data: keep the points with the top

k

scores. We argue that top-

k

by a fixed, mode-blind per-point score is the wrong selection rule precisely in the regime where data selection matters. The reason is that a point's value is not a fixed number but depends on what else has been selected -- a near-duplicate of an already-chosen point adds almost nothing, however high its score -- and a per-point scalar cannot express this interaction. The theory quantifies this failure exactly: in an equal latent-mode model, fixed top-

k

selection incurs a closed-form excess risk over adaptive coverage because it repeatedly spends budget on already-covered modes. The gap is largest when the budget is tight, the learner representation is redundant, and missing a mode is costly. The practical rule is simple: use balance when the labels available at selection identify the learner's modes, and use explicit coverage when they do not. On standard classification tasks with strong pretrained features, class-balancing the same per-point scores recovers the missing coverage and matches or outperforms generic set coverage across our datasets. Without labels, label-free coverage improves accuracy by up to +0.20 while raw top-

k

can fall below random; when labels are coarser than the modes, coverage within coarse strata performs best. We turn this distinction into a pre-specified diagnostic and validate it across both favorable and stress-test settings. The limits are part of the claim: the advantage is representation-relative. Coverage fails when the relevant modes are not expressed in the learner geometry, as with spurious subgroups, or when data quality varies independently of coverage -- exactly as the theory predicts.