Training Data Selection

Training data selection asks which examples, tokens, or data sources best help a model learn a target task, generalize, or retain existing capabilities. Methods include quality and difficulty filtering, diversity-aware coresets, gradient or influence scoring, active learning, and dynamic curricula or data mixtures. Researchers test these approaches in settings such as language-model pretraining and instruction tuning, vision, speech, and scientific learning, measuring both downstream performance and the cost of selecting and training on data. A central tension is that carefully chosen subsets can save resources or improve results, but selection may be expensive, depend on the model and data budget, or harm coverage and future learning; in some settings, random or full-data training remains competitive.