cs.LGNov 5, 2026

Rethinking Data Selection: The Importance of Coverage over Difficulty in LLM Fine-Tuning

Authors: Lalchand Pandia, Kanishka Misra, Allyson Ettinger

Organizations: Department of Computer Science University of Chicago · Department of Linguistics The University of Texas at Austin · Allen Institute for Artificial Intelligence

Abstract

Selecting high-quality training data can reduce computation cost for LLM fine-tuning. Prior work on data selection developed a variety of difficulty-based scores, motivated by the intuition that harder examples are more informative. While effective for smaller classification models, we find that such methods often fail to outperform random selection for LLM fine-tuning. Our analysis reveals that random selection performs better because it is more representative of the underlying input space---i.e., has better coverage. To validate the importance of coverage, we implement a simple clustering-based selection method which selects data that is more representative of the underlying input distribution. On a set of diverse single-domain tasks, this method yields performance at or above that of random selection. On the diverse, multi-domain Tülu 3 mixture, where random sampling no longer guarantees a balanced subset, our method outperforms all methods. Overall, our results indicate that prioritizing coverage over difficulty leads to more effective data selection for LLM fine-tuning.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs

    Sep 30, 2026Luyang Zhang, Jialu Wang, Bebei LiData SelectionData Selection for Language Model Fine-Tuning

  2. The Long-Term Effects of Data Selection in LLM Fine-Tuning

    May 28, 2026Yuxin Yang, Aoxiong Zeng, Xiangquan YangContinual Learning for LLMsLLM Fine-Tuning

  3. Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuning

    May 13, 2026Siyuan Liu, Tinghong Chen, Xinghan Li +2Supervised Fine-TuningTraining Data Selection