cs.CLOct 25, 2026

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

Authors: Vladimir Necula, David Gvadzabia, Sofia Serrano

Organizations: University of Cambridge Department of Physics · Lafayette College Department of Computer Science

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

"Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which simple example-level metrics, drawn from prior work on data selection, text quality, and instruction tuning, best predict training utility under a fixed finetuning budget. For each metric, we sort instruction-response examples by score, partition them into low, middle, and high bins with roughly equal token mass, finetune separately on each bin, and compare validation perplexity and LLM-as-judge scores for the models' generated answers as a function of cumulative training tokens. Our results suggest that answer-dense supervision, as captured by response ratio, is a stronger and more portable predictor of instruction-tuning utility than commonly used alternative proxies.

Figures & tables

Figure 1: Validation perplexities for Qwen3.5-9B-Base finetuned on Natural Instructions. Of the 66-100 bins, only response ratio’s results in perplexities significantly below its other bins. Confidence intervals for these graphs are so small as to not be visible in this figure.
Figure 2: LLM-as-judge scores for Qwen3.5-9B-Base finetuned on Natural Instructions, where higher scores are better . Confidence intervals for all bins heavily overlap, but for response ratio, the distribution of scores for high-bin instances shifts higher shortly into training.
 Natural InstructionsFLAN
Metric0– 3333– 6666– 1000– 3333– 6666– 100
Alignment18.932.448.730.949.220.0
Semantic density34.026.839.217.629.952.5
Length68.824.66.685.59.94.5
Table 1: Percentages of highest-response-ratio examples (tokenized by Qwen3.5) in other metrics’ bins.
Figure 3: Cross-ablation perplexities for Qwen3.5-9B-Base on FLAN.
Figure 4: Cross-ablation LLM-as-judge scores for Qwen3.5-9B-Base on FLAN.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

ComponentSetting
ModelsQwen3.5-9B-Base, Llama-3.1-8B
DatasetsNatural Instructions, FLAN
MetricsLength, alignment, response ratio, semantic density
BinningLow / mid / high by roughly equal token mass
Training data10% sampled training split
Validation/testFull validation and test splits retained
Appendix
Table 2: Core experimental setup for the main comparison grid.
Sample groupMedian full- prompt tokensMedian answer tokens
Natural Instructions,134.59
66-100 bin
Natural Instructions,162.51
0-33 and 33-66 bins
FLAN, 66-100 bin4424.5
FLAN,113.53
Appendix
Table 3: Median token counts, using Qwen3.5’s tokenizer, for different response ratio bins across the two different datasets.
Figure 5: Perplexity curves for Llama-3.1-8B on Natural Instructions 10%.
Figure 6: LLM-as-judge curves for Llama-3.1-8B on Natural Instructions 10%.
Figure 7: Perplexity curves for Qwen3.5-9B-Base on Natural Instructions 10%.
Figure 8: LLM-as-judge score curves for Qwen3.5-9B-Base on Natural Instructions 10%.
Figure 9: Perplexity curves for Llama-3.1-8B on FLAN 10%.
Figure 10: LLM-as-judge score curves for Llama-3.1-8B on FLAN 10%.
Figure 11: Perplexity curves for Qwen3.5-9B-Base on FLAN 10%.
Figure 12: LLM-as-judge score curves for Qwen3.5-9B-Base on FLAN 10%.
Figure 13: Perplexity curves for cross-ablation with Llama-3.1-8B on FLAN 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 14: LLM-as-judge curves for cross-ablation with Llama-3.1-8B on FLAN 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 15: Perplexity curves for cross-ablation with Qwen-3.5-9B-Base on FLAN 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 16: LLM-as-judge score curves for cross-ablation with Qwen-3.5-9B-Base on FLAN 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 17: Perplexity curves for cross-ablation with Llama-3.1-8B on Natural Instructions 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 18: LLM-as-judge score curves for cross-ablation with Llama-3.1-8B on Natural Instructions 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 19: Perplexity curves for cross-ablation with Qwen3.5-9B-Base on Natural Instructions 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
Figure 20: LLM-as-judge score curves for cross-ablation with Qwen3.5-9B-Base on Natural Instructions 10%, comparing alignment after removing high-response-ratio examples with response ratio after removing high-alignment examples.
More in this topic

Instruction-Tuning Data Selection

Instruction-tuning data selection chooses and organizes training examples to improve model behavior while reducing fine-tuning cost.

See all 43 papers

Explore similar work

CardsList

  1. What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

    Apr 28, 2026Guangzeng Han, Xiaolei HuangInstruction-Tuning Data SelectionTraining Data Selection

    University of Memphis

  2. How Output Format Confounds Data Quality and Capability in Instruction Tuning

    Sep 2, 2026Chengguang Gan, Hanjun Wei, Yunhao Liang +3LLM EvaluationInstruction Tuning

    1Independent Researcher · University of Chinese Academy of Sciences · 3Pusan National University +2

  3. One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

    Apr 14, 2026Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu +1Language Model RobustnessInstruction Tuning

    University of Southern California, Los Angeles, USA · Intel AI, USA