Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction

Zile Yang; Ling Li; Na Di; Jinlong Pang; Yao Zhou; Hao Cheng; Bo Han; Jiaheng Wei

cs.CLOct 27, 2025

Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction

Authors: Zile Yang, Ling Li, Na Di, Jinlong Pang, Yao Zhou, Hao Cheng, Bo Han, Jiaheng Wei

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · University of California, Santa Cruz · Hong Kong Baptist University

Abstract

Data selection for supervised fine-tuning (SFT) often discards low-scoring instruction examples that may retain useful supervision. We propose a rule-guided LLM reconstruction framework to reclaim this information. The framework identifies low-scoring samples through LLM-based rating and score correction, selects semantically representative sources, and reconstructs instruction-response pairs through strategy-guided generation, criteria-based validation, and failure-guided repair. From 123,786 low-scoring samples, it constructs 54,888 training examples. Across three 7B/8B base LLMs, the reconstructed data achieves higher average downstream performance than the low-scoring source set and all evaluated data-selection baselines. Despite using substantially fewer examples than the full source dataset (approximately 300K), it achieves higher reported average scores than full-data training across all three base models. Stepwise ablations identify rule-guided reconstruction as the largest source of observed improvements over direct fusion, while a manual audit reveals remaining limitations in source integration and answer faithfulness. These findings support reconstruction as a complementary approach to selection-based SFT data curation.

Figures & tables

Figure 1: Comparison between selection-based data curation and rule-guided LLM reconstruction. (a) Selection-based methods retain existing samples according to quality or utility estimates, typically discarding lower-scoring examples. (b) Our framework reconstructs new training examples from low-scoring data via representative selection, LLM reconstruction, criteria-based validation, and failure-guided repair.

Figure 2: Overview of our three-stage rule-guided LLM reconstruction framework. Step (1) identifies low-scoring data through LLM-based rating and score correction. Step (2) forms one-hop clusters and selects representatives under three cluster cases. Step (3) reconstructs instruction-response pairs through multi-stage LLM operations within and across clusters.

Figure 3: Detailed workflow of rule-guided LLM reconstruction. Preparation performs structure-dependence filtering and Domain Analysis (DA). Cycle 1 generates and validates candidates through strategy-guided Merged Corpus Generation (MCG), Information Completeness Detection (ICD), and rule-guided prompt updates. Cycle 2 refines responses through Final Answer Check (FAC), Final Answer Update (FAU), and further prompt updates. Purple denotes LLM-based operations, orange denotes rule-based operations and condition checks, and red/green arrows indicate revision/progression upon validation.

Dataset	MMLU (EM $\uparrow$ )	TruthfulQA (I $\times$ T $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Mistral-7B-v0.3
Vanilla Base Model	59.7	30.4	38.0	47.6	54.8	46.1
LQ Set (123786)	47.5	43.7	43.5	52.7	41.1	45.7
HQ Set (131247)	58.4	39.2	46.0	55.5	52.5	50.3
Full Set (300932)	52.6 $\pm$ 10.5	43.1 $\pm$ 0.6	43.5 $\pm$ 0.0	53.2 $\pm$ 0.9	51.4 $\pm$ 2.9	48.7 $\pm$ 2.6
Completion Length	25.4	43.5	43.0	55.7	45.8	42.7

Table 1: Performance comparison across five benchmarks (Mistral-7B-v0.3). The

\Delta

row reports the difference between the two-seed mean of Ours and the single-run LQ Set baseline. Bold red and bold black indicate the best and second-best Average scores, respectively.

Dataset	MMLU (EM $\uparrow$ )	TruthfulQA (I $\times$ T $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Llama-3.1-8B
Vanilla Base Model	64.1	32.9	58.0	55.3	22.1	46.5
LQ Set (123786)	52.7	44.3	57.0	61.0	43.9	51.8
HQ Set (131247)	62.3	41.8	57.5	59.3	58.9	56.0
Full Set (300932)	62.1 $\pm$ 2.0	40.7 $\pm$ 1.9	60.8 $\pm$ 0.4	59.2 $\pm$ 0.1	61.4 $\pm$ 2.1	56.8 $\pm$ 1.3
Completion Length	59.5	45.8	60.0	58.6	61.2	57.0

Table 2: Performance comparison across five benchmarks (Llama-3.1-8B). The

\Delta

row reports the difference between the two-seed mean of Ours and the single-run LQ Set baseline. Bold red and bold black indicate the best and second-best Average scores, respectively.

Dataset	MMLU (EM $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Mistral-7B-v0.3
LQ Set (123786)	47.5	43.5	52.7	41.1	46.2
HQ Set (131247)	58.4	46.0	55.5	52.5	53.1
Full Set (300932)	52.6 $\pm$ 10.5	43.5 $\pm$ 0.0	53.2 $\pm$ 0.9	51.4 $\pm$ 2.9	50.1 $\pm$ 3.1
Ours-20%	56.7 $\pm$ 3.7	42.8 $\pm$ 1.8	52.2 $\pm$ 3.4	53.2 $\pm$ 3.2	51.2 $\pm$ 2.1
Ours-60%	56.0 $\pm$ 4.4	45.8 $\pm$ 0.4	52.5 $\pm$ 0.3	53.2 $\pm$ 4.2	51.9 $\pm$ 2.0

Table 3: Data-size ablation of reconstructed training data. Results with

\pm

indicate mean

\pm

sample standard deviation over two seeds (42 and 47); others are from single runs. The Average is computed over the four benchmarks. Best and second-best Average scores are highlighted in bold red and bold black , respectively.

Criterion	Passed	Rate
Source integration	20/30	66.7%
Context completeness	28/30	93.3%
Question clarity	28/30	93.3%
Answer faithfulness	18/30	60.0%
Overall usability	14/30	46.7%
All five criteria	14/30	46.7%

Table 4: Manual quality audit of 30 reconstructed samples across five criteria.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Datasets	Derived From	Data size	$\bar{N}_{\text{rounds}}$	$\bar{L}_{\text{prompt}}$	$\bar{L}_{\text{response}}$
Stanford Alpaca	Generated w/ Davinci-003	52K	1.0	23.5	56.4
Flan V2	Human Annotation	100K	1.0	304.1	27.7
Open-Assistant 1	Human Annotation	33K	1.6	32.3	189.1
WizardLM	ChatGPT Annotation	100K	1.0	122.3	352.5
Dolly	Human Annotation	15K	1.0	99.5	79.3

Appendix

Table 5: Comprehensive overview of the source corpora used in this work. We report three additional descriptive dimensions, the average number of conversation turns (

\bar{N}_{\text{rounds}}

), the average prompt length (

\bar{L}_{\text{prompt}}

), and the average response length (

\bar{L}_{\text{response}}

), to provide a more nuanced understanding of the composition of our source corpora.

Stage	Hyperparameter or Setting	Value
Steps 1, 3	External LLM	gpt-4o-mini-2024-07-18
Steps 1–2	Embedding model	BAAI/bge-large-en-v1.5
Step 1	Mapped quality-score range	$0$ – $5$
Step 1	Low-/high-quality score partition	LQ: $0$ – $2$ ; HQ: $3$ – $5$
Step 2	One-hop cosine-similarity threshold	$0.9$
Step 2	Candidate number of sub-clusters ( $k$ )	$2\leq k\leq\min(10,M-1)$

Appendix

Table 6: Corpus-construction hyperparameters and predefined settings. Step 1 covers quality rating and score curation; Step 2 covers semantic grouping and representative selection; and Step 3 covers rule-guided LLM reconstruction. For sub-clustering,

M

denotes the number of samples remaining after excluding the one-hop seed.

Hyperparameter	Value
LoRA rank	64
LoRA scaling parameter	16
Batch size	128
Learning rate	$1\times 10^{-4}$
Training epochs	5
LoRA dropout	0.1

Appendix

Table 7: LoRA fine-tuning hyperparameters used across the three base models.

	LESS ( 2024 )	MathFusion ( 2025 )	Evol-Instruct ( 2024 )	Ours
Average API Cost (in USD)	0	0.004	0.003	0.005
GPU Runtime (in GPU-hours)	152.5	17	17	17
Validation Set	Required	Not Required	Not Required	Not Required

Appendix

Table 8: Representative comparison of average API cost per generated output sample, GPU runtime, and task-specific validation-set requirements. The methods have different computational profiles, so GPU usage and external-LLM API cost should be considered jointly rather than as interchangeable resource measures. GPU runtime refers to downstream fine-tuning and excludes external-LLM inference and other offline data-construction operations, which are accounted for separately where applicable.

Dataset	MMLU (EM $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Qwen2.5-7B
Vanilla Base Model	71.8	83.5	58.1	25.3	59.7
LQ Set (123786)	69.3	77.5	58.6	55.4	65.2
HQ Set (131247)	72.2	79.0	60.4	60.2	68.0
Full Set (300932)	72.0	78.0	59.8	65.2	68.8
Completion Length	67.2	75.5	57.9	62.7	65.8

Appendix

Table 9: Performance comparison across four downstream benchmarks using Qwen2.5-7B as the fine-tuning base model. Unless otherwise specified, the training dataset contains 54,888 samples. The best and second-best average scores are highlighted in bold red and bold black , respectively. Values in parentheses for our method indicate the absolute performance differences relative to the LQ Set on each benchmark and the overall average.

Figure 4: Comparison of curated overall quality-score distributions between the original low-scoring source pool before structure-dependence filtering and the reconstructed dataset. Scores range from 0 to 5, with larger values indicating higher model-estimated quality.

Manual quality criterion	Passed	Rate
Source integration	20/30	66.7%
Context completeness	28/30	93.3%
Question clarity	28/30	93.3%
Answer faithfulness and consistency	18/30	60.0%
Overall transformation usability	14/30	46.7%
All five criteria	14/30	46.7%

Appendix

Table 10: Manual source-grounded quality audit of 30 randomly sampled reconstructed instruction–response samples. Each criterion is assigned a binary pass/fail label. The last row requires all five criteria to pass simultaneously.

Reconstruction setting	Audited	Passed	Rate
Pairwise cross-cluster	13	11/13	84.6%
Multi-source intra-cluster	17	9/17	52.9%

Appendix

Table 11: Source-integration pass rates by reconstruction setting within the 30-sample manual audit. The comparison is descriptive and is not controlled for source count, topic, or input complexity.

Example	SI	CC	QC	AF	OU
Sick-leave email	1	1	1	1	1
La Paz / botany	0	0	1	0	0
Apps as companions	0	0	0	0	0

Appendix

Table 12: Case-level manual quality judgments for the illustrated reconstructions. The sick-leave example belongs to the random audit, while the other two are reassessed failure cases. SI: Source Integration; CC: Context Completeness; QC: Question Clarity; AF: Answer Faithfulness and Consistency; OU: Overall Transformation Usability.

Figure 5: Reassessed reconstruction failure: unsupported bridging in the La Paz example. The generated botanical-conference scenario introduces information not supported by the source samples and omits major source concepts. The manual quality vector is

[0,0,1,0,0]

Figure 6: Reassessed reconstruction failure: task drift in the app-review example. The generated discussion of apps as companions does not preserve the original review-writing task and introduces unsupported abstractions. The manual quality vector is

[0,0,0,0,0]

Dataset	MMLU (EM $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Mistral-7B-v0.3
LQ Set (123786)	47.5	43.5	52.7	41.1	46.2
HQ Set (131247)	58.4	46.0	55.5	52.5	53.1
Full Set (300932)	60.0	43.5	52.5	53.4	52.4
Ours-20%	59.3	41.5	54.6	55.4	52.7 (+6.5)
Ours-40%	58.9	42.0	50.5	56.8	52.1 (+5.9)

Appendix

Table 13: The fine-tuning base models are Mistral-7B-v0.3 and Llama-3.1-8B. We report single-run results for 20%, 40%, 60%, 80%, and 100% of the reconstructed dataset, with seed 42 used for these configurations. The Average is computed over the four reported benchmarks. Best and second-best Average scores for each base model are highlighted in bold red and bold black , respectively. Parenthesized values indicate absolute Average-score differences relative to the corresponding single-run LQ Set baseline.

Dataset	MMLU (EM $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Qwen2.5-7B
Vanilla Base Model	71.8	83.5	58.1	25.3	59.7
LQ Set (123786)	69.3	77.5	58.6	55.4	65.2
HQ Set (131247)	72.2	79.0	60.4	60.2	68.0
Full Set (300932)	72.0	78.0	59.8	65.2	68.8
Intra-20%	71.2	86.5	58.1	67.8	70.9 (+5.7)

Appendix

Table 14: Data-size ablation of intra-cluster reconstructed samples across four downstream benchmarks using Qwen2.5-7B, Mistral-7B-v0.3, and LLaMA-3.1-8B as fine-tuning base models. Intra-

x

% denotes the use of

x

% of the 15,488 intra-cluster reconstructed samples. The best and second-best average scores within each base-model setting are highlighted in bold red and bold black , respectively. Values in parentheses indicate the absolute differences in average performance relative to the corresponding LQ Set.

Dataset	MMLU (EM $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Mistral-7B-v0.3
Vanilla Base Model	59.7	38.0	47.6	54.8	50.0
LQ Set (123786)	47.5	43.5	52.7	41.1	46.2
HQ Set (131247)	58.4	46.0	55.5	52.5	53.1
Full Set (300932)	60.0	43.5	52.5	53.4	52.4
Inter-20%	59.4	37.0	54.3	57.5	52.1 (+5.9)

Appendix

Table 15: Data-size ablation of inter-cluster reconstructed samples across four downstream benchmarks using Mistral-7B-v0.3 and LLaMA-3.1-8B as fine-tuning base models. Inter-

x

% denotes the use of

x

% of the 39,400 inter-cluster reconstructed samples. The best and second-best average scores within each base-model setting are highlighted in bold red and bold black , respectively. Values in parentheses indicate the absolute differences in average performance relative to the corresponding LQ Set.

Dataset	MMLU (EM $\uparrow$ )	TruthfulQA (I $\times$ T $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Llama-3.1-8B
Vanilla Base Model	64.1	32.9	58.0	55.3	22.1	46.5
LQ Set (123786)	52.7	44.3	57.0	61.0	43.9	51.8
HQ Set (131247)	62.3	41.8	57.5	59.3	58.9	56.0
Full Set (300932)	63.5	42.0	61.0	59.1	62.8	57.7
HQ+Inter-40%	62.0	45.3	58.0	59.9	55.0	56.0 (+0.0)

Appendix

Table 16: Performance comparison across five downstream benchmarks using LLaMA-3.1-8B as the fine-tuning base model. HQ+Inter-

x

% denotes the combination of the complete HQ Set (131,247 samples) with

x

% of the 39,400 inter-cluster reconstructed samples. The best and second-best average scores are highlighted in bold red and bold black , respectively. Values in parentheses indicate the absolute differences in average performance relative to the HQ Set.

Dataset	MMLU (EM $\uparrow$ )	TruthfulQA (I $\times$ T $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Qwen2.5-7B
Vanilla Base Model	71.8	11.1	83.5	58.1	25.3	50.0
LQ Set (123786)	69.3	43.5	77.5	58.6	55.4	60.9
HQ Set (131247)	72.2	38.2	79.0	60.4	60.2	62.0
Full Set (300932)	72.0	41.4	78.0	59.8	65.2	63.3
HQ+Intra-40%	72.0	36.3	72.5	58.7	63.6	60.6

Appendix

Table 17: Performance comparison across five downstream benchmarks using Qwen2.5-7B as the fine-tuning base model. HQ+Intra-

x

% denotes the combination of the complete HQ Set (131,247 samples) with

x

% of the 15,488 intra-cluster reconstructed samples. The best and second-best average scores are highlighted in bold red and bold black , respectively.

Dataset	MMLU (EM $\uparrow$ )	TruthfulQA (I $\times$ T $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Qwen2.5-7B
Vanilla Base Model	71.8	11.1	83.5	58.1	25.3	50.0
LQ Set (123786)	69.3	43.5	77.5	58.6	55.4	60.9
HQ Set (131247)	72.2	38.2	79.0	60.4	60.2	62.0
Full Set (300932)	72.0	41.4	78.0	59.8	65.2	63.3
HQ+Inter-20%	71.6	37.3	73.0	59.5	64.0	61.1

Appendix

Table 18: Performance comparison across five downstream benchmarks using Qwen2.5-7B as the fine-tuning base model. HQ+Inter-

x

% denotes the combination of the complete HQ Set (131,247 samples) with

x

% of the 39,400 inter-cluster reconstructed samples. The best and second-best average scores are highlighted in bold red and bold black , respectively.

Dataset	MMLU (EM $\uparrow$ )	TruthfulQA (I $\times$ T $\uparrow$ )	GSM8K (EM $\uparrow$ )	BBH (EM $\uparrow$ )	TydiQA (1-Shot F1 $\uparrow$ )	Average $\uparrow$
Base Model: Mistral-7B-v0.3
Vanilla Base Model	59.7	30.4	38.0	47.6	54.8	46.1
LQ Set (123786)	47.5	43.7	43.5	52.7	41.1	45.7
HQ Set (131247)	58.4	39.2	46.0	55.5	52.5	50.3
Full Set (300932)	60.0	43.5	43.5	52.5	53.4	50.6
HQ+Ours-40%	57.8	42.4	45.0	54.4	55.2	51.0

Appendix

Table 19: Performance comparison across five downstream benchmarks using Mistral-7B-v0.3, LLaMA-3.1-8B, and Qwen2.5-7B as fine-tuning base models. HQ+Ours-

x

% denotes the combination of the complete HQ Set (131,247 samples) with

x

% of our reconstructed dataset (54,888 samples), comprising both intra-cluster and inter-cluster reconstructions. The best and second-best average scores within each base-model setting are highlighted in bold red and bold black , respectively.

Panel of Figure 7 — Figure 7: Source inputs and domain analysis. Top: the original source pair, containing noisy Russian-language and translation examples. Bottom: the DA report, which identifies key terms from each source and classifies their relationship as related-domain .

Panel of Figure 8 — Figure 8: Initial candidate and first ICD evaluation. Top: initial MCG output, showing the selected term_bridging candidate. Bottom: ICD flags insufficient question context and an answer that does not directly address the requested framework, despite reporting no missing key terms.

Panel of Figure 9 — Figure 9: First MCG regeneration and subsequent ICD evaluation. Top: the candidate produced by the first qa -guided repair, with a more specific question and revised response. Bottom: the second ICD evaluation identifies insufficient integration of sentence construction , while the question-context and answer-regeneration checks no longer flag deficiencies.

Panel of Figure 10 — Figure 10: Second MCG regeneration and subsequent ICD evaluation. Top: the candidate produced by the second missing_terms -guided repair, with an explicit explanation of sentence construction and its role in the proposed translation framework. Bottom: the third ICD evaluation reports no missing source terms and no remaining question-context or answer-regeneration requirements.

Panel of Figure 11 — Figure 11: Cycle 2 validation and retained reconstruction. Top: FAC identifies a direct answer and finds no content requiring deletion. Bottom: the final term_bridging record, retaining the revised User content from Generation 2 and the Assistant response from Generation 3.

Figure 12: LLM quality-rating prompt template adopted from DS 2 ( Pang et al., 2025 ) .

Figure 13: Nine predefined reconstruction strategies grouped by the semantic relationship between source samples. Each relationship category is associated with three strategies that guide subsequent LLM-based reconstruction.

Figure 14: Structure-dependence filtering prompt used to identify source samples requiring strict preservation of their original format.

Figure 15: Prompt template for Domain Analysis (DA).

Figure 16: Initial prompt template for Merged Corpus Generation (MCG).

Figure 17: Prompt template for Information Completeness Detection (ICD).

Figure 18: Conditional MCG repair prompt template for missing key technical terms.

Figure 19: Conditional MCG repair prompt template for missing key technical terms and a direct answer.

Figure 20: Conditional MCG repair prompt template for missing key technical terms and essential contextual knowledge.

Figure 21: Conditional MCG repair prompt template for missing key technical terms, essential contextual knowledge, and a direct answer.

Figure 22: Conditional MCG repair prompt template for missing key technical terms and question-related deficiencies.

Figure 23: Conditional MCG repair prompt template for question-related deficiencies.

Figure 24: Conditional MCG repair prompt template for missing essential contextual knowledge.

Figure 25: Conditional MCG repair prompt template for missing essential contextual knowledge and a direct answer.

Figure 26: Conditional MCG repair prompt template for a missing direct answer.

Figure 27: Prompt template for Final Answer Check (FAC).

Figure 28: Conditional FAU repair prompt template for generating a missing direct answer.

Figure 29: Conditional FAU repair prompt template for removing irrelevant or redundant answer content.

Explore similar work

Feb 16, 2026

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets, models, and candidate pools. While no single method dominates, gradient-based representations paired with greedy round-robin selection often perform best on average at low budgets, but these gains diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.

Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2

Harvard University · MIT · Kempner Institute

Instruction-Tuning Data Selection Training Data Selection LLM Fine-Tuning

Ask PDF

May 13, 2026

From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuning

Supervised fine-tuning (SFT) data selection is commonly formulated as instance ranking: score each example and retain a top-

k

subset. However, effective SFT training subsets are often produced through ordered curation recipes, where filtering, mixing, and deduplication operators jointly shape the final data distribution. We formulate this problem as fixed-pool data recipe search: given a raw instruction pool and a library of grounded operators, the goal is to discover an executable recipe that constructs a high-quality selected subset under a limited budget of full SFT evaluations, without generating, rewriting, or augmenting training samples. We introduce AutoSelection, a two-layer solver that decouples fixed-pool materialization based on cached task-, data-, and model-side signals from expensive full evaluation, using warmup probes, realized subset states, local recipe edits, Gaussian-process-assisted ranking, and stagnation-triggered reseeding. Experiments on a 90K instruction pool show that AutoSelection achieves the strongest in-distribution reasoning average across three base models, outperforming full-data training, random recipe search, random top-

k

, and single-operator selectors. Additional Out-of-distribution graph-reasoning results, search-stability analyses, structural ablations, and 1.5B-to-7B transfer checks further show that recipe structure matters beyond individual selection operators. Code is available at https://github.com/w253/AutoSelection.

Haodong Wu, Jiahao Zhang, Lijie Hu +1

The Hong Kong University of Science and Technology (Guangzhou) · Mohamed bin Zayed University of Artificial Intelligence

Supervised Fine-Tuning Training Data Curation Instruction-Tuning Data Selection Data Selection Training Data Selection Selective Fine-Tuning

Ask PDF

Jan 29, 2026

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

Ajay Patel, Colin Raffel, Chris Callison-Burch

University of Pennsylvania · University of Toronto · Vector Institute +1

Language Model Pretraining Synthetic Data Pretraining Synthetic Data Generation Instruction Tuning Synthetic Data Generation for Language Models

Ask PDF