Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Tatsuya Ichinose; Youmi Ma; Masanari Oi; Ryuto Koike; Naoaki Okazaki

cs.CLApr 15, 2026

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Authors: Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki

Organizations: Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII

Abstract

Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

Figures & tables

Figure 1: Comparison of CoDIT with direct response generation. Existing methods use responses directly generated by the post-trained model (e.g., "In recent months, there are"). In contrast, CoDIT isolates the instruction-following behavior via contrastive decoding, which prioritizes tokens with high likelihood under the post-trained model and low likelihood under the pre-trained model.

Teacher	Qwen3-8B			Qwen3-30B-A3B			Gemma-3-27B
Student	Llama	Qwen	Gemma	Llama	Qwen	Gemma	Llama	Qwen	Gemma
WildBench WB-Score
Vanilla	43.95	60.23	30.42	42.32	57.32	30.70	48.30	60.53	34.29
Best-of-N	44.67	60.12	32.82	46.44	59.30	32.66	48.97	59.90	34.96
CoDIT	48.34	63.18	31.84	48.28	60.23	33.07	54.12	62.21	38.89
Alpaca Eval 2.0 Win Rates (%)

Table 1: Evaluation scores of instruction-following performance for models trained on instruction-tuning datasets constructed by the proposed method and baselines. We confirm that CoDIT yields consistently better instruction-tuning effectiveness than the baselines.

Dataset	# Instances	Instruction	Response
WildChat	778,380	WildChat	GPT-3.5, GPT-4
Llama-3.1-LMSYS	453,737	LMSYS-Chat-1M	Llama-3.1-405B-Instruct
Gemma-2-LMSYS	453,861	LMSYS-Chat-1M	gemma-2-27b-it
Magpie-Pro-300K-Filtered	299,912	Llama-3-70B-Instruct	Llama-3-70B-Instruct
WebR-Basic	87,490	Rewrite web text using Llama-3-70B-Instruct
WebR-Pro	87,498	Rewrite web text using GPT-4o-mini

Table 2: Summary of instruction-tuning datasets. “-Chat-1M-Synth” is omitted from Llama-3.1-LMSYS and Gemma-2-LMSYS for brevity.

	Student Model
	Llama-3.1-8B				Qwen3-8B-Base
Dataset	Wild Bench	Alpaca Eval2.0		MT Bench	Wild Bench	Alpaca Eval2.0		MT Bench
Dataset	WB Score	LC	WR	AVG	WB Score	LC	WR	AVG
WildChat	21.69	15.07	9.54	62.24	29.99	23.66	14.49	70.04
Llama-3.1-LMSYS-Chat-1M-Synth	30.94	27.81	27.84	72.11	42.05	34.58	31.32	80.05
Gemma-2-LMSYS-Chat-1M-Synth	36.20	40.74	31.87	69.51	46.84	45.52	33.80	74.56

Table 3: Performance comparison between publicly available instruction-tuning datasets and our synthesized datasets. Our dataset outperforms existing datasets.

Figure 2: Score distribution of the synthetic dataset evaluated by gpt-oss-120b on a scale of 1 to 10. A comparison between responses generated with and without CoDIT reveals that CoDIT does not affect output quality.

Figure 3: Cosine similarity between the teacher’s chat vector

\Delta\theta

and the update vector

\Delta\theta^{\prime}

. Compared to Vanilla, CoDIT exhibits consistently higher similarity as the data scale grows, with the performance gap widening at larger scales. This indicates that CoDIT more effectively distills the teacher model’s chat vector into the text-space.

		Similarity with Teacher’s chat vector
Teacher Model	Response Generation	IT ( $\Delta\theta_{\mathrm{inst}}$ )	IT+RL ( $\Delta\theta_{\mathrm{post}}$ )
IT ( $\Delta\theta^{\prime}_{\mathrm{inst}}$ )	Vanilla	0.1121	0.1120
IT ( $\Delta\theta^{\prime}_{\mathrm{inst}}$ )	CoDIT	0.1331	0.1331
IT+RL ( $\Delta\theta^{\prime}_{\mathrm{post}}$ )	Vanilla	0.1323	0.1323
IT+RL ( $\Delta\theta^{\prime}_{\mathrm{post}}$ )	CoDIT	0.1493	0.1494

Table 4: Cosine similarity between parameter update induced by teacher’s synthesized responses at different stages and the teacher’s chat vectors. Our results suggest that RL models are more effective at distilling chat vectors.

Evaluation Set	Size	Exact (%)	$n$ -gram (%)	Cosine Similarity (%)
MT-Bench	80	32.50	51.25	46.25
AlpacaEval 2.0	805	23.73	39.01	32.05
WildBench	1,024	0.20	2.05	0.68

Table 5: Overlap between LMSYS-Chat-1M training instructions and evaluation prompts.

Dataset	AlpacaEval 2.0		WildBench
Dataset	LC	WR	WB-Score
WildChat	13.42	8.46	19.81
Gemma-2-LMSYS-Chat-1M-Synth	37.52	30.13	33.75
Magpie-Pro-300K-Filtered	25.21	32.20	26.52
WebR-Pro	32.71	35.93	39.39
CoDIT-Gemma3	47.34	69.60	52.05

Table 6: Performance comparison after removing overlap between LMSYS-Chat-1M and evaluation benchmarks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Score distribution of synthesized responses using Qwen3-8B. Consistent with the results in Section 4.1 , CoDIT maintains a text quality comparable to the Vanilla teacher model, indicating that the method’s effectiveness does not rely on enhancing general text fluency.

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}=0.90,\beta_{2}=0.95$ )
Learning rate scheduler	Cosine with 0.1 warmup ratio
Peak learning rate	$2.5\times 10^{-5}$ ( $1.0\times 10^{-5}$ for gemma-3-4b-pt)
Minimum learning rate	$2.5\times 10^{-6}$ ( $1.0\times 10^{-6}$ for gemma-3-4b-pt)
Training epochs	2
Effective batch size	512

Appendix

Table 7: Hyperparameters for Supervised Fine-Tuning

Figure 5: Hyperparameter tuning for

\alpha

based on MT-Bench scores. The average performance of student models (Qwen3-8B-Base and Llama-3.1-8B) is plotted against different

\alpha

values for each teacher model. The optimal

\alpha

is selected where the MT-Bench score is maximized.

Figure 6: Robustness of CoDIT across various

\alpha

settings. Performance on WildBench remains stable and superior to the baselines regardless of the specific choice of

\alpha

, indicating that the gains are driven by the core CoDIT mechanism rather than sensitive hyperparameter tuning.

Teacher	Qwen3-8B			Qwen3-30B			Gemma-3-27B
Student	Llama	Qwen	Gemma	Llama	Qwen	Gemma	Llama	Qwen	Gemma
Vanilla	43.45	60.12	30.21	43.01	57.55	31.37	47.40	59.63	34.08
CoDIT	48.50	62.97	34.70	48.19	60.43	34.06	54.96	62.92	38.91

Appendix

Table 8: WB-Score on WildBench using the original official prompts. It can be observed that CoDIT consistently yields performance improvements compared to the direct use of teacher model outputs (Vanilla), even when evaluated with the official prompts.

Method	AlpacaEval 2.0		WildBench
Method	LC	WR	WB-Score
Vanilla	37.65	46.99	42.61
CoDIT	40.47	51.86	45.35

Appendix

Table 9: Effect of applying CoDIT’s response-generation procedure to the WebR-Pro instruction source, with Qwen3-8B as the teacher and Llama-3.1-8B as the student.

Teacher	Qwen3-8B			Qwen3-30B-A3B			Gemma-3-27B
Student	Llama	Qwen	Gemma	Llama	Qwen	Gemma	Llama	Qwen	Gemma
WildBench WB-Score
Vanilla	1.31	0.97	1.43	1.35	1.10	1.42	1.30	0.97	1.36
Best-of-N	1.28	0.99	1.41	1.23	0.97	1.44	1.22	0.98	1.40
CoDIT	1.25	0.93	1.42	1.24	0.97	1.41	1.13	0.95	1.39
Alpaca Eval 2.0 Win Rates (%)

Appendix

Table 10: Standard errors for WildBench, MT-Bench, and AlpacaEval metrics

Method	gemma-3-27b-it			Qwen3-8B			Qwen3-30B-A3B
Method	Llama	Qwen	gemma	Llama	Qwen	gemma	Llama
Vanilla	4.07	11.38	$-$ 13.15	$-$ 5.23	9.29	$-$ 55.77	$-$ 6.80
CoDIT	6.19	11.40	$-$ 11.29	$-$ 3.46	9.29	$-$ 54.92	$-$ 5.54

Appendix

Table 11: WB-Scores on a 300-question subset of WildBench evaluated with GPT-5.5 as the judge.

Method	#Responses	Teacher		Judge		Relative Cost
Method	#Responses	Time	Memory	Time	Memory	Relative Cost
Vanilla	1	940.8s	91.70GB	–	–	1.00 $\times$
Best-of- $N$ ( $N=5$ )	5	940.8s	91.70GB	462.0s	187.32GB	10.02 $\times$
CoDIT	1	1477.2s	183.41GB	–	–	3.14 $\times$

Appendix

Table 12: Dataset synthesis cost measured on 5,000 instructions from LMSYS-Chat-1M with Qwen3-8B as the teacher model. Time denotes wall-clock time, and Memory denotes peak GPU memory consumption. Best-of-

N

additionally requires judge annotation using gpt-oss-120b.

Datasets	Llama-3.1-8B			Qwen3-8B
	WildBench	AlpacaEval 2.0		WildBench	AlpacaEval 2.0
	WB-Score	LC	WR	WB-Score	LC	WR
Vanilla (IT)	32.57	28.39	25.34	39.84	35.79	31.88
CoDIT(IT)	34.65	41.48	37.71	44.35	46.43	41.02
Vanilla (IT + RL)	47.96	41.01	49.20	55.17	47.00	55.16
CoDIT(IT + RL)	50.04	50.84	58.79	58.13	54.48	62.72

Appendix

Table 13: Performance comparison of student models (Llama-3.1-8B and Qwen3-8B-Base) trained on synthetic datasets generated by teacher models at different post-training stages (IT vs. IT + RL). LC stands for Length-Controlled win rate, and WR stands for Win Rate.

Figure 7: Cosine similarity between the parameter updates and the teacher’s chat vector for Qwen3-8B. While the performance gain of CoDIT over the baseline is consistent with the trends observed in Llama models, the lower absolute similarity suggests that certain dimensions of the chat vector—specifically those associated with reasoning processes—are not fully captured, as thinking tokens were outside the scope of our current data construction.

Explore similar work

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
Token Utility Is Selection-Conditioned: Coupled Selection of Prompt Context and Response Supervision for Instruction Tuning
Sep 19, 2026Can Wu, Xinrui Chen, Ou Wu +1LLM Fine-Tuning Instruction-Tuning Data Selection
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism
Apr 7, 2026Elisabetta Rocchetti, Alfio FerraraLLM Interpretability Instruction Following
Department of Computer Science, Università degli Studi di Milano

Jan 29, 2026J/K move · Enter open · S save

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Ajay Patel, Colin Raffel, Chris Callison-Burch

University of Pennsylvania · University of Toronto · Vector Institute+1

Language Model Pretraining Synthetic Data Pretraining Synthetic Data Generation Instruction Tuning Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

Abstract

Explore similar work

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Token Utility Is Selection-Conditioned: Coupled Selection of Prompt Context and Response Supervision for Instruction Tuning

How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism