DS$^2$-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning

Ruiyao Xu; Noelle I. Samia; Han Liu

cs.CLMar 13, 2026

DS $^2$ -Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning

Authors: Ruiyao Xu, Noelle I. Samia, Han Liu

Organizations: Northwestern University

Abstract

Adapting Large Language Models (LLMs) to specialized domains requires high-quality instruction tuning datasets, which are expensive to create through human annotation. Existing data synthesis methods focus on general-purpose tasks and fail to capture domain-specific terminology and reasoning patterns. To address this, we introduce DS $^2$ -Instruct, a zero-shot framework that generates domain-specific instruction datasets without human supervision. Our approach first generates task-informed keywords to ensure comprehensive domain coverage. It then creates diverse instructions by pairing these keywords with different cognitive levels from Bloom's Taxonomy. Finally, it uses self-consistency validation to ensure data quality. We apply this framework to generate datasets across seven challenging domains, such as mathematics, finance, and logical reasoning. Comprehensive evaluation demonstrates that models fine-tuned on our generated data achieve substantial improvements over existing data generation methods.

Figures & tables

Figure 1: Top : Traditional methods require human seed data with manual annotation. Middle : Document-based approaches requiring domain-specific corpora and preprocessing. Bottom : Our DS 2 -Instruct framework generates data using only task definitions through systematic keywords expansion and instruction generation.

Figure 2: Overview of the DS 2 -Instruct framework. Given only a task definition, our approach systematically generates domain-specific instruction-tuning datasets through three stages: ❶ keyword generation using bi-directional expansion and retrieval augmentation, ❷ cognitive-level instruction generation based on Bloom’s Taxonomy, and ❸ self-consistency filtering for quality assurance.

Metric	CFA	PubMedQA	MedQA	GPQA	LogiQA	GSM8K	MATH
Avg. Instruction Length (words) $\uparrow$	154.92	119.73	140.65	144.98	169.39	108.49	116.24
Unique Verb-Noun Pairs $\uparrow$	5,706	2,572	6,834	6,684	5,211	1,855	1,630
Avg. Pair Occurrences $\downarrow$	2.69	2.32	2.16	2.02	2.33	2.25	6.19
Std. Dev. of Occurrences $\downarrow$	17.01	5.35	7.54	6.27	18.94	9.94	148.77

Table 1: Statistics of generated instruction datasets across different domains.

Panel of Figure 3 — Figure 3: Root verb-noun pairs in instructions of (a) Domain-Aware Self-Instruct and (b) DS 2 -Instruct in the math domain. The inner circle represents root verbs and the outer circle represents direct nouns. For each verb, we show the top 4 verb-noun pairs.

	Methods	Finance	Medicine		Science	Logic	Math		Avg.
	Methods	CFA	PubMedQA	MedQA	GPQA	LogiQA	GSM8K	MATH	Avg.
Qwen2.5	Zero-Shot	53.71	60.34	48.35	26.32	32.77	71.53	49.44	48.92
	Self-Instruct	55.83	61.47	48.92	27.18	42.56	72.38	53.67	51.72
	InstructMix	57.91	62.15	47.83	26.09	43.88	75.42	56.24	53.07
	ExploreInstruct	56.47	60.89	47.15	25.93	41.19	76.67	51.38	51.10
	DS 2 -Instruct	58.34	63.62	50.98	26.35	44.29	78.94	60.12	54.95

Table 2: Performance comparison across domain-specific benchmarks. Results show accuracy (%) for multiple-choice tasks and exact match (%) for problem-solving tasks. For each model family, the best results are bold and second-best are underlined .

Quality Metric	GSM8K (%)	MedQA (%)
Valid instruction	96.83	92.42
Valid response	91.26	94.89
Domain-appropriate	94.91	90.87

Table 3: Quality assessment of generated instruction-response pairs.

Figure 4: Impact of Training Data Size on Model Performance.

Figure 5: Performance on Models of Different Sizes. We evaluate the Qwen model family on Math and GPQA datasets.

Figure 6: Ablation study evaluating the effectiveness of keyword generation components.

Figure 7: Ablation study evaluating the effectiveness of cognitive level integration in instruction generation.

Figure 8: Ablation study comparing model performance with and without self-consistency filtering.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Test Size	Question Type	Evaluation Metric
CFA Xie et al. (2023)	Finance	1,506	Multiple Choice	Accuracy
PubMedQA Jin et al. (2019b)	Biomedical	1,000	Yes/No/Maybe	Accuracy
MedQA Jin et al. (2021)	Medical	1,273	Multiple Choice	Accuracy
GPQA Rein et al. (2024)	Graduate Science	448	Multiple Choice	Accuracy
LogiQA Liu et al. (2020)	Logical Reasoning	651	Multiple Choice	Accuracy
GSM8K Cobbe et al. (2021)	Mathematics	1,319	Problem-Solving	Exact Match

Appendix

Table 4: Overview of benchmark datasets used for evaluation.

Metric	CFA	PubMedQA	MedQA	GPQA	LogiQA	GSM8K	MATH
Avg. Length (words) $\uparrow$	85.65	53.33	91.92	41.98	79.91	46.39	37.69
Unique Verb-Noun Pairs $\uparrow$	505	45	333	287	196	117	68
Avg. Occurrences $\downarrow$	28.56	63.60	12.36	3.96	18.60	27.73	8.00
Std. Dev. Occurrences $\downarrow$	160.31	129.81	49.70	8.66	88.05	115.53	8.77

Appendix

Table 5: Statistical analysis of Self-Instruct generated datasets across different domains, including verb-noun (V-N) pair diversity metrics.

Hyperparameter	Value
LoRA Rank ( $r$ )	8
LoRA Alpha ( $\alpha$ )	16
Learning Rate	2e-5
Training Epochs	5
Batch Size	16
Weight Decay	0.01

Appendix

Table 6: Fine-tuning hyperparameters for LoRA adaptation.

Cognitive Level	Generated Instruction
Remembering	What is the definition of satisfiability in the context of logic and problem-solving? Options: A) The ability to prove that a statement is true in all possible scenarios. B) The process of finding a solution that meets all specified constraints. C) The method of determining the truth value of a logical expression. D) The technique of simplifying complex logical expressions to their basic components.
Understanding	Which statement best describes the roles of formal proof techniques and symbolic logic in constructing logical arguments? Options: A) Formal proof techniques and symbolic logic are both essential tools in constructing logical arguments, each offering unique advantages in terms of clarity and rigor. B) Symbolic logic is the preferred method for constructing logical arguments, as it eliminates the need for formal proofs. C) Formal proof techniques are less precise than symbolic logic, but they are faster and easier to use in logical arguments. D) Neither formal proof techniques nor symbolic logic provide significant value in constructing logical arguments.
Applying	In a logical reasoning test, you are given the statements: ‘If the system is secure, then the data is protected’ (P $\rightarrow$ Q) and ‘The data is not protected’ ( $\neg$ Q). Using modus tollens, what can you conclude about the security of the system? Options: A) The system is secure. B) The data is protected. C) The system is not secure. D) The system’s security status cannot be determined from the given statements.
Analyzing	In the context of logical reasoning, compare and contrast the application of “basic set operations” and “verified observation” in constructing a valid argument. How do these concepts interact when evaluating the strength and validity of an argument? Options: A) Basic set operations are used to verify observations, ensuring that the logical structure of an argument is sound. B) Verified observation is a prerequisite for applying basic set operations, as it provides the necessary empirical data. C) Basic set operations and verified observation are independent processes that do not influence each other in logical reasoning. D) Basic set operations can be used to manipulate sets of verified observations to test the consistency and validity of an argument.
Evaluating	Evaluate the following statement: “All bachelors are unmarried men, which is a truth known a priori.” How does this statement relate to the concept of synthetic a priori knowledge, and what is the nature of the conclusion drawn? Options: A) The statement is an example of analytic a priori knowledge, not synthetic a priori, because the predicate ‘unmarried’ is contained within the subject ‘bachelor.’ B) The statement is an example of synthetic a priori knowledge because it combines a priori reasoning with empirical facts about bachelors. C) The statement is neither analytic nor synthetic a priori because it relies on empirical observation to be true. D) The statement is an example of synthetic a posteriori knowledge because it requires empirical verification to confirm that all bachelors are indeed unmarried.
Creating	Design a logical reasoning problem where two events, A and B, always occur together (constant conjunction), but it is not clear whether one causes the other. Present a scenario and ask students to identify which additional information would best help determine if there is a causal relationship between A and B, or if their conjunction is coincidental. Example Scenario: In a small town, every time the town clock strikes 12, a flock of birds flies over the town square. This has been observed consistently for the past year. Options: A) The type of birds observed flying over the town square. B) The time of day when the birds are observed flying over the town square. C) Whether the birds still fly over the town square when the clock is temporarily stopped. D) The weather conditions when the birds are observed flying over the town square.

Appendix

Table 7: Case study demonstrating DS 2 -Instruct ’s generation of cognitively diverse questions across all six levels of Bloom’s Taxonomy for the logical reasoning domain.

Bloom’s Taxonomy Level	GPQA (%)	GSM8K (%)	MedQA (%)
Remembering	5.5	9.2	7.8
Understanding	36.0	16.0	18.3
Applying	20.8	20.4	32.9
Analyzing	24.0	25.2	23.3
Evaluating	9.7	15.8	10.4
Creating	4.0	13.4	7.3

Appendix

Table 8: Bloom’s Taxonomy distribution across benchmarks based on human annotation of 500 randomly sampled questions. GPQA emphasizes conceptual understanding (36.0%), GSM8K shows balanced higher-order demands (29.2%), and MedQA concentrates on procedural application (32.9%).

Explore similar work

Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning
May 28, 2026Tong Ye, Hang Yu, Tengfei Ma +6Domain Adaptation Synthetic Data Generation
vivo AI Lab, Ant Group, Zhejiang University
Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain
Mar 2, 2026Yuma Okochi, Fabio Milentiansen Sim, Tomoyasu OkadaSynthetic Data Generation for Language Models Domain Adaptation for LLMs
Nomura Research Institute, Ltd. · NRI Indonesia
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Apr 15, 2026Tatsuya Ichinose, Youmi Ma, Masanari Oi +2Instruction Tuning Synthetic Data Generation for Language Models
Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII

May 28, 2026J/K move · Enter open · S save

Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

Tong Ye, Hang Yu, Tengfei Ma, Xuhong Zhang+5

vivo AI Lab, Ant Group, Zhejiang University

Domain Adaptation Synthetic Data Generation Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Large Language Models have demonstrated remarkable progress in general-purpose capabilities and can achieve strong performance in specific domains through fine-tuning on domain-specific data. However, acquiring high-quality data for target domains remains a significant challenge. Existing data synthesis approaches follow a deductive paradigm, heavily relying on explicit domain descriptions expressed in natural language and careful prompt engineering, limiting their applicability in real-world scenarios where domains are difficult to describe or formally articulate. In this work, we tackle the underexplored problem of domain-specific data synthesis through an inductive paradigm, where the target domain is defined only through a set of reference examples, particularly when domain characteristics are difficult to articulate in natural language. We propose a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data. DOMINO integrates prompt tuning with a contrastive disentanglement objective to separate domain-level patterns from sample-specific noise, mitigating overfitting while preserving core domain characteristics. Theoretically, we prove that DOMINO expands the support of the synthetic data distribution, ensuring greater diversity. Empirically, on challenging coding benchmarks where domain definitions are implicit, fine-tuning on data synthesized by DOMINO improves Pass@1 accuracy by up to 4.63% over strong, instruction-tuned backbones, demonstrating its effectiveness and robustness. This work establishes a new paradigm for domain-specific data synthesis, enabling practical and scalable domain adaptation without manual prompt design or natural language domain specifications.

Abstract

Explore similar work

Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding