Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain

Yuma Okochi; Fabio Milentiansen Sim; Tomoyasu Okada

cs.LGMar 2, 2026

Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain

Authors: Yuma Okochi, Fabio Milentiansen Sim, Tomoyasu Okada

Organizations: Nomura Research Institute, Ltd. · NRI Indonesia

Abstract

In adapting LLMs to specific domains, achieving both domain expertise and reasoning ability remains an urgent challenge. This study proposes a general method for constructing high-quality synthetic instruction data for any domain, starting from domain-specific vocabulary. As a demonstration, we applied this method to the financial domain and constructed a large-scale instruction dataset totaling approximately 9.5 billion tokens with Chain-of-Thought reasoning traces. Evaluation results confirmed performance improvements over baseline models on financial benchmarks, demonstrating the effectiveness of our approach. We also report findings on the impact of reasoning trace length on performance and its limitations. Lastly, we open-source our models and datasets on https://huggingface.co/nri-ai .

Figures & tables

Figure 1: Proposed dataset construction pipeline

Model	CoT	japanese-lm-fin-harness						pfmt-bench-fin-ja
Model	CoT	Avg.	chabsa	cma	cpa	fp2	ss1	Avg.	turn1	turn2
Qwen3-14B	No	66.50	91.73	87.58	41.83	47.03	64.31	7.781	7.819	7.743
Qwen3-14B	Yes	71.04	91.96	93.26	49.37	53.37	67.22	8.104	8.211	7.997
Qwen3-14B-Base + SFT (Ours)	Yes	70.69	91.48	91.20	47.96	55.00	67.81	8.415	8.472	8.358
Qwen3-14B-Base + CPT + SFT (Ours)	Yes	71.78	91.62	91.45	48.59	60.00	67.27	8.455	8.514	8.395
gpt-oss-20b	No	61.17	91.09	81.33	33.45	41.82	58.17	7.480	7.425	7.534

Table 1: Evaluation results of each model on financial benchmarks (CoT = presence of reasoning traces)

Figure 2: Changes in task accuracy with reasoning trace length

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Parameter	Value
1. Topic word selection	Financial domain seed words	135
1. Topic word selection	General domain (for mixing) seed words	20
2. User question generation	Generation count / sub-topic (multiple-choice)	8
2. User question generation	Generation count / sub-topic (others)	10
3. Instruction expansion	Expansion variations / question (multiple-choice)	3
3. Instruction expansion	Expansion variations / question (others)	5

Appendix

Table 2: Dataset construction pipeline configuration parameters

Perspective	Criteria (5 levels)
Accuracy (factual correctness, presence of misinformation)	5: Completely accurate / 3: Some errors / 1: Major misinformation
Relevance/Instruction-following in response	5: Completely appropriate / 3: Partially unaddressed / 1: Off-topic
Usefulness/Comprehensiveness of answer/information	5: Extremely useful / 3: Average / 1: Insufficient
Reasoning/Depth quality, analysis, logical consistency	5: Excellent reasoning / 3: Average / 1: Logical breakdown
Safety/Appropriateness, ethics of expression	5: Completely safe / 3: Minor issues / 1: Dangerous

Appendix

Table 3: Quality evaluation perspectives and criteria

Dataset Name	Category	Samples	Total Tokens	Avg. Tokens (/sample)				Multi-turn
Dataset Name	Category	Samples	Total Tokens	user	assistant	reasoning	total	Multi-turn
nri-fin-reasoning	Finance/General	632,636	6,352,341,377	191	8,996	855	10,041	Yes
NuminaMath-CoT-modified	Math	88,727	141,678,621	83	508	1,006	1,596	No
Nemotron-Post-Training-math-modified	Math	694,168	2,999,298,386	169	640	3,512	4,320	No
smol-constraints-modified	Instruction-following	24,411	12,803,102	71	199	254	524	No
Total	—	1,439,942	9,506,121,486	171.7	4,295.4	2,134.6	6,601.7	—

Appendix

Table 4: Statistics of each dataset used in training

Explore similar work

DS $^2$ -Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning
Mar 13, 2026Ruiyao Xu, Noelle I. Samia, Han LiuSynthetic Data Generation for Language Models LLM Fine-Tuning
Northwestern University
CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning
Mar 1, 2026Xinyu Zhu, Yihao Feng, Yanchao Sun +5LLM Reasoning Synthetic Data Generation for Language Models
Department of Computer Science, University of Virginia, Charlottesville, VA, USA · Apple, Cupertino, CA, USA · Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training
Mar 7, 2026Chuxue Cao, Honglin Lin, Zhanping Zhong +5LLM Training Language Modeling
Shanghai Artificial Intelligence Laboratory, OpenDataLab, OpenDataArena · Hong Kong University of Science and Technology

Mar 13, 2026J/K move · Enter open · S save

DS $^2$ -Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning

Ruiyao Xu, Noelle I. Samia, Han Liu

Northwestern University

Synthetic Data Generation for Language Models LLM Fine-Tuning Instruction Tuning

Open paper AskPDF arXiv

Adapting Large Language Models (LLMs) to specialized domains requires high-quality instruction tuning datasets, which are expensive to create through human annotation. Existing data synthesis methods focus on general-purpose tasks and fail to capture domain-specific terminology and reasoning patterns. To address this, we introduce DS

^2

-Instruct, a zero-shot framework that generates domain-specific instruction datasets without human supervision. Our approach first generates task-informed keywords to ensure comprehensive domain coverage. It then creates diverse instructions by pairing these keywords with different cognitive levels from Bloom's Taxonomy. Finally, it uses self-consistency validation to ensure data quality. We apply this framework to generate datasets across seven challenging domains, such as mathematics, finance, and logical reasoning. Comprehensive evaluation demonstrates that models fine-tuned on our generated data achieve substantial improvements over existing data generation methods.

Abstract

Explore similar work

DS2^22-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning

CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning

Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training

DS $^2$ -Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning