Efficient Embedding-based Synthetic Data Generation for Complex Reasoning Tasks

Srideepika Jayaraman; Achille Fokoue; Dhaval Patel; Jayant Kalagnanam

cs.LGMar 15, 2026

Efficient Embedding-based Synthetic Data Generation for Complex Reasoning Tasks

Authors: Srideepika Jayaraman, Achille Fokoue, Dhaval Patel, Jayant Kalagnanam

Organizations: IBM Research, Yorktown Heights, NY, USA

Abstract

Synthetic Data Generation (SDG), leveraging Large Language Models (LLMs), has recently been recognized and broadly adopted as an effective approach to improve the performance of smaller but more resource and compute efficient LLMs through fine-tuning. A key challenge in SDG is ensuring the quality and diversity of the generated data. In this paper, we analyze the diversity and distribution of generated data in the embedding space, and demonstrate a strong correlation between the density of examples within a specific neighborhood and the accuracy of predictions on examples drawn from that region. Building on this insight, we present a targeted pipeline for embedding-based sampling that enhances data diversity and consistently improves performance across several benchmarks.

Figures & tables

Figure 1: Pipeline to perform targeted embedding driven SDG

Figure 2: Grid density distribution of embedding space

Figure 3: Red box represents potential gap (sparsity) in embedding space

Figure 4: Motivation: Correlation between performance and density in embedding space

Figure 5: Interpolation: select area in sparsity between two topics

Model	Random Seed Selection			EmbedSDG			Base Model
Model	500	1000	4500	500	1000	4500	Base Model
Math - GSM8k
Granite 3 8b code instruct	0.761	0.761	0.773	0.782	0.79	0.79	0.55
Granite 3.1 8b instruct	0.786	0.786	0.785	0.824	0.806	0.81	0.74
Mistral 7B	0.354	0.558	0.723	0.62	0.725	0.746	0.354
Math - MATH

Table 1: Experimental results comparing accuracy for random vs Embedding-driven seed selection on math reasoning

Figure 7: Correlation between density in ES and Model Accuracy

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

EmbGen: Teaching with Reassembled Corpora
May 19, 2026Arun K Lenin, Kai Rouse, Andrea Nicastro +1Synthetic Instruction Generation Domain Adaptation for LLMs
Commonwealth Bank of Australia Sydney, Australia
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Apr 15, 2026Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9Synthetic Data Pretraining Synthetic Data Generation for Language Models
Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan
Synthetic Data for any Differentiable Target
Apr 9, 2026Tristan Thrush, Sung Min Park, Herman Brunborg +5Synthetic Data Generation for Language Models RL for Generative Models
Stanford University

May 19, 2026J/K move · Enter open · S save

EmbGen: Teaching with Reassembled Corpora

Arun K Lenin, Kai Rouse, Andrea Nicastro, Anna Leontjeva

Commonwealth Bank of Australia Sydney, Australia

Synthetic Instruction Generation Domain Adaptation for LLMs Synthetic Data Generation for Language Models Language Model Post-Training Large Language Model-Based Data Augmentation

Open paper AskPDF arXiv

Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets (5 and 20 million tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12.5% at 5M and 88.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity.

Abstract

Synthetic Data Generation

Explore similar work

EmbGen: Teaching with Reassembled Corpora

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Synthetic Data for any Differentiable Target