SEDGE: Structural Extrapolated Data Generation

Kun Zhang; Jiaqi Sun; Yiqing Li; Ignavier Ng; Namrata Deka; Shaoan Xie

cs.LGApr 2, 2026

SEDGE: Structural Extrapolated Data Generation

Authors: Kun Zhang, Jiaqi Sun, Yiqing Li, Ignavier Ng, Namrata Deka, Shaoan Xie

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University

Abstract

This paper aims to address the challenge of data generation beyond the training data and proposes a framework for Structural Extrapolated Data GEneration (SEDGE) based on suitable assumptions on the underlying data-generating process. We provide conditions under which data satisfying novel specifications can be generated reliably, together with the approximate identifiability of the distribution of such data under certain ``conservative" assumptions, as well as the inherent non-identifiability of this distribution without such assumptions. On the algorithmic side, we develop practical methods to achieve extrapolated data generation, based on a structure-informed optimization strategy or diffusion posterior sampling, respectively. We verify the extrapolation performance on synthetic data and also consider extrapolated image generation as a real-world scenario to illustrate the validity of the proposed framework.

Figures & tables

Panel of Figure 1 — Figure 1: The two generating processes as initial thoughts.

Panel of Figure 2 — Figure 2: The generating processes where features generate the specification and the observed data satisfy the constraint that the selection variable $S=1$ .

Figure 3: Synthetic experiment results with given $\mathbf{X}$ and $\mathbf{Z}$ . Panels (a–b) illustrate the data split in a two-dimensional view over the $X_{1}$ and $X_{3}$ axes. The novel specifications induce a previously unseen joint distribution over $(X_{1},X_{3})$ , ensuring that successful performance requires true extrapolation rather than interpolation. Panels (c–h) present the optimization-based generation (OPT) results for all five models. Panel (h) shows the three-dimensional view of the Oracle $\mathbf{X}$ . Panels (i–p) visualize the three-dimensional generated $\mathbf{X}$ using OPT and diffusion posterior sampling (DPS) for models A, B, and C, and OPT for models D and E, with the corresponding MMD values to the Oracle $\mathbf{X}$ reported.

Panel of Figure 4 — Figure 4: Comparison of image generation results under different compositional prompts. Baseline models include SANA ( Xie et al., 2024 ) , Aligner ( Xie et al., 2025b ) , Stable Diffusion 3.5-Large ( Esser et al., 2024 ) , QwenImage ( Wu et al., 2025 ) , Z-Image ( Cai et al., 2025 ) , and GPT5.2. w/o Spar and w/o CI denote SEDGE without sparsity loss and conditional independence structure, respectively. Each row displays generated image for a prompt, and each column presents results from either SEDGE or baseline models. Despite using only 1.6B parameters, SEDGE achieves performance comparable to GPT5.2 and outperforms the 20B QwenImage model, demonstrating its effectiveness for extrapolation. Without the proposed sparsity and conditional independence structure, the model struggles to identify distinct concepts, often producing entangled representations (e.g., a full bird instead of a bird wing) and failing to adhere to the prompt.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

\mathbf{x}^{(t-1)}\leftarrow\begin{cases}\boldsymbol{\mu}^{(t)}+\sqrt{\beta_{t}}\cdot\boldsymbol{\epsilon},&\text{if }t>1,\quad\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ \boldsymbol{\mu}^{(t)},&\text{if }t=1\end{cases}

Appendix

Algorithm 2 Structure-Informed Extrapolated Sampling (Diffusion Posterior Sampling) [ 6 ]

Figure 5: Data Split Based on Specifications

Figure 6: Complete 2-d view of the extrapolated data points compared with baselines (model B, C, D, E) on optimization-based generation method

Figure 7: Complete 2-d view of the extrapolated data points compared with baselines (model B, C) on diffusion posterior sampling method

Figure 8: Identification of Specifications ( $Z[0]$ and $Z[1]$ corresponds to $Z1$ and $Z2$ , respectively.)

Figure 9: Identification of Features ( $X[0]$ , $X[1]$ , and $X[2]$ correspond to $X1$ , $X2$ , and $X3$ , respectively.)

$X_{1}$	$X_{2}$	$X_{3}$	$Z_{1}$	$Z_{2}$
0.4346	0.9017	0.7097	0.9088	0.9393

Appendix

Table 1: MCC scores of features and specifications for verification of identifiability

MMD (OPT)	MMD (DPS)
$0.0332\pm 0.0024$	$0.0346\pm 0.0029$

Appendix

Table 2: MMD scores of extrapolated generation by the two proposed methods

Method	(A) Original ( $X\to Z$ )	(B) $Z\to X$ , no shared	(C) $Z\to X$ , shared
OPT	$0.0394\pm 0.0068$	$0.0389\pm 0.0090$	$0.0588\pm 0.0106$
DPS	$0.0362\pm 0.0024$	$0.0405\pm 0.0122$	$0.0856\pm 0.0247$

Appendix

Table 3: MMD under causal-direction violations. (A) is the original setting; (B) and (C) reverse the direction to

Z\to X

, without and with shared features, respectively.

Method	Original	$Z_{1}\to Z_{2}$ added	Dense graph
OPT	$0.0394\pm 0.0068$	$0.1033\pm 0.0150$	$0.0595\pm 0.0132$
DPS	$0.0362\pm 0.0024$	$0.0973\pm 0.0002$	$0.0738\pm 0.0243$

Appendix

Table 4: MMD under violations of conditional independence among

Z

. “Dense” refers to a fully connected graph among

Z

variables.

Model variant	DPS MMD
Full model	$0.0468\pm 0.0133$
Without sparsity	$0.0472\pm 0.0151$
Without cond. independence	$0.0690\pm 0.0237$
Without sparsity or cond. indep.	$0.0699\pm 0.0240$

Appendix

Table 5: Ablation study: DPS MMD for four model variants removing conditional independence (Cond. Indep.) and/or sparsity from the estimated model.

Method	Test 0	Test 1	Test 2	Test 3	Test 4
OPT	$0.0554\pm 0.0269$	$0.0490\pm 0.0230$	$0.0510\pm 0.0169$	$0.0551\pm 0.0253$	$0.0425\pm 0.0164$
DPS	$0.0431\pm 0.0121$	$0.0578\pm 0.0207$	$0.0672\pm 0.0207$	$0.0570\pm 0.0204$	$0.0668\pm 0.0178$

Appendix

Table 6: MMD across five settings of increasing violation of the conservative assumption (Assumption 3). Tests 0–4 correspond to increasing severity of violation.

Method	Original	Remove $X_{1}\!\to\!Z_{1}$	Add $X_{1}\!\to\!Z_{2}$	Remove + Add
OPT	$0.0346\pm 0.0090$	$0.0734\pm 0.0150$	$0.0385\pm 0.0096$	$0.0505\pm 0.0133$
DPS	$0.0374\pm 0.0048$	$0.0380\pm 0.0040$	$0.0370\pm 0.0049$	$0.0381\pm 0.0045$

Appendix

Table 7: MMD under misspecified

X

–

Z

graph structures.

Model configuration	OPT MMD	DPS MMD
Deterministic AE	$0.0295\pm 0.0037$	$0.0345\pm 0.0031$
Frozen extractor (pretrained VAE)	$0.0281\pm 0.0032$	$0.0316\pm 0.0073$
Linear extractor	$0.0806\pm 0.0048$	$0.0557\pm 0.0078$
DDPM prior	$0.0307\pm 0.0040$	$0.0351\pm 0.0037$
Flow Matching prior	$0.0307\pm 0.0040$	$0.0310\pm 0.0026$

Appendix

Table 8: The MMD by OPT and DPS for different extractor and prior choices. “Frozen extractor” denotes a pretrained VAE extractor kept fixed during likelihood training.

Panel of Figure 10 — Figure 10: More qualitative comparisons under different prompts.

Panel of Figure 11 — Figure 11: Consistency of our generation. Given prompt a camera with insect atennae , we generated figures using both ours and concept aligner models acorss $8$ different seeds. As we can see in the figure, the generation of concept aligner has mixed up concepts like legs and bodies. Our generation results (especially with larger guidance values) makes consistent generation.

Models	SANA	Aligner	Ours
GPT-4o alignment	51.9%	50.0%	53.0%

Appendix

Table 9: Model alignment comparison

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization
Aug 12, 2026Ellen Su, Andres Potapczynski, Shikai Qiu +2Synthetic Data Generation OOD Generalization
New York University · Inherent
Synthesis Without Training: An Inference-Only Pipeline for Tabular, Temporal, and Relational Synthetic Data
Sep 29, 2026Zilong Zhao, Abdul Raheem, Jiayu Li +5Constrained Generative Modeling Synthetic Data Generation
Betterdata AI · National University of Singapore · University of Illinois Urbana-Champaign +1
Synthics: Synthetic Physics-like Datasets for Machine Learning
Jun 4, 2026Jari VepsäläinenSynthetic Data Generation Scientific ML
Mechatronics Department of Energy and Mechanical Engineering School of Engineering Aalto University Finland

Aug 12, 2026J/K move · Enter open · S save

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization

Ellen Su, Andres Potapczynski, Shikai Qiu, Edward Hughes+1

New York University · Inherent

Synthetic Data Generation OOD Generalization Synthetic Data Selection

Open paper AskPDF arXiv

Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization in these new, unanticipated settings? One hypothesis is that data with more structural information could contain shared circuits and subprograms that could be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, provides a mechanism to reason about this relationship. In this paper, we show how to operationalize epiplexity as an online training signal for data selection and synthetic data generation. For selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, we define a generator's reward as the change in learner epiplexity over a buffer of previously generated data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning based tasks, supporting the hypothesis that data rich in structural information yield representations that transfer across domains.

Abstract

Synthetic Data Generation

Explore similar work

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization

Synthesis Without Training: An Inference-Only Pipeline for Tabular, Temporal, and Relational Synthetic Data

Synthics: Synthetic Physics-like Datasets for Machine Learning