This paper aims to address the challenge of data generation beyond the training data and proposes a framework for Structural Extrapolated Data GEneration (SEDGE) based on suitable assumptions on the underlying data-generating process. We provide conditions under which data satisfying novel specifications can be generated reliably, together with the approximate identifiability of the distribution of such data under certain ``conservative" assumptions, as well as the inherent non-identifiability of this distribution without such assumptions. On the algorithmic side, we develop practical methods to achieve extrapolated data generation, based on a structure-informed optimization strategy or diffusion posterior sampling, respectively. We verify the extrapolation performance on synthetic data and also consider extrapolated image generation as a real-world scenario to illustrate the validity of the proposed framework.
Figures & tables
Figure 1: The two generating processes as initial thoughts.
Figure 2: The generating processes where features generate the specification and the observed data satisfy the constraint that the selection variable S=1 .
Figure 3: Synthetic experiment results with given X and Z . Panels (a–b) illustrate the data split in a two-dimensional view over the X1 and X3 axes. The novel specifications induce a previously unseen joint distribution over (X1,X3) , ensuring that successful performance requires true extrapolation rather than interpolation. Panels (c–h) present the optimization-based generation (OPT) results for all five models. Panel (h) shows the three-dimensional view of the Oracle X . Panels (i–p) visualize the three-dimensional generated X using OPT and diffusion posterior sampling (DPS) for models A, B, and C, and OPT for models D and E, with the corresponding MMD values to the Oracle X reported.
Figure 4: Comparison of image generation results under different compositional prompts. Baseline models include SANA ( Xie et al., 2024 ) , Aligner ( Xie et al., 2025b ) , Stable Diffusion 3.5-Large ( Esser et al., 2024 ) , QwenImage ( Wu et al., 2025 ) , Z-Image ( Cai et al., 2025 ) , and GPT5.2. w/o Spar and w/o CI denote SEDGE without sparsity loss and conditional independence structure, respectively. Each row displays generated image for a prompt, and each column presents results from either SEDGE or baseline models. Despite using only 1.6B parameters, SEDGE achieves performance comparable to GPT5.2 and outperforms the 20B QwenImage model, demonstrating its effectiveness for extrapolation. Without the proposed sparsity and conditional independence structure, the model struggles to identify distinct concepts, often producing entangled representations (e.g., a full bird instead of a bird wing) and failing to adhere to the prompt.
Figure 6: Complete 2-d view of the extrapolated data points compared with baselines (model B, C, D, E) on optimization-based generation method
Figure 7: Complete 2-d view of the extrapolated data points compared with baselines (model B, C) on diffusion posterior sampling method
Figure 8: Identification of Specifications ( Z[0] and Z[1] corresponds to Z1 and Z2 , respectively.)
Figure 9: Identification of Features ( X[0] , X[1] , and X[2] correspond to X1 , X2 , and X3 , respectively.)
X1
X2
X3
Z1
Z2
0.4346
0.9017
0.7097
0.9088
0.9393
Appendix
Table 1: MCC scores of features and specifications for verification of identifiability
MMD (OPT)
MMD (DPS)
0.0332±0.0024
0.0346±0.0029
Appendix
Table 2: MMD scores of extrapolated generation by the two proposed methods
Method
(A) Original ( X→Z )
(B) Z→X , no shared
(C) Z→X , shared
OPT
0.0394±0.0068
0.0389±0.0090
0.0588±0.0106
DPS
0.0362±0.0024
0.0405±0.0122
0.0856±0.0247
Appendix
Table 3: MMD under causal-direction violations. (A) is the original setting; (B) and (C) reverse the direction to Z→X , without and with shared features, respectively.
Method
Original
Z1→Z2 added
Dense graph
OPT
0.0394±0.0068
0.1033±0.0150
0.0595±0.0132
DPS
0.0362±0.0024
0.0973±0.0002
0.0738±0.0243
Appendix
Table 4: MMD under violations of conditional independence among Z . “Dense” refers to a fully connected graph among Z variables.
Model variant
DPS MMD
Full model
0.0468±0.0133
Without sparsity
0.0472±0.0151
Without cond. independence
0.0690±0.0237
Without sparsity or cond. indep.
0.0699±0.0240
Appendix
Table 5: Ablation study: DPS MMD for four model variants removing conditional independence (Cond. Indep.) and/or sparsity from the estimated model.
Method
Test 0
Test 1
Test 2
Test 3
Test 4
OPT
0.0554±0.0269
0.0490±0.0230
0.0510±0.0169
0.0551±0.0253
0.0425±0.0164
DPS
0.0431±0.0121
0.0578±0.0207
0.0672±0.0207
0.0570±0.0204
0.0668±0.0178
Appendix
Table 6: MMD across five settings of increasing violation of the conservative assumption (Assumption 3). Tests 0–4 correspond to increasing severity of violation.
Method
Original
Remove X1→Z1
Add X1→Z2
Remove + Add
OPT
0.0346±0.0090
0.0734±0.0150
0.0385±0.0096
0.0505±0.0133
DPS
0.0374±0.0048
0.0380±0.0040
0.0370±0.0049
0.0381±0.0045
Appendix
Table 7: MMD under misspecified X – Z graph structures.
Model configuration
OPT MMD
DPS MMD
Deterministic AE
0.0295±0.0037
0.0345±0.0031
Frozen extractor (pretrained VAE)
0.0281±0.0032
0.0316±0.0073
Linear extractor
0.0806±0.0048
0.0557±0.0078
DDPM prior
0.0307±0.0040
0.0351±0.0037
Flow Matching prior
0.0307±0.0040
0.0310±0.0026
Appendix
Table 8: The MMD by OPT and DPS for different extractor and prior choices. “Frozen extractor” denotes a pretrained VAE extractor kept fixed during likelihood training.
Figure 10: More qualitative comparisons under different prompts.
Figure 11: Consistency of our generation. Given prompt a camera with insect atennae , we generated figures using both ours and concept aligner models acorss 8 different seeds. As we can see in the figure, the generation of concept aligner has mixed up concepts like legs and bodies. Our generation results (especially with larger guidance values) makes consistent generation.
Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization in these new, unanticipated settings? One hypothesis is that data with more structural information could contain shared circuits and subprograms that could be recycled in a wider array of downstream settings. Epiplexity, a recently proposed measure of the structural information a compute-bounded learner can extract from data, provides a mechanism to reason about this relationship. In this paper, we show how to operationalize epiplexity as an online training signal for data selection and synthetic data generation. For selection, we fit scaling laws to the training loss curves of natural data domains to predict the expected epiplexity gain as a function of training tokens, and use this signal to adaptively determine the sampling weights over domains during training. For synthetic data generation, we define a generator's reward as the change in learner epiplexity over a buffer of previously generated data and use REINFORCE policy gradients to guide the generator toward an epiplexity-maximizing distribution. In both cases, higher epiplexity predicts improved downstream performance on zero-shot and fine-tuning based tasks, supporting the hypothesis that data rich in structural information yield representations that transfer across domains.
Synthetic data generation is dominated by the fit-then-sample paradigm: a generative model is trained on a private dataset and then sampled from. Despite its widespread adoption, this paradigm faces three challenges: (1) a new training run is required for every dataset; (2) different data modalities, such as single tables, time series, and relational databases, require task-specific models and feature engineering; and (3) the resulting model is opaque, making its behavior under data constraints difficult to inspect. We propose GENSCRIPT, an inference-only pipeline that eliminates model training. GENSCRIPT computes a deterministic statistical profile of the source data (column types, ranges, missingness, categories, correlations, etc.) and passes it--rather than raw rows--to a language model to infer field semantics and cross-column integrity constraints. A coding agent then compiles the profile and constraints into an executable, auditable sampler. This unified approach supports single-table, temporal, and relational data without task-specific modeling. Across four single-table benchmarks, GENSCRIPT builds generators in 2 minutes and samples 50k rows within 6 seconds, while remaining within a few points of leading methods in marginal fidelity. Notably, it is the only method that perfectly preserves a 1-to-1 mapping between columns in the Adult dataset. On a smart-building dataset, it produces conditional time series that more closely match the real distribution than two baselines and perfectly preserves primary- and foreign-key relationships in the corresponding relational database.
Zilong Zhao, Abdul Raheem, Jiayu Li +5
Betterdata AI · National University of Singapore · University of Illinois Urbana-Champaign +1
Representative data is fundamental in machine learning, as limited data hinders generalisation. Collecting sufficient real-world samples is often infeasible. Synthetic data generation offers a practical solution, but only if the generated data faithfully reflects the structure of real observations. In this paper, a method for generating synthetic regression datasets that structurally resemble physics equations from a given equation corpus is presented. The approach uses a Bayesian Probabilistic Context-Free Grammar to capture the underlying algebraic structure of the corpus, from which novel equations are sampled. To ensure the generated inputs lie within a physically meaningful domain, the applicability domain is characterised for each equation through non-intrusive probing, also recovering inter-variable constraints. Input sampling further mimics realistic experimental conditions by drawing from random sub-ranges of the valid domain with mixed uniform and truncated normal distributions. The generated data is statistically validated against the Feynman equation corpus using Kolmogorov-Smirnov tests. The generated equations match the corpus on all of the eight studied structural features, compared to only two for an unsmoothed purely probabilistic grammar, demonstrating that the Bayesian prior is essential for structural fidelity given the size of the corpus. In a downstream hyperparameter-tuning task, a gradient-boosted regressor tuned on the synthetic data picks, on average, the 6th-best configuration out of 20 on real data, matching the result of tuning on real data itself and substantially outperforming random expression trees (10th) and noise (19th).
Jari Vepsäläinen
Mechatronics Department of Energy and Mechanical Engineering School of Engineering Aalto University Finland