Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data

Zhenyu Tao; Wei Xu; Xiaohu You; Petar Popovski; Osvaldo Simeone

cs.AIAug 28, 2026

Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data

Authors: Zhenyu Tao, Wei Xu, Xiaohu You, Petar Popovski, Osvaldo Simeone

Organizations: National Mobile Communications Research Lab, Southeast University, Nanjing 210096, China · Pervasive Communication Research Center, Purple Mountain Laboratories, Nanjing 211111, China · Department of Electronic Systems, Aalborg University, 9220 Aalborg, Denmark · Institute for Intelligent Networked Systems, Northeastern University London, E1 8PH London, U.K. ·

Abstract

Digital twins (DTs) and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems. Owing to the inevitable simulation-to-reality (sim-to-real) gap, however, augmentation may fail to improve the performance of the trained model on the real data distribution. This paper addresses the resulting decision problem: Given a real dataset, a candidate synthetic dataset, and a fixed learning algorithm, decide whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible. Two formulations are considered: a direct test on the mean loss difference between the two trained models, and a symmetry-based test on the paired loss difference, which trades a stronger null assumption for faster evidence accumulation. For the latter, we introduce the {adaptive e-process sign-flip test} (aeSFT), a doubly adaptive procedure that adapts both the number of Monte Carlo sign-flip rounds, and hence the computational cost, and the amount of real test data consumed. aeSFT yields anytime-valid Type-I error control, with no need to pre-specify the test-set size. Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing, matches the power of fixed-sample sign-flip testing and the paired $t$ -test, while keeping the false-positive rate below the target level.

Figures & tables

Fig. 1: Illustration of the problem setting: Given a real dataset $\mathcal{D}$ and a synthetic dataset $\mathcal{D}^{\mathrm{syn}}$ , we wish to decide whether training on the augmented dataset $\mathcal{D}\cup\mathcal{D}^{\mathrm{syn}}$ reduces the population loss of the trained model.

Fig. 2: Betting interpretation of SFT [ 27 ] , eSFT [ 29 ] , and aeSFT (this work).

Symbol	Meaning
$P$ , $P^{\mathrm{syn}}$	Real and synthetic data distributions.
$\mathcal{D}$ , $\mathcal{D}^{\mathrm{syn}}$	Real and synthetic training datasets.
$f$ , $f^{\mathrm{syn}}$	Models trained on $\mathcal{D}$ and $\mathcal{D}\cup\mathcal{D}^{\mathrm{syn}}$ .
$\mathbb{L}(f;Z)$	Sample-level loss.
$\mathbb{L}(f)$	Population loss.
$\delta$ , $\Delta$	Paired sample-level loss difference and its population mean.

TABLE I: Notation used throughout the paper.

Fig. 3: Illustration of SFT [ 27 ] , where the test-set size

M

and the number of sign-flip sequences

B

are fixed in advance.

Fig. 4: Illustration of eSFT [ 29 ] : the process stops and rejects the null at the first step

b

for which the e-process satisfies

W^{b}\geq 1/\alpha

. The size of the test dataset,

M

, is fixed in advance.

Fig. 5: Illustration of aeSFT. Within each batch

k

, the test uses a fresh test set

\mathcal{D}_{k}^{\mathrm{test}}

of size

m_{k}

together with an adaptive number

B_{k}

of sign-flip rounds.

Fig. 6: Evidence accumulation versus consumed test data in a test instance where synthetic data are useful (

\Delta>0

). The reciprocal

t

-test p -value is shown only as a visual score. The vertical axis is logarithmic.

Fig. 7: Empirical CDF of stopping time

m_{\mathrm{stop}}

under the alternative hypothesis

H_{1}

(synthetic data are useful) for aMT, eSFT,

t

-test, and aeSFT in the binary classification task.

Fig. 8: TPR and FPR over

1{,}000

independent synthetic data tests with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Fig. 9: aeSFT parameter ablation in the binary classification experiment. Solid curves show TPR and FPR, diamonds show

\mathbb{E}_{H_{1}}[m_{\mathrm{stop}}]

, black dashed lines mark the selected values, and horizontal dotted lines mark

\alpha=0.1

Fig. 10: Three-class packet-scheduling task [ 35 ] . The scheduling policy allocates a shared bandwidth pool for packets to maximize the expected network reward.

Fig. 11: Empirical CDF of stopping time

m_{\mathrm{stop}}

under the alternative hypothesis

H_{1}

(synthetic data are useful) for aMT, eSFT,

t

-test, and aeSFT in the wireless packet scheduling task.

Fig. 12: TPR and FPR versus average consumption of real-network test episodes for the packet-scheduling task, with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Fig. 13: Sionna RT [ 11 ] radio maps for a fixed $512\,\mathrm{m}\times 512\,\mathrm{m}$ scene imported from OpenStreetMap [ 41 ] and a base-station location (star marker) under different reflection depths, i.e., maximum numbers of reflections, and rays/Tx settings.

Fig. 14: Empirical CDF of stopping time

m_{\mathrm{stop}}

under the alternative hypothesis

H_{1}

(synthetic data are useful) for aMT, eSFT,

t

-test, and aeSFT in the radio-map prediction task.

Fig. 15: TPR and FPR over the 196 settings versus the average number of consumed high-fidelity radio maps, with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Fig. 16: TPR and FPR for the IoU metric over the 196 radio-map settings versus the average number of consumed high-fidelity radio maps, with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Environment	Arrival rates $\bm{\mu}=(4.0,1.8,0.8)$ ; deadlines $\bm{\tau}=(6,3,1)$ ; bandwidths $\bm{b}=(1,2,4)$ ; rewards $\bm{R}=(1,4,10)$ ; shared capacity $C=10$ .
State and action	Three deadline-bucket buffers with $D_{\max}=8$ and per-bucket cap $N_{\max}=30$ , giving a $24$ -dimensional normalized state; $66$ integer bandwidth allocations satisfying $\sum_{k}u_{k}=C$ .
Episode and dataset	$T=60$ slots; $\|\mathcal{D}\|=10$ real trajectories ( $600$ transitions); $\|\mathcal{D}^{\mathrm{syn}}\|=30$ synthetic trajectories ( $1{,}800$ transitions).
Behavior data policy	Greedy immediate-reward action with probability $0.55$ and a uniformly selected feasible action with probability $0.45$ .
DQN architecture	State and three normalized action counts as input; two fully connected ReLU hidden layers of width $64$ ; scalar Q-value output.
DQN training	Adam with learning rate $10^{-3}$ , Smooth- $L_{1}$ loss, discount $0.97$ , minibatch size $128$ , target update every $25$ gradient steps, and gradient-norm clipping at $5$ . The real-only model uses $120$ updates and the pooled-data model uses $480$ updates, preserving the number of passes over the data.

Appendix

TABLE II: Configuration of the wireless packet-scheduling experiment.

Component	Setting
Spatial grid	A $512\,\mathrm{m}\times 512\,\mathrm{m}$ window with $4\,\mathrm{m}$ cells, giving a two-channel $128\times 128$ input and a one-channel $128\times 128$ RSS prediction.
Input encoding	Channel 1 is the 3D geometry represented as a building-height grid and normalized to $[0,1]$ . Channel 2 is zero everywhere except at the grid cell containing the base station, whose positive value is the base-station height normalized to $[0,1]$ .
Transmitter setting	Carrier frequency $3.5\,\mathrm{GHz}$ ; transmit power $40\,\mathrm{dBm}$ ; $1\times 1$ planar transmit array with pattern=iso and vertical polarization.
Wireless propagation	Radio-map orientation $(0,0,0)$ . Line-of-sight, specular reflection, and diffraction are enabled. All buildings use itu_concrete , and the ground uses itu_very_dry_ground ; material parameters are fixed across all datasets.
Predictor	A compact version of the PMNet encoder–decoder architecture [ 31 ] . The encoder uses a $16$ -channel stem and four bottleneck residual stages with $(1,1,3,1)$ blocks and output widths $(64,128,128,256)$ . Atrous spatial pyramid pooling (ASPP) is used to combine five parallel $64$ -channel branches and projects into $128$ channels. Six skip-connected decoder blocks with widths $(128,128,64,64,64,32)$ reconstruct the $128\times 128$ map; the final head has widths $(34,16,16,1)$ and a sigmoid output.
Training	AdamW with learning rate $3\times 10^{-4}$ , weight decay $10^{-5}$ , batch size $16$ , and $30$ epochs; a ReduceLROnPlateau scheduler is adopted with factor $0.25$ and patience $3$ . All models share the same initialization and training seed.

Appendix

TABLE III: Configuration of the radio-map prediction experiment.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Mar 31, 2026

Reasoning-Driven Synthetic Data Generation and Evaluation

Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce or inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data as a scalable alternative. However, existing synthetic data generation methods often rely on manual prompts, evolutionary algorithms, or extensive seed data from the target distribution - limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning-driven framework for data generation and evaluation. It employs a seedless, agentic approach to generate synthetic datasets at scale, allowing users to define desired dataset characteristics through an explainable and controllable process that enables fine-grained resource allocation. We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scale, and (3) unlocks new opportunities for developing and deploying AI in domains where data scarcity or privacy concerns are paramount.

Tim R. Davidson, Benoit Seguin, Enrico Bacis +2

EPFL · Google · Google Deepmind

Synthetic Data Generation Synthetic Data Evaluation

Ask PDF

Oct 20, 2026

When Does Training on Synthetic Data Provably Work?

High-quality training data are a major bottleneck in machine learning, motivating increasing use of artificially generated data. We study when a binary classifier trained on synthetic data transfers successfully to a real distribution, under a shared conditional label distribution. First, we derive finite-sample excess-error bounds when the real covariate distribution is absolutely continuous with respect to the synthetic distribution under a controlled density ratio. Second, we study the partially and fully singular case, where the real distribution has mass outside the synthetic one. If the hypothesis class is sufficiently expressive, synthetic risk alone cannot uniformly control real-world error: a classifier can be nearly optimal on synthetic data while approaching worst-case error on the uncovered real region, even when the distributions are arbitrarily close in Wasserstein distance. For function classes where hypotheses and the log-odds of the conditional label probability are Lipschitz, we recover a bound in terms of the probability that a real sample lies outside the reach of the synthetic data, restoring a transfer guarantee even in the fully singular case. Experiments confirm this in controlled low-dimensional settings and on 32,200 images generated as counterparts to real ImageNet images by four state-of-the-art generators: an adversarial classifier reaches 0.5% error on the synthetic domain but 99.25% error on the real domain, while ordinarily trained architectures show positively correlated synthetic and real test errors. Synthetic data should be budgeted at high sample counts and models validated on real data.

Leroy Odunlami, Tobias Riedlinger, Hanno Gottschalk

Institute of Mathematics, Technische Universität Berlin, Berlin, Germany

Synthetic-to-Real Domain Adaptation Covariate Shift Distribution Matching Synthetic Data Generation

Ask PDF

May 7, 2026

Does Synthetic Data Help? Empirical Evidence from Deep Learning Time Series Forecasters

Synthetic data has transformed language model training, yet its role in time series forecasting remains poorly understood. We present a large-scale empirical study: nine experiment groups, 4,218 runs systematically evaluating synthetic time series augmentation across five architectures, four synthetic signals and seven datasets. The effect is sharply architecture-conditional: channel-mixing models (TimesNet, iTransformer) benefit in the majority of trials, while channel-independent models (DLinear, PatchTST) are consistently degraded. In selected low-resource settings the gains are striking: TimesNet trained on only 10% of Weather data with synthetic augmentation surpasses the full-data baseline (4 of 16 sparsity-dataset combinations). Averaged across all architectures, augmentation hurts in 67% of trials. We further find that only the Seasonal-Trend generator reliably helps across the tested benchmarks, and that hard curriculum switching is actively harmful (+24% MSE degradation). These results provide concrete, actionable guidelines on how to use synthetic data: use synthetic augmentation with channel-mixing architectures, use gradual annealing schedules, and treat low-resource augmentation as architecture- and dataset-dependent. Code is available at \href{https://github.com/hugoiscracked/synthetic-ts/tree/main}

Hugo Cazaux, Eyjólfur Ingi Ásgeirsson, Hlynur Stefánsson

Department of Engineering Reykjavík University Iceland, Menntavegur 1, 102

Synthetic Data Augmentation Synthetic Data Generation Time Series Forecasting Time Series Data Augmentation Synthetic Data Evaluation

Ask PDF

Abstract

Synthetic Data Evaluation

Explore similar work

Reasoning-Driven Synthetic Data Generation and Evaluation

When Does Training on Synthetic Data Provably Work?

Does Synthetic Data Help? Empirical Evidence from Deep Learning Time Series Forecasters