Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data

Zhenyu Tao; Wei Xu; Xiaohu You; Petar Popovski; Osvaldo Simeone

cs.AIAug 28, 2026

Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data

Authors: Zhenyu Tao, Wei Xu, Xiaohu You, Petar Popovski, Osvaldo Simeone

Organizations: National Mobile Communications Research Lab, Southeast University, Nanjing 210096, China · Pervasive Communication Research Center, Purple Mountain Laboratories, Nanjing 211111, China · Department of Electronic Systems, Aalborg University, 9220 Aalborg, Denmark · Institute for Intelligent Networked Systems, Northeastern University London, E1 8PH London, U.K. ·

Abstract

Digital twins (DTs) and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems. Owing to the inevitable simulation-to-reality (sim-to-real) gap, however, augmentation may fail to improve the performance of the trained model on the real data distribution. This paper addresses the resulting decision problem: Given a real dataset, a candidate synthetic dataset, and a fixed learning algorithm, decide whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible. Two formulations are considered: a direct test on the mean loss difference between the two trained models, and a symmetry-based test on the paired loss difference, which trades a stronger null assumption for faster evidence accumulation. For the latter, we introduce the {adaptive e-process sign-flip test} (aeSFT), a doubly adaptive procedure that adapts both the number of Monte Carlo sign-flip rounds, and hence the computational cost, and the amount of real test data consumed. aeSFT yields anytime-valid Type-I error control, with no need to pre-specify the test-set size. Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing, matches the power of fixed-sample sign-flip testing and the paired $t$ -test, while keeping the false-positive rate below the target level.

Figures & tables

Fig. 1: Illustration of the problem setting: Given a real dataset $\mathcal{D}$ and a synthetic dataset $\mathcal{D}^{\mathrm{syn}}$ , we wish to decide whether training on the augmented dataset $\mathcal{D}\cup\mathcal{D}^{\mathrm{syn}}$ reduces the population loss of the trained model.

Fig. 2: Betting interpretation of SFT [ 27 ] , eSFT [ 29 ] , and aeSFT (this work).

Symbol	Meaning
$P$ , $P^{\mathrm{syn}}$	Real and synthetic data distributions.
$\mathcal{D}$ , $\mathcal{D}^{\mathrm{syn}}$	Real and synthetic training datasets.
$f$ , $f^{\mathrm{syn}}$	Models trained on $\mathcal{D}$ and $\mathcal{D}\cup\mathcal{D}^{\mathrm{syn}}$ .
$\mathbb{L}(f;Z)$	Sample-level loss.
$\mathbb{L}(f)$	Population loss.
$\delta$ , $\Delta$	Paired sample-level loss difference and its population mean.

TABLE I: Notation used throughout the paper.

Fig. 3: Illustration of SFT [ 27 ] , where the test-set size

M

and the number of sign-flip sequences

B

are fixed in advance.

Fig. 4: Illustration of eSFT [ 29 ] : the process stops and rejects the null at the first step

b

for which the e-process satisfies

W^{b}\geq 1/\alpha

. The size of the test dataset,

M

, is fixed in advance.

Fig. 5: Illustration of aeSFT. Within each batch

k

, the test uses a fresh test set

\mathcal{D}_{k}^{\mathrm{test}}

of size

m_{k}

together with an adaptive number

B_{k}

of sign-flip rounds.

Fig. 6: Evidence accumulation versus consumed test data in a test instance where synthetic data are useful (

\Delta>0

). The reciprocal

t

-test p -value is shown only as a visual score. The vertical axis is logarithmic.

Fig. 7: Empirical CDF of stopping time

m_{\mathrm{stop}}

under the alternative hypothesis

H_{1}

(synthetic data are useful) for aMT, eSFT,

t

-test, and aeSFT in the binary classification task.

Fig. 8: TPR and FPR over

1{,}000

independent synthetic data tests with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Fig. 9: aeSFT parameter ablation in the binary classification experiment. Solid curves show TPR and FPR, diamonds show

\mathbb{E}_{H_{1}}[m_{\mathrm{stop}}]

, black dashed lines mark the selected values, and horizontal dotted lines mark

\alpha=0.1

Fig. 10: Three-class packet-scheduling task [ 35 ] . The scheduling policy allocates a shared bandwidth pool for packets to maximize the expected network reward.

Fig. 11: Empirical CDF of stopping time

m_{\mathrm{stop}}

under the alternative hypothesis

H_{1}

(synthetic data are useful) for aMT, eSFT,

t

-test, and aeSFT in the wireless packet scheduling task.

Fig. 12: TPR and FPR versus average consumption of real-network test episodes for the packet-scheduling task, with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Fig. 13: Sionna RT [ 11 ] radio maps for a fixed $512\,\mathrm{m}\times 512\,\mathrm{m}$ scene imported from OpenStreetMap [ 41 ] and a base-station location (star marker) under different reflection depths, i.e., maximum numbers of reflections, and rays/Tx settings.

Fig. 14: Empirical CDF of stopping time

m_{\mathrm{stop}}

under the alternative hypothesis

H_{1}

(synthetic data are useful) for aMT, eSFT,

t

-test, and aeSFT in the radio-map prediction task.

Fig. 15: TPR and FPR over the 196 settings versus the average number of consumed high-fidelity radio maps, with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Fig. 16: TPR and FPR for the IoU metric over the 196 radio-map settings versus the average number of consumed high-fidelity radio maps, with target level

\alpha=0.1

for aMT, eSFT,

t

-test, and aeSFT.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Environment	Arrival rates $\bm{\mu}=(4.0,1.8,0.8)$ ; deadlines $\bm{\tau}=(6,3,1)$ ; bandwidths $\bm{b}=(1,2,4)$ ; rewards $\bm{R}=(1,4,10)$ ; shared capacity $C=10$ .
State and action	Three deadline-bucket buffers with $D_{\max}=8$ and per-bucket cap $N_{\max}=30$ , giving a $24$ -dimensional normalized state; $66$ integer bandwidth allocations satisfying $\sum_{k}u_{k}=C$ .
Episode and dataset	$T=60$ slots; $\|\mathcal{D}\|=10$ real trajectories ( $600$ transitions); $\|\mathcal{D}^{\mathrm{syn}}\|=30$ synthetic trajectories ( $1{,}800$ transitions).
Behavior data policy	Greedy immediate-reward action with probability $0.55$ and a uniformly selected feasible action with probability $0.45$ .
DQN architecture	State and three normalized action counts as input; two fully connected ReLU hidden layers of width $64$ ; scalar Q-value output.
DQN training	Adam with learning rate $10^{-3}$ , Smooth- $L_{1}$ loss, discount $0.97$ , minibatch size $128$ , target update every $25$ gradient steps, and gradient-norm clipping at $5$ . The real-only model uses $120$ updates and the pooled-data model uses $480$ updates, preserving the number of passes over the data.

Appendix

TABLE II: Configuration of the wireless packet-scheduling experiment.

Component	Setting
Spatial grid	A $512\,\mathrm{m}\times 512\,\mathrm{m}$ window with $4\,\mathrm{m}$ cells, giving a two-channel $128\times 128$ input and a one-channel $128\times 128$ RSS prediction.
Input encoding	Channel 1 is the 3D geometry represented as a building-height grid and normalized to $[0,1]$ . Channel 2 is zero everywhere except at the grid cell containing the base station, whose positive value is the base-station height normalized to $[0,1]$ .
Transmitter setting	Carrier frequency $3.5\,\mathrm{GHz}$ ; transmit power $40\,\mathrm{dBm}$ ; $1\times 1$ planar transmit array with pattern=iso and vertical polarization.
Wireless propagation	Radio-map orientation $(0,0,0)$ . Line-of-sight, specular reflection, and diffraction are enabled. All buildings use itu_concrete , and the ground uses itu_very_dry_ground ; material parameters are fixed across all datasets.
Predictor	A compact version of the PMNet encoder–decoder architecture [ 31 ] . The encoder uses a $16$ -channel stem and four bottleneck residual stages with $(1,1,3,1)$ blocks and output widths $(64,128,128,256)$ . Atrous spatial pyramid pooling (ASPP) is used to combine five parallel $64$ -channel branches and projects into $128$ channels. Six skip-connected decoder blocks with widths $(128,128,64,64,64,32)$ reconstruct the $128\times 128$ map; the final head has widths $(34,16,16,1)$ and a sigmoid output.
Training	AdamW with learning rate $3\times 10^{-4}$ , weight decay $10^{-5}$ , batch size $16$ , and $30$ epochs; a ReduceLROnPlateau scheduler is adopted with factor $0.25$ and patience $3$ . All models share the same initialization and training seed.

Appendix

TABLE III: Configuration of the radio-map prediction experiment.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Reasoning-Driven Synthetic Data Generation and Evaluation
Mar 31, 2026Tim R. Davidson, Benoit Seguin, Enrico Bacis +2Synthetic Data Generation Synthetic Data Evaluation
EPFL · Google · Google Deepmind
When Does Training on Synthetic Data Provably Work?
Oct 20, 2026Leroy Odunlami, Tobias Riedlinger, Hanno GottschalkSynthetic-to-Real Domain Adaptation Covariate Shift
Institute of Mathematics, Technische Universität Berlin, Berlin, Germany
Does Synthetic Data Help? Empirical Evidence from Deep Learning Time Series Forecasters
May 7, 2026Hugo Cazaux, Eyjólfur Ingi Ásgeirsson, Hlynur StefánssonSynthetic Data Augmentation Synthetic Data Generation
Department of Engineering Reykjavík University Iceland, Menntavegur 1, 102

Mar 31, 2026J/K move · Enter open · S save

Reasoning-Driven Synthetic Data Generation and Evaluation

Tim R. Davidson, Benoit Seguin, Enrico Bacis, Cesar Ilharco+1

EPFL · Google · Google Deepmind

Synthetic Data Generation Synthetic Data Evaluation

Open paper AskPDF arXiv

Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce or inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data as a scalable alternative. However, existing synthetic data generation methods often rely on manual prompts, evolutionary algorithms, or extensive seed data from the target distribution - limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning-driven framework for data generation and evaluation. It employs a seedless, agentic approach to generate synthetic datasets at scale, allowing users to define desired dataset characteristics through an explainable and controllable process that enables fine-grained resource allocation. We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scale, and (3) unlocks new opportunities for developing and deploying AI in domains where data scarcity or privacy concerns are paramount.

Abstract

Synthetic Data Evaluation

Explore similar work

Reasoning-Driven Synthetic Data Generation and Evaluation

When Does Training on Synthetic Data Provably Work?

Does Synthetic Data Help? Empirical Evidence from Deep Learning Time Series Forecasters