Trustworthy synthetic data for campaign decision support: strategy simulation fidelity and the PolicySynth framework
Authors: Tung Dang, The Hung Phung, Son Lam Nguyen, Tu Nguyen
Organizations: aGraduate School of Science, The University of Tokyo, 7-3-1 Hongo Bunkyo-ku, Tokyo, 113-0033, Japan · bFaculty of Human Resource Management, Trade Union University, 169 Tay Son, Kim Lien, Hanoi, Vietnam · cInstitute for Business Management Development, Academy Of Finance, 58 Le Van Hien Street, Dong Ngac Ward, Hanoi, Vietnam · dMinistry of Industry and Trade, 23 Ngo Quyen, Hoan Kiem, Hanoi, Vietnam
Abstract
Decision support systems (DSS) increasingly run retention what-if analysis on synthetic customer populations, because privacy constraints preclude unrestricted use of real data. Such a system is trustworthy only if the synthetic data lead managers to the same decisions as the real data would; yet prevailing criteria certify distributional similarity, not decision alignment, so a synthetic population can match every marginal distribution while still steering a marketing team toward the wrong campaigns. We close this decision-alignment gap with three contributions: strategy simulation fidelity (SSF), a criterion measuring how often the synthetic population yields the same go/no-go campaign decision as the real population; PolicySynth, a DSS framework whose generator is conditioned on the production churn scorer to align decision-relevant structure; and a three-axis reporting standard of decision alignment, membership-inference resistance, and novel-record rate as the minimum deployment quality gate. On a telecommunications churn corpus and a banking acquisition corpus, PolicySynth attains a mean SSF of 0.923 and 0.960, with seed-to-seed variance roughly ten times tighter than CTGAN on telecommunications and 2.5 times on banking. This stability is the deployable property: go/no-go recommendations shift by at most 1.2 percentage points between monthly retraining cycles, against 11.5 for CTGAN, a reversed recommendation on one campaign in nine. A bootstrap baseline matches PolicySynth on SSF yet copies real records verbatim and fails membership inference, evidence that no single axis suffices. PolicySynth reliably supports directional go/no-go screening; its ROI estimates diverge from real outcomes by 70 to 78% and require the volume correction we document.
Synthetic data is widely used in healthcare to create datasets that preserve statistical properties of real data without exposing sensitive patient information. Generating and evaluating synthetic data across privacy, utility, and fairness dimensions is crucial for enabling high-quality data availability in downstream prediction tasks and clinical decision making. We present \textbf{Memisis}, a tool that orchestrates and evaluates synthetic data by leveraging existing synthesis libraries, large language models (LLMs), and state-of-the-art evaluation metrics. Our tool creates a unified workflow for data generation, validation, and evaluation. Users can control training size, training epochs, and the number of synthetic rows to sample. Beyond manual configuration, an interactive agent mode allows users to specify data generation goals in natural language, and the tool orchestrates the full pipeline by invoking existing synthesizers while performing the requisite evaluation. For the demo, we use an open-source schizophrenia dataset with protected attributes related to race and gender, evaluate six synthesizers spanning GANs, VAEs, diffusion models, and normalizing flows, and use a local LLM to orchestrate the workflow. The system affords users flexibility and control over the data generation and evaluation process.
Synthetic and distilled student data are increasingly used to enable privacy-conscious learning analytics, yet their suitability for decision-facing institutional support remains uncertain. In dropout support, generated data must preserve not only predictive utility or distributional resemblance, but also the financial-status evidence used to guide advising, payment-plan assistance, and scholarship-related decisions. Method: This study introduces CaP-Eval, a decision-facing causal-privacy audit workflow for evaluating generated student data under a fixed estimand, timing-aware adjustment design, estimator set, and empirical privacy-governance screen. The workflow compares original, distilled, adversarial synthetic, statistical synthetic, and DPGNet privacy-oriented generated data on predictive utility, treatment-effect fidelity, robustness to alternative estimators, and local training-record proximity. Results: DPGNet and distilled data preserved the original financial-status treatment-effect structure more reliably than the adversarial and Gaussian Copula baselines. DPGNet preserved full direction and rank agreement across epsilon levels; epsilon = 10 produced the smallest non-original IPW and DML deviations, while epsilon = 1 and epsilon = 5 amplified several financial-status contrasts. Distilled data remained highly faithful but retained the strongest local training-record proximity signal. TabularGNet preserved qualitative directions with moderate attenuation, and Gaussian Copula compressed effect magnitudes. Conclusions: Predictive utility, privacy orientation, empirical disclosure signals, and causal fidelity diverged; generated student data require joint audits of direction, magnitude, overlap, and release-governance risk before decision use.
We revisit the fairness notion of disparate impact for synthetic data generation (SDG), that assesses whether the utility of generated records is the same across sensitive groups. Our approach departs from existing work on fair SDG, that address the problem of correcting for undue biases in the observed distribution, hence redefining SDG as learning a distribution that is not that of the real data. By contrast, non-disparate impact is notably achieved when the synthetic and real distributions are the same. We expose reasons why SDG may fail to reach that solution and discuss why approximation and estimation errors occur and can be disparate across groups. We notably look into the expressive power of SDG methods relative to distribution complexity, sampling errors due to group proportions, and estimation errors induced by differential privacy mechanisms. We illustrate cases of disparate impact on both artificial and real-world data, focusing on SDG methods that rely on probabilistic graphical models. We also introduce a strategy of learning group-wise SDG models, that constitutes a simple baseline for analyzing and in some settings mitigating cases of disparate impact.