Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization

Yefeng Yuan; Zhan Shi; Liang Cheng; Yuhong Liu

cs.CLAug 26, 2026

Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization

Authors: Yefeng Yuan, Zhan Shi, Liang Cheng, Yuhong Liu

Organizations: Santa Clara University Santa Clara, USA · eBay Inc. San Jose, USA

Abstract

Multimodal synthetic datasets combine structured attributes with free text, but are often evaluated separately. Such metrics can remain high after tabular--text pairings are disrupted. We present a projection-based evaluator for tabular--text synthetic data. A fixed sentence encoder maps text to embeddings, $k$ -means converts them to cluster states, and tabular variables are represented as categorical or quantile-binned states. Real and synthetic contingency tables are compared using Jensen--Shannon divergence (JSD), normalized mutual information (NMI), conditional JSD (cJSD), and joint-state entropy. We also report text-to-attribute (T2A) utility and a holdout-calibrated proximity flag rate (PFR) as a representation-level diagnostic. A text-permutation control preserves both marginal distributions while disrupting their pairing. Experiments on Amazon Reviews, Kiva Loans, and the Employment Scam Aegean Dataset show that modality-specific scores remain high under this control. The projection diagnostics detect disruption when real projected dependence exceeds a permutation baseline, but are less informative for weak or sparse projections. Some conditioned LLM baselines also exhibit stronger measured dependence than the corresponding real-data projections. These results support explicit cross-modal evaluation with permutation baselines and coverage reporting.

Figures & tables

Fig. 1: The “stitching assumption” in multimodal synthetic data evaluation. Separate tabular and text metrics can assign high scores even when the two components of the same record contradict each other.

Fig. 2: Architectural overview of the proposed framework. Semantic quantization maps text embeddings and selected tabular features into a joint discrete representation, enabling coarse-grained joint evaluation of multimodal synthetic data across fidelity, dependence, utility, diversity, and record proximity.

Dataset	Method	Tabular marginal	BERTScore F1	Embedding cosine
Amazon	$R$	1.000	1.000	1.000
	$S_{\mathrm{tilt}}$	1.000	0.840	–
	$S_{\mathrm{ind}}$	0.510	0.844	–
	$S_{\mathrm{seq}}$	0.510	0.852	–
	$S_{\mathrm{joint}}$	0.635	0.846	–
	TabSyn-SBERT	0.794	–	0.441

TABLE I: Modality-specific baseline scores. Higher values indicate greater similarity within each individual metric column. BERTScore-F1 and embedding cosine use different inputs and scales and must not be compared across columns.

Method	Projected JSD
$R$	0.0000
$S_{\mathrm{tilt}}$	0.1017
$S_{\mathrm{ind}}$	0.5778
$S_{\mathrm{seq}}$	0.7369
$S_{\mathrm{joint}}$	0.5865
TabSyn-SBERT	0.3219

TABLE II: Projected JSD for the Amazon rating–text boundary case. Lower is closer to the real reference; weak real dependence limits the permutation control’s discriminative power.

Dataset	Method	Proj. JSD $\downarrow$	NMI ratio / $\Delta_{\mathrm{NMI}}$	cJSD $\downarrow$	T2A $\uparrow$	$H_{\mathrm{rel}}$	PFR 1%
Kiva	$R$	0.000	1.00 / 0.000	0.000	–	1.000	–
	$S_{\mathrm{tilt}}$	0.698	0.016 / 0.600	0.687	0.076	1.432	0.186
	$S_{\mathrm{ind}}$	0.844	0.075 / 0.565	0.773	0.062	1.601	0.000
	TabSyn-SBERT	0.727	0.791 / 0.128	0.363 †	0.427	1.093	0.000
	$S_{\mathrm{seq}}$	0.602	0.948 / 0.032	0.448 ‡	0.345	1.120	0.010
	$S_{\mathrm{joint}}$	0.729	1.100 / -0.061	0.623	0.331	0.957	0.000

TABLE III: Projection-level results for Kiva and Fake Jobs. Lower JSD/cJSD and smaller

\lvert\Delta_{\mathrm{NMI}}\rvert

are preferred; higher T2A is better;

H_{\mathrm{rel}}

is interpreted relative to 1. PFR is a proximity flag rate, not a quality or privacy score. NMI is ratio / signed gap. Bold marks best directly comparable synthetic values; partial-coverage cJSD is excluded.

Method	Pos. F1	Balanced Acc.	AUROC	AUPRC	MCC	Min. Recall
Majority	0.000	0.500	–	–	–	0.000
$S_{\mathrm{ind}}$	0.372	0.604	0.604	0.286	0.166	0.741
$S_{\mathrm{seq}}$	0.343	0.598	0.622	0.401	0.239	0.277
$S_{\mathrm{joint}}$	0.341	0.599	0.611	0.314	0.260	0.259
TabSyn-SBERT	0.555	0.760	0.845	0.679	0.439	0.741
$S_{\mathrm{tilt}}$	0.215	0.462	0.463	0.185	-0.061	0.321

TABLE IV: Fake Jobs T2A utility under class imbalance. Higher is better for all reported utility metrics. Bold values mark the best value in each metric column among non-reference methods; ties are bolded. Min. Recall denotes minority-class recall. Values are rounded; “–” denotes a score-based metric that was not computed for the hard-label majority baseline.

Dataset	$\lambda^{\star}$	$\tau_{5\%}$	$\tau_{10\%}$	Max learned PFR	Tilted PFR
Amazon	1.138	0.553	0.643	0.000 / 0.014 / 0.032	0.388 / 0.490 / 0.508
Kiva	1.315	0.000	0.000	0.010 / 0.010 / 0.010	0.186 / 0.186 / 0.186
Fake Jobs	1.165	0.000	0.000	0.000 / 0.000 / 0.000	0.580 / 0.580 / 0.580

TABLE V: Holdout-calibrated DCR thresholds and PFR values. Thresholds are shown to three decimals, but PFR is computed before rounding using

\tilde{\tau}_{\alpha}=\max(\tau_{\alpha},\delta_{\mathrm{num}})

. For Kiva and Fake Jobs, the displayed

\tau_{5\%}

and

\tau_{10\%}

remain 0.000 under the selected DCR representation, so PFR 1% , PFR 5% , and PFR 10% coincide.

Check	Setting	Main finding
$K$ -sensitivity	$K\in\{5,10,15,20,30\}$	Fake Jobs remains stable, with tilted-to-TabSyn-SBERT JSD ratios from 2.54 to 1.49. Amazon remains weak because rating–text dependence is small. Kiva is sparse under the primary sector projection, with occupied-cell ratios 20–51% and minimum cell count 1.
$\lambda$ -sensitivity	$\lambda/\lambda^{\star}\in\{0.25,0.5,1,2,4\}$	Raw DCR changes with $\lambda$ , but PFR-based flagging pattern is unchanged across this 16-fold range: learned baselines remain unflagged or nearly unflagged, while $S_{\mathrm{tilt}}$ remains flagged.
XOR/parity stress test	$X_{1},X_{2},X_{3}\in\{0,1\}$ , $C_{T}=X_{1}\oplus X_{2}\oplus X_{3}$ ; $S_{\mathrm{tilt}}$ shuffles $C_{T}$ while preserving the tabular marginal	Each pair $(X_{j},C_{T})$ is marginally independent, so the mean pairwise JSD remains near zero in the finite-sample evaluation (0.0075). Shuffling destroys the parity support constraint, and the targeted 3-way projection detects the violation with empirical JSD 0.5542. This value is a finite-sample diagnostic statistic rather than the closed-form JSD of an ideal uniform parity distribution.

TABLE VI: Robustness and boundary-condition summary.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Mar 10, 2026

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.

Lukáš Eigler, Jindřich Libovický, David Hurych

Faculty of Mathematics and Physics, Charles University, Czech Republic

Synthetic Data Evaluation Synthetic Benchmark Generation Multilingual Language Model Evaluation

Ask PDF

May 18, 2026

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Synthetic data is widely used in healthcare to create datasets that preserve statistical properties of real data without exposing sensitive patient information. Generating and evaluating synthetic data across privacy, utility, and fairness dimensions is crucial for enabling high-quality data availability in downstream prediction tasks and clinical decision making. We present Memisis, a tool that orchestrates and evaluates synthetic data by leveraging existing synthesis libraries, large language models (LLMs), and state-of-the-art evaluation metrics. Our tool creates a unified workflow for data generation, validation, and evaluation. Users can control training size, training epochs, and the number of synthetic rows to sample. Beyond manual configuration, an interactive agent mode allows users to specify data generation goals in natural language, and the tool orchestrates the full pipeline by invoking existing synthesizers while performing the requisite evaluation. For the demo, we use an open-source schizophrenia dataset with protected attributes related to race and gender, evaluate six synthesizers spanning GANs, VAEs, diffusion models, and normalizing flows, and use a local LLM to orchestrate the workflow. The system affords users flexibility and control over the data generation and evaluation process.

Nitish Nagesh, Pengbao Zhou, Atchuth Naveen Chilaparasetti +8

University of California, Irvine, CA, USA · PathAI, USA

Healthcare Synthetic Tabular Data Generation Privacy-Preserving ML Synthetic Data Evaluation Synthetic Clinical Data Generation

Ask PDF

Mar 31, 2026

Reasoning-Driven Synthetic Data Generation and Evaluation

Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce or inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data as a scalable alternative. However, existing synthetic data generation methods often rely on manual prompts, evolutionary algorithms, or extensive seed data from the target distribution - limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning-driven framework for data generation and evaluation. It employs a seedless, agentic approach to generate synthetic datasets at scale, allowing users to define desired dataset characteristics through an explainable and controllable process that enables fine-grained resource allocation. We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scale, and (3) unlocks new opportunities for developing and deploying AI in domains where data scarcity or privacy concerns are paramount.

Tim R. Davidson, Benoit Seguin, Enrico Bacis +2

EPFL · Google · Google Deepmind

Synthetic Data Generation Synthetic Data Evaluation

Ask PDF

Abstract

Synthetic Data Evaluation

Explore similar work

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Reasoning-Driven Synthetic Data Generation and Evaluation