Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization

Yefeng Yuan; Zhan Shi; Liang Cheng; Yuhong Liu

cs.CLAug 26, 2026

Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization

Authors: Yefeng Yuan, Zhan Shi, Liang Cheng, Yuhong Liu

Organizations: Santa Clara University Santa Clara, USA · eBay Inc. San Jose, USA

Abstract

Multimodal synthetic datasets combine structured attributes with free text, but are often evaluated separately. Such metrics can remain high after tabular--text pairings are disrupted. We present a projection-based evaluator for tabular--text synthetic data. A fixed sentence encoder maps text to embeddings, $k$ -means converts them to cluster states, and tabular variables are represented as categorical or quantile-binned states. Real and synthetic contingency tables are compared using Jensen--Shannon divergence (JSD), normalized mutual information (NMI), conditional JSD (cJSD), and joint-state entropy. We also report text-to-attribute (T2A) utility and a holdout-calibrated proximity flag rate (PFR) as a representation-level diagnostic. A text-permutation control preserves both marginal distributions while disrupting their pairing. Experiments on Amazon Reviews, Kiva Loans, and the Employment Scam Aegean Dataset show that modality-specific scores remain high under this control. The projection diagnostics detect disruption when real projected dependence exceeds a permutation baseline, but are less informative for weak or sparse projections. Some conditioned LLM baselines also exhibit stronger measured dependence than the corresponding real-data projections. These results support explicit cross-modal evaluation with permutation baselines and coverage reporting.

Figures & tables

Fig. 1: The “stitching assumption” in multimodal synthetic data evaluation. Separate tabular and text metrics can assign high scores even when the two components of the same record contradict each other.

Fig. 2: Architectural overview of the proposed framework. Semantic quantization maps text embeddings and selected tabular features into a joint discrete representation, enabling coarse-grained joint evaluation of multimodal synthetic data across fidelity, dependence, utility, diversity, and record proximity.

Dataset	Method	Tabular marginal	BERTScore F1	Embedding cosine
Amazon	$R$	1.000	1.000	1.000
	$S_{\mathrm{tilt}}$	1.000	0.840	–
	$S_{\mathrm{ind}}$	0.510	0.844	–
	$S_{\mathrm{seq}}$	0.510	0.852	–
	$S_{\mathrm{joint}}$	0.635	0.846	–
	TabSyn-SBERT	0.794	–	0.441

TABLE I: Modality-specific baseline scores. Higher values indicate greater similarity within each individual metric column. BERTScore-F1 and embedding cosine use different inputs and scales and must not be compared across columns.

Method	Projected JSD
$R$	0.0000
$S_{\mathrm{tilt}}$	0.1017
$S_{\mathrm{ind}}$	0.5778
$S_{\mathrm{seq}}$	0.7369
$S_{\mathrm{joint}}$	0.5865
TabSyn-SBERT	0.3219

TABLE II: Projected JSD for the Amazon rating–text boundary case. Lower is closer to the real reference; weak real dependence limits the permutation control’s discriminative power.

Dataset	Method	Proj. JSD $\downarrow$	NMI ratio / $\Delta_{\mathrm{NMI}}$	cJSD $\downarrow$	T2A $\uparrow$	$H_{\mathrm{rel}}$	PFR 1%
Kiva	$R$	0.000	1.00 / 0.000	0.000	–	1.000	–
	$S_{\mathrm{tilt}}$	0.698	0.016 / 0.600	0.687	0.076	1.432	0.186
	$S_{\mathrm{ind}}$	0.844	0.075 / 0.565	0.773	0.062	1.601	0.000
	TabSyn-SBERT	0.727	0.791 / 0.128	0.363 †	0.427	1.093	0.000
	$S_{\mathrm{seq}}$	0.602	0.948 / 0.032	0.448 ‡	0.345	1.120	0.010
	$S_{\mathrm{joint}}$	0.729	1.100 / -0.061	0.623	0.331	0.957	0.000

TABLE III: Projection-level results for Kiva and Fake Jobs. Lower JSD/cJSD and smaller

\lvert\Delta_{\mathrm{NMI}}\rvert

are preferred; higher T2A is better;

H_{\mathrm{rel}}

is interpreted relative to 1. PFR is a proximity flag rate, not a quality or privacy score. NMI is ratio / signed gap. Bold marks best directly comparable synthetic values; partial-coverage cJSD is excluded.

Method	Pos. F1	Balanced Acc.	AUROC	AUPRC	MCC	Min. Recall
Majority	0.000	0.500	–	–	–	0.000
$S_{\mathrm{ind}}$	0.372	0.604	0.604	0.286	0.166	0.741
$S_{\mathrm{seq}}$	0.343	0.598	0.622	0.401	0.239	0.277
$S_{\mathrm{joint}}$	0.341	0.599	0.611	0.314	0.260	0.259
TabSyn-SBERT	0.555	0.760	0.845	0.679	0.439	0.741
$S_{\mathrm{tilt}}$	0.215	0.462	0.463	0.185	-0.061	0.321

TABLE IV: Fake Jobs T2A utility under class imbalance. Higher is better for all reported utility metrics. Bold values mark the best value in each metric column among non-reference methods; ties are bolded. Min. Recall denotes minority-class recall. Values are rounded; “–” denotes a score-based metric that was not computed for the hard-label majority baseline.

Dataset	$\lambda^{\star}$	$\tau_{5\%}$	$\tau_{10\%}$	Max learned PFR	Tilted PFR
Amazon	1.138	0.553	0.643	0.000 / 0.014 / 0.032	0.388 / 0.490 / 0.508
Kiva	1.315	0.000	0.000	0.010 / 0.010 / 0.010	0.186 / 0.186 / 0.186
Fake Jobs	1.165	0.000	0.000	0.000 / 0.000 / 0.000	0.580 / 0.580 / 0.580

TABLE V: Holdout-calibrated DCR thresholds and PFR values. Thresholds are shown to three decimals, but PFR is computed before rounding using

\tilde{\tau}_{\alpha}=\max(\tau_{\alpha},\delta_{\mathrm{num}})

. For Kiva and Fake Jobs, the displayed

\tau_{5\%}

and

\tau_{10\%}

remain 0.000 under the selected DCR representation, so PFR 1% , PFR 5% , and PFR 10% coincide.

Check	Setting	Main finding
$K$ -sensitivity	$K\in\{5,10,15,20,30\}$	Fake Jobs remains stable, with tilted-to-TabSyn-SBERT JSD ratios from 2.54 to 1.49. Amazon remains weak because rating–text dependence is small. Kiva is sparse under the primary sector projection, with occupied-cell ratios 20–51% and minimum cell count 1.
$\lambda$ -sensitivity	$\lambda/\lambda^{\star}\in\{0.25,0.5,1,2,4\}$	Raw DCR changes with $\lambda$ , but PFR-based flagging pattern is unchanged across this 16-fold range: learned baselines remain unflagged or nearly unflagged, while $S_{\mathrm{tilt}}$ remains flagged.
XOR/parity stress test	$X_{1},X_{2},X_{3}\in\{0,1\}$ , $C_{T}=X_{1}\oplus X_{2}\oplus X_{3}$ ; $S_{\mathrm{tilt}}$ shuffles $C_{T}$ while preserving the tabular marginal	Each pair $(X_{j},C_{T})$ is marginally independent, so the mean pairwise JSD remains near zero in the finite-sample evaluation (0.0075). Shuffling destroys the parity support constraint, and the targeted 3-way projection detects the violation with empirical JSD 0.5542. This value is a finite-sample diagnostic statistic rather than the closed-form JSD of an ideal uniform parity distribution.

TABLE VI: Robustness and boundary-condition summary.

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
Mar 10, 2026Lukáš Eigler, Jindřich Libovický, David HurychSynthetic Data Evaluation Synthetic Benchmark Generation
Faculty of Mathematics and Physics, Charles University, Czech Republic
Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets
May 18, 2026Nitish Nagesh, Pengbao Zhou, Atchuth Naveen Chilaparasetti +8Healthcare Synthetic Tabular Data Generation
University of California, Irvine, CA, USA · PathAI, USA
Reasoning-Driven Synthetic Data Generation and Evaluation
Mar 31, 2026Tim R. Davidson, Benoit Seguin, Enrico Bacis +2Synthetic Data Generation Synthetic Data Evaluation
EPFL · Google · Google Deepmind

Mar 10, 2026J/K move · Enter open · S save

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Lukáš Eigler, Jindřich Libovický, David Hurych

Faculty of Mathematics and Physics, Charles University, Czech Republic

Synthetic Data Evaluation Synthetic Benchmark Generation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.

Abstract

Synthetic Data Evaluation

Explore similar work

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Reasoning-Driven Synthetic Data Generation and Evaluation