Synthetic Data Evaluation

Follow this topic Ask about this topic

Synthetic data evaluation asks whether generated examples are realistic and suitable for their intended use, such as model training, scientific inference, or benchmarking. Researchers combine distributional and dependence metrics with train-on-synthetic, test-on-real experiments, human or expert review, domain-specific checks, and privacy attacks. Controlled benchmarks with known structure can test whether methods preserve causal, temporal, subgroup, or task-relevant properties that ordinary summary statistics may miss. A central open challenge is that no single fidelity score reliably predicts utility or privacy: results depend on the task, data domain, and evaluation protocol.

Latest papers

236 papers26 papers in the last four weeks, down 19% on the four weeks before. 0.2% of all new papers.

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
Mar 10, 2026Lukáš Eigler, Jindřich Libovický, David HurychSynthetic Data Evaluation Synthetic Benchmark Generation
Faculty of Mathematics and Physics, Charles University, Czech Republic
Evaluating Synthetic Data for Baggage Trolley Detection in Airport Logistics
Mar 8, 2026Abdeldjalil Taibi, Mohmoud Badlis, Amina Bensalem +2Synthetic Data Evaluation Object Detection
The National School of Artificial Intelligence (ENSIA), Algiers 16000, Algeria · Information Systems and Development Department, Algiers Airport, Algiers 16000, Algeria
xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truth
Mar 6, 2026Gregor BaerSynthetic Benchmark Generation Interpretable Time-Series Classification
Information Systems, Industrial Engineering & Innovation Sciences Eindhoven University of Technology Eindhoven, The Netherlands
From Statistical Fidelity to Clinical Consistency: Scalable Generation and Auditing of Synthetic Patient Trajectories
Mar 6, 2026Guanglin Zhou, Armin Catic, Motahare Shabestari +4Synthetic Data Generation Synthetic Clinical Data Generation
The University of Queensland, Brisbane, QLD, Australia · The University of New South Wales, Sydney, NSW, Australia · Shahid Sadoughi University of Medical Sciences and Health Services, Yazd, Iran +1
Harnessing Synthetic Data from Generative AI for Statistical Inference
Mar 5, 2026Ahmad Abdel-Azim, Ruoyu Wang, Xihong LinSynthetic Data Generation Synthetic Data Evaluation
Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts 02115, USA · Harvard University, Boston, Massachusetts 02115, USA
ConTSG-Bench: A Unified Benchmark for Conditional Time Series Generation
Mar 5, 2026Shaocheng Lan, Shuqi Gu, Zhangzhi Xiong +1Time Series Generation Synthetic Data Evaluation
School of Information Science and Technology, ShanghaiTech University, Shanghai, China.
Balancing Fidelity, Utility, and Privacy in Synthetic Cardiac MRI Generation: A Comparative Study
Mar 4, 2026Madhura Edirisooriya, Dasuni Kawya, Ishan Kumarasinghe +5Medical Image Synthesis Synthetic Data Augmentation
University of Peradeniya, Sri Lanka · University of Maryland, College Park, USA · Tulane University School of Medicine, USA; Simula Research Laboratory, Norway +1
Scalable Evaluation of the Realism of Synthetic Environmental Augmentations in Images
Mar 4, 2026Damian J. Ruck, Paul Vautravers, Oliver Chalkley +1Synthetic Data Evaluation Image Editing Evaluation
Advai Ltd, 20–22 Wenlock Road, London, N1 7GU, United Kingdom
CausalWrap: Model-Agnostic Causal Constraint Wrappers for Tabular Synthetic Data
Mar 2, 2026Amir Asiaee, Zhuohui J. Liang, Chao YanSynthetic Data Generation Synthetic Tabular Data Generation
Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA · Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
Data-driven Synthesis of Magnetic Resonance Spectroscopy Data using a Variational Autoencoder
Feb 28, 2026Dennis M. J. van de Sande, Julian P. Merkofer, Sina Amirrajab +4Magnetic Resonance Spectroscopy Synthetic Data Generation
Department of Biomedical Engineering Eindhoven University of Technology Eindhoven, The Netherlands · Department of Electrical Engineering Eindhoven University of Technology Eindhoven, The Netherlands · Department of Radiology & Nuclear Medicine Maastricht University Medical Center Maastricht, The Netherlands
When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators
Feb 23, 2026Krzysztof Adamkiewicz, Brian Bernhard Moser, Stanislav Frolov +3Synthetic-to-Real Domain Adaptation Distribution Shift
1RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · 2German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany
RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning
Feb 21, 2026Seungku Kim, Suhyeok Jang, Byungjun Yoon +3Synthetic Data Generation for Robotics Synthetic Data Augmentation
KAIST · RLWRLD
Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity
Feb 20, 2026Vasile Marian, Yong-Bin Kang, Alexander BudderySynthetic Data Evaluation Synthetic Data Augmentation
School of Mechanical and Mining Engineering, The University of Queensland, Level 4, Mansergh Shaw Building (45), St Lucia, QLD 4072, Australia · Swinburne University of Technology, John Street, Hawthorn, VIC 3122, Australia
LAND: A Longitudinal Analysis of Neuromorphic Datasets
Feb 17, 2026Gregory Cohen, Alexandre MarcireauNeuromorphic Computing Synthetic Data Evaluation
International Centre for Neuromorphic Systems (ICNS) Western Sydney University Sydney, Australia · International Centre for Neuromorphic Systems (ICNS) The University of Manchester Manchester, United Kingdom
SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
Feb 16, 2026David Chanin, Adrià Garriga-AlonsoSparse Autoencoders Synthetic Benchmark Generation
University College London · 2MATS · 3Decode Research
Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data
Feb 10, 2026Jason Miklian, Kristian Hoelscher, John E. KatsosHuman Behavior Simulation Synthetic Data Generation
University of Oslo · Peace Research Institute Oslo · American University of Sharjah
Synthetic Time Series Generation via Complex Networks
Jan 30, 2026Jaime Vale, Vanessa Freitas Silva, Maria Eduarda Silva +1Time Series Generation Synthetic Data Evaluation
Faculdade de Ciˆencias, Universidade do Porto · INESC TEC-CRACS · Faculdade de Economia, Universidade do Porto +1
Interpretable Similarity of Synthetic Image Utility
Dec 18, 2025Panagiota Gatoula, George Dimas, Dimitris K. IakovidisMedical Image Synthesis Synthetic Data Evaluation
University of Thessaly, Department of Computer Science and Biomedical Informatics, Lamia 35131 Greece
From Simulation to Experiment: Benchmarking Machine-Learning Models for Quantum-Dot State Recognition
Nov 27, 2025Valeria Díaz Moreno, Ryan P Khalili, Daniel Schug +3Sim-to-Real Transfer Synthetic Data Evaluation
Department of Physics, University of Wisconsin-Madison, Madison, Wisconsin 53706, USA · Department of Computer Science, University of Maryland, College Park, MD 20742, USA · Department of Applied Physics, Stanford University, Stanford, California 94305, USA +3
Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
Oct 18, 2025Bingji Yi, Qiyuan Liu, Yuwei Cheng +1Synthetic Data Pretraining Model Collapse
Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago. +1
Generating Individual Travel Diaries Using Large Language Models Informed by Census and Land-Use Data
Sep 7, 2025Sepehr Golrokh Amin, Devin Rhoads, Fatemeh Fakhrmoosavi +2Synthetic Data Generation Synthetic Data Evaluation
School of Civil and Environmental Engineering, University of Connecticut, United States
MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions
Sep 4, 2025Aishik Mandal, Tanmoy Chakraborty, Iryna GurevychMental Health Counseling Mental Health
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Electrical Engineering, Indian Institute of Technology Delhi, India +1
Synthetic data for ratemaking: imputation-based methods vs adversarial networks and autoencoders
Sep 2, 2025Yevhen Havrylenko, Meelis Käärik, Artur TuttarSynthetic Data Augmentation Synthetic Tabular Data Generation
Department of Actuarial Science, Faculty of Business and Economics, University of Lausanne, Switzerland · Institute of Mathematics and Statistics, Faculty of Science and Technology, University of Tartu, Estonia
Can synthetic data reproduce real-world findings in epidemiology? A replication study using adversarial random forests
Aug 19, 2025Jan Kapar, Kathrin Günther, Lori Ann Vallis +27Synthetic Tabular Data Generation Synthetic Data Evaluation
Leibniz Institute for Prevention Research and Epidemiology—BIPS, Bremen, Germany · Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Human Health and Nutritional Sciences, University of Guelph, Guelph, Ontario, Canada +23
OmniFall: From Staged Through Synthetic to Wild, A Unified Multi-Domain Dataset for Robust Fall Detection
May 26, 2025David Schneider, Zdravko Marinov, Moritz Mistol +6Synthetic-to-Real Domain Adaptation Fall Detection
Karlsruhe Institute of Technology · Mercedes-Benz Tech Innovation
Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework
Jun 12, 2024Zineb Senane, Axel Karlsson, Lele Cao +6Synthetic Benchmark Generation Causal Inference
KTH Royal Institute of Technology · Stockholm, Sweden · King AI Labs, Microsoft +2

Mar 10, 2026J/K move · Enter open · S save

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Lukáš Eigler, Jindřich Libovický, David Hurych

Faculty of Mathematics and Physics, Charles University, Czech Republic

Synthetic Data Evaluation Synthetic Benchmark Generation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.