Synthetic Benchmark Generation

Ask about this topic

Overview

Synthetic benchmark generation creates evaluation tasks and datasets when real examples are scarce, costly, private, contaminated, or too fixed to expose specific weaknesses. Approaches include procedural generators and simulators, LLM- or agent-driven task synthesis, and solver-backed construction with executable checks for answers, feasibility, and difficulty. These methods support fresh, adjustable tests across areas such as reasoning, coding, robotics, retrieval, and scientific workflows, often with ground truth known by construction. A central challenge is balancing control and verifiability with realistic, diverse tasks that transfer to real settings, while avoiding artifacts or shared generation and evaluation rules that inflate scores.

Latest papers

CardsList

289 papers26 papers in the last four weeks, down 16% on the four weeks before. 0.2% of all new papers.

  1. ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments

    Mar 30, 2026Pragat Wagle, Zheng Chen, Lantao LiuAutonomous Driving DatasetsSynthetic Benchmark Generation

    Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN 47408, USA

  2. SynMVCrowd: A Large Synthetic Benchmark for Multi-view Crowd Counting and Localization

    Mar 25, 2026Qi Zhang, Daijie Chen, Yunfei Gong +1Crowd CountingSynthetic Benchmark Generation

    Visual Computing Research Center, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.

  3. OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection

    Mar 23, 2026Jeffrey FlyntLLMs for CybersecuritySynthetic Benchmark Generation

    Independent Researcher

  4. BenchBench: Benchmarking Automated Benchmark Generation

    Mar 21, 2026Yandan Zheng, Haoran Luo, Zhenghong Lin +2Benchmark DesignSynthetic Benchmark Generation

    Nanyang Technological University Singapore, Singapore

  5. TAB-AUDIT: Detecting AI-Fabricated Scientific Tables via Multi-View Likelihood Mismatch

    Mar 20, 2026Shuo Huang, Yan Pen, Lizhen QuAI-Generated Text DetectionSynthetic Benchmark Generation

    Monash University

  6. Toward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection

    Mar 18, 2026Amine LbathAutomated Exploit GenerationSoftware Vulnerability Detection

    Univeristy of Grenoble Grenoble, France

  7. DiscoGen: Procedural Generation of Algorithm Discovery Tasks in Machine Learning

    Mar 18, 2026Alexander D. Goldie, Zilin Wang, Adrian Hayler +17Automated Algorithm DiscoverySynthetic Benchmark Generation

    University of Oxford · University of California, Santa Barbara · University College London +1

  8. ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments

    Mar 17, 2026Ziyang Cai, Amir Saeidi, Harkirat BehlAutonomous Research AgentsLLM Agent Training

    Arizona State University · Microsoft Research

  9. Infinite Problem Generator: Verifiably Scaling Physics Reasoning Data with Agentic Workflows

    Mar 15, 2026Aditya Sharan, Sriram Hebbale, Dhruv KumarSynthetic Data Generation for Language ModelsMathematical Problem Generation

    BITS Pilani, Pilani Campus, India

  10. BenDFM: A taxonomy and synthetic CAD dataset for manufacturability assessment in sheet metal bending

    Mar 13, 2026Matteo Ballegeer, Dries F. BenoitSynthetic Benchmark Generation

    Ghent University CVAMO Core Lab Research Group Data Analytics

  11. daVinci-Env: Open SWE Environment Synthesis at Scale

    Mar 13, 2026Dayuan Fu, Shenyu Wu, Yunze Wu +11Software EngineeringLLM Agent Training

    SII · GAIR · SJTU

  12. CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

    Mar 12, 2026Zi-Han Wang, Lam Nguyen, Zhengyang Zhao +4Computational CreativityCreativity Assessment

    Southern University of Science and Technology · Tsinghua University · Xi’an Jiaotong University +3

  13. LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

    Mar 10, 2026Lukáš Eigler, Jindřich Libovický, David HurychSynthetic Data EvaluationSynthetic Benchmark Generation

    Faculty of Mathematics and Physics, Charles University, Czech Republic

  14. MultiGraSCCo: A Multilingual Anonymization Benchmark with Annotations of Personal Identifiers

    Mar 9, 2026Ibrahim Baroud, Christoph Otto, Vera Czehmann +4Synthetic Benchmark GenerationClinical NLP

    1German Research Center for Articial Intelligence (DFKI) · 3Technische Universität Berlin · University of Potsdam +3

  15. xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truth

    Mar 6, 2026Gregor BaerSynthetic Benchmark GenerationInterpretable Time-Series Classification

    Information Systems, Industrial Engineering & Innovation Sciences Eindhoven University of Technology Eindhoven, The Netherlands

  16. SynthCharge: An Electric Vehicle Routing Instance Generator with Feasibility Screening to Enable Learning-Based Optimization and Benchmarking

    Mar 3, 2026Mertcan Daysalilar, Fuat Uyguroglu, Gabriel Nicolosi +1Synthetic Benchmark GenerationVehicle Routing

    Industrial and Systems Engineering, University of Miami, Coral Gables, FL, USA · Faculty of Engineering, Cyprus International University 99258 Nicosia, North Cyprus, via Mersin 10, Turkey · Engineering Management and Systems Engineering, Missouri University of Science and Technology, Rolla, MO, USA

  17. Code2Math: Can Your Code Agent Evolve Math Problems Through Exploration?

    Mar 3, 2026Dadi Guo, Yuejin Xie, Qingyu Liu +11Mathematical Problem GenerationSynthetic Benchmark Generation

    Hong Kong University of Science and Technology · Tsinghua University · Zhejiang University +4

  18. Tide: A Customisable Dataset Generator for Anti-Money Laundering Research

    Mar 2, 2026Montijn van den Beukel, Jože Martin Rožanec, Ana-Lucia VarbanescuAnti-Money Laundering DetectionSynthetic Data Generation

    University of Amsterdam, The Netherlands · Jožef Stefan Institute, Slovenia · University of Amsterdam/University of Twente, The Netherlands

  19. Adversarial Query Synthesis via Bayesian Optimization

    Mar 2, 2026Jeffrey Tao, Yimeng Zeng, Haydn Thomas Jones +4Synthetic Benchmark GenerationDatabase Systems

    University of Pennsylvania

  20. The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents

    Feb 28, 2026Shrey Shah, Levent OzgurAdversarial Robustness of Language ModelsAI Agent Benchmarks

  21. BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languages

    Feb 28, 2026Jason Lucas, Matt Murtagh-White, Adaku Uchendu +6Fake News DetectionRobustness of Machine-Generated Text Detection

    Penn State University, USA · Trinity College Dublin, Ireland · MIT Lincoln Lab, USA +2

  22. SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks

    Feb 28, 2026Yucheng Zeng, Shupeng Li, Daxiang Dong +11Synthetic Benchmark GenerationSoftware Engineering Benchmarks

    Qianfan Team, Baidu Inc.

  23. TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models

    Feb 27, 2026Daniel Nobrega MedeirosVisual ReasoningSpatial Reasoning Benchmarks

    Independent Researcher

  24. MMKG-RDS: Reasoning Data Synthesis via Deep Mining of Multimodal Knowledge Graphs

    Feb 27, 2026Lun Zhan, Feng Xiong, Huanyong Liu +2Synthetic Data GenerationSynthetic Data Generation for Language Models

    AI Research Institute, Qihoo 360

  25. LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning

    Feb 27, 2026Yu Zhu, Kai YangSynthetic Benchmark GenerationLLM Reasoning

    Department of Computer Science and Technology, Tongji University

  26. SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

    Feb 26, 2026Sungho Park, Jueun Kim, Wook-Shin HanTable QAQuestion Answering

    POSTECH, Pohang, Republic of Korea

  27. SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

    Feb 26, 2026Peiyao Xiao, Xiaogang Li, Xinyi Gao +7LLM EvaluationMultimodal Model Evaluation

    Alibaba Group Beijing, China

  28. EvolveGen: Algorithmic Level Hardware Model Checking Benchmark Generation through Reinforcement Learning

    Feb 26, 2026Guangyu Hu, Xiaofeng Zhou, Wei Zhang +1Synthetic Benchmark GenerationFormal Hardware Verification

    The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

  29. CaDrift: A Time-dependent Causal Generator of Drifting Data Streams

    Feb 23, 2026Eduardo V. L. Barboza, Jean Paul Barddal, Robert Sabourin +1Synthetic Data GenerationSynthetic Benchmark Generation

    LIVIA, École de Technologie Supérieure (ÉTS), Université du Québec, Montreal, QC, Canada · Graduate Program in Informatics (PPGIa), Pontifícia Universidade Católica do Paraná (PUCPR), Curitiba, PR, Brazil

  30. Toward Trustworthy Evaluation of Sustainability Rating Methodologies: A Human-AI Collaborative Framework for Benchmark Dataset Construction

    Feb 19, 2026Xiaoran Cai, Wang Yang, Xiyu Ren +3Synthetic Benchmark GenerationHuman-in-the-Loop Evaluation

    Columbia University, USA · Case Western Reserve University, USA · Hong Kong University of Science and Technology, Hong Kong +3