Synthetic Benchmark Generation

Overview

Synthetic benchmark generation creates evaluation tasks and datasets when real examples are scarce, costly, private, contaminated, or too fixed to expose specific weaknesses. Approaches include procedural generators and simulators, LLM- or agent-driven task synthesis, and solver-backed construction with executable checks for answers, feasibility, and difficulty. These methods support fresh, adjustable tests across areas such as reasoning, coding, robotics, retrieval, and scientific workflows, often with ground truth known by construction. A central challenge is balancing control and verifiability with realistic, diverse tasks that transfer to real settings, while avoiding artifacts or shared generation and evaluation rules that inflate scores.

Latest papers

CardsList

302 papers17 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. BugGen 2.0: Rethinking the Roles of LLMs for Scalable RTL Bug Synthesis

    Nov 17, 2026Minh Luu, Michael Quinn, Aakash Tyagi +2Synthetic Benchmark GenerationLLM Code Generation

  2. ScenarioForge: Resource-Grounded Generation and Execution Validation of Cyber Exercises

    Nov 16, 2026Jaime C. Acosta, Anantaa Kotal, Aritran PiplaiAdversarial Scenario GenerationScenario-Based Testing

  3. Synthetic Neurosymbolic Task Generation with Controlled Constraint Complexity

    Nov 14, 2026Mykhailo Buleshnyi, Zsolt Zombori, Michael BenediktSynthetic Benchmark GenerationNeurosymbolic Learning

  4. Generating Diverse and Verified Synthetic Clinical Notes with Labels by Construction

    Nov 7, 2026Matthias Lehmann, Madeleine WerthebachSynthetic Clinical Data GenerationSynthetic Benchmark Generation

  5. TSSDA: An Autonomous Agent for Constructing a Time-Series Text-to-SQL Benchmark

    Nov 1, 2026Zhihan Zheng, Shaoxu Song, Haitao Yuan +1Text-to-SQLSynthetic Benchmark Generation

  6. Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks

    Nov 1, 2026Yu Jiang, Yue Huang, Hang Hua +6Terminal-Agent BenchmarksLong-Horizon Agent Evaluation

  7. When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications

    Oct 29, 2026Yishuo Cai, Minzheng Wang, Liang Zhao +5Code GenerationLong-Horizon Agent Tasks

  8. Deogen: Automatic Benchmark Generation for Reasoning over Policies

    Oct 28, 2026Sam Kouteili, William Liang, Keeran Murray +4Synthetic Benchmark GenerationLanguage Model Reasoning Evaluation

  9. GNBG-C: A Unified and Configurable Benchmark Generator for Constrained Continuous Optimization

    Oct 27, 2026Ali Ghasemzadeh, Erfan Babakhani, Armin Khosravi +4Synthetic Benchmark GenerationConstrained Optimization

  10. SMEBENCH: Benchmarking Neural Music Editing with Synthetized MIDI Edits

    Oct 27, 2026Matthieu Cervera, Francesco Paissan, Micael Antunes +3Audio EditingSynthetic Benchmark Generation

  11. A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

    Oct 25, 2026Peter Romero, Haotian Li, Hugo Save +8Generative AI EvaluationSynthetic Benchmark Generation

  12. Beyond Copying: Lifelong Smart Contract Security Task Construction via Protocol Mutation

    Oct 23, 2026Zifan Peng, Peng Huang, Jingyi Zheng +5Software SecuritySynthetic Benchmark Generation

  13. IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

    Oct 21, 2026Rabindra Lamsal, Sisi Zlatanova, Xuesong ShenSynthetic Benchmark GenerationAI Agent Benchmarks

  14. Hard Instances on Demand

    Oct 15, 2026Jamie Fravel, Martin Koutecký, Pavlo TsitseiSynthetic Benchmark Generation

  15. Data World: Scaling Data Environments from Scratch

    Oct 15, 2026Zhanli Li, Zeyuan Xu, Chaoxu Pang +16Synthetic Data Generation for LLM AgentsData Analysis Agents

  16. GRADS: Graph-Regulated Agentic Data Synthesis with Controllable Difficulty

    Oct 15, 2026Gaocheng Yang, Ruijie Tao, Kai Ye +3Synthetic Data Generation for LLM AgentsSynthetic Data Generation for Language Models

  17. Exhaustive Canonical Skeletons Generation: A Formal Dataset for Decomposable Symbolic Regression

    Oct 14, 2026Giorgio Morales, Anikó EkártSymbolic RegressionSynthetic Benchmark Generation

  18. SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

    Oct 14, 2026Songwei Yu, Ziqi Miao, Jingxuan Xu +10Software Engineering BenchmarksCode Generation Benchmarks

  19. CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

    Oct 11, 2026Anahita Sharma, Apaar Shanker, Rishav Chakravarti +6Synthetic Benchmark GenerationIR Evaluation

  20. CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

    Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark GenerationBenchmark Design

  21. Studying the Effects of Task Diversity Using Small Mathematical Worlds

    Oct 9, 2026Emily Xia Meng, Xuanyu Yang, Junaid Hasan +2Multi-Task LearningLLM Training

  22. SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

    Oct 8, 2026Tian Jin, Wenhao Wang, Jing Kang +4Computer-Use AgentsLLM Agent Training