Synthetic Benchmark Generation

Overview

Synthetic benchmark generation creates evaluation tasks and datasets when real examples are scarce, costly, private, contaminated, or too fixed to expose specific weaknesses. Approaches include procedural generators and simulators, LLM- or agent-driven task synthesis, and solver-backed construction with executable checks for answers, feasibility, and difficulty. These methods support fresh, adjustable tests across areas such as reasoning, coding, robotics, retrieval, and scientific workflows, often with ground truth known by construction. A central challenge is balancing control and verifiability with realistic, diverse tasks that transfer to real settings, while avoiding artifacts or shared generation and evaluation rules that inflate scores.

Latest papers

CardsList

302 papers17 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. HardCore Generation: Generating Hard UNSAT Problems for Data Augmentation

    Sep 27, 2024Joseph Cotnareanu, Zhanguang Zhang, Hui-Ling Zhen +2Synthetic Data AugmentationSynthetic Benchmark Generation

  2. Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework

    Jun 12, 2024Zineb Senane, Axel Karlsson, Lele Cao +6Synthetic Benchmark GenerationCausal Inference