Synthetic Benchmark Generation

Follow this topic Ask about this topic

Synthetic benchmark generation creates evaluation tasks and datasets when real examples are scarce, costly, private, contaminated, or too fixed to expose specific weaknesses. Approaches include procedural generators and simulators, LLM- or agent-driven task synthesis, and solver-backed construction with executable checks for answers, feasibility, and difficulty. These methods support fresh, adjustable tests across areas such as reasoning, coding, robotics, retrieval, and scientific workflows, often with ground truth known by construction. A central challenge is balancing control and verifiability with realistic, diverse tasks that transfer to real settings, while avoiding artifacts or shared generation and evaluation rules that inflate scores.

Latest papers

289 papers26 papers in the last four weeks, down 16% on the four weeks before. 0.2% of all new papers.

SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
Feb 16, 2026David Chanin, Adrià Garriga-AlonsoSparse Autoencoders Synthetic Benchmark Generation
University College London · 2MATS · 3Decode Research
LogicEnvGen: Task-Logic Driven Generation of Diverse Simulated Environments for Embodied AI
Jan 20, 2026Jianan Wang, Siyang Zhang, Bin Li +4Robotics Simulation AI Agent Evaluation
College of Computer Science and Technology, National University of Defense Technology · Intelligent Game and Decision Lab (IGDL), Beijing · School of Artifical Intelligence, Shanghai Jiao Tong University
An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models
Jan 19, 2026Yuhiro Ono, Tomohiro Harada, Yukiya MiuraEvolutionary Optimization Benchmark Design
Graduate School of Systems Design, Tokyo Metropolitan University, 6-6, Asahigaoka, Hino, 1910065, Tokyo, Japan. · Graduate School of Science and Engineering, Saitama University, 2552026 Shimo-okubo, Sakura-ku, Saitama, 3388570, Saitama, Japan.
SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models
Jan 16, 2026Turhan Can Kargin, Wojciech Jasiński, Adam Pardyl +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland. · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland. · AGH University of Krakow, Kraków, Poland. +3
Kinship Data Benchmark for Multi-hop Reasoning
Jan 12, 2026Tianda Sun, Dimitar KazakovLLM Evaluation Synthetic Benchmark Generation
Department of Computer Science University of York, UK
Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
Jan 5, 2026Chenghao Yin, Da Huang, Di Yang +16Robotic Data Collection Robotics Simulation
Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks
Nov 22, 2025Jie Li, Hongyi Cai, Mingkang Dong +4Synthetic Benchmark Generation Video Anomaly Detection
Shanghai Jiao Tong University · University of Science & Technology Beijing · Universiti Malaya +2
Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Nov 6, 2025Alberto Cattaneo, Carlo Luschi, Daniel JustusGraph Retrieval-Augmented Generation Synthetic Benchmark Generation
Graphcore AI Research
SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
Aug 21, 2025Xiangyang Zhu, Yuan Tian, Chunyi Li +3LLM Safety Benchmarks Synthetic Benchmark Generation
Shanghai AI Lab · East China Normal University
Instance Generation for Patient-to-room Assignment and Admission Scheduling Based on Real Hospital Data
Jul 4, 2025Tabea Brandt, Christina Büsing, Johanna Leweke +2Healthcare Dynamic Programming
Chair of Combinatorial Optimization, RWTH Aachen University, Im Süsterfeld 9, 52072 Aachen, Germany · Chair of Algorithms and Complexity, RWTH Aachen University, Ahornstraße 55, 52074 Aachen, Germany
ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese
Jun 11, 2025Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Farber +2Synthetic Data Generation Hate Speech Detection
Advanced Knowledge Center for Immersive Technologies (AKCIT) · Federal University of Goiás
PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics
May 29, 2025Atharva Naik, Prakam, Yash Mathur +6Program Synthesis Inductive Inference
Carnegie Mellon University · Independent Researcher · Ohio State University
Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data
May 21, 2025Guilherme de Oliveira, Matheus M. dos Santos, Paulo L. J. Drews-JrSynthetic-to-Real Domain Adaptation Underwater Imaging
NAUTEC - Intelligent robotics and automation group, FURG - Federal University of Rio Grande, Rio Grande, RS, Brazil · ICMC - Institute of Mathematics and Computer Sciences, USP - University of S˜ao Paulo, S˜ao Carlos, SP, Brazil
Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings
May 19, 2025Adrien Lagesse, Marc LelargeGraph Neural Networks Synthetic Benchmark Generation
INRIA, École Normale Supérieure - PSL, Paris, France.
AneumoBench: A Source-Linked Benchmark for Synthetic-Geometry Transfer in Aneurysm CFD
May 19, 2025Xigui Li, Yuanye Zhou, Feiyang Xiao +16Neural Surrogate Modeling Synthetic Data Pretraining
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China · Human Phenome Institute, Fudan University, Shanghai, China +1
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
Jan 20, 2025Zijin Hong, Hao Wu, Su Dong +8Mathematical Reasoning Benchmarks Language Model Generalization
The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Electronic Science and Technology of China, Chengdu, China · Tencent Youtu Lab, Shanghai, China +2
Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions
Nov 15, 2024Yutao Hou, Yajing Luo, Zhiwen Ruan +4Multi-Hop QA LLM Evaluation
Shanghai University of Finance and Economics · Southern University of Science and Technology · University of Edinburgh +1
HardCore Generation: Generating Hard UNSAT Problems for Data Augmentation
Sep 27, 2024Joseph Cotnareanu, Zhanguang Zhang, Hui-Ling Zhen +2Synthetic Data Augmentation Synthetic Benchmark Generation
McGill University Montreal, Canada · Huawei Noah’s Ark Lab Montreal, Canada · Huawei Noah’s Ark Lab Hong Kong, China +1
Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework
Jun 12, 2024Zineb Senane, Axel Karlsson, Lele Cao +6Synthetic Benchmark Generation Causal Inference
KTH Royal Institute of Technology · Stockholm, Sweden · King AI Labs, Microsoft +2

Feb 16, 2026J/K move · Enter open · S save

SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data

David Chanin, Adrià Garriga-Alonso

University College London · 2MATS · 3Decode Research

Sparse Autoencoders Synthetic Benchmark Generation Synthetic Data Evaluation

Open paper AskPDF arXiv

Improving Sparse Autoencoders (SAEs) requires benchmarks that can precisely validate architectural innovations. Current LLM-based SAE benchmarks are too noisy to differentiate architectural improvements, while commonly used synthetic-data experiments are too small-scale, unstandardized, and unrealistic to be meaningful. We introduce SynthSAEBench, a benchmark and toolkit for evaluating SAEs against large-scale synthetic data with realistic feature characteristics including correlation, hierarchy, and superposition, while providing ground-truth features and firings. SynthSAEBench acts as a controlled lower-bound test: SAE architectures that fail when the Linear Representation Hypothesis holds by construction have little hope on real LLMs. The benchmark reproduces known LLM SAE phenomena including the disconnect between reconstruction and latent quality, poor SAE probing, and a precision-recall trade-off mediated by L0, demonstrating that SynthSAEBench findings reproduce results on LLM SAEs. We further identify a novel failure mode: Matching Pursuit SAEs exploit superposition noise to improve reconstruction without learning ground-truth features, suggesting more expressive encoding procedures can easily overfit. SynthSAEBench complements LLM benchmarks with ground-truth features and controlled ablations for diagnosing SAE failure modes, while providing a clear target for SAE architecture work.