Synthetic Benchmark Generation

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 213

All topics

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Jul 9, 2026Andrej Leban, Yuekai SunStructural Causal Models AI Agent Benchmarks
Department of Statistics University of Michigan Ann Arbor, MI, United States
MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
Jul 8, 2026Charbel Al Bateh, Samer SaabMathematical Reasoning Benchmarks Synthetic Benchmark Generation
Department of Electrical and Computer Engineering Lebanese American University Byblos, Lebanon
Measuring Intelligence Beyond Human Scale
Jul 8, 2026Jerry Han, Rafael Moschopoulos, Ella Colby +6LLM Evaluation AI Agent Evaluation
Princeton Superalignment
GraphBU: MILP Instance Generation with Graph-Native Block Units
Jul 7, 2026Xiaolei Guo, Chenyu Zhou, Jianghao Lin +1Mixed-Integer Linear Programming Synthetic Benchmark Generation
Shanghai Jiao Tong University
PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation
Jul 7, 2026Yuhang Wu, Shuxiang Zhang, Wee Hian Ching +2Image Generation Synthetic Benchmark Generation
College of AI, Tsinghua University · Sun Yat-sen University · Shanghai Qi Zhi Institute
Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music
Jul 7, 2026Tomáš Sourada, Katia Vendrame, Jan HajičAudio-Language Model Evaluation Synthetic Benchmark Generation
Charles University · Brno University of Technology
Doppelganger: Sound Effects and Their Synthetic Twins
Jul 5, 2026Elliott AshAudio Representation Learning Audio Deepfake Detection
ETH Zürich
Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation
Jul 3, 2026Alberto Foresti, Ivan Butakov, Alexander Tolmachev +3Copula Models Benchmark Design
Department of Data Science EURECOM 450 Route des Chappes, 06410 Biot, France · Applied AI Institute Presnenskaya naberezhnaya, 12, 121205 Moscow, Russia · Moscow Independent Research Institute of Artificial Intelligence Bolshaya Cheremushkinskaya Street, 20, Building 4, Premises 3/1, 117218 Moscow, Russia
A $^{2}$ utoLPBench: An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Construction
Jul 2, 2026Shuo Ren, Yaohui Han, Yifan Shi +6AI Agent Benchmarks Synthetic Benchmark Generation
The Chinese University of Hong Kong
AgenticDataBench: A Comprehensive Benchmark for Data Agents
Jul 2, 2026Zhaoyan Sun, Shan Zhong, Daizhou Wen +10LLM Agent Evaluation AI Agent Benchmarks
Tsinghua University · Ant Digital Technologies, Ant Group
Aionoscope: Debugging Latent-State Accessibility in Time-Series Representations
Jul 1, 2026Alexander Chemeris, Ming Jin, Randall BalestrieroSynthetic Benchmark Generation Time Series Representation Learning
Langotime South Africa · Griffith University Australia · Brown University United States
Timesynth: A Temporal Fidelity Framework for Health Signal Digital Twins
Jul 1, 2026Md Rakibul Haque, Shireen Elhabian, Warren Woodrich PettineTime Series Forecasting Forecasting Benchmarks
Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, 84112, UT, USA. · Kahlert School of Computing, University of Utah, Salt Lake City, 84112, UT, USA. · Department of Psychiatry, University of Utah, Salt Lake City, UT, USA.
AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation
Jun 30, 2026Xinyuan Song, Zekun Cai, Liang ZhaoSynthetic Benchmark Generation Code Generation Evaluation
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
SFBench: The SciFy Scientific Feasibility Benchmark
Jun 28, 2026Cash Costello, James Mayfield, Elsbeth Turcan +7LLM Evaluation Benchmark Design
Johns Hopkins Applied Physics Laboratory
Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking
Jun 27, 2026Balázs Szalontai, Ábel Szauter, Balázs Márton +3Software Engineering Automated Program Repair
Eötvös Loránd University · Faculty of Informatics
NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
Jun 25, 2026Henry Shaowu Yuchi, Michal Kucer, Benjamin H. Sims +2LLM Evaluation Scientific QA
Los Alamos National Laboratory
ProfileFoundry: A Synthetic Person-Object Substrate for Privacy, Memory, and Tool-Use Evaluation in LLM Agent
Jun 24, 2026Sriram Selvam, Anneswa GhoshSynthetic Data Generation Synthetic Benchmark Generation
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Jun 23, 2026Yuru Wang, Lejun Cheng, Yuxin Zuo +14AI for Science Coding Agents
Horizon Research, Frontis.AI Tsinghua University
Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners
Jun 23, 2026Anirban Das, Joanne Boisson, Irtaza Khalid +2Relational Reasoning Synthetic Benchmark Generation
Cardiff University · University of Pennsylvania
PROTECT-90: A Fault Dataset for Power System Protection
Jun 23, 2026Julian Oelhaf, Georg Kordowich, Christian Bergler +3Benchmark Design Synthetic Benchmark Generation
1Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Institute of Electrical Energy Systems, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Electrical Engineering, Media and Computer Science, Ostbayerische Technische Hochschule Amberg-Weiden, Amberg, Germany
A Synthetic Reliability-Aware PINN Benchmark for Offshore Wind Turbine Support-Structure Monitoring with Bayesian Inverse Identification
Jun 23, 2026Puneet Kant, Monika TanwarNeural Surrogate Modeling Bayesian Inverse Problems
aSchool of Artificial Intelligence and Data Science, Indian Institute of Technology Jodhpur · bSchool of Management and Entrepreneurship, Indian Institute of Technology Jodhpur
Fabric Image Demoiréing Benchmark from Synthesis to Restoration
Jun 23, 2026Pengchao Wei, Xiaojie GuoSynthetic Benchmark Generation Image Restoration
Tianjin University, Tianjin, China
The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence
Jun 19, 2026David NordforsLLM Evaluation LLM-as-a-Judge
ELADO: Elliptic PDE Assessment Datasets for Operator Learning
Jun 18, 2026Frank Ehebrecht, Toni Scharle, Martin AtzmuellerSynthetic Benchmark Generation PDE Operator Learning
Osnabrück University, Semantic Information Systems Group, Osnabrück, Germany · ROSEN Technology and Research Center GmbH, Lingen, Germany · German Research Center for Artificial Intelligence (DFKI), Osnabrück, Germany
Critical Percolation as a Synthetic Data Model for Interpretability
Jun 18, 2026Aryeh Brill, Tom Ingebretsen CarlsonSynthetic Data Generation Neural Network Interpretability
Principles of Intelligence
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
Jun 18, 2026Xinyi Zheng, Ling Shi, Tianlong Yu +3Logical Reasoning LLM Evaluation
Huazhong University of Science and Technology Wuhan, China · Nanyang Technological University Singapore, Singapore · Hubei University Wuhan, China +2
Source-Grounded Data Generation for Text-to-JSON Learning
Jun 18, 2026Sunghee Ahn, Guijin Son, Youngjae YuGrounded Text Generation Synthetic Data Generation
1Seoul National University
SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents
Jun 17, 2026Qiao Zhao, JianYing Qu, Jun Zhang +3Software Engineering Agents Software Engineering Benchmarks
Baidu Inc.
ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Jun 17, 2026Jaeho Lee, Nick Merrill, Ezra KargerForecasting Benchmarks Probabilistic Forecasting
1Forecasting Research Institute · 2UC Berkeley
DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
Jun 17, 2026Patrick Cooper, Alvaro VelasquezLLM Evaluation Abductive Reasoning
University of Colorado Boulder

Jul 9, 2026, cs.AIJ/K move · Enter open · S save

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Andrej Leban, Yuekai Sun

Department of Statistics University of Michigan Ann Arbor, MI, United States

Structural Causal Models AI Agent Benchmarks Causal Reasoning in Language Models Synthetic Benchmark Generation Causal Reasoning Data Analysis Agents

Open paper AskPDF arXiv

Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.

Momentum

Related topics

Latest papers 213