Synthetic Benchmark Generation

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 213

All topics

Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
Aug 26, 2026Qiankai Xu, Qiguang Chen, Zixin Su +4Long-Form Document Generation Synthetic Benchmark Generation
ByteDance Seed · Nanjing University · Evolvent AI
Compositional Benchmark Synthesis for Hierarchical Human Action Recognition
Aug 11, 2026Farnaz Soleimani, Abdelghani Chibani, Yacine Amirat +1Human Activity Recognition Synthetic Benchmark Generation
LISSI
$π$ -SUB: A Physics-Informed Synthetic Underwater Benchmark Dataset for Underwater Image Enhancement
Aug 11, 2026Namritha Lasyapriya Maddali, Rajini Makam, Suresh Sundaram +1Underwater Image Enhancement Underwater Imaging
Department of Aerospace Engineering, Indian Institute of Science, Bangalore, India. · Nanyang Technological University, Singapore.
Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction
Aug 9, 2026Zhuowen Liang, Zhengxuan Zhang, Jiayang Wang +2Document Information Extraction Synthetic Benchmark Generation
1The Hong Kong University of Science and Technology (Guangzhou)
Back to the Future: A workbook time machine for spread sheet creation benchmarks
Aug 8, 2026Mansi Uniyal, Agamdeep Singh, Ananya Singha +5LLM Agent Evaluation Synthetic Benchmark Generation
Microsoft
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
Aug 7, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Robustness
University of Chicago · Stony Brook University · Toyota Technological Institute at Chicago
Ask-E: An Environment for Calibrated Question Generation
Aug 7, 2026Sarah Pratt, Jae Sung Park, Scott Geng +1Mathematical Reasoning Benchmarks Synthetic Benchmark Generation
University of Washington · 2Allen Institute for AI
Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints
Aug 6, 2026Omid Bazgir, Md Nasir, Jacob Hoffman +6Synthetic Benchmark Generation Benchmark Validity
Oracle Health and Life Sciences United States
Recursive Synthesis for Long-Horizon Terminal Tasks
Aug 5, 2026Zhongzhi Li, Yucheng Shi, Zongxia Li +8Terminal Agents Synthetic Benchmark Generation
Tencent HY LLM Frontier · University of Georgia · University of Maryland, College Park +5
LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4LLM Evaluation LLM Personalization
1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1
From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution
Aug 3, 2026Can Wang, Haoran Chen, Haowen Gao +3LLM Evaluation Benchmark Design
Shandong Key Laboratory of Digital Service Computing Technology and Systems · Alibaba Token Hub · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences +1
Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models
Aug 2, 2026Fali Wang, Ali Al-Lawati, Iliyas Bektas +5LLM Evaluation LLM Reasoning with Graphs
The Pennsylvania State University, University Park, PA, USA · Rensselaer Polytechnic Institute, Troy, NY, US
AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents
Aug 1, 2026Alina Kapanova, Arun Kanhai, Natan Vidra +1AI Agent Benchmarks Synthetic Benchmark Generation
Anote AI · Cornell University · CUNY +1
SDDMO-Bench: A Benchmark Suite for Streaming Data-Driven Dynamic Multi-Objective Optimization
Aug 1, 2026Wenjie Xiao, Hui Bai, Junhao ChenBenchmark Design Multi-Objective Evolutionary Optimization
School of Xiangtan University Xiangtan, China
InferQ: A Database-Oriented Benchmark for Quantum Circuits Simulation
Jul 31, 2026Andrei Ilinescu, Aadi Patwardhan, Rihan HaiBenchmark Design Synthetic Benchmark Generation
Delft University of Technology, Delft, The Netherlands
Library Reachability in LSR-Synth: How Anti-Memorization Design Changes the Measurement of Symbolic Discovery
Jul 30, 2026Zhan'ao Yao, Liang Yin, Zhihao Gao +9Synthetic Benchmark Generation Data Contamination in Language Models
State Key Laboratory of High Performance Ceramics, Shanghai Institute of Ceramics, Chinese Academy of Sciences, 1295 Dingxi Road, Shanghai 200050, China · Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing 100049, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China +4
PAUSE: A User-Centric Benchmark for Personal AI Assistants in Unified Service Environments
Jul 29, 2026Haoyu Chen, Xirui Shi, Yuyao Wang +2Long-Horizon Agent Evaluation AI Agent Benchmarks
Electrical and Computer Engineering
DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series
Jul 29, 2026Dennis Thumm, Billy Tim Anthony, Ying ChenCounterfactual Evaluation Structural Causal Models
National University of Singapore, Singapore, Singapore
E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
Jul 26, 2026Weihuang Zheng, Tianyuan Zou, Eileen Ye +5Computer-Use Agent Benchmarks Tool-Use Evaluation
Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University
Benchmarking Text-to-SQL under Role-Based Access Control
Jul 24, 2026Yang Fei, Yangfan Jiang, Yin Yang +1Text-to-SQL Synthetic Benchmark Generation
National University of Singapore · Hamad Bin Khalifa University
SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data
Jul 22, 2026Zenghui Zhou, Xiaoyang Li, Xiaoxuan Qiao +2Synthetic Data Generation Agent Evaluation
ByteDance Inc., Beijing, China
Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator
Jul 21, 2026Yuxiang JiSynthetic Benchmark Generation Data Contamination in Language Models
Sunway College
SAGA: Synthetic Agentic Graph Architecture for Temporal Benchmark Generation
Jul 19, 2026Jiacheng Ding, Xiaofei ZhangSynthetic Data Generation Synthetic Benchmark Generation
University of Memphis · Memphis, Tennessee, USA · Tennessee
ReqGenX: An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents
Jul 18, 2026Ragib Shahariar Ayon, Rayed Fahmi, Sumon Biswas +1Requirements Engineering Synthetic Data Generation
Texas State University, San Marcos, TX, USA · Case Western Reserve University, Cleveland, OH, USA
Privacy-Aware Synthetic Video Benchmarking and Relational Evaluation for Worker-Under-Suspended-Load Detection
Jul 17, 2026Anshu Singh, Alejandro SeifSynthetic Benchmark Generation Privacy-Preserving ML
Government Technology Agency of Singapore
MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
Jul 16, 2026Goktug OzkanSynthetic Benchmark Generation AI Safety Evaluation
Department of Internal Medicine, Kutahya Emet Dr. Fazil Dogan State Hospital, Kutahya, Turkiye
ChronoQG: Towards a Temporally Expressive and Hop-Bounded Benchmark for Temporal Knowledge Graph Question Generation
Jul 16, 2026Xuemeng Liu, Zhengpin Li, Wanpeng Tang +2Synthetic Benchmark Generation Temporal KGs
Peking University Beijing, China · Shanghai University of Finance and Economics Shanghai, China
The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol
Jul 15, 2026Serkan BallıMultilingual Language Model Evaluation LLM-as-a-Judge
Department of Software Engineering, Mehmet Akif Ersoy University, Burdur, Türkiye · Department of Software Engineering, Mehmet Akif Ersoy University, Türkiye
SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
Jul 11, 2026Abhigya Verma, Khyati Mahajan, Amit Kumar Saha +4VLM Evaluation Vision-Language Models
ServiceNow AI · Mila · Université de Montréal
SYNRARE: Synthetic Rare Disease EHR Generation for ML Benchmarking
Jul 10, 2026Nicolai Dinh Khang Truong, Richard RöttgerRare Disease Diagnosis Synthetic Data Generation
Department of Mathematics and Computer Science, University of Southern Denmark, Campusvej 55, 5230, Odense, Denmark

Aug 26, 2026, cs.CLJ/K move · Enter open · S save

Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

Qiankai Xu, Qiguang Chen, Zixin Su, Wenhao Huang+3

ByteDance Seed · Nanjing University · Evolvent AI

Long-Form Document Generation Synthetic Benchmark Generation Continued Pretraining Synthetic Data Generation for Language Models Text Generation

Open paper AskPDF arXiv

A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leaves the structure of whole documents untouched. Scientific papers are written to a clear and largely uniform structure and make a natural substrate for lifting this paradigm to the document level. We present a pipeline that unfolds each paper into a multi-turn generation trajectory in which a teacher model reconstructs the writing process of the whole paper: a writing request, a global plan, and pre-writing deliberation for each section. All section texts and the abstract are kept verbatim from the source paper. We apply the pipeline to 1.8M quality-filtered arXiv papers and obtain a 60B-token corpus for continued pre-training (CPT) that is roughly twice the source text. The same reverse construction extends to instruction data and evaluation. We build an SFT dataset of 200K samples using answers derived from paper text. We also use held-out papers to construct PAW-Bench, a benchmark of 2,940 academic writing tasks with per-task rubrics and checklists. In controlled experiments, CPT on our corpus followed by SFT on public datasets improves writing performance while preserving general reasoning and improving long document reading. Replacing part of the writing SFT data with our synthetic instruction data further improves performance on PAW-Bench.

Momentum

Related topics

Latest papers 213