Synthetic Benchmark Generation

Latest papers 213

All topics
CardsList
  1. DrugTargetWorld: A Synthetic Biobank for Training and Benchmarking AI Scientists

    Oct 7, 2026Samuel Margolis, Paul Schmiedmayer, Alan Huang +12Drug DiscoveryAI Agent Benchmarks

  2. RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement

    Oct 7, 2026Renxiong Wang, Darvin Yi, Abril Herrlein +16Self-Improving AgentsRecursive Self-Improvement

  3. MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

    Oct 5, 2026Abdul Basit, Muhammad Abdullah Hanif, Muhammad ShafiqueLLM EvaluationSynthetic Benchmark Generation

  4. Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI

    Sep 30, 2026Nishtha N. Vaidya, Stephan Grimm, Thomas Hubauer +1LLM EvaluationSynthetic Benchmark Generation

  5. Teaching LLMs to Generate Challenging MILP Instances via Solver Feedback

    Sep 29, 2026Jitin Singla, Parikshit Pareek, Pratik Jawanpuria +1Mixed-Integer Linear ProgrammingLarge Language Model-Guided Optimization

  6. MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

    Sep 27, 2026Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu +3LLM EvaluationBenchmark Design

  7. A Living Benchmark for Information Retrieval from Electronic Health Records

    Sep 24, 2026Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani +23Synthetic Benchmark GenerationClinical QA

  8. SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback

    Sep 24, 2026Chenxi Li, Wenxuan Zeng, Yun Luo +4Execution-Guided Code GenerationScientific Code Generation

  9. SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation

    Sep 21, 2026Fatih Deniz, Yazan Boshmaf, Issa KhalilLLM Safety BenchmarksLLM Evaluation

  10. A paired synthetic construction-site image dataset for robust computer vision under adverse conditions

    Sep 21, 2026Viet Huy Duong, Ruoxin Xiong, Md Abdullah Al Forhad +1Synthetic Data GenerationSynthetic Benchmark Generation

  11. AURORA: A Natural Language-Driven Agentic Framework for Understanding, Reasoning, and Orchestrating Reliable Air-Ground Co-Simulation

    Sep 17, 2026Keshu Wu, Hao Zhang, Rui Gan +4LLM Agent OrchestrationSynthetic Benchmark Generation

  12. WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

    Sep 14, 2026Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy +1Synthetic Data GenerationAI Agent Evaluation

  13. Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

    Sep 14, 2026Baoyang Jiang, Fengchun Zhang, Leyuan Wang +9Benchmark ConstructionEmbodied QA

  14. SynCo: Synthetic Community-Aware Attributed Graph Generator for Graph Neural Network Benchmarking

    Sep 9, 2026Guilherme Henrique Messias, Mariana Caravanti de Souza, Sylvia Iasulaitis +1Synthetic Data AugmentationGraph Generation

  15. FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?

    Sep 7, 2026Jingpu Yang, Fengxian Ji, Jinri Guo +7Computer-Use Agent BenchmarksFinancial Services

  16. SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

    Sep 2, 2026Michael J. BommaritoLLM EvaluationLibrary and Information Science

  17. Peg-in-Bench: A Modular Benchmark for High-Precision Robotic Insertion

    Sep 1, 2026Yosel Delgado, José G. Buenaventura-Carreón, Floris Erich +4Contact-Rich Robotic ManipulationGeneralization in Robotic Manipulation

  18. Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs

    Aug 31, 2026Jonathan Zheng, Zirui Shao, Alan Ritter +1Synthetic Data PretrainingKnowledge Editing

  19. Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

    Aug 26, 2026Qiankai Xu, Qiguang Chen, Zixin Su +4Long-Form Document GenerationSynthetic Benchmark Generation

  20. ππ-SUB: A Physics-Informed Synthetic Underwater Benchmark Dataset for Underwater Image Enhancement

    Aug 11, 2026Namritha Lasyapriya Maddali, Rajini Makam, Suresh Sundaram +1Underwater Image EnhancementUnderwater Imaging

  21. Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction

    Aug 9, 2026Zhuowen Liang, Zhengxuan Zhang, Jiayang Wang +2Document Information ExtractionSynthetic Benchmark Generation

  22. SABRE: Scalable and Automated Benchmarking of VLMs under Stress

    Aug 7, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM EvaluationVLM Robustness

  23. Ask-E: An Environment for Calibrated Question Generation

    Aug 7, 2026Sarah Pratt, Jae Sung Park, Scott Geng +1Mathematical Reasoning BenchmarksSynthetic Benchmark Generation

  24. Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints

    Aug 6, 2026Omid Bazgir, Md Nasir, Jacob Hoffman +6Synthetic Benchmark GenerationBenchmark Validity

  25. Recursive Synthesis for Long-Horizon Terminal Tasks

    Aug 5, 2026Zhongzhi Li, Yucheng Shi, Zongxia Li +8Terminal AgentsSynthetic Benchmark Generation

  26. LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

    Aug 5, 2026Jiahao Zhang, Yongzhi Tong, Zelin Fu +4LLM EvaluationLLM Personalization

  27. From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

    Aug 3, 2026Can Wang, Haoran Chen, Haowen Gao +3LLM EvaluationBenchmark Design

  28. Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models

    Aug 2, 2026Fali Wang, Ali Al-Lawati, Iliyas Bektas +5LLM EvaluationLLM Reasoning with Graphs

  29. AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents

    Aug 1, 2026Alina Kapanova, Arun Kanhai, Natan Vidra +1AI Agent BenchmarksSynthetic Benchmark Generation

  30. InferQ: A Database-Oriented Benchmark for Quantum Circuits Simulation

    Jul 31, 2026Andrei Ilinescu, Aadi Patwardhan, Rihan HaiBenchmark DesignSynthetic Benchmark Generation

  31. Library Reachability in LSR-Synth: How Anti-Memorization Design Changes the Measurement of Symbolic Discovery

    Jul 30, 2026Zhan'ao Yao, Liang Yin, Zhihao Gao +9Synthetic Benchmark GenerationData Contamination in Language Models

  32. DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series

    Jul 29, 2026Dennis Thumm, Billy Tim Anthony, Ying ChenCounterfactual EvaluationStructural Causal Models

  33. E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

    Jul 26, 2026Weihuang Zheng, Tianyuan Zou, Eileen Ye +5Computer-Use Agent BenchmarksTool-Use Evaluation

  34. Benchmarking Text-to-SQL under Role-Based Access Control

    Jul 24, 2026Yang Fei, Yangfan Jiang, Yin Yang +1Text-to-SQLSynthetic Benchmark Generation

  35. SenWorld: A Digital-Twin Simulation for Generating Context-Rich Evaluation Data

    Jul 22, 2026Zenghui Zhou, Xiaoyang Li, Xiaoxuan Qiao +2Synthetic Data GenerationAgent Evaluation

  36. SAGA: Synthetic Agentic Graph Architecture for Temporal Benchmark Generation

    Jul 19, 2026Jiacheng Ding, Xiaofei ZhangSynthetic Data GenerationSynthetic Benchmark Generation

  37. ReqGenX: An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents

    Jul 18, 2026Ragib Shahariar Ayon, Rayed Fahmi, Sumon Biswas +1Requirements EngineeringSynthetic Data Generation

  38. MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection

    Jul 16, 2026Goktug OzkanSynthetic Benchmark GenerationAI Safety Evaluation

  39. ChronoQG: Towards a Temporally Expressive and Hop-Bounded Benchmark for Temporal Knowledge Graph Question Generation

    Jul 16, 2026Xuemeng Liu, Zhengpin Li, Wanpeng Tang +2Synthetic Benchmark GenerationTemporal KGs

  40. The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

    Jul 15, 2026Serkan BallıMultilingual Language Model EvaluationLLM-as-a-Judge

  41. SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

    Jul 11, 2026Abhigya Verma, Khyati Mahajan, Amit Kumar Saha +4VLM EvaluationVision-Language Models

  42. SYNRARE: Synthetic Rare Disease EHR Generation for ML Benchmarking

    Jul 10, 2026Nicolai Dinh Khang Truong, Richard RöttgerRare Disease DiagnosisSynthetic Data Generation