Specification-Based Testing

Latest papers

18 papers6 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications
Oct 29, 2026Yishuo Cai, Minzheng Wang, Liang Zhao +5Code Generation Long-Horizon Agent Tasks
LLM Core, Xiaomi · Peking University · University of Hong Kong
Formal Specifications with Data-Driven Signal Analysis
Oct 28, 2026Roderick Bloem, Benedikt Maderbacher, Cristinel Mateis +3Fault Detection Anomaly Detection
Graz University of Technology, Graz, Austria · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · AIT Austrian Institute of Technology, Vienna, Austria +1
Skeleton Coding: Program Structure as a Specification for Agents
Oct 28, 2026Aditya Golatkar, Alessandro Achille, Renos Zabounidis +9Code Generation Agentic Code Generation
AWS Agentic AI
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
Oct 27, 2026Binhang Qi, Yiming Liu, Yun Lin +3Automated Test Generation LLM Agent Evaluation
National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1
What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks
Oct 22, 2026Fei DingSoftware Engineering Benchmarks Benchmark Auditing
Alibaba Group
SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Oct 12, 2026Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5AI Coding Agents Specification-Based Testing
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
Jul 24, 2026Junda Zhao, Shurui Zhou, Eldan CohenLLM Prompting Specification-Based Testing
University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada · University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada
Specification Grounding Drives Test Effectiveness for LLM Code
Jul 7, 2026Amin Haeri, Mahdi GhelichiLLM Grounding Automated Software Testing
Model Development Innovation TD Bank, Toronto, Canada
MAS-Lab: A Specification-Driven Validation Framework for Reliable Multi-Agent Systems
Jun 29, 2026Jordan Augé, Giovanna Carofiglio, Giulio Grassi +1Multi-Agent LLM Systems Automated Software Testing
Cisco Systems Paris, France
SPADE: Structure-Prior Adaptive Decision Estimation
Jun 22, 2026Yifan WangModel Selection Scientific ML
Department of Mechanical Engineering, McGill University · QC, H3A 2T7, Canada
Cluster-Aware Dual-Level Test Specification Generation for Large-Scale Automotive Software Requirements
Jun 15, 2026Hazem Ayman, Menna Sedik, Kareem Mostafa +3Requirements Engineering Automated Software Testing
CairoMotive Cairo, Egypt
Inferring Code Correctness from Specification
May 28, 2026Tambon Florian, Papadakis MikeAutomated Software Testing Code Generation Evaluation
University of Luxembourg Luxembourg, Luxembourg
STAB: Specification-driven Testing for Algorithmic Bottlenecks
May 27, 2026Soohan Lim, Joonghyuk Hahn, Hyundong Jin +1Competitive Programming Automated Test Generation
Yonsei University, Seoul, Republic of Korea
Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications
May 24, 2026Xiaoyue Lu, Xianglin Yang, Haijun Liu +4Automated Software Testing Language Model Safety Evaluation
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · National University of Singapore, Singapore · Independent Researcher
STADA: Specification-based Testing for Autonomous Driving Agents
Mar 11, 2026Joy Saha, Trey Woodlief, Sebastian Elbaum +1Specification-Based Testing Automated Test Generation
University of Virginia, Virginia, USA · William & Mary, Virginia, USA
Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications
Mar 9, 2026Tzafrir RehanTool-Using Agents AI Agent Evaluation
Fiverr Labs
A Governance and Evaluation Framework for Deterministic, Rule-Based Clinical Decision Support in Empiric Antibiotic Prescribing
Feb 24, 2026Francisco José Gárate, Paloma Chausa, Diego Moreno +3Specification-Based Testing Clinical Decision Support
Grupo de Bioingeniería y Telemedicina, ETSI Telecomunicación, Centro de Tecnología Biomedica, Universidad Politécnica de Madrid, Avenida Complutense 30, 28040 Madrid, Spain · Instituto de Investigación Hospital 12 de Octubre (imas12), Hospital Universitario 12 de Octubre, 28041 Madrid, Spain · Innovation Unit, Parc Sanitari Sant Joan de Deu, Sant Boi de Llobregat, Spain; Institut de Recerca Sant Joan de Déu, Esplugues de Llobregat, Spain +2
Requirement-Based Testing: Enhancing Reinforcement Learning with Game Theory
Jul 26, 2024Ocan Sankur, Thierry Jéron, Nicolas Markey +2Automated Software Testing Automated Test Generation
DEVINE · MERCE-France · Mitsubishi Electric Corporation, Tokyo, Japan

Oct 29, 2026J/K move · Enter open · S save

When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications

Yishuo Cai, Minzheng Wang, Liang Zhao, Linghao Zhang+4

LLM Core, Xiaomi · Peking University · University of Hong Kong

Code Generation Long-Horizon Agent Tasks Synthetic Benchmark Generation Coding Agent Benchmarks Specification-Based Testing

Open paper AskPDF arXiv

To succeed in real-world long-horizon software development, coding agents must satisfy multi-dimensional task specifications, such as interface compatibility and error handling, and do so without breaking existing dependencies. However, current works lack specification in both quantity and diversity, often emphasizing primary goals while ignoring complex constraints. Synthesizing tasks to bridge this gap presents two challenges: naively adding specifications risks reducing tasks to trivial checklists rather than compelling joint reasoning, and verifying these subtle interactions risks overlooking hidden flaws or rejecting valid alternatives. To address these challenges, we introduce CodeSpecWeave, a framework for synthesizing tasks with rich, interdependent specifications. Grounded in real developer seeds, a synthesis agent iteratively develops both coupled task requirements and executable tests, while a review agent audits solution trajectories from multiple models to guarantee test reliability. Using this pipeline, we construct 4,670 training tasks across 2,865 repositories. Applying Reinforcement learning on these tasks significantly improves performance over the base model on DeepSWE-v1.1, largely outperforming existing baselines. Furthermore, we curate CodeSpecWeave-Bench, a rigorous benchmark comprising 100 tasks and 1,044 specifications, which clearly separates frontier models and reveals significant headroom in handling interdependent engineering constraints.