Specification-Based Testing

Follow this topic Ask about this topic

Latest papers

13 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Oct 12, 2026Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5AI Coding Agents Specification-Based Testing
The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software +2
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
Jul 24, 2026Junda Zhao, Shurui Zhou, Eldan CohenLLM Prompting Specification-Based Testing
University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada · University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada
Specification Grounding Drives Test Effectiveness for LLM Code
Jul 7, 2026Amin Haeri, Mahdi GhelichiLLM Grounding Automated Software Testing
Model Development Innovation TD Bank, Toronto, Canada
MAS-Lab: A Specification-Driven Validation Framework for Reliable Multi-Agent Systems
Jun 29, 2026Jordan Augé, Giovanna Carofiglio, Giulio Grassi +1Multi-Agent LLM Systems Automated Software Testing
Cisco Systems Paris, France
SPADE: Structure-Prior Adaptive Decision Estimation
Jun 22, 2026Yifan WangModel Selection Scientific ML
Department of Mechanical Engineering, McGill University · QC, H3A 2T7, Canada
Cluster-Aware Dual-Level Test Specification Generation for Large-Scale Automotive Software Requirements
Jun 15, 2026Hazem Ayman, Menna Sedik, Kareem Mostafa +3Requirements Engineering Automated Software Testing
CairoMotive Cairo, Egypt
Inferring Code Correctness from Specification
May 28, 2026Tambon Florian, Papadakis MikeAutomated Software Testing Code Generation Evaluation
University of Luxembourg Luxembourg, Luxembourg
STAB: Specification-driven Testing for Algorithmic Bottlenecks
May 27, 2026Soohan Lim, Joonghyuk Hahn, Hyundong Jin +1Competitive Programming Automated Test Generation
Yonsei University, Seoul, Republic of Korea
Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications
May 24, 2026Xiaoyue Lu, Xianglin Yang, Haijun Liu +4Automated Software Testing Language Model Safety Evaluation
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · National University of Singapore, Singapore · Independent Researcher
STADA: Specification-based Testing for Autonomous Driving Agents
Mar 11, 2026Joy Saha, Trey Woodlief, Sebastian Elbaum +1Specification-Based Testing Automated Test Generation
University of Virginia, Virginia, USA · William & Mary, Virginia, USA
Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications
Mar 9, 2026Tzafrir RehanTool-Using Agents AI Agent Evaluation
Fiverr Labs
A Governance and Evaluation Framework for Deterministic, Rule-Based Clinical Decision Support in Empiric Antibiotic Prescribing
Feb 24, 2026Francisco José Gárate, Paloma Chausa, Diego Moreno +3Specification-Based Testing Clinical Decision Support
Grupo de Bioingeniería y Telemedicina, ETSI Telecomunicación, Centro de Tecnología Biomedica, Universidad Politécnica de Madrid, Avenida Complutense 30, 28040 Madrid, Spain · Instituto de Investigación Hospital 12 de Octubre (imas12), Hospital Universitario 12 de Octubre, 28041 Madrid, Spain · Innovation Unit, Parc Sanitari Sant Joan de Deu, Sant Boi de Llobregat, Spain; Institut de Recerca Sant Joan de Déu, Esplugues de Llobregat, Spain +2
Requirement-Based Testing: Enhancing Reinforcement Learning with Game Theory
Jul 26, 2024Ocan Sankur, Thierry Jéron, Nicolas Markey +2Automated Software Testing Automated Test Generation
DEVINE · MERCE-France · Mitsubishi Electric Corporation, Tokyo, Japan

Oct 12, 2026J/K move · Enter open · S save

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

Zhiqing Zhong, Mingyu Chen, Aoyang Fang, Yuzhong Zhang+4

The Chinese University of Hong Kong, Shenzhen · Aalto University · Lero, the Research Ireland Centre for Software+2

AI Coding Agents Specification-Based Testing Automated Test Generation Software Engineering Benchmarks

Open paper AskPDF arXiv

As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenarios that thoroughly exercise the code, but current benchmarks provide limited insight into whether agents can construct such evidence themselves. We introduce SpecGuardBench, a benchmark of 291 tasks from 13 real-world repositories, covering 8 types of verification logic such as functional postconditions and interaction protocols. Given a natural language specification, repository context, and a reference implementation, an agent constructs a validation harness comprising executable checks and test scenarios. The checks should reject implementations that violate the specification while accepting those that satisfy it, and the scenarios should thoroughly exercise the code. Across four proprietary and open-weight coding agents, none constructs a harness scored as faithful on the benchmark's implementations for more than 63% of the tasks. The bottleneck lies less in exploring test scenarios than in constructing checks that faithfully reflect the specification. Specifically, the agents show a clear tendency to write checks that mirror the reference implementation rather than the specification, and therefore reject implementations that the specification permits.