Organizations: School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute · School of Computer Science and Engineering, University of Washington · School of Artificial Intelligence, Shanghai Jiao Tong University
Formal verification of memory-manipulating programs critically depends on precise function specifications that capture memory states written by experts. This requirement has become a major bottleneck as large language models (LLMs) increasingly generate low-level systems code whose correctness cannot be assumed. To enable scalable formal verification, we focus exclusively on function specification generation, deliberately avoiding the synthesis of complex loop invariants that are central to traditional verification pipelines. We propose a neuro-symbolic framework for automatically generating memory-aware formal function specifications for C programs from natural language problem descriptions and function signatures. The pipeline first produces candidate specifications via in-context learning, and then iteratively refines them using compiler diagnostics from symbolic provers and the verification toolchain. In particular, we validate candidate specifications by constructing a proof for the negation of the specification with concrete examples, enabling machine-checked rejection of plausible-but-incorrect specifications. To support systematic evaluation, we introduce LeetCode-C-Spec, a new benchmark of 200 C programming problems for generating memory-aware formal function specifications. Experiments show that iterative refinement substantially improves syntactic validity, while symbolic prover-based refutation significantly enhances correctness assessment by filtering false positives that LLM-only judges frequently accept. Our results demonstrate that combining neural generation with symbolic feedback provides an effective approach to formal specification synthesis for memory-safe systems software.
Figures & tables
Figure 2: System architecture for LLM-based generation and refinement of memory-aware formal specifications. The pipeline consists of two main phases: (1) Specification Generation using LLM with in-context learning, and (2) Iterative Refinement guided by compiler feedback, symbolic execution, and counterexample-guided refutation to ensure both syntactic correctness.
Figure 4: Counterexample-guided refutation workflow. Given a candidate specification, we negate the postcondition and attempt to find a test case satisfying the precondition but violating the postcondition. If such a counterexample exists, symbolic execution in Coq can concretely demonstrate the specification’s incorrectness by exhibiting the violating heap state.
Category
Count
Percentage (%)
Difficulty
Easy
62
31.0
Medium
97
48.5
Hard
41
20.5
Data Structure
Integer/Number
147
73.5
Table 1: Distribution of the LeetCode-C-Spec-200 benchmark across difficulty levels (as assigned by LeetCode) and data structures. Note that a single problem may involve multiple data structures.
Institute of Software, Chinese Academy of Sciences · The Hong Kong University of Science and Technology · Guangzhou Institute of Technology, Xidian University