Coding agents are typically provided end-to-end behavioral specifications that the code they produce should satisfy, such as tests and input--output examples. How many such specifications must a developer provide, and which ones, for the agent to produce the developer's intended program? We argue that this cannot be determined a priori, for it depends on the structure of the implementation, which is left to the agent to decide. An agent first commits to a structure---a decomposition into data types, modules, and functions---which fixes a class of candidate programs; behavioral specifications then select a program within that class. A set of tests that suffices for one structure may leave another badly underdetermined, requiring different edge cases or exponentially more examples. Since the structure is chosen by the agent, developers cannot ensure their specification is complete without reverse-engineering it from thousands of lines of generated code. To resolve this, we introduce Skeleton Coding, in which developers jointly specify the structure---in high-level, code-like format---alongside behavioral specifications. Theoretically, we model AI-assisted coding as a control problem, and show that skeletons can reduce exponentially the number of tests developers must supply to steer an agent toward an acceptable program. Empirically, across four models and three benchmarks, the best-performing skeleton specifications reduce mean test error by 48.4% relative to the best end-to-end behavioral specifications. Even when the agent can query a complete reference implementation, adding a skeleton reduces mean test error relatively by up to 27.0%. Skeletons also narrow the average gap between small and large models by about 52.4% while lowering implementation cost by 15--26x, and reduce code erosion by 24.7% on average.
Figures & tables
Figure 1: Behavioral specifications do not pinpoint correct programs. (Left) We ask a coding agent to implement a code base from ProgramBench given its documentation and a fraction of its test suite. We then measure error rate across the hidden suite. On average, no number of tests provided results in a program that passes all hidden tests. (Middle) Even providing a full behavioral specification is not enough: We measure the pass rate with access to an oracle providing the full and exact input-output behavior. Even so, adding a skeleton with local contracts increases the pass rate averaged across models by 10.9% relative to the same setup without a skeleton on ProgramBench. (Right) A skeleton specifying the local contract for slicing a strided view.
Figure 2: Value of different types of specifications. We generate specifications of different kinds and measure the error rate of a coding agent using them. No matter the format, behavioral specifications that describe only end-to-end behavior underperform specifications that describe the developer’s intended structure. Of the structural specifications, code-like specifications that describe function signatures and invariants outperform design files in natural language. Adding overly detailed specifications decreases the accuracy of smaller models.
Figure 3: Skeletons and tests shrink the set of programs the agent considers : (a) Without a skeleton, the agent’s prior spreads over all programs. (b) Fixing a skeleton restricts the agent to programs with that skeleton. (c) Each set of end-to-end tests further keeps only the programs that agree with the reference on the tested inputs.
Figure 4: A strong skeleton makes better use of tests. Visualization of how tests affect the two skeletons of Theorem 2.1 .
Figure 5: Skeletons bridge the gap between large and small models : Skeleton specifications significantly improve performance of smaller models, leading to accuracy similar to that of larger models at a lower cost. In particular, the entire cost–performance Pareto frontier (dashed) is formed mostly by small models with structural specifications.
Figure 6: (Left) More architecture detail does not match skeletons. Adding architecture sections to the markdown file slightly raises mean pass rate, but the resulting pass rate remains below that achieved with skeleton-based specifications. (Right) Accuracy improvement with additional behavioral observations. On ProgramBench, we show the average test pass rate increases as we increase the number of observations provided to the model (unlimited lets the agent decide), but in all cases end-to-end behavioral specifications perform worse than skeletons with just 20 queries.
Figure 7: Structural specifications produce less eroded code than behavioral specifications : Across all benchmarks and models, structural specifications reduce code erosion by up to 24.7%, compared to the task-only prompt. Meanwhile, behavioral specifications increase it. Verbosity varies little across conditions.
Figure 8: Skeletons reduce query effort and cost during iterative repair : For iterative repair tasks, using skeletons reduces the total cost of implementation, and also the number of user queries. We model user queries with the number of calls to the program binary. Additionally, final code has lower erosion and higher maintainability.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Skeleton coverage vs. pass rates : (Left, Middle) We select a subset of the functions to add to the skeleton based on their complexity. Providing additional coverage increases pass-rate. (Right) Adding per-folder descriptions to a directory sketch improves the pass rates.
Figure 10: Value of different types of skeletons : Providing skeletons improves the pass-rates on ProgramBench despite query access to the oracle executable.
A coding agent can bootstrap itself. Starting from a 926-word specification and a first implementation produced by an existing agent (Claude Code), a newly generated agent re-implements the same specification correctly from scratch. This reproduces, in the domain of AI coding agents, the classical bootstrap sequence known from compiler construction, and instantiates the meta-circular property known from Lisp. The result carries a practical implication: the specification, not the implementation, is the stable artifact of record. Improving an agent means improving its specification; the implementation is, in principle, regenerable at any time.
Martin Monperrus
KTH Royal Institute of Technology, Stockholm, Sweden.
Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts. Yet existing methods fall short in agent pipelines: much of the reusable text is missing from their corpora or stored in a form that differs from what the agent emits, and their draft lengths ignore that accept length varies across agents and drifts over turns. We present AgSpec, a framework that supplies the corpus and draft-length policies that existing retrieval engines lack in coding-agent pipelines. AgSpec retrieves from session, workspace, and global corpora, retaining the ongoing session trajectory and indexing opened files in the agent's emission format. It bounds each agent's draft length with an offline-profiled cap and adapts the length online from verification feedback. On two repository-level multi-agent coding benchmarks, AgSpec outperforms five retrieval-based drafters and EAGLE-3 in most evaluated settings, raising generation throughput over autoregressive decoding up to 4.37× at batch size 1 and 4.76× at batch size 16. AgSpec also remains effective on benchmarks without a repository or a multi-agent pipeline, showing that its gains generalize to coding agents broadly.
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
Gabriel Orlanski, Devjeet Roy, Alexander Yun +7
University of Wisconsin–Madison · Washington State University · MIT