cs.SESep 25, 2026
SaveCompact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer
Organizations: Daffodil International University, Dhaka, Bangladesh · HawAII, University of Hawai‘i at M¯anoa, Honolulu, HI, USA
Abstract
We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.
Figures & tables
| Work | Direction | Granularity | Oracle | Optimizes writer? |
|---|---|---|---|---|
| RTC ( Allamanis et al. 2024 ) | code NL code | function | unit tests | no |
| IdentityChain ( Min and others 2024 ) | code NL | function | test-output match | no |
| RTCE ( Maveli et al. 2026 ) | invertible round trip | snippet | exact match | no |
| SWE-bench ( Jimenez and others 2024 ) | NL code | repo | fail/pass tests | no |
| MRG-Bench ( Li and others 2025 ) | context code | repo | runnable tests | no |
| CoderEval ( Yu and others 2024 ) | NL code | function | unit tests | no |
Table 1: Positioning against related benchmarks. Prior roundtrip and execution-based benchmarks measure a model or a description; ours additionally uses the roundtrip score to optimize the system that writes the description.
| Condition | No desc. | With desc. |
|---|---|---|
| Contract only in description | 0/3 | 3/3 |
| Contract visible in code | 3/3 | 3/3 |
| 1 of 5 facts present | 0/5 | 1/5 |
| 3 of 5 facts present | 0/5 | 3/5 |
| 5 of 5 facts present | 0/5 | 5/5 |
| Complete, compact (38w) | 0/5 | 5/5 |
Table 2: When and why documentation helps a weak agent, as pass counts without versus with the description. Documentation helps only when it carries a contract the code does not expose; success rises with the number of facts the description includes; and a compact and a verbose description carrying the same facts give the same result.
| Fixture | Lines | Density | Fidelity |
|---|---|---|---|
| contains | 48 | 1.57 | 1.00 0.00 |
| point | 1376 | 0.56 | 0.67 0.09 |
| unitsystem | 205 | 1.22 | 0.63 0.19 |
| unitsystem_v2 | 205 | 1.30 | 0.61 0.54 |
| homomorphisms | 549 | 0.94 | 0.33 0.00 |
| ast | 1869 | 0.39 | 0.18 0.32 |
Table 3: Per-fixture roundtrip results on the eleven SWE-bench Verified fixtures. Fidelity is the fraction of original tests passed by code regenerated from the description alone, reported as mean standard deviation over three independent runs; density is tokens in the description divided by tokens in the code, measured on the same runs. The bottom row aggregates across fixtures.
| Failure mode | Example |
|---|---|
| Lost output literal | omits an exact string the tests check |
| Lost imports | drops a needed import |
| Lost constraint | an invariant is not restated |
| Recitation blocked | declines to reproduce known source |
Table 4: Failure modes when a description fails to regenerate its file.
Figure 1: Roundtrip fidelity (mean SD over three runs) against description density (tokens in the description over tokens in the code) across the eleven Verified fixtures.
| Fixture | OpenWiki | Our description |
|---|---|---|
| saferepr | 0.00 | 0.73 |
| contains | 0.67 | 1.00 |
| unitsystem | 0.00 | 0.85 |
Table 5: Roundtrip fidelity of documentation produced by an external tool (OpenWiki) against our own descriptions, scored through the identical regenerate-and-evaluate pipeline.
| Documentation given to Flash-lite | Passes |
|---|---|
| None | 0/4 |
| OpenWiki | 0/4 |
| Our description | 4/4 |
Table 6: The deny-override task under three documentation conditions. Each row reports how many of four independent attempts by Flash-lite implement a resolve method that satisfies the priority and deny-override contract. OpenWiki’s documentation, which reports the code’s observed order-dependent behavior, leaves the agent exactly where no documentation leaves it.
Figure 2: The prompt optimization loop. The optimizer proposes a describe prompt; the benchmark scores the descriptions it produces; the prompt is kept if the score improves. Optimization uses a training set of fixtures, and the discovered prompt is then evaluated on a held-out set never seen during optimization.
Figure 3: Optimizing the describe prompt. Left: the objective and mean fidelity climb across iterations; the loop reaches fidelity 1.0, then reduces length. Right: the discovered prompt raises held-out fidelity from 0.5 to 1.0 on fixtures it never optimized on.
| Fixture | Issue | Baseline | Optimized |
|---|---|---|---|
| contains | 0.33 | 0.56 | 0.83 |
| point | 0.00 | 0.00 | 0.77 |
| unitsystem | 0.00 | 1.00 | 0.99 |
| unitsystem_v2 | 0.00 | 0.00 | 0.97 |
| homomorphisms | 0.00 | 0.00 | 0.22 |
| ast | 0.18 | 0.30 | 0.88 |
Table 7: Issue resolution with the source file withheld. Mean fraction of the instance’s tests passed over three attempts. Issue gives the agent the issue text alone; Baseline and Optimized additionally supply a description of the pre-fix file from the baseline and optimized describe prompts.
| Setting | Model | Paired | Issue only | +Compact | +Full | +Context |
|---|---|---|---|---|---|---|
| Kind-B multi-file | Qwen 3.6 | 43 | 14 | – | 11 | – |
| Kind-B django | Gemini 3.1 Pro | 15 | 9 | 8 | 4 | – |
| SCB django, control | Gemini 3.1 Pro | 29 | 14 | – | – | 15 |
| SCB django | Gemini 3.1 Pro | 32 | 15 | 16 | – | – |
| SCB Lite | Gemini 3.8 Flash | 58 | 33 | 29 | – | 30 |
Table 8: Issue resolution with the source present, across models, fixture sets, and documentation types. Each cell is the number of resolved tasks out of the Paired total, the tasks that produced a scorable edit under every condition compared in that row; a dash means the condition was not run in that setting. +Compact is the roughly eighty-word static description of the target file, +Full the full-length description from the optimized prompt, and +Context the benchmark’s retrieved past-task experience. For Kind-B multi-file, 43 of 50 attempted fixtures paired. The SCB django control and the SCB django compact comparison come from separate runs with different attrition, so their issue-only counts (14 and 15) are taken over different task sets. No documentation condition improves on issue-only by more than a single task, and the full-length description is the worst condition wherever it was run.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent model | No desc. | Compact (38w) | Verbose (664w) |
|---|---|---|---|
| Flash-lite | 0/8 | 8/8 | 8/8 |
| Gemma 4 31B | 0/8 | 8/8 | 8/8 |
| Gemma 3 4B | 0/8 | 8/8 | 8/8 |
Table 9: Five-fact task. A compact and a verbose description carrying the same facts give identical uplift, from Flash-lite down to a 4B model.
| Agent model | Compact | Verbose | Verbose |
|---|---|---|---|
| (listed) | (described) | (buried) | |
| Flash-lite | 8/8 | 0/8 | 8/8 |
| Gemini 2.5 Flash | 4/4 | 0/4 | 4/4 |
| Gemma 3 4B | 8/8 | 0/8 | 8/8 |
Table 10: Harder task with a confound control. The verbose description fails only when it omits the exact key strings the task needs. Once those strings are present, even buried in long filler, it passes as well as the compact one.
| Repository | Resolved / Scored |
|---|---|
| matplotlib | 8/9 |
| django | 14/22 |
| scikit-learn | 3/5 |
| psf (requests) | 2/3 |
| sympy | 6/21 |
| astropy | 1/2 |
Table 11: Issue-only resolution on SWE-ContextBench Lite by repository (Gemini 3.8 Flash, official Docker evaluation), over the 68 of 99 tasks that produced a scorable edit. pylint-dev (one task) and pytest-dev (two tasks) produced no scorable edit and are omitted.
More in this topic
Software Engineering
Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.
See all 265 papers