From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation

Zhijun Gao; Jing Chen

cs.SEAug 20, 2026

From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation

Authors: Zhijun Gao, Jing Chen

Organizations: Peking University Beijing, China

Abstract

Technical documentation is written for human developers, but an increasing share of software changes is now authored by autonomous coding agents. Which documents they consult, when, and what follows remain unknown. We conduct a behaviour-grounded study of agent-documentation interaction across two public datasets: 557 agentic coding sessions from SWE-chat, yielding 94,813 development events including 3,033 documentation interactions; and 33,097 agentic pull requests from AIDev, with 690,260 classified file-level change records. Four findings challenge current documentation practice. First, agents' documentation work is dominated by agent-facing artefacts: instruction files and working notes account for 60.5% of all documentation interactions, versus 10.6% for classical technical documentation and 1.3% for API references. Second, the link between consultation and code editing is unresolved: the adjacent transition probability is 0.002 and the unadjusted three-event lift 1.05, whereas a stage-adjusted model places it above unity (OR 1.33 [1.09, 1.62]); documentation creation is elevated unadjusted (lift 1.67) but its adjusted interval includes unity. Third, no explicit documentation-based validation sequence was observed, and consultation is associated with less immediate testing (lift 0.23, cluster CI 0.08-0.45; adjusted OR 0.39 [0.25, 0.60]). Fourth, consultation is self-initiated (70.2%) far more often than failure-driven (7.5%), and documentation trails code: among multi-commit pull requests changing both, code is touched first 4.7x more often. From these traces we derive a descriptive model of agent-documentation interaction as a two-lobed cycle rather than a linear journey, and show that two widely assumed properties of "agent-friendly" documentation - actionability and verifiability - lack consistent behavioural support. We release our pipeline, coding scheme, and event-level data.

Figures & tables

Figure 1. Agents’ documentation interactions across 557 SWE-chat sessions (panels a–d, $n=3{,}033$ documentation events) and 33,097 AIDev agentic pull requests (panel e). (a) Agent-facing documents — instruction files and agent working notes — account for 60.5% of interactions, whereas API references account for 1.3%; the right-hand column uses a magnified axis. (b) Testing and building are less frequent within three events of a consultation and remain so after adjustment; the two authoring outcomes differ between the unadjusted lift and the stage-adjusted odds ratio, so those associations remain unresolved. Intervals are session-level cluster-bootstrap estimates. (c) Interaction is predominantly self-initiated (70.2% of all 3,033 interactions; 62.5% of the 1,615 consultation events) rather than failure-driven (7.5%; 9.7%); the axis counts all interactions. (d) Of ten candidate stages, Validate and Escalate have zero events under our operational definitions (open markers); stages overlap and do not sum to 3,033. (e) Among pull requests that change both code and documentation, code is touched first 4.7 $\times$ more often, although 42.6% first touch both in a single commit, for which no order is observable; panel e uses a different dataset and unit of analysis (pull requests), and its error bars are Wilson intervals — the repository cluster-bootstrap equivalents are in Table 6 . Five panels summarising document types, post-consultation actions, interaction triggers, candidate interaction stages, and code–documentation ordering in pull requests.

Document type	Events	Share
Agent instructions †	1,074	35.4%
Agent working notes †	760	25.1%
Task / requirements	301	9.9%
Configuration	205	6.8%
README	197	6.5%
Other prose (residual)	165	5.4%

Table 1. Documentation-interaction events by document type (

n=3{,}033

events across 557 sessions). Agent-facing categories are marked

\dagger

. Together, the two agent-facing categories account for 60.5% of interactions; the nine categories constituting classical technical documentation (API reference, troubleshooting, architecture, schema, installation, examples, testing, contributing, and changelog) account for 10.6%.

Trigger	Events	Share
Agent initiative	1,236	40.8%
Implementation need	893	29.4%
User instruction	618	20.4%
Tool failure	217	7.2%
Planning	58	1.9%
Test failure	7	0.2%

Table 2. RQ2: what triggers documentation interaction (

n=3{,}033

interactions; consultation events alone are

n=1{,}615

, for which the self-initiated share is 62.5%). Self-initiated triggers outnumber failure-driven ones by 9.3

\times

Action	After consult.	Base	$\Delta p$ [95% CI]	Lift [95% CI]	Adj. OR [95% CI]
Create documentation	0.044	0.026	+0.0176 [+0.0040, +0.0320]	1.67 [1.14, 2.31]	1.41 [0.98, 2.02]
Edit code	0.232	0.221	+0.0113 [-0.0325, +0.0587]	1.05 [0.86, 1.27]	1.33 [1.09, 1.62]
Revise plan	0.015	0.019	-0.0044 [-0.0124, +0.0042]	0.77 [0.35, 1.24]	0.75 [0.43, 1.30]
Run test	0.005	0.022	-0.0169 [-0.0250, -0.0094]	0.23 [0.08, 0.45]	0.39 [0.25, 0.60]
Build	0.004	0.025	-0.0214 [-0.0292, -0.0143]	0.15 [0.02, 0.33]	0.25 [0.14, 0.44]

Table 3. Actions within three events of a documentation consultation (

n_{c}=1{,}615

anchors) compared with the non-anchor base rate (

n_{b}=93{,}198

), with session cluster-bootstrap intervals (2,000 resamples, percentile method) for the risk difference

\Delta p

and the lift ratio, and an adjusted association from a logistic GEE with exchangeable working correlation clustered by session and adjusted for development stage, within-session position, session length, and agent family. Unadjusted and adjusted estimates differ for the two authoring outcomes: documentation creation is elevated before adjustment, but its adjusted interval includes 1; code editing is indistinguishable from the baseline before adjustment, but is elevated after controlling for stage. Only the lower frequencies of testing and building are robust to both analyses. We therefore treat these reductions as the finding and the authoring associations as unresolved.

First recovery action	$n$ (%)	Resolved	95% CI
Read code	631 (31.0)	39/159	[.18, .32]
Retry directly	404 (19.9)	48/192	[.19, .32]
No action within horizon	318 (15.6)	5/67	[.03, .16]
Edit directly	312 (15.3)	45/160	[.22, .36]
Search code	251 (12.3)	17/73	[.15, .34]
Read documentation	109 (5.4)	7/11	[.35, .85]

Table 4. Failure recovery: the first action after each of 2,034 failure episodes and the fraction of episodes subsequently resolved. Resolution can be computed only for episodes followed by an observable test or build outcome, so the resolution denominators are smaller than the frequency counts. Documentation-based recovery has the highest point estimate but the widest interval; it overlaps every other strategy, so we draw no conclusion about the relative ranking.

	$k/n$	Prop. [95% CI]
(a) Documentation involvement
Code only	17,988/33,097	.543 [.538, .549]
Code + documentation	10,586/33,097	.320 [.315, .325]
Documentation only	3,164/33,097	.096 [.092, .099]
Neither (build/config/data)	1,359/33,097	.041 [.039, .043]
(b) First artefact touched

Table 5. Artefact-level results for RQ3 (AIDev pull requests): (a) documentation involvement among all analysable agentic PRs; (b) the artefact touched first, restricted to multi-commit PRs that change both artefact types and for which order is observable; and (c) merge rate by documentation involvement, reported as an association rather than a causal effect. All intervals are Wilson 95% CIs.

Statistic	$k/n$	Prop.	Cluster-bootstrap 95% CI	Wilson 95% CI	Width ratio
Sessions as clusters (SWE-chat)
Agent-facing, all interactions	1,834/3,033	0.605	[0.539, 0.665]	[0.587, 0.622]	3.6
consultation events	927/1,615	0.574	[0.508, 0.634]	[0.550, 0.598]	2.6
production events	893/1,401	0.637	[0.553, 0.713]	[0.612, 0.662]	3.2
Self-initiated, all interactions	2,129/3,033	0.702	[0.667, 0.733]	[0.685, 0.718]	2.0
Failure-driven, all interactions	228/3,033	0.075	[0.060, 0.093]	[0.066, 0.085]	1.8

Table 6. Cluster-bootstrap intervals (2,000 resamples) for the headline proportions, resampling sessions for SWE-chat statistics and repositories for AIDev statistics, alongside Wilson intervals that assume independent trials. The ratio column gives cluster-interval width divided by Wilson-interval width. Clustering widens every interval, especially for AIDev, where the largest repository contributes 8,911 pull requests and the ten largest contribute 44.7%. We report cluster intervals as the primary uncertainty estimates.

Document type	Cons.	%	Prod.	%
Agent instructions †	545	33.7	526	37.5
Agent working notes †	382	23.7	367	26.2
Task / requirements	129	8.0	170	12.1
Configuration	195	12.1	10	0.7
README	89	5.5	107	7.6
Other prose	72	4.5	93	6.6

Table 7. Consultation (Read, Search, Discover;

n_{c}=1{,}615

) versus production (Edit, Create;

n_{p}=1{,}401

) events by document type. Agent-facing categories are marked

\dagger

; they account for 57.4% of consultation and 63.7% of production. Configuration files are consulted far more than they are produced; agent working notes are consulted and produced in nearly equal measure. “Other low-frequency” pools troubleshooting, license/legal, and contributing, so the columns sum to

n_{c}

and

n_{p}

Statistic	Event	Sess.-eq.	Agent-wt.
Agent-facing, all interactions	60.5%	54.7%	55.1%
consultation events only	57.4%	50.5%	50.1%
production events only	63.7%	67.1%	66.3%
Self-initiated, all interactions	70.2%	63.2%	61.9%
Sessions w/ doc event	—	56.7%	59.8%

Table 8. Sensitivity of headline proportions to weighting. The event-weighted estimate pools all events (long sessions weigh more); the session-equal estimate averages per-session shares; and the agent-weighted estimate reweights sessions to the corpus agent distribution (83.8% Claude Code, 4,852/5,790 labelled sessions; 61 of the 5,851 released sessions are excluded from the reweighting — 52 carry no agent label and 9 belong to agents with fewer than 20 sessions), correcting our deliberate oversampling of minority agents. Because the sample was also stratified by session length, this corrects agent-level oversampling only and is not a full joint-stratum estimate. All three statistics describe the stratified sample; the broad pattern is stable across all three weightings.

Agent	With docs	Sessions	Rate [95% CI]
Gemini CLI	9	12	.750 [.468, .911]
Agent	8	12	.667 [.391, .862]
Claude Code	238	380	.626 [.577, .673]
OpenCode	45	99	.455 [.360, .552]
Codex	16	43	.372 [.244, .521]
Cursor	0	11	.000 [.000, .259]

Table 9. Fraction of sessions containing at least one documentation interaction, by agent. Sampling was deliberately non-proportional, so these per-agent estimates must not be pooled. The zero estimate for one agent is based on 11 sessions and does not constitute evidence of absence.

Candidate stage	Events	Status
Contribute/Update	1,401	strongly attested
Retrieve	1,344	strongly attested
Orient	462	attested
Interpret	413	attested
Revisit	360	attested
Discover	287	attested

Table 10. Evidence for each candidate stage of the documentation-interaction cycle (

n=3{,}033

documentation events). Two stages in the initial scheme are entirely unattested.

Software Engineering

Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.

See all 265 papers

Sep 25, 2026

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.

Md Shohel Arman, Igor Molybog

Daffodil International University, Dhaka, Bangladesh · HawAII, University of Hawai‘i at M¯anoa, Honolulu, HI, USA

Software Engineering AI Coding Agents Code Generation Benchmarks Text Summarization

Ask PDF

Sep 30, 2026

DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?

We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation. It asks whether an agent can produce documentation that experienced technical writers would accept in review. The benchmark contains 292 items from open source projects, including Helm, PostHog, and Mautic. Each item gives the agent a pre-change repository and a trigger, such as a code pull request or a reported documentation gap. The agent must first decide whether the documentation needs an update. For items that need one, the agent must produce an acceptable patch in one attempt. For items that do not need updates, the agent must abstain. Task-specific rubrics, validated with project maintainers, score each patch on accuracy, completeness, reader guidance, placement, and repository conventions. The composite score combines patch quality with correct abstention, and a score of 100 means an agent meets every requirement for the task. Scores should not be interpreted as a percentage of an expert's capability. We evaluated seven agents. The highest-scoring agent reached 47.3 out of 100 on the 117-item held-out split. In a separate audit of 1,267 patches, the most common failure modes were task-completion gaps (45.5%), technical inaccuracies (36.6%), and incomplete conceptual or reference coverage (32.5%). Analysis of the corresponding trajectories identified three key patterns associated with these failures: (1) describing interfaces without examining how readers use them (36.0%), (2) missing decisive evidence and filling the gaps with plausible assumptions (33.1%), and (3) stopping after finding the first plausible documentation surface and leaving other affected pages stale (30.1%).

Frances Liu, Manny Silva, Paige Calvert +2

PROMPTLESS · PROMPTLESS / DOC DETECTIVE

Software Engineering LLM Agent Evaluation Text Generation

Ask PDF

Apr 22, 2026

SWE-chat: Coding Agent Interactions From Real Users in the Wild

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains almost 18,000 sessions, comprising more than 229,000 user prompts and 2 million agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 25%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Only 59% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs - through corrections, failure reports, and interruptions - in 50% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.

Joachim Baumann, Vishakh Padmakumar, Xiang Li +3

Stanford University

Coding Agents Human-AI Collaboration LLM Agent Reliability

Ask PDF

Abstract

Software Engineering

Explore similar work

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?

SWE-chat: Coding Agent Interactions From Real Users in the Wild