Mind the Gap: A Deployed Multi-Agent System for Documentation Maintenance
Organizations: Amazon
Abstract
The "AI revolution" has enabled teams across the technology industry to deliver software features at unprecedented velocity. While code evolves faster than ever, customer-facing documentation often struggles to stay up to speed. As users increasingly rely on LLM-based agents that consume documentation as their primary interface, the impact of this documentation gap worsens: stale information propagates and is treated as authoritative. Signals for identifying where documentation has fallen behind already exist in abundance, both explicit (e.g., resolved support tickets revealing what documentation lacks) and implicit (e.g., code-documentation discrepancies detectable from automated loops). We deploy a multi-agent system, built around a triage agent we call "TriAgent," that continuously extracts these signals and repairs documentation drift. Over 43 days of shadow-mode operation, our system triaged 499 tickets and surfaced 92 gaps in the documentation. Using a panel of LLM judges from independent model families, 81.1% of TriAgent's verifiable assertions were correct (245 of the 302 assertions the human resolution addresses; majority of three judges); on length-comparable pairs the panel preferred TriAgent's response in 70.5% of pairs and the human engineer's in 23.2% (the rest had no judge majority). Importantly, agent- and human-sourced replies tended to be complementary, with the human often surfacing insights that were either buried within the existing documentation or missing from it entirely; our system turns each such insight into a documentation gap and, after human review, a fix. Repairing a gap measurably improved the agent's later answer: when the agent re-answered the same ticket after the repair, blinded judges preferred the post-repair answer over its own pre-repair answer 71.4% of the time (n=35). A blinded human audit confirms strong alignment on self-improvement.
Figures & tables
| Agent | Stage | Mode | Role |
|---|---|---|---|
| TriAgent | Triage | Live | First-response classification/drafting for incoming tickets |
| Feedback Loop | Detection | Live | Blind re-triage of resolved tickets, structured learning |
| Archivist | Detection | Batch | Gap discovery (harvest) and remediation orchestrator |
| Student | Detection | Proactive | Adversarial and speculative probing of KB topics |
| Code-Merge Watcher | Detection | Proactive | Post-merge documentation-drift detection |
| Documentator | Remediation | Batch | Research draft self-review code review |
| Metric | Value |
|---|---|
| Tickets triaged | 499 |
| Documentation gaps discovered | 92 |
| Median agent response availability lead time | 4.6 h |
| Agent drafting time (gap submitted fix) | 5 min |
| Outcome | All pairs | Comparable length |
|---|---|---|
| Agent preferred | 78.1% | 70.5% |
| Human preferred | 18.2% | 23.2% |
| No majority | 3.6% | 6.2% |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Definition | Example |
|---|---|---|
| missing | No KB coverage for the topic | A feature ships with no documentation page |
| stale | Coverage exists but references outdated behavior | A threshold changed in code but docs still show the old value |
| contradictory | Two KB sources conflict | Tutorial says “method A”; reference guide says “method B” |
| ambiguous | Coverage exists but is underspecified | Docs say “may take some time” with no latency bound |
| broken_xref | Link or pointer is dead or incorrect | A “see also” pointer targets a moved page |
| Stage | Count |
|---|---|
| Tickets triaged (deployment scale) | 499 |
| Pairs with both agent & human response | 333 |
| after mechanical + length filters | 199 |
| non-answers (agent correctly declined) | |
| Answered pairs (evaluation set) | 192 |
| Panel | Probe | |
| Correctness (1121 assertions) | ||
| Correct | 245 | 279 |
| Incorrect | 57 | 164 |
| Unverifiable | 813 | 678 |
| Correct by this judge only | 29 | 63 |
| Usefulness (192 pairs) | ||