What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

Yikai Zhao; Saurabh Pandey; Pradeep Kumar Misra

cs.CLOct 21, 2026

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

Authors: Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra

Organizations: Amazon

Abstract

Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF into tokens - and that layer is where an agent deployed in the wild silently loses information. We show that one layer is lost by every mainstream converter - the meaning an author encodes in how text looks - and that the loss makes agents confidently wrong rather than visibly incomplete. We build a task from 76 real policy documents: recover the requirements the author singled out, scored against the file's own OOXML properties by a deterministic matcher with no LLM judge. Formatting-blind ingestion collapses from micro F1 0.690 to 0.150 (docling) and 0.075 (plain text), a gap that replicates across six answering models from two vendors - five Claude models spanning the Opus, Sonnet and Haiku tiers, and OpenAI's GPT-5.6. An agent that cannot see the cue guesses, and docling's false-positive rate on requirements the author did not flag is four times ours. A span-level leave-one-out localises the loss to two cues - highlight (+0.302) and colour (+0.262), both Holm-adjusted p < 1e-4 - while bold, italic, monospace and strikethrough each sit within +/-0.003 of zero. The same asymmetry appears in PDF. Colour is also the cue no mainstream converter preserves and no parsing benchmark scores, and it is common: 32.8% of 6,460 documents carry a deliberate colour or highlight, 10.3% on an obligation. A pre-registered test shows that typing a cue for the agent adds +0.085 beyond merely preserving it (Holm-adjusted p = 0.049). We validate the labels against rendered page images through an independent modality (kappa = 0.72) and bound what wording alone can achieve (AUC 0.664). For agent builders the practical finding is that ingestion, not reasoning, is where this reliability is won or lost, and that the fix is cheap and measurable.

Figures & tables

Arm	micro	prec.	FP	$\Delta$ macro	$p_{\text{Holm}}$
	F1		rate	F1
ours	0.690	0.824	0.011	—	—
ours , salience roles	0.687	0.834	0.010	$-0.005$	1.00
ours , untyped markup	0.605	0.811	0.010	$+0.092$	0.025
docx2python	0.586	0.797	0.010	$+0.110$	0.022
docling	0.150	0.196	0.043	$+0.578$	$<10^{-4}$

Table 1: Marked-requirement set retrieval on real documents. Upper panel: all six arms, 76 documents, 3 repeats, 1,368 calls, paired per-document macro F1 against ours . Lower panel: the pre-registered confirmatory test of the typing claim, run once on 79 documents with 6 repeats (1,422 calls) and three arms only, so the correction family is two hypotheses. Both arms in H1 show the model identical inline cue markers over byte-identical content, differing only in whether a typed role token is attached. Tests are exact two-sided paired sign tests, Holm-corrected, with 10,000-resample bootstrap CIs over documents.

Figure 1: Loss in per-document macro F1 when exactly one cue class is removed at the span level, with 10,000-resample bootstrap CIs. Dark bars are significant after Holm correction. Colour and highlight account for essentially the entire effect; the other four cues sit within

\pm 0.003

of zero with intervals straddling it.

Feature	$n$	plain	docling	unstr.	dx2py	ours
Table	143	0.916	0.939	0.932	0.986	0.986
Text	59	0.977	0.977	0.977	0.977	1.000
Font	24	0.083	0.312	0.292	0.604	0.750
Hyperlink	16	0.062	1.000	0.062	1.000	1.000
Layout	14	0.357	0.196	0.071	0.714	0.821
Metadata	14	0.000	0.000	0.000	0.000	1.000

Table 2: Microsoft OCB, File-Fidelity Word track: mean weighted assertion score over 326 items in 59 documents (1,630 scored rows, none dropped), with questions written by Microsoft. The 5 features with

n\!\leq\!7

are aggregated; the full 14-feature breakdown is in Appendix S2 . Bold marks the best system per row. Lower block: paired over items against ours, exact sign test, Holm-corrected.

Figure 2: Recovery of author-marked requirements under six answering models spanning two vendors and three capability tiers. Plain text collapses under every model (

0.007

–

0.075

), so the failure is a property of the representation and not of a particular model. docling is omitted because it was not run under every model; it appears for all arms in Table 1 .

Parser	micro F1	macro F1	retention	$\Delta$	$p_{\text{Holm}}$
ours	0.503	0.526	0.665	—	—
docx2python	0.258	0.347	0.634	$+0.178$	0.0001
docling	0.084	0.104	0.607	$+0.422$	$<10^{-4}$

Table 3: Full pipeline run per arm on 68 documents, 2 repeats, 174 packages built and 408 scored calls. Every arm receives the same extraction and rendering treatment; only the parsed text differs. “retention” is the share of gold requirements whose text is still present anywhere in the rendered package, and is near-identical across arms.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Formatting	Semantic Role	Signal
Bold, some spans only	Key_Term	Key terms
Bold, standalone block	Implicit_Heading	Section heads
Italic/underline, not all-bold	Emphasis	Emphasis
Red text or red highlight	Critical_Warning	Critical alerts
Any other visible highlight	Warning	Cautions
Strikethrough	Deprecated	Obsolete content

Appendix

Table 4: Semantic role assignments. Only HIGH-confidence formatting-to-semantics mappings are included. Medium-confidence rules (e.g., italic

\rightarrow

document reference, green text

\rightarrow

status marker) are excluded to avoid misleading the extraction model with ambiguous signals. “Visible highlight” means any hue other than WHITE / NONE , which render identically to unhighlighted text (§ S4.1 ); “standalone” means a single line under 200 characters. Appendix S13 gives the full parse-time disambiguation rule, serialization, and extraction signal for each role.

cue	$n$	ours	pymupdf4llm	plain	$p$
highlight	18	11 / 11	4 / 12	0 / 15	0.039
bold	34	23 / 23	23 / 23	0 / 31	1.00
italic	47	14 / 22	19 / 25	0 / 40	0.125
strikethrough	27	17 / 19	19 / 22	0 / 21	0.500
total	126	65 / 75	65 / 82	0 / 107	1.00

Appendix

Table 5: All 126 born-digital formatting rules of ParseBench. Cells are cue recovered / text located ; cue

\leq

text necessarily. Plain text locates almost every target and recovers no cue at all.

p

is an exact McNemar test over discordant rules against pymupdf4llm .

Answering model	ours	untyped	dx2py	plain	untyped	dx2py
	micro F1				$\Delta$ vs. ours ( $p_{\text{Holm}}$ )
Claude Opus 4.6	0.687	0.605	0.586	0.075	$+.097^{*}$	$+.115^{*}$
Claude Haiku 4.5	0.629	0.531	0.499	0.048	$+.108^{**}$	$+.168^{***}$
Claude Sonnet 4.6	0.615	0.577	0.544	0.007	$+.085^{*}$	$+.122^{**}$
GPT-5.6-terra	0.534	0.468	0.510	0.045	$+.019$	$+.014$
Claude Opus 5	0.358	0.346	0.263	0.068	$+.015$	$+.064^{*}$

Appendix

Table 6: Cross-model generalisation, 76–79 documents per model.

\Delta

columns are paired per-document macro F1 against the typed representation, exact sign test, Holm-corrected within each model. Plain text collapses under every model; beating docx2python replicates on four of six; the typing benefit replicates on three of six and is positive on five, never significantly negative. Claude Opus 4.6 values are from the six-arm run of Table 1 .

	Pattern	Passages	Documents
A	superseded value (struck + live)	71	8
B	struck content	360	35
C	colour/highlight on a constraint	2,860	359
D	tracked replacement	68	21
D	tracked deletion	423	31
any pattern		—	414 (6.4%)

Appendix

Table 7: Prevalence of cue-dependent passages in 6,462 real documents. Detection is at OOXML run level. Pattern D was invisible to earlier scans: a tracked deletion is a w:r nested inside w:del carrying w:delText , which neither paragraph.runs nor iter_inner_content reaches.

Figure 3: Three-stage pipeline. Stage 1 (Parse & Annotate, the focus of this paper ) recovers character-level formatting—including content unreachable from a body walk, such as headers, footers, footnotes, document properties, tracked changes and equations—and serializes it into a hybrid Markdown+tag format. Stage 2 auto-selects a tier: a single call under 25K tokens, one call plus faithfulness verification to 50K, and above that segmentation with binary hierarchical merge at $O(\log N)$ depth, costing $2N{-}1$ calls for $N\leq 16$ segments; a parallel vision pass (dashed) handles embedded diagrams. Stage 3 routes the resulting typed entities into a five-file progressive-disclosure package. Stages 2–3 are reported as downstream utility (§ S7 ), not as the object of measurement.

Word Attr.	Disambiguation Rule	SemanticRole	LLM Serialization	Extraction Signal
Bold span	Some but not all spans in the block are bold	Key_Term	text (Markdown)	Definitional term or key concept; strong candidate for entity name
Bold paragraph	Block is standalone (single line, $<$ 200 chars) and every span is bold	Implicit_Heading	[PARAGRAPH roles= "IMPLICIT_HEADING"]	Section boundary; treat following content as a sub-section even without an explicit heading style
ALL-CAPS line	Every character run in the paragraph is all-caps; no mixed-case text present	Implicit_Heading	[PARAGRAPH roles= "IMPLICIT_HEADING"]	Implicit section label used by some authors instead of styled headings; same signal as above
Italic / underline	No disambiguation needed; single HIGH-confidence mapping	Emphasis	text (Markdown)	Emphasized qualifier; often an exception trigger, a deadline phrase, or a key condition
Red text	Fuzzy threshold: $\mathit{is\_red}(r,g,b)=(r{>}180)\wedge(g{<}80)\wedge(b{<}80)$ , plus a set of known red hex literals (#FF0000, #C00000, #CC0000). Tolerates RGB variance from custom author palettes	Critical_Warning	[COLOR:red]text[/COLOR] (custom tag, block attr. if whole paragraph)	Critical prohibition, safety constraint, or immediate-consequence warning; always extract as Warning with CRITICAL severity
Yellow highlight	Highlight attribute present and not WHITE / NONE ; RED escalates to Critical_Warning	Warning	[HIGHLIGHT: hue ] text[/HIGHLIGHT]	Caution statement; extract as Warning with MAJOR severity if an explicit consequence is stated

Appendix

Table 8: Complete formatting-to-LLM pipeline for all 11 HIGH-confidence SemanticRoles. Disambiguation Rule is the parse-time algorithm; Serialization is what the LLM receives verbatim; Extraction Signal is the semantic intent conveyed. Block-level roles wrap the entire paragraph via [PARAGRAPH roles=…] attributes; span-level roles wrap individual runs inline. Medium-confidence rules (italic

\to

document reference, green text

\to

status marker) are excluded: a wrong hint biases the LLM toward an incorrect type.

Diagram Type	Primary Entity	Secondary
Flowchart	Procedure	Rule , Role
Decision Tree	Decision_Point	Rule
Org Chart	Enumeration	Role
Matrix (RACI)	Enumeration	Role
Timeline	Timeline	Rule
Data Chart	Metric	–

Appendix

Table 9: Diagram type categories with entity mappings.

Figure 4: Single-pass diagram understanding protocol. Context from surrounding text (caption, heading path, adjacent paragraphs) is provided alongside the image in a single vision call. The model classifies the diagram type and extracts entities simultaneously. Vision calls run sequentially after text extraction, with decorative images skipped early.

Figure 5: Faithfulness verification flow. A 20% random sample (capped at 80 entities) is verified against source quotes using Opus 4.6. Score 0 or 1 downgrades confidence to Low ; score 0 does not discard (to avoid false-positive losses from truncation). Relationships referencing downgraded entities are also updated.

Type	Count	Type	Count
Rule	5	Definition	1
Prerequisite	3	Metric	3
Procedure	1	Exception	1
Warning	1	Scope	1
Glossary_Entry	3	Concept	1

Appendix

Table 15

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 838 papers

Apr 9, 2026

ParseBench: A Document Parsing Benchmark for AI Agents

AI agents are changing the requirements for document parsing. What matters is semantic correctness: parsed output must preserve the structure and meaning needed for autonomous decisions, including correct table structure, precise chart data, semantically meaningful formatting, and visual grounding. Existing benchmarks do not fully capture this setting for enterprise automation, relying on narrow document distributions and text-similarity metrics that miss agent-critical failures. We introduce ParseBench, a benchmark of

{\sim}2{,}000

human-verified pages from enterprise documents spanning insurance, finance, and government, organized around five capability dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. Across 14 methods spanning vision-language models, specialized document parsers, and LlamaParse, the benchmark reveals a fragmented capability landscape: no method is consistently strong across all five dimensions. LlamaParse Agentic achieves the highest overall score at 84.9%, and the benchmark highlights the remaining capability gaps across current systems. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ParseBench and https://github.com/run-llama/ParseBench.

Boyang Zhang, Sebastián G. Acosta, Preston Carlson +4

Document Parsing Benchmarks Document Parsing Document Information Extraction AI Agent Benchmarks Visual Grounding

Ask PDF

Apr 30, 2026

ObjectGraph: From Document Injection to Knowledge Traversal -- A Native File Format for the Agentic Era

Every document format in existence was designed for a human reader moving linearly through text. Autonomous LLM agents do not read - they retrieve. This fundamental mismatch forces agents to inject entire documents into their context window, wasting tokens on irrelevant content, compounding state across multi-turn loops, and broadcasting information indiscriminately across agent roles. We argue this is not a prompt engineering problem, not a retrieval problem, and not a compression problem: it is a format problem. We introduce OBJECTGRAPH (.og), a file format that reconceives the document as a typed, directed knowledge graph to be traversed rather than a string to be injected. OBJECTGRAPH is a strict superset of Markdown - every .md file is a valid .og file - requires no infrastructure beyond a two-primitive query protocol, and is readable by both humans and agents without tooling. We formalize the Document Consumption Problem, characterise six structural properties no existing format satisfies simultaneously, and prove OBJECTGRAPH satisfies all six. We further introduce the Progressive Disclosure Model, the Role-Scoped Access Protocol, and Executable Assertion Nodes as native format primitives. Empirical evaluation across five document classes and eight agent task types demonstrates up to 95.3 percent token reduction with no statistically significant degradation in task accuracy (p > 0.05). Transpiler fidelity reaches 98.7 percent content preservation on a held-out document benchmark.

Mohit Dubey, Open Gigantic

Open Gigantic

LLM Agents Knowledge Graphs

Ask PDF

Jul 31, 2026

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on HuggingFace and GitHub.

Boyang Zhang, Adrian Lyjak, Eli Stewart +2

LLM Grounding LLM Agent Evaluation Document Information Extraction

Ask PDF

Abstract

LLM Agent Reliability

Explore similar work

ParseBench: A Document Parsing Benchmark for AI Agents

ObjectGraph: From Document Injection to Knowledge Traversal -- A Native File Format for the Agentic Era

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction