One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity -- whether an extracted value matches the source -- is the standard measure for agentic document extraction, and it scores that run a success. We therefore log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all. From that dispatch record we build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value. The detector raises no flag on 207 clean fidelity-passing extractions across three model families, and recovers all 50 planted faults that withhold exactly the tools its rules check. The two results are not symmetric: the first bounds the false-positive rate, the second is recall by construction, and detection power against runs that call their tools and still answer wrongly is unmeasured. A second, independent oracle, a causal chamber that tests whether the datasheet's claims hold under physical measurement, is intentionally partial: it confirms only what the apparatus can exercise, a verifiable envelope of 2 of those 37 claims, and we give a taxonomy of why the rest are not physically gradable. Under a controlled perturbation, fidelity passes throughout while the chamber verdict flips exactly at the measurement uncertainty. Across three deployed model stacks (one destabilised by its serving stack, not by any capability gap) the tool layer buys portability and observability rather than accuracy, and earns its premium only once a document outgrows the context window.
Figures & tables
Model
Fidelity
Eng. err.
Latency
Claude Sonnet 4.6
25.0 ± 0.0
0/0/0
76 ± 5 s
GPT-5.1
25.0 ± 0.0
0/0/0
236 ± 78 s
Qwen3.6-27B
19.0 ± 4.0
1/6/10
130 ± 2 s
Table 1: Agentic-engine results across three deployable model-stack configurations, over three repeated runs. Fidelity is the pass count of 25; the engine-error column gives the per-repeat counts (a timeout, or a turn ending without the submit call). Qwen’s instability is reasoning-mode-dependent: 25/24/24 with reasoning disabled (Appendix N ). Latency is a wall-clock mean over cells that returned a result, dominated by time outside local tool execution (Section 6.3 ); ± values are standard deviations across the three repeats.
Signal logged
Failure it exposes
Action enabled
Dispatch record
fidelity pass, 0 calls
output via tool call
Reasoning trace
reasoning dropped
provider-native API
HTTP vs. call
200 OK, no call
alert on backend
Rubric label
tool vs. reasoning
fix tool or prompt
Cross-check call
verification skipped
flag for human review
Table 2: Industry lessons. The first two rows are the silent portability failures. All five are warning signals that route a run to human review, not automatic release gates; Section 6.5 reports what the cross-check signal does and does not certify.
Planted fault
N
Fidelity-only
Detector
Tool-bypass
25
0
25
Verification-skipped
25
0
25
Table 3: Planted silent failures recovered (Claude Sonnet 4.6). Fidelity-only scoring catches none; the detector catches all 50 at zero false positives on 207 fidelity-passing cells spanning all three model families (the post-audit reference run plus two further repeats). Recall on these rule-aligned injections is by construction (Section 6.5 ).
Figure 1: Reproducibility verdict under controlled perturbation of the DPS310 operating-range claim: the real cross-sensor measurement ( 945.285±0.080 hPa) is held fixed while the datasheet-stated bound is shrunk through it; the verdict flips exactly at the combined uncertainty (shaded). The perturbation is a controlled vehicle, not a claim that the datasheet is in error.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Engine / backend
Failure mode
Detect.
Baseline, GPT-5.1 passthrough
PDF block rejected
Loud
Agentic, GPT-5.1 passthrough
reasoning dropped
Silent
Baseline, Qwen (vLLM)
PDF not ingestible
Loud
Agentic, Qwen (vLLM)
tool tokens masked
Silent
Agentic, DeepSeek (vLLM)
tool tokens absent
Loud
Agentic, Llama (vLLM)
replies unparseable
Loud
Appendix
Table 4: Six cross-provider portability failures surfaced on one inference gateway. Two are silent (guided-decoding tool masking and dropped reasoning) and are the ones fidelity-only scoring would miss. The two GPT-5.1 rows are artefacts of the Anthropic-shaped passthrough and are resolved by the provider-native Responses API (Section 4 ); the rest are backend limitations.
Artefact
Fixed
Rel. fix
Claim set, bounds, conditions
05-05 – 05-12
before
Required-substring needles
05-05 – 05-12
before
Confidence floors
05-05 – 05-12
before
Attribution rubric, classifier
05-06
before
Prompt-leak fix
05-18
—
Silent-failure detector rules
05-22
after
Appendix
Table 5: When each artefact was fixed, relative to the prompt-leak fix of 2026-05-18 (all dates 2026). The three grading-surface rows are the exposure: they predate the fix and were never revisited. The detector rules do not share it, having been authored four days afterwards, and the 2026-06-05 revision made the predicate stricter by removing the chamber-side cross-check.
Model
Fid.
Exact
Flags
Nav/claim
Claude Sonnet 4.6
12/12
12/12
0
5.25 (8.43)
GPT-5.1
12/12
12/12
0
7.25 (9.11)
Appendix
Table 6: The fourth component, first run, no engine errors. Navigation calls per claim are given with the model’s own clean-corpus mean in brackets. The comparison must be like for like: an all-tool count against a navigation-only mean reverses the conclusion.
Arm
Model
Compl.
Err.
Ans.
Right
Closed-book
Claude
25
0
17
7
GPT-5.1
25
0
2
1
Qwen
19
6
5
0
Null-tool
Claude
25
0
1
0
GPT-5.1
25
0
1
0
Qwen
8
17
0
0
Appendix
Table 7: Probe arms, 25 claims per model and arm. “Ans.” counts runs that returned a value rather than declining; “Right” counts those that also passed fidelity. Qwen3.6-27B is scored for closed-book only; its two tools-callable rows are shown for completeness but lose most runs to engine errors.
N
Population
Reported in
75
agentic cells, one repeat
§ 6.3
74
storing an extraction
App. D
73
and fidelity-passing
App. K
207
fidelity-passing, three repeats
§ 6.5
74
claim-by-model groups
§ 6.5
Appendix
Table 8: The cell populations this paper reports over. Each block nests within itself; the two 74s are unrelated counts that happen to coincide.
Per-repeat metric
Reas. ON
Reas. OFF
Fidelity / 25
23 / 19 / 15
25 / 24 / 24
Engine errors
1 / 6 / 10
0 / 0 / 1
Engine-error rate
23%
1.3%
Mean latency (s)
130
64
Stable claims / 25
13
24
Appendix
Table 9: Qwen3.6-27B variance under the two reasoning-mode configurations, three repeated runs of the same 25 claims. Disabling reasoning cuts the engine-error rate by an order of magnitude and roughly halves per-cell latency while preserving fidelity. The single residual engine error ( si115x-adc-bit-depth , max-turn exhaustion) is a different failure mode (tool over-triggering rather than tool-call drop) and matches the secondary effect that Qwen3 issue #1817 (2026) reports for the enable_thinking=false workaround.
Work
Oracle
Domain
ModelGen
MLLM judge
Semiconductor models
AMSbench
Mixed
AMS circuits
DocDancer
Doc-internal
Long-context docs
DCI
QA correctness
Web / multi-hop QA
RWE-bench
Medical DB
Clinical evidence
Causal Chambers
Physical meas.
Causal discovery
Appendix
Table 10: Positioning against related benchmarks. Ours is, to our knowledge, the first to pair document-extraction fidelity with independent physical measurement of the same claims (Section 6.6 ).
Figure 2: Per-claim agentic fidelity by model (post-audit). Green is a fidelity pass, orange a fail, grey an engine error; the number is the per-claim confidence.
Figure 3: Mean calls per cell for each tool, by model (agentic engine, post-audit). The means cover the document-side tools only: submit tools and the entire chamber phase are excluded, so these are not the counts the cross-check predicate of Section 6.5 reads against a chamber-side call. All three models exercise the tool surface; this dispatch record is what made the silent zero-tool-call failure of Section 5 detectable.
Figure 4: Mean navigation-tool calls per cell across three development revisions. Qwen issues zero tool calls until the submit-tool channel replaces the output_format constraint, after which it engages the datasheet—the silent tool-bypass of Section 5 and its fix. This development period predates the Qwen3.5-to-3.6 gateway swap, so the figure shows Qwen3.5-27B; the cross-model results in Section 6 use Qwen3.6-27B. It also predates the oracle-leak fix (Appendix C ): the first two revisions are pre-audit measurements carried forward as constants, and only the third is recomputed from the post-audit matrix. Nothing here is a fidelity number—the axis is navigation calls—but the boundary is worth naming, since Section 3.3 says every result in Section 6 is post-audit.
Figure 5: Cost and latency per cell under the agentic engine, by model (cost on a log scale; large markers are per-model means, with ± std error bars over the three repeated runs). At list prices agentic extraction costs about 0.38perclaimforClaudeand0.16 for GPT-5.1, 1.2× and 1.8× their single-pass baselines; Qwen’s 3.9× ratio comes from a cheap baseline rather than an expensive agent, its page-image input being ≈ 24k tokens against Claude’s ≈ 98k. Fidelity is close across these points: the models separate far more sharply on cost and latency.
Figure 6: Per-model distribution of agent-reported confidence over the status-ok cells of the post-audit baseline; medians and quartiles marked.
Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.
Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on HuggingFace and GitHub.
Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply one. We isolate this post-failure decision with a benchmark of 1,024 items spanning 16 internal-system domains and eight tool-failure types, in which a tool call is enforced and the returned payload is guaranteed to be unusable. Under a deployment-style system prompt, 14.10% of responses are dishonest: the model either asserts a value the payload cannot support or declines while citing a fabricated policy or capability limit. The rate is governed almost entirely by whether the failure is signalled. When the tool returns status:error, dishonesty is absent (0.0%); when it returns status:ok with a redacted, corrupted, stale, malformed, empty or truncated value, dishonesty reaches 45.3%. The behaviour is not an artefact of our prompts: it appears under a neutral prompt (10.17%) and under the shipped prompt of every production agent framework we evaluate, reaching 24.67% under CrewAI's, and none of the nine frameworks we audit specifies what the model should do when a tool fails. Comparing prompt-level defences, we find that the operative variable is not deference to tool output but the absence of a named failure state. Appending a single sentence that requires the model to emit retrieval_status: OK or FAILED before answering reduces dishonesty from 14.10% to 0.87%, with one item of 688 worsening against 92 improving, and transfers unchanged into three foreign agent scaffolds. The emitted flag is faithful in 99.7-99.9% of declarations, giving a runtime detector that needs only a regular expression.
Arham Sethi, Arsen Kenzhebayev, Saanvi Paturi +3
The Shishukunj International School · Haileybury Astana · UWCSEA East Campus +2