LLM agents answer questions by calling APIs and trusting the results. An API can fail silently. It ignores part of a request, such as a misnamed filter, yet returns HTTP 200 and a plausible but wrong result. Current fixes, such as checklists, reviewer models and LLM judges, ask the model to catch it. But models often see the problem and answer anyway. We present SilentGuard, which moves the decision from the model to the harness running tool calls. It tests each result against invariants that any correct API must satisfy and withholds values that break one. We find silent failures in 11 of 15 public research APIs, with counts inflated up to 691 times. Current fixes leave over 70% of answers wrong, and agents report 77.4% of counts a judge flags. Across 7 open models, SilentGuard cuts wrong answers on its target faults from 74.0% to 1.7%, with no loss on fault-free tasks. On three frontier models they fall from 70.2% to 0.5%. A warning from the same check leaves 70.1% wrong, so enforcement is the key. On new fault types, the harness alone cuts wrong answers from 61.1% to 24.5%. It avoids wrong answers by abstaining rather than recovering.
Figures & tables
Figure 1: SilentGuard in the harness. Every tool result is checked, and a result that fails is re-checked once. If it fails again, the harness withholds it when the failed check is exact and warns when the check is learned and may itself be wrong (Eq. 2 ). The model never decides alone whether a result can be trusted.
Operation
Silent failure
Check that must hold
Tested
Exact checks
Count, filtered
filter ignored, joined with OR, capped
count identity at every position (Eq. 1 )
✓
List, filtered
records outside the filter, repeats
each record meets the filter, no repeats, size = total
✓
Documented API
unknown parameter accepted
parameter is in the published spec
✓
One filter
filter ignored
value swap (a different value changes the result)
✓
Pages, ranges, sorting, writes
records skipped, or bound, order or update ignored
pages do not overlap, results in bound and in order, a read returns the write
Table 1: Checks by kind of tool operation. Exact checks are invariants that hold on every correct API. Learned checks are mined from clean responses and can raise false alarms. Of 84 documented operations in 13 public APIs, 61 allow an exact check and all allow the spec check (Table 14 ).
(a) One LLM judge
Advises
Blocks
Without the check’s results
71.7
70.8
With the check’s results
71.2
6.1
(b) Acting on the check
Wrong ↓
Clean ↑
No guard (no check)
74.0 (32–87)
84.9
Warning
70.1 (19–87)
86.3
Warning + may abstain
18.2 (0–59)
83.5
Table 2: Evidence is necessary, and enforcement makes it reliable. % wrong answers under silent faults on AgentOrchBench. The same 7 open models and 2 seeds in every cell. (a) One judge (gpt-oss-120b) in four settings. It does or does not see the results of the count check’s extra queries, and its verdict is advice to the agent (advises) or is enforced by the harness (blocks). (b) Three ways to act on the full count check. May abstain: the prompt allows the answer that the count cannot be determined. In brackets: lowest and highest model. Clean: % correct with no fault. All methods: Table 3 .
Method
Wrong ↓
Corr. ↑
Unk.
Clean ↑
Baselines and controls
No guard
74.0 ± 7.1
16.4
9.6
84.9 ± 10.6
Caution prompt
73.3 ± 7.3
16.4
10.3
86.1 ± 9.1
Self-verification
56.2 ± 9.2
14.2
29.6
76.6 ± 12.8
Re-query
74.9 ± 5.0
17.0
8.1
84.7 ± 8.6
Blind retry
74.4 ± 5.1
17.3
8.3
88.3 ± 7.4
Table 3: SilentGuard gives the fewest wrong answers on AgentOrchBench. Mean over 7 open models ( ± standard error), % of runs with a silent fault (Wrong, Corr., Unk.) or with no fault (Clean). Unk.: the answer says the count cannot be determined. Bold is best and underlined is second. The LLM judge sees the check results and advises. Published: checklist ( Sun et al., 2024 ) , reviewer ( Ta et al., 2026 ) , Outcome Monitors (OM) ( Panthi and Abdelfattah, 2026 ) . Blocking: a flagged result is withheld and only accepted counts may be reported. No evidence: the judge does not see the check results.
Figure 2: Risk against coverage on AgentOrchBench with clean and faulted calls mixed. Lower is safer and further right answers more. Each line runs from a 10% fault rate (right end) to 50% (left end, higher risk), with a large marker at 25%. Mean over 7 open models. Gray lines are methods without evidence of the fault, or with advice alone.
Retail domain
Wrong ↓
Correct ↑
Clean ↑
No guard
78.6 ± 9.9
7.2 ± 2.4
36.2 ± 11.5
Auto-repair
15.0 ± 4.0
30.5 ± 10.0
35.7 ± 11.8
Enforce
12.3 ± 3.8
6.5 ± 2.3
33.8 ± 11.5
SilentGuard (ours)
11.0 ± 2.8
7.1 ± 2.0
38.5 ± 11.5
Real APIs (9)
Wrong ↓
Correct ↑
Right docs ↑
No guard
89.8 ± 5.3
3.5 ± 2.8
92.1 ± 6.5
Table 4: The other benchmarks , each with the methods run on it (mean over 7 open models, ± standard error, %). Retail: our count benchmark on a second schema. Real APIs: agents given slightly wrong documentation. Right docs: % correct with the right documentation. Typed: tool calls with a strict parameter schema, as in function calling. ToolMaze ( Zhu et al., 2026 ) : % task success under silent faults, loud (explicit) errors and no fault. OM: Outcome Monitors. Blind retry was not run on loud errors. Bold: best. Underlined: second.
Model
None
Self-v.
Warn
W+A
Enf.
Ours
Wrong answers under silent faults (%, ↓ )
Open models
74.0
56.2
70.1
18.2
8.8
1.7
GPT-5.6 Sol
89.4
2.5
11.3
2.3
31.5
0.0
GPT-5.6 Terra
62.3
3.9
53.9
35.9
34.0
0.2
GPT-5.6 Luna
59.0
6.9
63.2
45.1
36.1
1.4
Correct on clean tasks (%, ↑ )
Table 5: Frontier models fall for silent faults too. SilentGuard cuts wrong answers the most and keeps clean accuracy. AgentOrchBench, six methods on three frontier models (2 seeds) and the mean of the 7 open models. Self-v.: self-verification, the model runs the count check itself. Warn: the harness warns. W+A: warning plus permission to abstain. Enf.: Enforce. None: no guard. Ours: SilentGuard. Bold: best in the row. Means are computed before rounding.
Unseen fault
Fired
No guard
SilentGuard
Auto-repair
Word-stem match
0/36
32/4/0
32/4/0
32/4/0
NOT drops empty
0/36
0/36/0
0/36/0
0/36/0
Partial shard
36/36
20/16/0
0/0/36
34/2/0
Tag-join double
15/36
36/0/0
21/0/15
36/0/0
Cursor skips ties
36/36
9/27/0
0/0/36
0/36/0
Lossy index
35/36
35/1/0
0/1/35
0/1/35
Table 6: Auto-repair returns wrong answers marked as verified. Six fault types written after the method was frozen, run through the harness alone (36 count tasks each, cells: wrong / correct / unknown). Fired: tasks where the count check fired. When its check fires, SilentGuard (bold) never releases a wrong answer.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Fault
API behavior
Modeled on
Check
Drop last
drops last filter
wrong filter name
✓
Cap total
reports at most 5
page size as total
✓
AND → OR
uses OR
query builder
✓
Ignore field
ignores any filter
unindexed field
✓
Ignore NOT
drops negation
unsupported NOT
FA
Off by one
count +1
range bug
✓
Appendix
Table 7: Silent fault types in AgentOrchBench and the kind of API bug each is modeled on. The first is the failure we measure in real APIs (§ 5.1 ). The Check column says whether the exact count check (Eq. 1 ) fires (✓), fires although the count may be right (FA, false alarm), or cannot see the fault (–), because the fault keeps the identity true.
API
Returned
Correct
Over
Caught
Misnamed filter ignored (HTTP 200)
ChEMBL
2,921,148
4,225
691 ×
3/3
UniProtKB
55,460
666
83 ×
3/3
arXiv
186,698
2,799
67 ×
2/2
NCBI gene
37,885
1,567
24 ×
3/3
Europe PMC
291,095
15,436
19 ×
3/3
Appendix
Table 8: Real-API probes. Of 15 public APIs, 10 answer a request with a plausibly misnamed filter with HTTP 200 and the unfiltered count (ignored). The others reject the unknown parameter with an error (rejected). Returned: the count with the misnamed filter, or the unfiltered count when the API rejected it. Correct: the count with the documented filter. Over: returned / correct. Caught: misnamed filters flagged by the value-swap check (swap the filter’s value, an ignored filter returns the same count). –: not applicable, or not measured (rate limit).
Figure 3: How many times too large the returned count is when a real API silently ignores a misnamed filter (log scale). The overcount spans more than two orders of magnitude.
Wrong ↓
Correct ↑
API
None
Retry
Ours
None
Ours
NCBI gene
86
91
57
6
29
UniProtKB
80
83
40
6
51
Europe PMC
91
97
51
0
26
GBIF
94
91
6
0
94
Zenodo
100
91
3
0
83
Appendix
Table 9: Real-API agent study per API, wrong documentation (% of runs, 7 models pooled, 35 runs per API and arm). The four APIs above the line were chosen before the method was frozen. The five below were added later, with the method unchanged. On NCBI, UniProt, Europe PMC and arXiv the correct filter goes inside a query-language string that the wrong documentation never mentions, so after the spec check rejects the misnamed parameter the agent must find that syntax itself. On the others the correct parameter is in the list of allowed parameters that the spec check returns, which is the most likely reason for the gap. None: no guard. Retry: blind retry. Ours: SilentGuard (bold).
No guard
SilentGuard
Right docs
Model
Wrong ↓
Wrong ↓
Corr. ↑
Corr. ↑
Llama-4-Scout
97.8
35.6
48.9
97.8
Llama-4-Maverick
93.3
6.7
93.3
100.0
Llama-3.3-70B
100.0
28.9
68.9
97.8
gpt-oss-120b
80.0
11.1
88.9
100.0
Nemotron-3-Ultra
100.0
20.0
80.0
100.0
Appendix
Table 10: Agents on nine real public APIs, per model (45 runs per model and cell, %). No guard and SilentGuard: agents given slightly wrong documentation. Right docs: % correct with no guard and the right documentation. The frontier models ran these two arms only.
Misnamed
Bad filter
Page over
API
filter
value
limit
NCBI gene
ignored
applied
honored
UniProtKB
ignored
rejected
rejected
ClinicalTrials.gov
rejected
rejected
capped
Europe PMC
ignored
ignored
rejected
OpenAlex
rejected
rejected
rejected
Appendix
Table 11: Three kinds of silent fault in real public APIs (read-only probes). Misnamed filter: a plausibly misnamed filter parameter. Rejected means the API returned an error. Bad filter value: a typo in the value of a valid filter (for example a bad year or date). Ignored means the count equals the unfiltered one, applied means the value was used as given (usually zero results), which is correct. Page over limit: a request for more records per page than the API allows. Capped means HTTP 200 with fewer records than asked and no error. Honored means the API returned the larger page. Bold red: silent. –: rate-limited, not measured.
API
Typo in
Ignored
Over
Fired
FA
Europe PMC
query string
5/5
19–40 ×
5/5
0/5
GitHub search
query string
5/5
3.2–4.4 ×
5/5
0/5
iNaturalist
parameter
5/5
1.1–1.5 ×
5/5
0/5
Appendix
Table 12: A typo in the value of a valid filter, on the three APIs that silently ignore it (five probes each, read-only, no model). The parameter name is valid, so the spec check passes. Ignored: the API returned the unfiltered count. Over: returned / correct. Fired: the count identity (Europe PMC, GitHub, where the value sits inside the query string) or the value-swap check (iNaturalist) flagged the result. FA: the same check fired on the correctly spelled query.
Method
Wrong ↓
Correct ↑
Unknown
No guard
97.1
0.0
2.9
Spec check alone
100.0
0.0
0.0
SilentGuard
20.0
2.9
77.1
Appendix
Table 13: A wrong value format in the documentation of a real API (iNaturalist date filter, 5 tasks, mean over 7 open models, % of runs). The documentation gives the date as DD/MM/YYYY, which the API silently ignores. The parameter name is valid, so the spec check passes. With the right documentation an unguarded agent is correct 88.6% of the time. Predictions for this run are V1 to V5 in Table 31 .
Operation kind
Operations
Exact check
Search or list
43
43
Get by ID
16
5
Other read
9
4
Aggregate
6
5
Write
5
0
Pagination
4
3
Appendix
Table 14: Which checks real API operations allow, read from the documentation of 13 public APIs. Exact check: the operation allows a record-membership check (47 operations), the count identity (25) or a value swap (56). All 84 have a published parameter list, so the spec check applies to every one. The APIs are those of Table 8 without GitHub and openFDA. Write operations were not counted as allowing the write-then-read check of Table 1 . These are well-documented APIs. The share will be lower for undocumented or free-text tools.
Measure
Against
Difference [95% interval]
AgentOrchBench
wrong
No guard
−72.3 [ −74.0 , −70.5 ]
wrong
Self-verification
−54.5 [ −56.6 , −52.3 ]
wrong
LLM judge
−69.5 [ −71.3 , −67.7 ]
wrong
Outcome Monitors
−70.4 [ −72.3 , −68.3 ]
wrong
Blind retry
−72.7 [ −74.7 , −70.5 ]
Appendix
Table 15: Task-cluster bootstrap for the key differences (SilentGuard minus the other method, in percentage points, computed before rounding). We used 2,000 resamples of tasks, the same draw for every model and method, and the mean over 7 open models. An interval that excludes 0 means the difference holds across tasks, not only on average. OM: Outcome Monitors.
Against
Other
SilentGuard
λ∗
AgentOrchBench, seen faults (7 models)
No guard
16.4 / 74.0
2.4 / 1.7
0.2
Auto-repair
57.3 / 3.0
2.4 / 1.7
43.6
Lists, seen faults (7 models)
No guard
43.7 / 49.3
21.1 / 9.1
0.6
Auto-repair
47.9 / 14.6
21.1 / 9.1
4.9
Appendix
Table 16: Where withholding pays. Cells: % correct / % wrong of faulted runs. With score = correct −λ× wrong, λ∗ is the cost of a wrong answer, relative to a correct one, above which SilentGuard scores higher than the other method (computed from unrounded values). Against no guard λ∗ is small everywhere. Against auto-repair on count tasks it is large on the fault types repair was built for and small on fault types no method was built for.
Wrong answers by fault type ↓
All faults
Drop
Cap
AND
Ignore
Ignore
Off by
Method
last
total
→ OR
field
NOT
one
Wrong ↓
Correct ↑
Unknown
Clean ↑
No guard
89.7
92.9
89.3
77.4
6.2
88.9
74.0 ± 7.1
16.4 ± 1.8
9.6 ± 8.7
84.9 ± 10.6
Caution prompt
88.3
91.3
87.7
75.4
4.6
92.9
73.3 ± 7.3
16.4 ± 2.2
10.3 ± 9.3
86.1 ± 9.1
LLM judge
88.7
91.7
84.7
73.0
6.2
82.9
71.2 ± 7.8
17.4 ± 2.1
11.4 ± 9.7
86.5 ± 8.5
Checklist
87.7
90.7
87.7
73.4
4.8
89.3
72.3 ± 8.1
16.2 ± 2.2
11.5 ± 10.3
84.1 ± 10.6
Appendix
Table 17: AgentOrchBench, all methods (mean over 7 open models, %, ± standard error). Wrong, correct and unknown are shares of the runs with a silent fault. Clean is % correct with no fault. Auto-repair has the most correct answers but its wrong answers are released as verified (Table 6 ). Last condition: the earlier check of the last condition only. All other harness rows use the full check. No evidence: the judge does not see the consistency queries. OM: Outcome Monitors.
AgentOrchBench
Wrong ↓
Correct ↑
Unknown
Clean ↑
No guard
[56.8, 91.3]
[11.9, 20.9]
[0.0, 30.8]
[59.1, 100.0]
Caution prompt
[55.5, 91.2]
[11.1, 21.6]
[0.0, 33.1]
[63.9, 100.0]
Self-verification
[33.6, 78.8]
[9.2, 19.2]
[2.4, 56.7]
[45.4, 100.0]
Re-query
[62.6, 87.1]
[12.6, 21.4]
[0.0, 24.7]
[63.6, 100.0]
Blind retry
[62.0, 86.9]
[13.9, 20.7]
[0.0, 24.1]
[70.1, 100.0]
LLM judge
[52.1, 90.3]
[12.2, 22.5]
[0.0, 35.2]
[65.8, 100.0]
Appendix
Table 18: 95% confidence intervals for Tables 3 and 4 ( t interval over models, clipped to [0, 100]).
Model
None
Check.
Rev.
OM
Judge
Retry
Self-v.
Enf.
Ours
Llama-4-Scout
86.6
79.2
79.6
81.2
83.6
78.0
79.6
7.2
2.1
Llama-4-Maverick
80.8
80.3
79.9
79.6
80.1
80.1
36.3
0.5
0.5
Llama-3.3-70B
80.6
80.8
80.6
80.6
78.7
80.3
69.2
1.2
0.0
gpt-oss-120b
80.8
80.6
79.8
80.8
74.1
80.8
53.5
24.8
1.2
Nemotron-3-Ultra
80.6
80.6
80.6
80.6
79.9
80.6
78.5
12.0
3.7
Gemma-3-27B
76.9
81.0
76.2
78.9
77.3
77.3
63.9
6.5
3.5
Appendix
Table 19: AgentOrchBench, wrong answers per model and method (%). None: no guard. Check.: checklist. Rev.: reviewer. OM: Outcome Monitors. Judge: LLM judge. Retry: blind retry. Self-v.: self-verification. Enf.: Enforce. Ours: SilentGuard. Bold: best in the row. Underlined: second. The three older GPT models below the line finished only part of their runs (finished share in brackets) and are not ranked. The GPT-5.6 models are in Table 5 .
AgentOrchBench (wrong ↓ )
ToolMaze (success ↑ )
Model
None
Ours
Δ
C+OM
Ours
Δ
Llama-4-Scout
86.6
2.1
−84.5
31.7
38.6
+6.9
Llama-4-Maverick
80.8
0.5
−80.3
38.9
45.6
+6.7
Llama-3.3-70B
80.6
0.0
−80.6
36.9
38.6
+1.7
gpt-oss-120b
80.8
1.2
−79.6
38.6
35.6
−3.1
Nemotron-3-Ultra
80.6
3.7
−76.9
31.9
37.5
+5.6
Appendix
Table 20: The gain holds for every model on AgentOrchBench and for 5 of 7 on ToolMaze. AgentOrchBench: % wrong answers under silent faults, no guard against SilentGuard. ToolMaze: % task success under silent faults, the strongest published mix (Checklist + Outcome Monitors, C+OM) against SilentGuard. Δ : ours minus the other, computed before rounding (green: ours better, red: ours worse, bold: the better of the pair). p : exact sign-flip test over the 7 models.
Method
P3 ↑
P4 ↑
P3+P4 ↑
P0 ↑
P1+P2 ↑
Checklist
26.8 ± 4.3
19.5 ± 2.6
23.2 ± 3.3
55.5 ± 4.1
38.3 ± 4.2
Outcome Monitors
36.3 ± 2.2
24.8 ± 1.6
30.5 ± 1.3
57.1 ± 3.6
40.7 ± 2.2
Checklist + OM
39.9 ± 3.1
25.6 ± 2.0
32.7 ± 2.2
54.8 ± 4.1
39.8 ± 2.6
Blind retry
60.4 ± 3.1
27.0 ± 1.4
43.7 ± 1.7
54.2 ± 2.6
–
SilentGuard
47.6 ± 3.7
24.4 ± 1.5
36.0 ± 2.5
52.5 ± 4.2
39.8 ± 2.9
Appendix
Table 21: ToolMaze, all modes (% task success by its own judge, mean ± SE over 7 models, 180 fresh workflows). P3: silent fault on the first call. P4: silent fault on every call. P0: no fault. P1, P2: explicit errors. Blind retry was not run on P1/P2.
ToolMaze success ↑
Real APIs
Repeated
Harness
1-call fault
2-call fault
Lasting
wrong ↓
actions ↓
No re-check
39.9
32.5
25.6
–
–
Blind retry
60.4
56.0
27.0
89.5
2.36
SilentGuard
47.6
46.4
24.4
21.3
0.24
Appendix
Table 22: Blind retry helps only when the fault clears. Blind retry (the no re-check harness plus every call sent twice, second result kept) clears a short fault that lasts one or two calls. On a fault that lasts (ToolMaze P4) it adds little over the same harness without retry, and on real APIs, which ignore a misnamed filter every time, it does not reduce wrong answers. It also repeats bookings, payments and messages. No re-check: the checklist with Outcome Monitors, without retry. Mean over 7 open models. Values are % except repeated actions (repeated action calls per ToolMaze run). Bold: best. Underlined: second.
Model
No re-check
Blind retry
SilentGuard
Llama-4-Scout
32.8
60.0
50.6
Llama-4-Maverick
31.7
56.1
47.2
Llama-3.3-70B
32.2
66.1
57.2
gpt-oss-120b
36.4
51.2
43.4
Nemotron-3-Ultra
38.9
65.6
53.9
Gemma-3-27B
12.2
34.4
22.8
Appendix
Table 23: ToolMaze short fault lasting two calls, per model (% task success, 180 workflows, runs whose model calls failed at the provider are left out). No re-check: the checklist with Outcome Monitors.
Method
Wrong ↓
Correct ↑
Unknown
Right docs ↑
No guard
89.8 ± 5.3
3.5 ± 2.8
6.0 ± 4.6
92.1 ± 6.5
Blind retry
89.5 ± 5.8
3.2 ± 2.5
4.8 ± 3.7
93.0 ± 5.5
Value-swap check alone
17.8 ± 2.5
25.7 ± 11.4
55.2 ± 12.8
91.1 ± 7.8
Spec check alone
21.0 ± 4.6
70.8 ± 6.8
7.0 ± 4.6
92.1 ± 6.5
SilentGuard
21.3 ± 3.8
69.5 ± 7.5
7.9 ± 4.9
92.4 ± 7.3
Appendix
Table 24: Agents on nine real public APIs, mean ± standard error over 7 open models (45 runs per model, method and documentation). Wrong, correct, unknown: % of runs with slightly wrong documentation (the rest ran out of turns with no answer). Right docs: % correct with the right documentation. Value-swap check alone: the harness withholds a count whose filter had no effect, with no spec check. It leaves the fewest wrong answers but few correct ones. The spec check does most of the work, because it rejects the misnamed filter and the agent rewrites the query.
AgentOrchBench
Retail
Harness action
Wrong ↓
Correct ↑
Unknown
Clean ↑
Wrong ↓
Clean ↑
Warning (last condition)
63.1
11.4
25.4
85.5
71.5
35.7
Warning
70.1
14.1
15.9
86.3
–
–
Warning + tip
37.2
9.8
52.9
82.5
59.6
36.9
Auto-repair
3.0
57.3
39.7
83.3
15.0
35.7
Answer gate (last condition)
32.9
5.2
61.9
84.9
–
–
Appendix
Table 25: Ablation of the harness action after a failed check (mean over 7 open models, %). Warnings keep wrong answers high. Auto-repair gives the most correct answers but releases wrong ones marked as verified (Table 6 ). SilentGuard withholds only what an exact check proves wrong. Last condition: these two earlier arms check the last condition only. All other rows use the full check. Warning + tip: the warning also names the recovery tools. Enforce: only checked counts are accepted. Unknown: the answer says the value cannot be determined. –: not run on retail. Bold: best. Underlined: second.
Figure 4: All answers under silent faults on AgentOrchBench, mean over 7 open models. Prompts and model-based methods leave most answers wrong. Harness actions that go beyond a warning move them out of the wrong share, and under SilentGuard nearly all become unknown.
Figure 5: Wrong answers per model on AgentOrchBench, no guard against SilentGuard. Every model falls to below 4%.
Figure 6: ToolMaze task success under silent faults per model, for the checklist with Outcome Monitors against SilentGuard. SilentGuard is ahead on 5 of 7 models.
Figure 7: ToolMaze task success by fault type for no re-check, blind retry and SilentGuard. Blind retry leads while the fault clears and not when it lasts.
Seen faults
Unseen faults
Clean
Method
Wrong ↓
Corr. ↑
Wrong ↓
Corr. ↑
Corr. ↑
No guard
49.3
43.7
37.8
56.7
80.5
Auto-repair
14.6
47.9
16.9
37.5
81.0
Checked lists only
11.2
22.1
11.7
7.5
81.0
SilentGuard
9.1
21.1
11.5
7.1
84.8
Appendix
Table 26: The same rule gives the fewest wrong lists. The 30 tasks ask for the IDs of all matching records. Six fault types were seen in design and five were written later (unseen). The record checks test that every returned record meets the filter, that no record repeats, and that the list matches the reported total and the count identity. Checked lists only: the answer rule alone. SilentGuard adds the re-check, withholding and the result checklist. Corr.: correct. Bold: best. Underlined: second. Mean over 7 open models, % of runs.
Tool
Time
Extra LLM
Method
calls
Tokens
(s)
calls
No guard
2.0
3,183
9.2
0.0
Caution prompt
2.0
3,734
11.3
0.0
LLM judge
3.2
6,287
47.2
2.0
Checklist
1.9
3,429
8.7
0.0
Reviewer
1.9
9,276
25.0
2.6
Appendix
Table 27: Cost per faulted run on AgentOrchBench (mean over 7 open models). Columns give agent tool calls, LLM tokens, wall time and extra LLM calls (judge or reviewer) on top of the agent’s own. SilentGuard’s checks run in the harness and need no extra LLM call.
Greedy decoding
Sampling, T=0.7 , 3 seeds
Model
Training
Seen ↓
Unseen ↓
Consistent lie ↓
Reworded ↓
Clean ↑
Seen ↓
Unseen ↓
Clean ↑
Phi-4-mini
Base model
100.0
73.3
100.0
100.0
86.7
57.8
72.8
25.6
Phi-4-mini
SFT to refuse (trained on drop-last)
0.0
6.7
100.0
0.0
100.0
0.0
6.1
100.0
Phi-4-mini
SFT to repair
0.0
45.3
100.0
0.0
100.0
1.1
90.0
100.0
Phi-4-mini
RL (DAPO, false-equation penalty)
0.0
28.7
100.0
0.0
100.0
0.0
59.4
100.0
Mistral-7B
Base model
0.0
0.0
0.0
0.0
0.0
2.2
1.1
0.0
Appendix
Table 28: Training the model itself, full results (% wrong, clean: % correct). Seen: drop-last, the fault used in training. Unseen: cap-total, AND → OR, ignore-NOT, off-by-one and prefix. Consistent lie: a wrong answer the API backs up in every query. Reworded: the seen fault with reworded tasks. With sampling, unseen covers AND → OR and off-by-one only. SFT: supervised fine-tuning. RL: reinforcement learning with DAPO. Mistral-7B base cannot use the tools, so its 0% wrong is not a success.
Cutoff τ
Min. support
Withheld (%)
Changed
0.1%
≥ 998
0.0
0
0.5%
≥ 198
0.0
0
1%
≥ 98
0.0
0
2%
≥ 48
4.4
108
5%
≥ 18
34.6
847
10%
≥ 8
73.0
1,788
Appendix
Table 29: Sensitivity to the cutoff on ToolMaze. A learned check is treated as exact when 1/(n+2)≤τ , that is, when it held on at least the listed number of clean examples. Over all 2,448 withhold-or-warn decisions logged in our ToolMaze runs, the largest support is 52, so every cutoff below 1.85% (1/54) gives the same decisions as 1% (Changed: decisions that differ from 1%). On AgentOrchBench and real APIs every check is exact and the cutoff has no effect.
Extra queries
Seen faults
Check-aware
Positions checked
clean
faulted
caught ↑
wrong ↓
caught ↑
No check
0.0
0.0
0.0
80.6
0.0
Last condition only
2.0
2.0
88.0
8.8
0.0
1 random condition
2.0
2.0
88.4
8.6
0.9
2 random conditions
4.0
4.0
94.4
2.8
1.9
All, stop at first failure
5.2
2.3
97.2
0.0
2.8
Appendix
Table 30: How many conditions to check, measured on the harness alone (no model, 36 count tasks per fault type, detection averaged exactly over every choice of positions). Extra queries per filter call on clean and faulted runs. Caught: share of faulted results the check flags. Wrong: share released with a wrong count under the answer rule. No budget raised a false alarm on clean runs. Check-aware: the consistent lie, an API built to keep the identity true. No budget catches more than 2.8% of it.
ID
Prediction, written before the run
Observed
Outcome
A1
SilentGuard > blind retry on lasting faults (ToolMaze P4)
24.4 vs 27.0
failed
A2
SilentGuard ≥ blind retry −2 on one-time faults (P3)
47.6 vs 60.4
failed
A3
Blind retry repeats action calls
2.36 vs 0.24 per run
held
B1
Loud faults: SilentGuard within 3 points of Checklist + OM
39.8 vs 39.8
held
C1
AgentOrchBench wrong within 5 points of Enforce
1.7 vs 8.8
failed
C2
AgentOrchBench correct ≥ answer gate −5
2.4 vs 5.2
held
Appendix
Table 31: Predictions of the confirmatory experiments, written down before each run, and their outcomes (26 of 39 held). All were written, with the method frozen by file hashes, before the runs. Failed predictions (bold red) are reported and discussed in the text (blind retry, § 5.3 ). IDs are the labels used in the prediction log. Observed values are means over the 7 open models (%), or the number of models with fewer wrong answers. Rows L, K, P, X and Y use the 6 models the predictions were written for, so their no-guard values differ slightly from the 7-model tables. Within x points: the absolute difference is at most x . The rows from L down are control runs added after the first results. Each was written before its own run.
Evaluation frameworks for tool-augmented LLM agents focus overwhelmingly on capability metrics or explicit tool crashes, leaving silent infrastructure failures and HTTP 200 responses with empty, null, or malformed payloads largely unaudited. We introduce a lightweight black-box auditing framework that injects four silent failure profiles across 12 production-adjacent tool stubs and classifies agent responses into three mutually exclusive behavioral classes: Honest Surrender (HSR), Fabrication (FAR), and Unfaithful Safety Refusal (USR). Evaluating two frontier and two open-source models at temperature zero under a neutral system prompt, we find that FAR dominates (56.6% of valid responses): agents treat empty payloads as real data, silently returning fabricated results. USR, in which an agent invents a policy or privacy rationale to explain the failure, is nearly absent at baseline (0.25%, one instance across 396 valid trajectories). Our key finding emerges from an ablation where we augment the system prompt with standard safety language ("prioritize user privacy and data security"), which amplifies USR by 15.6x (from 0.25% to 3.95%; 95% CI on ablation rate: 2.2%-6.4%; Fisher's exact test, p < 0.001). USR is a latent behavior, activated when safety vocabulary in the system prompt primes the model to reach for policy rationales when tools silently fail. Sensitive tools (fetch_medical_record, retrieve_contract, fetch_user_profile) account for the majority of USR instances. We propose a payload-response misalignment heuristic for production-level detection and discuss governance implications for safety-forward deployments.
LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.
LLM agent systems increasingly run as long-lived autonomous runtimes: scheduling jobs, calling tools, maintaining memory, and pushing results to humans. We present a longitudinal study of silent failures in one such system: a personal-assistant agent runtime in continuous production since March 2026, with roughly 40 scheduled jobs, 8 LLM providers, a tool-governance proxy, and a knowledge-base memory plane, defended by 4,286 unit tests and 827 governance checks. Over eight weeks we documented 22 incidents with full root-cause postmortems, in which one meta-pattern -- a failure whose error signal never reaches a human in actionable form -- manifested at least 28 times. We derive a five-class, mechanism-oriented taxonomy: (A) environment and platform quirks, (B) design-assumption mismatches, (C) error swallowing and dilution, (D) chained hallucination and fabrication, (E) operational omission and forensic blind spots. Class D is unique to LLM systems and the most dangerous: the system does not merely fail to report an error -- the LLM transforms it into fluent, plausible narrative delivered to the user. We term this fail-plausible: gray failure's differential observability escalated -- the observer is not just blind, it is convincingly lied to by the failure itself. Three findings: about 70% of silent failures were caught by human user-view observation, not tests or audits; a retrospective audit of 15 incidents found 0% ex-ante prevention but 87% regression blocking -- audits are regression engines, not prediction engines; incident latency (13 hours to 60 days) tracks failure mechanism, not code complexity -- the longest-lived failures lived in the seams between components, where no test runs. We describe the resulting defense framework and distill design principles for agent systems whose failures are loud, attributable, and boring. All postmortems and artifacts are public.