Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries
Abstract
Modern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-relevant evidence never reaches its decision point. In one representative case, a regex detector returned a positive match, but the framework stranded the result before the block decision. This differs from jailbreaks and detector errors because the evidence is already recognized yet disconnected from enforcement. We study this repair-boundary signature with canary contracts: parent/fixed confirmation units that place a known payload only on the changed evidence surface and record monitor visibility together with the downstream decision for action-boundary cases. Applying the protocol to public framework repairs, we confirm 18 evidence-path failures across 11 LLM or agent safety project groups and four implementation ecosystems. Nine are established-boundary repairs, eight are new-surface policy-coverage gaps, and one is a boundary-only evidence case. Fourteen reach E1 enforcement boundaries: seven native action outcomes, six configured enforcement boundaries, and one synthetic-consumer consequence. The remaining cases affect verifier, safety-prompt, or observability channels. The cases form carrier, adapter, authority, and surface families and show how ordinary application behavior can remain intact while a safety-specific evidence path misses its decision. Exact revisions, deterministic outcomes, negative controls, and a frozen prospective audit document the confirmation rule. Monitor assumptions therefore need regression tests at the schemas, streams, tools, and adapters that frameworks evolve.
Figures & tables
| Contract field | Role in confirmation |
|---|---|
| Version pair | fix the parent/repaired revisions |
| Boundary | exclude new features or removed policies |
| Canary | place payload only on the changed surface |
| Visibility oracle | check monitor visibility of the payload |
| Decision oracle | check whether the intended action fires |
| Study item | Count |
|---|---|
| Confirmed before/after cases | 18 |
| Established / new-surface / boundary-only | 9 / 8 / 1 |
| Framework projects | 11 |
| Implementation ecosystems | 4 |
| Tier A / Tier B confirmations | 17 / 1 |
| E1 / E2 / E3 impact boundaries | 14 / 3 / 1 |
| Family | Hidden assumption | Canary focus |
|---|---|---|
| Carrier | payload location | new carrier only |
| Adapter | result-to-action map | positive result |
| Authority | channel priority | disagreement |
| Surface | materialization order | new surface only |
| Tier | Boundary type | Cases | Broken safety effect | Example |
|---|---|---|---|---|
| E1 | block / filter / rewrite | 5 | unsafe content passes in parent; repair blocks or rewrites it | NeMo regex |
| E1 | interrupt / HITL route | 3 | tool execution or routing bypasses the intended stop point | Haystack |
| E1 | hook / cap / policy state | 6 | configured enforcement hook, cap, or rule update does not fire | CrewAI |
| E2 | verifier / safety prompt | 3 | evidence is missing from evaluator or safety-prompt input | Promptfoo |
| E3 | observability channel | 1 | structured evidence is hidden from a policy-relevant consumer | AutoGen |
| Family | Py | TS | Java | C# | Projects |
|---|---|---|---|---|---|
| Carrier | 3 | 2 | 0 | 0 | 4 |
| Adapter | 4 | 0 | 0 | 1 | 3 |
| Authority | 2 | 2 | 0 | 0 | 4 |
| Surface | 3 | 0 | 1 | 0 | 4 |
| Project group | Cases | Boundary |
|---|---|---|
| LiteLLM | 4 | proxy, stream, policy |
| Promptfoo | 3 | verifier, tool-use |
| NeMo Guardrails | 2 | output, multimodal |
| Semantic Kernel | 2 | termination, filters |
| Single-case projects | 7 | analyzer, client, hook |
| Outcome | Evidence | Decision | Repair |
|---|---|---|---|
| Evidence invisible | absent | none | restore carrier |
| Enforcement skipped | visible | none | restore adapter |
| Wrong authority | stale | wrong | select channel |
| Policy state stale | stale | stale | rebind rule |
| Validation check | Reviewed | Result | Contribution |
|---|---|---|---|
| High-signal denominator | 195 refs | 9 newly promoted | candidate expansion |
| Random merged PR batch | 55 PRs | none confirmed | random-batch control |
| Held-out high-risk audit | 8 candidates | none confirmed | conservative promotion |
| Reviewed negatives | 13 candidates | excluded | boundary discipline |
| Known-pattern fixtures | 14 fixtures | 14 contracts | contract expressiveness |
| Rejected boundary cases | 3 cases | excluded | exclusion consistency |