Near-Miss: Latent Policy Failure Detection in Agentic Workflows

Ella Rabinovich; David Boaz; Naama Zwerdling; Ateret Anaby-Tavor

cs.CLMar 31, 2026

Near-Miss: Latent Policy Failure Detection in Agentic Workflows

Authors: Ella Rabinovich, David Boaz, Naama Zwerdling, Ateret Anaby-Tavor

Organizations: IBM Research

Abstract

Agentic systems for business process automation often require compliance with policies governing conditional updates to the system state. Evaluation of policy adherence in LLM-based agentic workflows is typically performed by comparing the final system state against a predefined ground truth. While this approach detects explicit policy violations, it may overlook a more subtle class of issues in which agents bypass required policy checks, yet reach a correct outcome due to favorable circumstances. We refer to such cases as near-misses or latent failures. In this work, we introduce a novel metric for detecting latent policy failures in agent conversations traces. Building on the ToolGuard framework, which converts natural-language policies into executable guard code, our method analyzes agent trajectories to determine whether agent's tool-calling decisions where sufficiently informed. We evaluate our approach on the $τ^2$ -verified Airlines benchmark across several contemporary open and proprietary LLMs acting as agents. Our results show that latent failures occur in 8-17% of trajectories involving mutating tool calls, even when the final outcome matches the expected ground-truth state. These findings reveal a blind spot in current evaluation methodologies and highlight the need for metrics that assess not only final outcomes but also the decision process leading to them.

Figures & tables

Figure 1: Canceling a reservation upon a customer request. Assuming accurate information, this workflow leads to the same outcome either with reservation details check (the tool call encircled by the dashed line) or without it (following the bypassing dashed path).

Figure 2: Schematic ReAct agentic flow using ToolGuard – cancellation eligibility is verified prior to cancel_reservation() invocation. Contrary to this approach, if policy adherence is left for the agent’s best-effort, three possible outcomes exist: (1) agent explicitly validates the policy by fetching and inspecting reservation details (the desired path), (2) agent bypasses policy validation, which results in policy violation (captured by the benchmark since differs from the ground-truth state), and (3) agent bypasses policy validation, yet the flow results in a valid outcome since the policy is not violated (near-miss, the subject of this study).

Figure 3: Near-miss detection in a completed task trajectory: mutating tool call with arguments args (MTC(args)) detected $\rightarrow$ load and simulate MTC’s guard code $\rightarrow$ if read-only tool call (RO) found, we search the trajectory history for precisely this, or any alternative read-only tool invocation, that obtains the same required information $\rightarrow$ if not found, assert latent failure to follow the policies.

agent	Claude-Sonnet4			Kimi-K2.5
model	NMR	P	R	NMR	P	R
generated-code-based search
CG-1	0.17	0.41	1.00	0.20	0.35	1.00
CG-2	0.07	1.00	1.00	0.07	1.00	1.00
LLM-based search
S-1	0.05	1.00	0.71	0.07	1.00	1.00

Table 1: Evaluation of near-miss detection accuracy: near-miss ratio (NMR), precision (P) and recall (R) are reported for two history search approaches: generated-code-based and LLM-based. Two agents (Claude-Sonnet4 and Kimi-K2.5) are evaluated against two SOTA LLMs and for code generation and search: CG-1 and CG-2 refer to code-generating models GPT5.1-Codex and Claude-Sonnet4, respectively; similarly S-1 and S-2 refer to models used for history search, again - GPT5.1-Codex and Claude-Sonnet4, respectively.

	trajectory outcome does not meet the gold state			trajectory outcome meets the gold state
model (agent)	total failure ratio	policy violation ratio	trajectories with MTCs	near-miss rate (NMR) (out of trajectories with MTCs)
GPT5-chat	0.480	0.120	98	0.085 (0.173)
Claude-Sonnet4	0.405	0.220	116	0.070 ( 0.121 )
Gemini-3-pro	0.229	0.205	85	0.060 (0.140)
GPT-oss-120b	0.399	0.182	93	0.040 ( 0.086 )
Qwen2.5-72b	0.611	0.207	144*	0.061 (0.084*)

Table 2: Performance results of three closed (top) and three open (bottom) models on the

\tau^{2}

-verified Airlines benchmark: total failure ratio is computed as the failure to meet the expected DB state after simulation completion; policy violation is detected by the ToolGuard library as explicit failure to follow guidelines. We further report the number of simulations involving mutating tool calls (out of the total of 200), near-miss ratio out of the 200 total, and, since latent failure is inherently relevant only to cases altering the system state – out of simulations with mutating tool calls, in brackets. All experiments are conducted with code generated by (the best performing, see Section 4.2 ) Claude-Sonnet4, and two best results in a column are boldfaced (the lower, the better).

Panel of Figure 4 — Figure 4: On the left: Latent failure distribution by mutating tool: update_reservation_flights() is the tool that is invoked most frequently without ensuring that the system state complies with policies, such as flight status and seats availability. Additional tool — update_reservation_passengers() — is missing from the chart since near-misses were not observed for this function. On the right: the distribution of read-only tools causing near-misses: get_flight_status() is the tool most frequently bypassed by agents; indeed, our manual inspection reveals that flights are often updated without verifying that their status is "available" as required by policies. Consistently with Table 2 , slightly higher amount of latent failures is observed in closed models (blue) than in open models (orange).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents
Jun 28, 2026Seongjae Kang, Taehyung Yu, Sung Ju HwangTool-Augmented Language Model Agents LLM Guardrails
KAIST
PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents
Aug 20, 2026Seongjae Kang, Taehyung Yu, Sung Ju HwangRuntime Enforcement for AI Agents AI Agent Governance
KAIST
Willful Disobedience: Automatically Detecting Failures in Agentic Traces
Mar 25, 2026Reshabh K Sharma, Shraddha Barke, Benjamin ZornAI Agent Evaluation AI Agent Auditing
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA

Jun 28, 2026J/K move · Enter open · S save

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Tool-Augmented Language Model Agents LLM Guardrails LLM Agent Evaluation LLM Agent Verification LLM Agent Safety

Open paper AskPDF arXiv

LLM agents handle user requests on behalf of organizations through tool calls and must follow the company policies stated in their system prompts. Prior work approaches this as a safeguarding problem -- external checks that block non-compliant agent actions. We argue that policy adherence is a broader problem: real workflows unfold across many turns, require explicit user confirmation and prerequisite reads, and hinge on the content of the dialogue rather than on any single argument value. Meeting this bar requires (i) full conversation context, (ii) self-reasoning over the policy and the current dialogue, and (iii) conversation-specific remediation that guides the agent's next turn -- three capabilities that prior safeguard work has often underestimated. We introduce POLICYGUARD, a sub-agent verifier that shares the agent's view of the dialogue, reasons over the policy in context, and provides actionable feedback for the agent's next turn. On tau^2-BENCH airline across three vendors (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro) with four trials per setting, POLICYGUARD improves PASS4 by +12.0 / +6.0 / +12.0 pp. Per-call analyses show POLICYGUARD achieves higher policy-violation recall while blocking roughly half as often as argument-level guards.

Abstract

LLM Agent Evaluation

Explore similar work

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

Willful Disobedience: Automatically Detecting Failures in Agentic Traces