Near-Miss: Latent Policy Failure Detection in Agentic Workflows

Ella Rabinovich; David Boaz; Naama Zwerdling; Ateret Anaby-Tavor

cs.CLMar 31, 2026

Near-Miss: Latent Policy Failure Detection in Agentic Workflows

Authors: Ella Rabinovich, David Boaz, Naama Zwerdling, Ateret Anaby-Tavor

Organizations: IBM Research

Abstract

Agentic systems for business process automation often require compliance with policies governing conditional updates to the system state. Evaluation of policy adherence in LLM-based agentic workflows is typically performed by comparing the final system state against a predefined ground truth. While this approach detects explicit policy violations, it may overlook a more subtle class of issues in which agents bypass required policy checks, yet reach a correct outcome due to favorable circumstances. We refer to such cases as near-misses or latent failures. In this work, we introduce a novel metric for detecting latent policy failures in agent conversations traces. Building on the ToolGuard framework, which converts natural-language policies into executable guard code, our method analyzes agent trajectories to determine whether agent's tool-calling decisions where sufficiently informed. We evaluate our approach on the $τ^2$ -verified Airlines benchmark across several contemporary open and proprietary LLMs acting as agents. Our results show that latent failures occur in 8-17% of trajectories involving mutating tool calls, even when the final outcome matches the expected ground-truth state. These findings reveal a blind spot in current evaluation methodologies and highlight the need for metrics that assess not only final outcomes but also the decision process leading to them.

Figures & tables

Figure 1: Canceling a reservation upon a customer request. Assuming accurate information, this workflow leads to the same outcome either with reservation details check (the tool call encircled by the dashed line) or without it (following the bypassing dashed path).

Figure 2: Schematic ReAct agentic flow using ToolGuard – cancellation eligibility is verified prior to cancel_reservation() invocation. Contrary to this approach, if policy adherence is left for the agent’s best-effort, three possible outcomes exist: (1) agent explicitly validates the policy by fetching and inspecting reservation details (the desired path), (2) agent bypasses policy validation, which results in policy violation (captured by the benchmark since differs from the ground-truth state), and (3) agent bypasses policy validation, yet the flow results in a valid outcome since the policy is not violated (near-miss, the subject of this study).

Figure 3: Near-miss detection in a completed task trajectory: mutating tool call with arguments args (MTC(args)) detected $\rightarrow$ load and simulate MTC’s guard code $\rightarrow$ if read-only tool call (RO) found, we search the trajectory history for precisely this, or any alternative read-only tool invocation, that obtains the same required information $\rightarrow$ if not found, assert latent failure to follow the policies.

agent	Claude-Sonnet4			Kimi-K2.5
model	NMR	P	R	NMR	P	R
generated-code-based search
CG-1	0.17	0.41	1.00	0.20	0.35	1.00
CG-2	0.07	1.00	1.00	0.07	1.00	1.00
LLM-based search
S-1	0.05	1.00	0.71	0.07	1.00	1.00

Table 1: Evaluation of near-miss detection accuracy: near-miss ratio (NMR), precision (P) and recall (R) are reported for two history search approaches: generated-code-based and LLM-based. Two agents (Claude-Sonnet4 and Kimi-K2.5) are evaluated against two SOTA LLMs and for code generation and search: CG-1 and CG-2 refer to code-generating models GPT5.1-Codex and Claude-Sonnet4, respectively; similarly S-1 and S-2 refer to models used for history search, again - GPT5.1-Codex and Claude-Sonnet4, respectively.

	trajectory outcome does not meet the gold state			trajectory outcome meets the gold state
model (agent)	total failure ratio	policy violation ratio	trajectories with MTCs	near-miss rate (NMR) (out of trajectories with MTCs)
GPT5-chat	0.480	0.120	98	0.085 (0.173)
Claude-Sonnet4	0.405	0.220	116	0.070 ( 0.121 )
Gemini-3-pro	0.229	0.205	85	0.060 (0.140)
GPT-oss-120b	0.399	0.182	93	0.040 ( 0.086 )
Qwen2.5-72b	0.611	0.207	144*	0.061 (0.084*)

Table 2: Performance results of three closed (top) and three open (bottom) models on the

\tau^{2}

-verified Airlines benchmark: total failure ratio is computed as the failure to meet the expected DB state after simulation completion; policy violation is detected by the ToolGuard library as explicit failure to follow guidelines. We further report the number of simulations involving mutating tool calls (out of the total of 200), near-miss ratio out of the 200 total, and, since latent failure is inherently relevant only to cases altering the system state – out of simulations with mutating tool calls, in brackets. All experiments are conducted with code generated by (the best performing, see Section 4.2 ) Claude-Sonnet4, and two best results in a column are boldfaced (the lower, the better).

Panel of Figure 4 — Figure 4: On the left: Latent failure distribution by mutating tool: update_reservation_flights() is the tool that is invoked most frequently without ensuring that the system state complies with policies, such as flight status and seats availability. Additional tool — update_reservation_passengers() — is missing from the chart since near-misses were not observed for this function. On the right: the distribution of read-only tools causing near-misses: get_flight_status() is the tool most frequently bypassed by agents; indeed, our manual inspection reveals that flights are often updated without verifying that their status is "available" as required by policies. Consistently with Table 2 , slightly higher amount of latent failures is observed in closed models (blue) than in open models (orange).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

Jun 28, 2026

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

LLM agents handle user requests on behalf of organizations through tool calls and must follow the company policies stated in their system prompts. Prior work approaches this as a safeguarding problem -- external checks that block non-compliant agent actions. We argue that policy adherence is a broader problem: real workflows unfold across many turns, require explicit user confirmation and prerequisite reads, and hinge on the content of the dialogue rather than on any single argument value. Meeting this bar requires (i) full conversation context, (ii) self-reasoning over the policy and the current dialogue, and (iii) conversation-specific remediation that guides the agent's next turn -- three capabilities that prior safeguard work has often underestimated. We introduce POLICYGUARD, a sub-agent verifier that shares the agent's view of the dialogue, reasons over the policy in context, and provides actionable feedback for the agent's next turn. On tau^2-BENCH airline across three vendors (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro) with four trials per setting, POLICYGUARD improves PASS4 by +12.0 / +6.0 / +12.0 pp. Per-call analyses show POLICYGUARD achieves higher policy-violation recall while blocking roughly half as often as argument-level guards.

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Tool-Augmented Language Model Agents LLM Guardrails LLM Agent Evaluation LLM Agent Verification LLM Agent Safety

Ask PDF

Aug 20, 2026

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

Customer-service LLM agents must follow organizational policy when acting on a user's behalf. Compliance failures arise from either forbidden actions, such as granting an ineligible change, or omitted procedural requirements, such as identification or confirmation. Runtime safeguards can intervene on risky actions, but action-local checks do not guide an agent through a multi-step procedure. Workflow-following systems support prescribed process execution, but primarily target workflow completion rather than safeguarding agent behavior. PolicyGuide instead compiles each domain policy into a workflow graph and invokes a proactive verifier at user-turn boundaries. From persisted graph state, the verifier reconciles open requests and returns step-specific remediation along a policy-compliant path. Across the

τ^2

-bench airline, retail, and telecom domains with a GPT-5.4 agent and verifier, PolicyGuide raises mean

\mathrm{Pass}^4

from

0.42

0.62

, with the largest gain on telecom (

0.19

0.61

), the most workflow-structured domain. The same workflows transfer to Claude Sonnet 4.6 and Gemini 2.5 Pro agents. Complementary evaluations find the lowest observed attack-success rate under adversarial users and the strongest procedural compliance in an author-designed workflow-level validation.

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Runtime Enforcement for AI Agents AI Agent Governance LLM Agent Verification LLM Guardrails

Ask PDF

Mar 25, 2026

Willful Disobedience: Automatically Detecting Failures in Agentic Traces

AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from

τ^2

-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.

Reshabh K Sharma, Shraddha Barke, Benjamin Zorn

University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA

AI Agent Evaluation AI Agent Auditing

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

Willful Disobedience: Automatically Detecting Failures in Agentic Traces