Policy-Invisible Violations in LLM-Based Agents

Jie Wu; Ming Gong

cs.AIApr 14, 2026

Policy-Invisible Violations in LLM-Based Agents

Authors: Jie Wu, Ming Gong

Organizations: Atlassian

Abstract

LLM-based agents can execute actions that are syntactically valid, user-sanctioned, and semantically appropriate, yet still violate organizational policy because the facts needed for correct policy judgment are hidden at decision time. We call this failure mode policy-invisible violations: cases in which compliance depends on entity attributes, contextual state, or session history absent from the agent's visible context. We present PhantomPolicy, a benchmark spanning eight violation categories with balanced violation and safe-control cases, in which all tool responses contain clean business data without policy metadata. We manually review all 600 model traces produced by five frontier models and evaluate them using human-reviewed trace labels. Manual review changes 32 labels (5.3%) relative to the original case-level annotations, confirming the need for trace-level human review. To demonstrate what world-state-grounded enforcement can achieve under favorable conditions, we introduce Sentinel, an enforcement framework based on counterfactual graph simulation. Sentinel treats every agent action as a proposed mutation to an organizational knowledge graph, performs speculative execution to materialize the post-action world state, and verifies graph-structural invariants to decide Allow/Block/Clarify. Against human-reviewed trace labels, Sentinel substantially outperforms a content-only DLP baseline (68.8% vs. 93.0% accuracy) while maintaining high precision, though it still leaves room for improvement on certain violation categories. These results demonstrate what becomes achievable once policy-relevant world state is made available to the enforcement layer.

Figures & tables

Category	Description	Invariant
Context boundary	Session originated in an INTERNAL context (e.g., internal channel); content derived from that context must not flow to EXTERNAL recipients.	I2: ContextBoundary
Text-output leakage	Confidential data provided verbally by the user (no file read) is included in an outbound action; data-source tracking cannot detect it.	I4: ContentFingerprint
Oversharing	Bulk or folder-level sharing includes a restricted document among permitted ones; the model shares all items without per-item policy checking.	I3: InformationFlow
Audience restriction	Document has an explicit audience policy (e.g., HR_ONLY, UNTRUSTED) that prohibits the target recipient regardless of scope.	I3: InformationFlow
Accumulated session leakage	Reading a confidential document during a session taints the session’s data sources; a subsequent outbound action carries the tainted data to a lower-scope recipient.	I3: InformationFlow
Cross-context dataflow	A confidential document is sent to the external party it directly concerns (e.g., internal legal strategy to the opposing vendor).	I3: InformationFlow

Table 1: Taxonomy of policy-invisible violation categories. Each category has one or more cases in PhantomPolicy and a corresponding Sentinel invariant (§ 4.6 ).

Category	Violation	Safe
Context boundary	8	5
Text-output leakage	8	9
Oversharing	8	10
Audience restriction	8	7
High-value resource protection	8	7
Accumulated session leakage	7	7

Table 2: PhantomPolicy case distribution. The benchmark contains 60 violation cases across eight categories and 60 safe cases.

Panel of Figure 1 — Figure 1: Sentinel architecture: Counterfactual Graph Simulation. Outbound tool calls are intercepted and translated into proposed graph mutations. Sentinel forks the world state graph, speculatively applies the mutations, checks seven declarative invariants on the mutated graph, and decides Allow / Block / Clarify based on invariant satisfaction. Read-only calls update the session’s taint state for subsequent verification. The five-phase pipeline (Translate $\to$ Fork $\to$ Mutate $\to$ Check $\to$ Decide) replaces ad-hoc verifier rules with a unified model-checking framework.

Tool call	Generated mutations
read_file(path)	$\mathcal{S}.\mathit{data\_sources}\mathrel{+}=\{\text{resolve}(\mathit{path})\}$ (lazy)
send_email(to, body)	$\forall d\in\mathcal{S}.\mathit{data\_sources}$ : AddEdge $(d,r,\text{{Data\_Flows\_To}})$ ;
	if fingerprint match: AddTaintNode $(\hat{d},d_{\mathit{src}})$ + AddEdge $(\hat{d},r,\text{{Data\_Flows\_To}})$
share_files(to, paths)	$\forall p\in\mathit{paths}$ : AddEdge $(\text{resolve}(p),r,\text{{Data\_Flows\_To}})$
delete_email_thread(id)	RemoveNode $(\text{resolve}(\mathit{id}))$
forward_email(thread, to)	AddEdge $(\text{resolve}(\mathit{thread}),r,\text{{Data\_Flows\_To}})$

Table 3: Action translation rules.

r=\text{resolve}(\theta.\textit{to})

denotes recipient resolution. Read operations use lazy materialization : taint is accumulated in

\mathcal{S}.\mathit{data\_sources}

and expanded into Data_Flows_To edges at the next write action.

Invariant	CB	TOL	OS	AR	ASL	TV	CCD
I1: ActiveRecipient						$\checkmark$
I2: ContextBoundary	$\checkmark$
I3: InformationFlow			$\checkmark$	$\checkmark$	$\checkmark$		$\checkmark$
I4: ContentFingerprint		$\checkmark$
I5: RecipientContext						$\checkmark$
I6: ScopeBoundary

Table 4: Invariant coverage across violation categories. CB = Context Boundary, TOL = Text Output Leakage, OS = Oversharing, HV = High-Value Resource, AR = Audience Restriction, ASL = Accumulated Session Leakage, TV = Temporal Validity, CCD = Cross-Context Dataflow.

Model	Violated	Self-avoided	Safe cases violated
GPT-5.4	58/60 (96.7%)	2	2/60
GPT-5 mini	59/60 (98.3%)	1	2/60
GPT-5.4 nano	59/60 (98.3%)	1	3/60
Claude Sonnet 4.6	54/60 (90.0%)	6	8/60
Claude Opus 4.6	56/60 (93.3%)	4	3/60

Table 5: Execution-oriented baseline model behavior on PhantomPolicy using human-reviewed trace labels (60 original violation-ground-truth cases and 60 original safe-control cases per model). Violated counts risky-case traces labeled as true policy violations under human review. Self-avoided counts risky-case traces labeled as safe under human review. Safe cases violated counts human-reviewed violations on the 60 safe controls.

Model	Violated	Self-avoided	Safe cases violated
GPT-5.4	18/60 (30.0%)	42	2/60
GPT-5 mini	19/60 (31.7%)	41	2/60
GPT-5.4 nano	51/60 (85.0%)	9	6/60
Claude Sonnet 4.6	15/60 (25.0%)	45	2/60
Claude Opus 4.6	19/60 (31.7%)	41	5/60

Table 6: Policy-in-prompt condition on PhantomPolicy using human-reviewed trace labels. Violated counts risky-case traces labeled as policy violations. Self-avoided counts risky-case traces labeled safe. Safe cases violated counts violations on the 60 safe controls.

Model	TP	TN	FP	FN	Prec.	Rec.
GPT-5.4	23	59	1	37	0.96	0.38
GPT-5 mini	27	58	1	34	0.96	0.44
GPT-5.4 nano	27	57	1	35	0.96	0.44
Claude Sonnet 4.6	23	57	1	39	0.96	0.37
Claude Opus 4.6	22	60	1	37	0.96	0.37

Table 7: Content-only DLP baseline on PhantomPolicy against human-reviewed trace labels. We count a DLP Block as a predicted violation and all other outcomes as predicted safe. The DLP baseline inspects visible outbound content, visible file contents, and visible thread previews, but does not access hidden world-model attributes.

Model	$n$	TP	TN	FP	FN	Prec.	Rec.
GPT-5.4	119	55	58	1	5	0.98	0.92
GPT-5 mini	120	56	57	2	5	0.97	0.92
GPT-5.4 nano	120	58	56	2	4	0.97	0.94
Claude Sonnet 4.6	120	46	58	0	16	1.00	0.74
Claude Opus 4.6	120	52	61	0	7	1.00	0.88

Table 8: Sentinel evaluation on PhantomPolicy against human-reviewed trace labels. We count Block as a predicted violation and Allow / Clarify as predicted safe for autonomous execution. Results reflect detection performance under complete benchmark world-model coverage.

Category	GPT-5 mini	GPT-5.4 nano	Son.	Opus	GPT-5.4
Accumulated session leakage	7/7	7/7	7/7	6/7	7/7
Audience restriction	8/8	8/8	8/8	7/8	8/8
Context boundary	8/8	8/8	5/8	7/8	8/8
Cross-context dataflow	5/6	6/6	6/6	6/6	6/6
High-value resource protection	8/8	7/8	7/8	8/8	6/8
Oversharing	8/8	8/8	7/8	8/8	8/8

Table 9: Human-reviewed violating risky-case traces by category on PhantomPolicy (numerator = risky-case traces labeled as true policy violations under human review; denominator = original violation-ground-truth cases in that category).

Coverage	Recall $\mu$	Recall $\sigma$	Min	Max
100%	100.0%	0.0%	100.0%	100.0%
90%	86.8%	7.8%	65.0%	100.0%
80%	79.6%	10.3%	56.7%	98.3%
70%	68.9%	11.5%	45.0%	93.3%
60%	62.4%	10.0%	35.0%	80.0%
50%	51.6%	11.8%	28.3%	78.3%

Table 10: Coverage degradation: Monte Carlo random entity removal (50 trials per level). Recall degrades monotonically with coverage; FP remains 0 at all levels. Even at 0% entity coverage, 20% of violations are still caught by context-boundary invariants that depend on session metadata rather than entity presence.

Strategy	Limitation vs. world-state-grounded enforcement
Baseline (no enforcement)	No policy awareness: violations whenever the model executes a risky action.
Content DLP	Against human-reviewed labels: 68.83% accuracy / 56.61 F 1 overall, with 96.06% precision but only 40.13% recall; misses cases whose decisive attributes never appear in visible content.
Policy-in-prompt	Under human-reviewed labels: reduces risky-case violations from 95.3% to 40.7% overall, but highly variable across models (25%–85% violation rate); safe-case errors at 5.7%.
Confirmation-only	User often uninformed; high friction; does not remove knowledge asymmetry about staged or directory-local policy.
Sentinel	Under full benchmark coverage and human-reviewed trace labels: 92.99% accuracy / 92.71 F 1 overall, with 5 false positives and 37 missed human-reviewed violations (Table 8 ).

Table 11: Comparison of enforcement strategies. All quantitative results (Sentinel, content-only DLP, and policy-in-prompt) are computed against human-reviewed trace labels on PhantomPolicy; the confirmation-only row summarizes qualitative trade-offs.

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 321 papers

Aug 20, 2026

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

Customer-service LLM agents must follow organizational policy when acting on a user's behalf. Compliance failures arise from either forbidden actions, such as granting an ineligible change, or omitted procedural requirements, such as identification or confirmation. Runtime safeguards can intervene on risky actions, but action-local checks do not guide an agent through a multi-step procedure. Workflow-following systems support prescribed process execution, but primarily target workflow completion rather than safeguarding agent behavior. PolicyGuide instead compiles each domain policy into a workflow graph and invokes a proactive verifier at user-turn boundaries. From persisted graph state, the verifier reconciles open requests and returns step-specific remediation along a policy-compliant path. Across the

τ^2

-bench airline, retail, and telecom domains with a GPT-5.4 agent and verifier, PolicyGuide raises mean

\mathrm{Pass}^4

from

0.42

0.62

, with the largest gain on telecom (

0.19

0.61

), the most workflow-structured domain. The same workflows transfer to Claude Sonnet 4.6 and Gemini 2.5 Pro agents. Complementary evaluations find the lowest observed attack-success rate under adversarial users and the strongest procedural compliance in an author-designed workflow-level validation.

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Runtime Enforcement for AI Agents AI Agent Governance LLM Agent Verification LLM Guardrails

Ask PDF

Jun 28, 2026

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

LLM agents handle user requests on behalf of organizations through tool calls and must follow the company policies stated in their system prompts. Prior work approaches this as a safeguarding problem -- external checks that block non-compliant agent actions. We argue that policy adherence is a broader problem: real workflows unfold across many turns, require explicit user confirmation and prerequisite reads, and hinge on the content of the dialogue rather than on any single argument value. Meeting this bar requires (i) full conversation context, (ii) self-reasoning over the policy and the current dialogue, and (iii) conversation-specific remediation that guides the agent's next turn -- three capabilities that prior safeguard work has often underestimated. We introduce POLICYGUARD, a sub-agent verifier that shares the agent's view of the dialogue, reasons over the policy in context, and provides actionable feedback for the agent's next turn. On tau^2-BENCH airline across three vendors (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro) with four trials per setting, POLICYGUARD improves PASS4 by +12.0 / +6.0 / +12.0 pp. Per-call analyses show POLICYGUARD achieves higher policy-violation recall while blocking roughly half as often as argument-level guards.

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Tool-Augmented Language Model Agents LLM Guardrails LLM Agent Evaluation LLM Agent Verification LLM Agent Safety

Ask PDF

Dec 3, 2025

Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs

As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority. Existing content moderation frameworks remain largely confined to the safety domain and lack the robustness to capture nuanced organizational policies. LLM-as-a-judge and fine-tuning approaches, though flexible, introduce significant latency and training cost. To address these limitations, we frame policy violation detection as an out-of-distribution (OOD) problem in the model's activation space. We propose a training-free method that operates directly on the LLM internal representations, leveraging prior evidence that decision-relevant information is encoded within them. Inspired by whitening techniques, we derive policy-violation scores directly from normalized representations of LLM hidden activations. Our method requires only the policy text and a small number of illustrative samples, making it lightweight and easily deployable. We extensively evaluate our method across multiple LLMs and challenging policy benchmarks. It achieves up to 86.0% F1, outperforming both fine-tuned and LLM-as-a-judge baselines while requiring substantially less computation. Our code is publicly available at: https://github.com/FujitsuResearch/LLM-policy-violation-detection

Oren Rachmil, Avishag Shapira, Roy Betser +5

Fujitsu Research of Europe · Ben-Gurion University of the Negev

Language Model Safety Evaluation LLM Auditing LLM Safety

Ask PDF

Abstract

LLM Agent Safety

Explore similar work

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs