Policy Compliance of User Requests in Natural Language for AI Systems

Pedro Cisneros-Velarde

cs.CLFeb 27, 2026

Policy Compliance of User Requests in Natural Language for AI Systems

Authors: Pedro Cisneros-Velarde

Organizations: VMware Research

Abstract

Consider an organization whose users send requests in natural language to an AI system that fulfills them by carrying out specific tasks. In this paper, we consider the problem of ensuring such user requests comply with a list of diverse policies determined by the organization with the purpose of guaranteeing the safe and reliable use of the AI system. We propose, to the best of our knowledge, the first benchmark consisting of annotated user requests of diverse compliance with respect to a list of policies. Our benchmark is related to industrial applications in the technology sector. We then use our benchmark to evaluate the performance of various LLM models on policy compliance assessment under different solution methods. We analyze the differences on performance metrics across the models and solution methods, showcasing the challenging nature of our problem.

Figures & tables

Figure 1: Examples of non-compliant (bad) and compliant (good) user requests for two given policies.

	TP $\uparrow$	FP $\downarrow$	FN $\downarrow$	FN* $\downarrow$	TN $\uparrow$	Accuracy $\uparrow$
Single Prompting
gpt-oss-120B	O 26.22 O	24.44	O 3.56 O	16.89	O 28.89 O	O 55.11 O
Llama 3.3 70B	10.22	37.78	1.78	34.67	15.56	25.78
Llama 3.1 8B	13.33	44.00	0.89	32.44	9.33	22.67
Mistral 7B	O 16.89 O	20.44	O 5.78 O	24.00	O 32.89 O	O 49.78 O
Gemma 3 4B	O 13.33 O	52.00	0.00	33.33	1.33	14.67

Table 1: Classification Performance of User Requests (%), Part I. For each solution method (Section 4 ), we highlight the highest accuracy, TP and TN with boldface, and the second highest with an underline. We also highlight with yellow the highest TP and TN achieved by each LLM model across all solution methods, and with gray the second highest ones. We highlight with red the highest accuracy achieved by each LLM model across all solution methods, and with green the second highest. Finally, we highlight with yellow the lowest relevant FN achieved by each model across all methods.

	TP $\uparrow$	FP $\downarrow$	FN $\downarrow$	FN* $\downarrow$	TN $\uparrow$	Accuracy $\uparrow$
Convincing Decision (C.D.)
gpt-oss-120B	4.00	36.00	1.78	40.89	17.33	21.33
Llama 3.3 70B	5.77	35.56	0.00	40.89	17.78	23.56
Llama 3.1 8B	0.00	53.33	0.00	46.67	0.00	0.00
Mistral 7B	0.89	50.67	0.00	45.78	2.67	3.56
Gemma 3 4B	0.00	53.33	0.00	46.67	0.00	0.00

Table 2: Classification Performance of User Requests (%), Part II. See the caption of Table 1 .

LLM Safety Benchmarks

Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.

See all 284 papers

PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
Sep 16, 2026Mika Okamoto, Ansel Kaplan ErolAI Agent Auditing LLM Agent Evaluation
Georgia Institute of Technology · Decagon AI · Baseten
EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents
Aug 20, 2026Seongjae Kang, Taehyung Yu, Sung Ju HwangRuntime Enforcement for AI Agents AI Agent Governance
KAIST

Sep 16, 2026J/K move · Enter open · S save

PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?

Mika Okamoto, Ansel Kaplan Erol

Georgia Institute of Technology · Decagon AI · Baseten

AI Agent Auditing LLM Agent Evaluation AI Agent Safety Benchmarks LLM Agent Reliability

Open paper AskPDF arXiv

As corporate AI adoption continues to grow, enterprise-grade LLM agents are being deployed into sensitive contexts such as hiring, healthcare, and finance. In these contexts, compliance with rules specified in an agent's system context is a first-order legal concern. Currently, no evaluation framework systematically measures which LLM models tend to violate compliance rules, especially under pressure from a persistent user, a hurried manager, or circumstances where violation is convenient or attractive. We introduce PACT (Pressure-Applied Compliance Testing), a benchmark for rule-following under pressure in AI agents assisting employees in daily tasks across twelve regulated enterprise domains and forty-eight scenarios, each set in a realistic multi-turn conversation. Each benchmark item pairs a standing rule against a rule-violating shortcut, and applies a battery of pressures across different wordings and system-prompt modes. We construct PACT component by component under strict LLM-as-judge auditing to ensure samples are unambiguous, ungameable, and realistic enough to avoid eliciting evaluation-aware behavior. We use PACT to profile LLM compliance across six complementary metrics that create a holistic picture of an AI assistant's robustness under pressure and throughout multi-turn conversations, its transparency, and ability to correctly discern where a rule applies. We aggregate this profile into PACTScore, a reliability-weighted compliance rate over all items and modes. Our results across 22 common LLM models spanning multiple providers and sizes show substantial variability in compliance across models and metric dimensions. Even the strongest assistants mis-apply a rule on 6 to 10% of items, and ordinary user pressure raises the violation rate by 65% on average. PACT highlights compliance risks in LLM assistants, motivating guardrails and careful model selection.

Abstract

LLM Safety Benchmarks

Explore similar work

PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents