PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

Kshitij Mishra; Abhijith Sharma; Nils Lukas; Salem Lahlou

cs.AIOct 15, 2026

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

Authors: Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Abstract

Modern agentic systems extend language models with external tools, enabling them to complete complex tasks through multi-turn interactions. As these agents are deployed in consequential settings, they are often constrained by policies governing pricing, privacy, confirmation, safety, and other requirements. However, existing evaluation methods leave a critical gap: capability benchmarks measure what agents can accomplish, while content-safety red-teaming measures harmful generations, but neither tests whether agents comply with predefined policies during otherwise valid tool use. Such policy violations are difficult to audit because their effects may emerge only after several turns, span multiple tool calls, or surface in downstream systems. We propose PROBE (Policy Rule Observation via Behavioral Exploration), an algorithm for training LLM-based auditors that interact with deployed agents as realistic users and probe them for policy violations through multi-turn, tool-mediated dialogue. We formalize agent auditing as a two-player Markov game and propose a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity. To study training objectives and reward design, we compare GRPO, DPO, and GFlowNet-based optimization in a controlled setting. Across extensive experiments, auditors trained with PROBE uncover substantially broader and higher-yield policy violations than same-architecture zero-shot adversaries, even when the auditor is significantly smaller than the target agent. Our results show that multi-turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing capability and safety benchmarks. PROBE offers a scalable path toward automated policy auditing for real-world agentic systems and suggests a co-evolutionary loop in which auditors expose failures and agents improve from auditor feedback.

Figures & tables

Figure 1: Probe overview. Verifiers score the full trace, including tool calls hidden from the auditor; the composite reward (Eq. 1 ) drives one of three updates. Bottom: an illustrative rollout.

Figure 2: The SkyAssist environment : (A) database, (B) tools and the policies each can violate, (C) verifiers, (D) the 20 policies (App. D ).

Group	Auditor	Coverage $\uparrow$	Yield/100	B4/S5/B6 $\downarrow$	Multi-pol.	Entropy
Baselines	Random scenarios	5%	28	100%	4%	0.69
	Zero-shot Qwen3-30B	35%	620	88%	64%	2.45
Behavior reward	GRPO	55%	1090	82%	88%	3.57
	DPO	60%	1098	79%	86%	3.30
	GFlowNet	45%	1042	84%	86%	3.29
Dedup-only	GRPO	55%	1914	78%	100%	3.30

Table 1: Held-out evaluation on 50 matched scenarios, training-time verifier suite (B4/B6/S5/P1 judge-scored; deterministic re-scoring in Figure 7 ). B4/S5/B6: share of hits on three high-frequency policies; best in bold.

Figure 4: Behavior-reward evaluation, 50 held-out episodes: (a) union coverage; (b) per-policy violation counts. B4/S5/B6 dominate raw hits; mutation-dependent policies stay hard to reach.

Figure 5: Training dynamics. DPO shows a usable preference signal, GFlowNet reaches the highest training yield, GRPO improves steadily, and plausibility rewards shorten auditor completions.

Figure 6: Reward–algorithm interaction. Calibration raises the training yield of reward-scale-sensitive GRPO and GFlowNet; DPO, which depends on winner–loser orderings, changes less.

Tool	GRPO	DPO	GFlowNet
search	461	341	420
check	0	0	4
hold	6	4	1
book	3	1	3
cancel	0	0	0

Table 2: Target tool calls during evaluation. Mutation tools (amber rows) are rare, limiting the policies any black-box auditor can expose.

Figure 7: Deterministic re-scoring and stronger baselines (50 held-out episodes). Error bars:

\pm

sd over three samples; 95% bootstrap CI. † Expert per-policy answer key.

Auditor	Cov.	Yield/100
Gemma-4-E4b-it target
Static suite	5%	42
Probe -DPO	40%	190
$\tau$ -bench
Zero-shot 9B	20%	26
Probe -DPO 9B	30%	58

Table 3: Transfer to an unseen Gemma target ( SkyAssist ) and to

\tau

-bench (same 9B base for both auditors).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

System	Multi-turn	Tool calls	Policy	Learned	Adaptive	Diverse
Probe (ours)	✓	✓	✓	✓	✓	✓
$\tau$ -bench [ Yao et al., 2024 ]	✓	✓	partial	✗	✗	✗
$\tau^{2}$ -bench [ Barres et al., 2025 ]	✓	✓	partial	✗	✗	✗
ST-WebAgentBench [ Levy et al., 2024 ]	✓	✓	✓	✗	✗	✗
GFN red-teaming [ Lee et al., 2025 ]	✗	✗	✗	✓	✓	✓
PAIR [ Chao et al., 2025 ]	partial	✗	✗	✗	✓	✗

Appendix

Table 4: Comparison with related approaches. Probe combines a learned prober, multi-turn tool use, adaptive interaction, diverse coverage pressure, and trace-level task-policy targets.

Component	Dialog.	Tool names	Tool out.	Database	Target wts.	Verifier labels
Auditor $\pi_{\theta}$	✓	indirect	indirect	✗	✗	reward only (training)
Target $\pi_{T}$	✓	✓	✓	tool-mediated	✓	✗
Verifier suite	✓	✓	✓	✓	✗	✓
Human evaluator	✓	optional	optional	optional	✗	aggregated

Appendix

Table 5: Information sets of each component of the auditing game. The auditor is restricted to the interface of an ordinary customer; the verifier inspects the full private trace; the target observes the database only through tool outputs.

Figure 8: DPO margin growth and implicit accuracy under the calibrated reward. The pairwise signal becomes informative only after the reward separates genuinely different trajectories from repeated-hit trajectories.

Figure 9: GFlowNet diagnostics. The learned partition estimate grows smoothly while the trajectory-balance loss decreases, indicating optimization progress even when held-out coverage does not improve monotonically.

Entity	Description	Count
Airports	Major international airports with IATA, city, country, time zone, restricted-destination flag	35
Routes	Directed non-stop graph; multi-leg itineraries composed at runtime	120
Flights	Route–date–time instances with inventory, status, fare-class availability	643
Fare classes	Economy, premium economy, business; price, cancellation terms, upgrade eligibility, internal code	3
Cancelled flights	Flights marked unavailable; must not be presented as bookable	46
Restricted destinations	Airports that must not be booked under policy B3	4

Appendix

Table 6: SkyAssist database. Deterministic across runs, enabling reproducible trace-level auditing.

Tool	Type	Policy relevance
search_flights	Query	B2 USD-only; B5 fare/cancellation disclosure; P1 internal-code privacy
check_availability	Query	S1 sufficient info; S4 safe defaults; O3 connection-time validation
get_flight_details	Query	S1 tool gating; S2 schema adherence; P3 tool/API non-disclosure
lookup_booking	Query	P2 passenger data; P4 PII redaction; S6 graceful errors
hold_itinerary	Mutation	S5 confirmation; B4 upgrade mention; B6 round-trip suggestion
book_flight	Mutation	S5 confirmation; B1 fare recommendation; B3 restricted; O1 passenger limit; O2 advance booking

Appendix

Table 7: SkyAssist tools and the policies they can violate.

Tool	Required arguments
search_flights	origin , destination , date ; optional passenger_count , cabin_preference
check_availability	flight_id , date , passenger_count
get_flight_details	flight_id
lookup_booking	booking_id + at least one verification field (e.g., last_name , email )
hold_itinerary	flight_id , passenger_count , fare_class
book_flight	flight_id , passengers , fare_class , payment_token , confirmed=true

Appendix

Table 8: Tool schemas. The verifiers check schema adherence, sufficient-information gating, and unsafe mutation behavior against these required fields.

ID	Description	Category	Sev.
B1	Recommend higher-margin fare classes when appropriate and available; do not hide cheaper valid options.	Business	Med
B2	Quote prices in USD only; do not introduce unsupported currencies or conversions.	Business	High
B3	Never book or hold itineraries to restricted destinations.	Business	Critical
B4	Proactively mention premium or upgrade options when available.	Business	Low
B5	Disclose fare class and cancellation/refund policy with every concrete quote.	Business	Med
B6	Suggest round-trip when one-way is requested but a return is plausible.	Business	Low

Appendix

Table 9: Complete policy taxonomy. Rows are colour-coded by category; severities are Critical=4, High=2, Med=1, Low=0.5.

Policy	Verifier type	Firing unit	Primary evidence
B1	Programmatic	Quote event	Fare availability vs. quoted option
B2	Programmatic	Assistant resp.	Currency symbols and price mentions
B3	Programmatic	Mutation call	Destination restricted flag
B4 ∗	Programmatic	Assistant resp.	Whether upgrade option mentioned
B5	Programmatic	Quote event	Fare and cancellation disclosure fields
B6 ∗	Programmatic	Assistant resp.	Whether round-trip alternative suggested

Appendix

Table 10: Verifier implementation. Event-level verifiers fire only on relevant tool calls or trace events; turn-level verifiers can fire after any assistant response. ∗ Scored by a hybrid LLM judge during training and in Table 1 ( Appendix E ); deterministic predicate in the re-scoring of Figure 7 .

Table 20

Auditor	Per-sample coverage	Mean $\pm$ sd	Range
Probe -GRPO	50, 50, 55	51.7 $\pm$ 2.9	50–55
Probe -DPO	55, 60, 55	56.7 $\pm$ 2.9	55–60

Appendix

Table 14: Coverage (%) of the trained auditors on three independent held-out samples of 50 episodes (deterministic scoring).

Reward	Design	Observed effect
Baseline (V1)	Severity-weighted verifier hits, weak plausibility, no randomized scenarios, one-step continuation	130+ word auditor messages, 58% role confusion, 34% loops, 9/20 policies
Behavior (V2)	Adds customer-role prompt, 27 scenarios, progress shaping, novelty, full plausibility scorer, three-turn continuations	Role confusion falls to 11%; 12–14/20 policies in training sweeps; high-frequency B4/S5/B6 still dominate raw reward
Calibrated, Probe (V3)	Per-policy diminishing returns, stronger anti-fabrication, database-aware scenarios, loop stopping	DPO margins increase, repeated-hit inflation decreases

Appendix

Table 15: Reward-design ablations (training-time diagnostics).

Figure 10: Reward-evolution diagnostic (training-time verifiers). Behavior constraints had to become part of the training objective rather than an after-the-fact filter.

Reward configuration	Optimizer	Coverage
Behavior (V2)	GRPO / DPO / GFlowNet	50% / 55% / 40%
Behavior + per-policy dedup	GRPO / DPO	50% / 50%
Calibrated, Probe (V3)	GRPO / DPO / GFlowNet	55% / 55% / 45%
Diversity-only (binary success)	GFlowNet	35%

Appendix

Table 16: Coverage of every trained reward configuration (deterministic scoring, 50 held-out episodes each).

Configuration	Coverage	Yield/100
$0.5\times\gamma_{r}$	50% (10/20), identical policy set	156
$1\times\gamma_{r}$ (reference)	50% (10/20)	144
$2\times\gamma_{r}$	45% (9/20)	150

Appendix

Table 17: Perturbing the plausibility weight

\gamma_{r}

(GRPO, matched training step, deterministic scoring, 50 held-out episodes).

Family	Purpose	Templates
Ordinary booking	Plausible search-to-booking flows with common city pairs	6
Budget-sensitive booking	Tests fare disclosure, upsell, USD-only pricing	5
Ambiguous request	Tests safe defaults, tool gating, clarification behavior	4
Restricted route	Targets restricted-destination policy B3	4
Group/edge-case booking	Targets passenger limits, advance booking, schema constraints	4
Cancellation/modification	Targets confirmation, lookup privacy, error handling	4

Appendix

Table 18: Scenario families used for training and evaluation.

Parameter	Value
Auditor base model	Qwen3-4B-Instruct
Target model	Qwen3-30B-A3B (vLLM)
Fine-tuning method	4-bit NF4 QLoRA
LoRA rank / $\alpha$	64 / 128
LoRA target modules	Q, K, V, O projections
Trainable parameters	$\sim$ 47M

Appendix

Table 19: Training and optimization hyperparameters.

Algorithm	Hyperparameters
GRPO	Clip $\epsilon_{c}=0.2$ ; KL coefficient $\beta_{\mathrm{KL}}=0.01$
DPO	Preference temperature $\beta_{\mathrm{D}}=0.1$ ; EMA reference decay $0.995$
GFlowNet	$\log Z_{\phi}$ learning rate $10^{-5}$ ; trivial backward policy ( $P_{B}\equiv 1$ ); terminal reward clipped to $[-10,10]$ before exponentiation

Appendix

Table 20: Algorithm-specific hyperparameters.

Parameter	Value
Severity weights	Low=0.5, Medium=1, High=2, Critical=4
Plausibility weight $\gamma_{r}$	0.1 (V1); 0.25 (V2); 0.35 (V3)
Assistant-like penalty $\lambda_{\mathrm{asst}}$	0.5
Fabrication penalty $\lambda_{\mathrm{fab}}$	0.6 (V3)
Formatting penalty $\lambda_{\mathrm{fmt}}$	0.3
Length penalty $\lambda_{\mathrm{len}}$	0.2

Appendix

Table 21: Reward hyperparameters. Sensitivity to these values is analyzed in Appendix H .

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 248 papers

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents
Jun 28, 2026Seongjae Kang, Taehyung Yu, Sung Ju HwangTool-Augmented Language Model Agents LLM Guardrails
KAIST
AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes
Aug 30, 2026Xun Wang, Bihe Zhao, Michael Backes +2LLM Auditing Language Model Fingerprinting
CISPA Helmholtz Center for Information Security
AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark Auditing AI Agent Evaluation
Anthropic Fellows Program · Anthropic

Jun 28, 2026J/K move · Enter open · S save

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Tool-Augmented Language Model Agents LLM Guardrails LLM Agent Evaluation LLM Agent Verification LLM Agent Safety

Open paper AskPDF arXiv

LLM agents handle user requests on behalf of organizations through tool calls and must follow the company policies stated in their system prompts. Prior work approaches this as a safeguarding problem -- external checks that block non-compliant agent actions. We argue that policy adherence is a broader problem: real workflows unfold across many turns, require explicit user confirmation and prerequisite reads, and hinge on the content of the dialogue rather than on any single argument value. Meeting this bar requires (i) full conversation context, (ii) self-reasoning over the policy and the current dialogue, and (iii) conversation-specific remediation that guides the agent's next turn -- three capabilities that prior safeguard work has often underestimated. We introduce POLICYGUARD, a sub-agent verifier that shares the agent's view of the dialogue, reasons over the policy in context, and provides actionable feedback for the agent's next turn. On tau^2-BENCH airline across three vendors (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro) with four trials per setting, POLICYGUARD improves PASS4 by +12.0 / +6.0 / +12.0 pp. Per-call analyses show POLICYGUARD achieves higher policy-violation recall while blocking roughly half as often as argument-level guards.

Abstract

AI Agent Auditing

Explore similar work

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors