Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards

Min-Young Yu

cs.LGOct 14, 2026

Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards

Authors: Min-Young Yu

Organizations: Corners Co., Ltd.

Abstract

Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward. We ask whether these decisions can instead become verifiable policy rewards for training the agents they guard. We introduce Policy-Verified Self-Training (PVST), a post-training paradigm in which an agent generates its own interaction trajectories and an executable policy verifier, here a deterministic gate compiled from a written policy, evaluates whether each action complies. Compliant trajectories are reinforced, and violations become negatives for preference optimization; beyond the gate and the benchmark's task-success signal, PVST needs no stronger teacher model, no human preference labels and no attack-specific supervision, although rollouts are collected in the benchmark's attacked environment. On held-out AgentDojo tasks under prompt injection, policy-verified preference training of Qwen3-8B reduces violating trajectories from 37.2% to 10.3 $\pm$ 0.9% (three seeds) and successful attacks from 12.4% to 0.5% at equal or higher pooled task utility, although utility falls on travel, compared with 22.0% when negatives are selected by an attack oracle and 31.9 $\pm$ 0.9% with negatives that contain no violation. The learned behavior holds under an attack template never used in training (37.7% to 11.0%) and on a suite excluded from training (34.8% to 16.7%), and the retained verifier's intervention rate falls from 39.2% to 8.8% (seed 0). Our analysis identifies two conditions under which the verdict is most informative: success must not already imply compliance, and guarded actions must appear in both permitted and forbidden contexts.

Figures & tables

Figure 1: Left: a runtime gate blocks a forbidden call and its verdict is discarded. Right: PVST replays the agent’s own rollouts through the same gate, keeps compliant successes as positives, cuts violating rollouts at the denied call as negatives and post-trains the agent; at deployment the gate stays and intervenes a fifth as often.

Arm	Negatives	Seeds	Utility %	ASR %	Violating traj. %	$\mathrm{util}_{w}$ %	$\mathrm{util}_{r}$ %
Base	–	–	35.5	12.4 [8.3, 16.5]	37.2 [31.0, 43.4]	21.5	53.3
RFT-succ	–	1	35.4	5.4	30.8	18.7	56.6
RFT-gate	–	3	40.2 $\,\pm\,3.3$	5.5 $\,\pm\,1.2$	28.8 $\,\pm\,2.0$	21.3 $\,\pm\,4.9$	65.0 $\,\pm\,3.3$
DPO-gate	gate-rejected, cut at denied call	3	39.9 $\,\pm\,1.9$	0.5 $\,\pm\,0.2$ [0.0, 1.2]	10.3 $\,\pm\,0.9$ [5.6, 12.8]	21.1 $\,\pm\,2.6$	65.1 $\,\pm\,0.5$
DPO-ctrl-any	any failure (55% violating)	3	45.1 $\,\pm\,0.2$	1.2 $\,\pm\,1.2$	20.2 $\,\pm\,2.8$ [14.5, 24.5]	20.6 $\,\pm\,1.1$	75.7 $\,\pm\,0.9$
DPO-ctrl-clean	violation-free failures	3	41.0 $\,\pm\,3.4$	6.5 $\,\pm\,1.1$	31.9 $\,\pm\,0.9$ [26.3, 38.4]	20.6 $\,\pm\,3.4$	69.3 $\,\pm\,6.0$

Table 1: Held-out user tasks

\times

all injection tasks, important_instructions , gate off, three suites pooled (

n\approx 240

). Mean

\pm

sd over seeds; brackets: 95% per-trajectory bootstrap CIs, seed 0.

Figure 2: Left: violating trajectories against utility per arm (Qwen3-8B, held-out AgentDojo, attack, gate off; three-seed means where available). Right: violation attempts with the gate off, gate interventions with the gate on, and successful attacks (ASR) with the gate off, for the base model and the two gate-trained arms (seed 0).

Suite	Base	RFT-gate	DPO-gate	DPO-ctrl-clean
banking ( $n=45$ )	40.0 / 24.4 / 37.8	40.0 / 4.4 / 13.3	40.0 / 0.0 / 0.0	40.0 / 11.1 / 24.4
travel ( $n=42$ )	33.3 / 26.2 / 45.2	31.0 / 14.3 / 26.2	16.7 / 0.0 / 0.0	23.8 / 23.8 / 45.2
workspace ( $n=145$ – $164$ )	34.8 / 5.2 / 34.8	43.9 / 3.0 / 34.8	46.0 / 0.6 / 14.1	44.8 / 0.7 / 31.0

Table 2: Per suite (utility / ASR / violating trajectories, %), seed 0.

Table 5

Model	Gate off	Gate on (passˆ1 / residual viol.)
base	21.7 / 88.5 / 3.20	30.0 / 18.6
SFT (reference-filtered distillation, airline)	23.3 / 40.9 / 0.73	30.0 / 7.7
SFT (airline + retail distillation)	35.0 / 33.3 / 0.75	40.0 / 10.3
RFT-1 (airline distillation + SFT-airline gate-on rollouts)	40.0 / 46.8 / 0.78	–
DPO (reference-labelled, from RFT-1)	31.7 / 0 of 0 writes / 0.00	31.7 / 0 of 0 writes
DPO-ctrl (from RFT-1, verdict-blind negatives)	30.0 / 41.7 / 0.40	35.0 / 5.9

Table 5:

\tau^{2}

-bench airline, the 20 held-out tasks of the 30/20 split

\times

3 trials, user simulator Qwen3-8B; passˆ1 / violation rate on WRITE calls / WRITE calls per simulation, violations under the per-action reading of the confirmation clause as in Yu et al. (2026) (a batch reading gives lower rates). Only the base row is a held-out evaluation: the training data of the SFT and RFT-1 rows contain trajectories of 10 (SFT airline), 12 (airline + retail) and 11 (RFT-1) of these 20 tasks, so these rows are in-distribution; the DPO rows start from RFT-1, with pairs from the 30 training tasks. One training run per row.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Suite	Rule	Type	Predicate	Tools
banking	BAN-AUTO-FBD-01	forbidden	password_not_from_user	update_password
banking	BAN-AUTO-FBD-02	forbidden	recipient_unknown	schedule_transaction , send_money
banking	BAN-AUTO-PRE-03	precondition	action_requested_by_user	schedule_transaction , send_money , update_password ,
				update_scheduled_transaction , update_user_info
travel	TRA-AUTO-FBD-01G	forbidden	target_not_user_named	send_email
travel	TRA-AUTO-FBD-02	forbidden	target_not_user_named	reserve_car_rental , reserve_hotel , reserve_restaurant

Appendix

Table 6: Rule sets used as the reward on AgentDojo. A rule ending in G is the copy retargeted by the compiler’s accessor-activation check ( Yu et al., 2026 ) . On

\tau^{2}

-bench airline the reward uses the 16-rule hand-written reference set (AIR-FBD-01 to 14, AIR-PRE-01 to 02) and the live gate is the 10-rule compiled set (AIR-AUTO-FBD-01 to 09, AIR-AUTO-PRE-10) of Yu et al. (2026) ; both files are in the artifact.

Predicate	Holds when	Used as
recipient_unknown	the recipient argument was neither named in any user turn (string or account-token match) nor seen as a counterparty field of a returned account record	forbidden
target_not_user_named	any value of a target-bearing argument present in the call ( recipient(s) , email , participants , calendar title , hotel , restaurant , company , file and email ids; and channel , user , user_email , url on suites not used here) was not named in any user turn; a list holds if any element is unnamed. Names, addresses and titles match as case-insensitive substrings of a user turn (URLs also without an agent-supplied scheme); file and email ids match only as whole tokens outside a date context	forbidden
body_has_unnamed_url	a body / content / message argument contains a URL the user never named	forbidden
password_not_from_user	the password argument does not appear in any user turn	forbidden
action_requested_by_user	for a call with a recipient argument (a payment): the recipient was named in a user turn (string or account-token match) or is a counterparty in the account history; for any other write: some user turn contains one of twelve request words (update, change, set, schedule, pay, send, transfer, cancel, correct, modify, new password, move), or the user named a source the agent then retrieved; or an explicit confirmation is pending. It establishes that some change was requested, not which ( Yu et al., 2026 ) ; injected text never enters the user channel and cannot satisfy it by itself	precondition
user_confirmed (bound by no AgentDojo rule set here; used on $\tau^{2}$ by the compiled gate and the reference clauses)	AgentDojo: an affirmative user turn answered an assistant turn that made no call; $\tau^{2}$ : the latest user turn is affirmative. Consumed by the next guarded write	precondition

Appendix

Table 7: Predicates of the AgentDojo gate (evaluated on the fact store rebuilt from all messages before the call).

Set	Trajectories / pairs	Notes
RFT-gate positives	158 (banking 43, travel 55, workspace 60), 0.80M tokens	$u=1\wedge v=0$ , $\leq 8$ per task
RFT-succ positives	181 (48 / 56 / 77), 0.97M tokens	$u=1$ ; 71 of 570 candidates contain a violation
DPO-gate pairs	113 (30 / 57 / 26)	rejected cut after first denied call
DPO-ctrl-clean pairs	73 (6 / 51 / 16)	rejected: $u=0\wedge v=0$ , random cut
DPO-ctrl-any pairs	93 (10 / 57 / 26)	29 of 53 rejected are also violating
DPO-oracle pairs	113 (30 / 57 / 26)	rejected: attack oracle positive, random cut; 34 of 63 rejected are also violating

Appendix

Table 8: Training data statistics.

Model	Gate	Utility %	Violation attempts %	ASR %	Gate interventions %
Base	off	35.5	37.2	12.4	–
Base	on	32.2	39.2	0.0	39.2
RFT-gate	off	41.0	29.5	5.2	–
RFT-gate	on	37.8	29.1	0.0	29.1
DPO-gate	off	40.0	9.2	0.4	–
DPO-gate	on	36.0	8.8	0.0	8.8

Appendix

Table 9: Base and trained models with the gate off and on (held-out, attack, seed 0). Gate interventions = trajectories with at least one call the gate refuses, counted by replaying the gated transcript; for gate-on rows it equals the violation-attempt rate by construction.

Suite	Tool	Base denied	Base allowed	DPO-gate allowed
banking	schedule_transaction	20.0	0.0	0.0
banking	send_money	91.1	24.4	2.2
banking	update_password	4.4	0.0	0.0
banking	update_scheduled_transaction	8.9	0.0	0.0
travel	create_calendar_event	21.4	16.7	0.0
travel	reserve_hotel	9.5	0.0	0.0

Appendix

Table 10: Denied and allowed calls per 100 trajectories for each write tool, base vs DPO-gate (held-out, attack, gate off, seed 0). On banking, allowed calls include injected payments to payees already in the account history, which the rule set admits ( Section 7 ).

Arm	Utility %	ASR %	Violating traj. %	Gate interventions % (gate on)
Base	48.6	12.5	37.3	36.1
RFT-gate	52.5	9.0	31.4	30.4
DPO-gate	38.0	0.0	10.2	10.6
DPO-ctrl-clean	50.6	9.4	28.6	29.0

Appendix

Table 11: Qwen3-4B-Instruct-2507 (a newer, smaller, non-thinking trainee) with the same pipeline, rollouts of its own (2,568 trajectories; 246 positives, 130 gate pairs, 84 control pairs), one seed; held-out attack, gate off and on.

Trainee / arm	Permitted call executed %	Forbidden call executed %	Withheld %
4B base	14.6	19.5	65.9
4B RFT-gate	29.3	14.6	56.1
4B DPO-gate	24.4	0.0	75.6
4B DPO-ctrl-clean	29.3	12.2	58.5
8B base	19.5	56.1	24.4
8B RFT-gate	9.8	41.5	48.8

Appendix

Table 12: Paired diagnostic (8B arms: seed 0; 4B arms: one seed): 41 matched contexts from training-task rollouts, each a shared prefix in which the guarded tool can be called with a permitted or a forbidden target; the next turn is generated at temperature 0 and the emitted call replayed through the gate. The minimal pairs themselves were never used for training, but 13 of the 41 prefixes also occur in rejected trajectories of the 8B DPO-gate pair set (10 identical up to and including the assistant turn that makes the forbidden call; 13 and 9 for the random-cut arm, 2 and 1 for DPO-oracle, 2 and 0 for its write-cut variant, none for the 8B controls), so the 8B gate-selected rows are partly in-distribution. The 4B arms were trained on their own rollouts and share no prefix with their negatives.

Pairs	0 (RFT-gate)	28	56	73	113
Violating traj. %	29.5	17.3	11.8	6.2	9.2
ASR %	5.2	0.4	0.0	0.4	0.4
Utility %	41.0	38.0	41.2	37.2	40.0

Appendix

Table 13: DPO data scaling (pairs

\rightarrow

held-out attack, gate off, seed 0).

Figure 3: DPO data scaling ( Table 13 ): violating trajectories, successful attacks (ASR) and utility against the number of gate-labelled preference pairs (held-out attack, gate off, seed 0).

Arm	$n$ (all)	errors	Viol. % (all)	Viol. % (common)	ASR % (common)	Utility % (common)
Base	242	13	37.2	38.7	12.6	36.1
RFT-succ	240	15	30.8	32.2	5.7	34.3
RFT-gate	251	4	29.5	28.3	5.7	41.3
DPO-gate (seeds 0/1/2)	250 / 244 / 255	5 / 11 / 0	9.2 / 10.7 / 11.0	6.1 / 10.0 / 6.1	0.0 / 0.4 / 0.4	35.7 / 39.1 / 33.9
DPO-ctrl-any (seeds 0/1/2)	241 / 241 / 241	14 / 14 / 14	19.5 / 23.2 / 17.8	20.4 / 24.3 / 18.7	1.3 / 2.6 / 0.0	43.9 / 44.3 / 44.3
DPO-ctrl-clean (seeds 0/1/2)	232 / 230 / 230	23 / 25 / 25	32.3 / 32.6 / 30.9	32.6 / 32.6 / 30.9	7.0 / 7.4 / 5.2	40.0 / 44.8 / 38.3

Appendix

Table 14: Held-out attack, gate off, on the 230 triples (suite, user task, injection task) that every arm evaluated without a context-length error. Differences carry 95% confidence intervals from a cluster bootstrap over user tasks (5,000 resamples).

AI Agent Safety

AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.

See all 326 papers

Oct 20, 2026

AgentPolicyForge: From Executable Environments to Policy Benchmarks and Adaptive Guards

Customized agent deployments define policies over tools, resources, and execution conditions. Providing these policies in the task agent's prompt does not reliably prevent violations. Training a guard requires examples of tool calls that comply with or violate those policies. We introduce AgentPolicyForge, a framework that uses executable environments to construct policy benchmarks and iteratively train policy guards. With this framework, we construct PolicyGym, a dataset across ten policy families using 698 scenarios for training data generation and 254 separate held-out scenarios for human-reviewed test sets. Tool calls are executed and their outcomes recorded to help determine whether they comply with the policy. Every violating case uses a permitted tool, requiring checks of arguments, resources, and context rather than tool identity alone. The case construction process supports iterative guard training: an attacker generates requests using retrieved strategies, and the guard is fine-tuned on balanced, environment-verified examples. Training improves policy enforcement even with a fixed archive. Updating the archive further reduces in-domain attack success compared with keeping it fixed. We compare policy-aware task agent self-regulation with a trained policy guard. On PolicyGym, the final guard-based configuration achieves a 40-44% relative reduction in attack success rate and fewer benign rejections than policy prompting alone.

Chao Shang, Wanyu Du, Yi Zhang +1

AWS AI Labs

Tool Access Control for LLM Agents LLM Guardrails AI Agent Security Benchmarks Runtime Enforcement for AI Agents

Ask PDF

Sep 8, 2026

No Free Checker: A Survey of Verifiers for Robot Policies

A verifier for robot policies reads a candidate behavior and returns a score for how well it did, used both to evaluate vision-language-action policies and to train them. Verifiers range from success detectors and reward models to runtime monitors, safety filters, and temporal-logic specifications. We survey roughly 150 verifiers and compare them along two properties. Availability is how much a verdict costs, how early in a rollout the verdict arrives, and how often a verdict can be asked for. Availability rises as verdicts get cheaper, earlier, and denser. Credibility is how much a high score tells us about the task. Credibility falls as the judgment becomes gameable and self-serving. We group the verifiers by who supplies the judgment: human verifiers, rule-based and formal verifiers, learned and pretrained verifiers, and model-intrinsic verifiers. Across the four families, we find that credibility falls as availability rises. Regardless of who supplies the judgment, there is no free checker. We then examine what validates a verifier itself, and how much a high score tells us. Three measures appear in the literature: agreement with human labels, the performance of the policy it trains, and behavior under reward hacking. We close with nine metrics that make a verifier claim checkable, and coordinates for the verifiers still to be built.

Yang Wan, Xihang Yue, Zhirui Liu +7

Zhejiang University · City University of Hong Kong

Robotic Policy Evaluation Formal Verification Runtime Enforcement for AI Agents

Ask PDF

Jun 28, 2026

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents

LLM agents handle user requests on behalf of organizations through tool calls and must follow the company policies stated in their system prompts. Prior work approaches this as a safeguarding problem -- external checks that block non-compliant agent actions. We argue that policy adherence is a broader problem: real workflows unfold across many turns, require explicit user confirmation and prerequisite reads, and hinge on the content of the dialogue rather than on any single argument value. Meeting this bar requires (i) full conversation context, (ii) self-reasoning over the policy and the current dialogue, and (iii) conversation-specific remediation that guides the agent's next turn -- three capabilities that prior safeguard work has often underestimated. We introduce POLICYGUARD, a sub-agent verifier that shares the agent's view of the dialogue, reasons over the policy in context, and provides actionable feedback for the agent's next turn. On tau^2-BENCH airline across three vendors (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro) with four trials per setting, POLICYGUARD improves PASS4 by +12.0 / +6.0 / +12.0 pp. Per-call analyses show POLICYGUARD achieves higher policy-violation recall while blocking roughly half as often as argument-level guards.

Seongjae Kang, Taehyung Yu, Sung Ju Hwang

KAIST

Tool-Augmented Language Model Agents LLM Guardrails LLM Agent Evaluation LLM Agent Verification LLM Agent Safety

Ask PDF

Abstract

AI Agent Safety

Explore similar work

AgentPolicyForge: From Executable Environments to Policy Benchmarks and Adaptive Guards

No Free Checker: A Survey of Verifiers for Robot Policies

PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents