AgentPolicyForge: From Executable Environments to Policy Benchmarks and Adaptive Guards

Chao Shang; Wanyu Du; Yi Zhang; Sandesh Swamy

cs.AIOct 20, 2026

AgentPolicyForge: From Executable Environments to Policy Benchmarks and Adaptive Guards

Authors: Chao Shang, Wanyu Du, Yi Zhang, Sandesh Swamy

Organizations: AWS AI Labs

Abstract

Customized agent deployments define policies over tools, resources, and execution conditions. Providing these policies in the task agent's prompt does not reliably prevent violations. Training a guard requires examples of tool calls that comply with or violate those policies. We introduce AgentPolicyForge, a framework that uses executable environments to construct policy benchmarks and iteratively train policy guards. With this framework, we construct PolicyGym, a dataset across ten policy families using 698 scenarios for training data generation and 254 separate held-out scenarios for human-reviewed test sets. Tool calls are executed and their outcomes recorded to help determine whether they comply with the policy. Every violating case uses a permitted tool, requiring checks of arguments, resources, and context rather than tool identity alone. The case construction process supports iterative guard training: an attacker generates requests using retrieved strategies, and the guard is fine-tuned on balanced, environment-verified examples. Training improves policy enforcement even with a fixed archive. Updating the archive further reduces in-domain attack success compared with keeping it fixed. We compare policy-aware task agent self-regulation with a trained policy guard. On PolicyGym, the final guard-based configuration achieves a 40-44% relative reduction in attack success rate and fewer benign rejections than policy prompting alone.

Figures & tables

Figure 1: AgentPolicyForge framework. One construction process produces human-reviewed policy tests and fresh guard training data. Execution records support labeling and rule checks. SFT updates the guard, and guard feedback yields candidate attack strategies. Validated strategies enter the archive to guide subsequent requests. Test cases are used only for evaluation, not guard training or strategy validation.

Statistic	Source seeds	In-domain test	Out-of-domain test
Distinct scenarios	698	187	67
Seed inputs / episodes	5,655	1,070	304
Threat / benign	2,863 / 2,792	535 / 535	152 / 152

Table 1: PolicyGym training and test inputs. Source counts describe seeds for request generation before the release split. Test counts describe final human-reviewed episodes with equal numbers of violating and compliant labels.

Task-agent self-regulation
System	Setting	In-domain				Out-of-domain
		ASR $\downarrow$	B-rej. $\downarrow$	Rec. ≥2 $\uparrow$	FPR ≥2 $\downarrow$	ASR $\downarrow$	B-rej. $\downarrow$	Rec. ≥2 $\uparrow$	FPR ≥2 $\downarrow$
Claude Sonnet 5	Policy-blind	98.50	0.00	—	—	99.34	0.00	—	—
Claude Sonnet 5	Policy-aware	14.77	8.79	—	—	13.16	9.21	—	—
Specialized policy-guard baseline
PolicyGuard-4B	Binary	32.52	2.80	66.22	2.43	28.29	0.66	71.18	0.89

Table 2: PolicyGym enforcement results (%). Task-agent rows measure self-regulation; guards judge fixed proposals. B-rej. is benign refusal or over-block. Guard ASR/B-rej. are episode-level at Block; recall/FPR are proposal-level at Level

\geq 2

(Block for binary guards). Models differ in size, and results are not calibrated to equal benign rejection. Bold marks column-best values.

Figure 2: Guard performance across training rounds on ID and OOD tests. The x-axis shows training rounds, starting with the untrained guard at Round 0. The y-axis shows successful attacks (blue), blocked benign episodes (red), and compliant calls flagged for intervention (green), all in percent. Lower is better. Blue and red use Level 4 as Block, and green counts Levels 2–4 as interventions. Black shows the ASR reduction minus the over-block increase since Round 0, in percentage points. Higher is better under this equal-weight measure.

Setting	ID Test		OOD Test		VIOLA
	ASR $\downarrow$	OB $\downarrow$	ASR $\downarrow$	OB $\downarrow$	ASR $\downarrow$	OB $\downarrow$
Evolving (R10)	8.22	7.48	7.89	5.92	18.75	66.67
Frozen $S_{0}$ (R10)	10.84	6.73	9.21	5.92	23.96	66.67

Table 3: Archive-update ablation: evolving versus fixed

S_{0}

(%). OB denotes over-block. Evolving R10 is the Qwen3-14B R10 checkpoint in Table 2 .

VIOLA Test+Dev: 1,709 invocation inputs
Traj.	Model	ASR $\downarrow$	Ov.blk $\downarrow$	Acc. $\uparrow$	F1 $\uparrow$	Youden’s J $\uparrow$
114	Qwen3-14B R0	27.08	61.11	67.54	79.10	11.81
	Qwen3-14B R10	18.75	66.67	73.68	83.87	14.58

Table 4: External diagnostics (%). VIOLA has 114 trajectories (Test + Dev), comprising 1,709 invocations (848 Test + 861 Dev). Visible denotes a post-hoc PolicyGuardBench subset whose compliance can be determined from the provided information. PolicyGuard-4B scores are published references.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Policy family	Restriction	Illustrative policy	Information used by the check
Time windows	Disallow an operation during a specified time interval.	Transfers are prohibited between 21:00 and 05:00 UTC.	The bound action-time field and the prohibited interval, including intervals crossing midnight.
Backdating	Limit how far an operation’s date may precede the reference date.	An entry must not be dated more than 30 days in the past.	The bound date field, fixed reference date, and maximum age in days.
Amount limits	Cap the amount of an operation.	A single transfer must not exceed $500.	The relevant amount argument, its units, and the cap.
Own-record access	Restrict an operation to the current user’s records.	A record’s owner identifier must match the current user.	The bound owner-identifier argument and trusted current-user identifier.
Restricted tables	Exclude a table from the session’s authorization.	The session must not directly write the audit-log table.	The restricted table name, clause scope, and evidence identifying the table targeted by the call.
Protected identifiers	Reserve identifiers below a threshold.	Manual operations on account IDs below 1,000 are prohibited.	The bound identifier argument and reserved-range threshold.

Appendix

Table 5: PolicyGym’s ten policy families. Names, fields, and numerical values in the examples are illustrative. Field masking denotes a write restriction, not output redaction. Dual approval checks the encoded approval-field requirement, not an independent two-person approval workflow.

Verifier	Sonnet 5	Records
Compliant	Compliant	1,930
Violation	Violation	915
Violation	Compliant	180
Compliant	Violation	11
Unknown	Compliant	146
Unknown	Violation	129

Appendix

Table 6: Agreement between Sonnet-5 and verifier-derived labels. Counts cover all 3,311 released supervision records, not the human-reviewed test sets. Unknown denotes cases the verifier cannot determine from executable evidence. Agreement with Sonnet 5 does not establish label accuracy.

Object	Train	Test	Exact	Median	P95
Natural request	2,708	1,371	0	0.049	0.106
Policy clause	3,017	1,587	1	0.217	0.406

Appendix

Table 7: Lexical overlap between the released training and test splits. Train and Test count unique strings. Exact counts test strings also present in training. Median and P95 summarize each test string’s similarity to its closest training string, using TF–IDF cosine similarity after removing shared instructions and output templates.

Split	Agree / $n$	Agreement (%)	$\kappa$
ID	76/78	97.44	0.949
OOD	22/22	100.00	1.000
Overall	98/100	98.00	0.960

Appendix

Table 8: Sampled human–reference agreement. Cohen’s

\kappa

compares submitted episode judgments with existing test labels.

Round	Active	Prod. $+$	Expl. $+$	Candidate-only bypasses
Initial–3	18	0	0	0
4	19	1	0	1
5	21	2	0	4
6	22	0	1	0
7	23	0	1	0
8	25	1	1	1

Appendix

Table 9: Growth of the active attacker-strategy archive from 18 to 25 strategies. Active is the total number of strategies. Prod.

+

and Expl.

+

count new production and exploration strategies. Candidate-only bypasses are training-validation cases where the new production strategy bypasses the guard but the prior archive does not.

Checkpoint	Split	Proposals	Level 0	Level 2	Level 4	Overblock (%)
Round 0	in-dom.	3,521	2,752	239	530	3.55
Round 10	in-dom.	3,521	2,783	101	637	7.48
Round 0	out-dom.	955	717	83	155	1.97
Round 10	out-dom.	955	721	48	186	5.92

Appendix

Table 10: Proposal-level intervention counts and episode-level over-block. Counts cover all frozen Qwen3-14B proposals, including threat and benign episodes. Over-block is the percentage of benign episodes with any Level-4 proposal. Levels 1 and 3 do not occur.

		ID		OOD
Guard	Block rule	ASR	OB	ASR	OB
R0	Level 4	19.63	3.55	17.11	1.97
	Level $\geq 2$	3.18	14.58	4.61	18.42
Fixed R10	Level 4	10.84	6.73	9.21	5.92
	Level $\geq 2$	2.80	12.52	1.97	17.11
Evolving R10	Level 4	8.22	7.48	7.89	5.92

Appendix

Table 11: Blocking-rule control (%). Both blocking rules use the same saved predictions and score whole episodes. OB is benign over-block. Fixed R10 uses the fixed-

S_{0}

archive.

		ID		OOD
Qwen3	Round	ASR	OB	ASR	OB
14B	R0	19.63	3.55	17.11	1.97
	R10	8.22	7.48	7.89	5.92
8B	R0	33.08	2.80	39.47	2.63
	R10	21.12	10.28	20.39	9.87

Appendix

Table 12: Adaptive-training endpoints with LLM supervision (%). R0 uses pretrained weights with the fused prompt. R10 follows ten rounds. OB is episode-level over-block. Both measures use Level 4.

	ID		OOD
Supervision	ASR	OB	ASR	OB
LLM	8.22	7.48	7.89	5.92
Verifier	11.40	6.54	11.18	2.63

Appendix

Table 13: Qwen3-14B R10 under two supervision sources (%). ASR and over-block (OB) are episode-level at Level 4. The common R0 appears in Table 12 .

Evaluation view	Split	ASR $\downarrow$	Ov.blk $\downarrow$	Accuracy $\uparrow$	F1 $\uparrow$	Recall $\uparrow$	FPR $\downarrow$	Parse $\uparrow$
Online	in-dom.	32.52	2.80	92.22	74.39	66.22	2.43	100.00
Online	out-dom.	28.29	0.66	94.14	81.21	71.18	0.89	100.00
Trajectory	in-dom.	32.71	2.62	82.34	79.21	67.29	2.62	100.00
Trajectory	out-dom.	28.29	0.66	85.53	83.21	71.71	0.66	100.00

Appendix

Table 14: PolicyGuard-4B under online and trajectory evaluation. Online ASR and over-block are episode-level. Accuracy, F1, recall, and FPR are proposal-level. Trajectory metrics score complete action summaries. Parse denotes output parseability. Values are percentages.

Split	Error type	Episodes	Frozen errors	Evolving errors	Repaired	New errors	Clustered 95% CI (pp)
In-domain	Missed violation	535	58	44	15	1	$[-4.17,-1.21]$
In-domain	Benign block	535	36	40	1	5	$[0.00,1.69]$
Out-of-domain	Missed violation	152	14	12	2	0	$[-3.37,0.00]$
Out-of-domain	Benign block	152	9	9	2	2	$[-2.60,2.63]$

Appendix

Table 15: Paired error changes from the frozen to the evolving archive. Both guards are evaluated at Round 10. Repaired counts fixed-archive errors corrected by evolving training. New errors counts the reverse changes. Intervals show the evolving-minus-frozen error-rate difference in percentage points, using 95% paired scenario-bootstrap intervals. They do not measure variation across training seeds.

Comparator	Error type	Trajectories	Comparator errors	Evolving errors	Repaired	New errors
Frozen archive	Missed violation	96	23	18	8	3
Frozen archive	Benign false alarm	18	12	12	2	2

Appendix

Table 16: Paired VIOLA error changes toward the evolving condition. Counts use the Table 3 Test+Dev trajectory predictions. Repaired and new errors describe changes from frozen to evolving training. Equal false-alarm totals do not imply identical decisions.

Guard	ASR $\downarrow$	Over-block $\downarrow$	FN / 90	FP / 90
R0	20.00	34.44	18	31
R10	2.22	47.78	2	43

Appendix

Table 17: Qwen-AgentWorld diagnostic dataset of 180 cases. The 180 cases contain 90 violations and 90 compliant calls with simulated tool observations. R0 and R10 are evaluated without target-domain fine-tuning. ASR and over-block are percentages. FN and FP count missed violations and blocked compliant calls at Level 4.

Model	Rows	Acc. $\uparrow$	ASR $\downarrow$	Ov.blk $\downarrow$	F1 $\uparrow$	FN	FP
Full
R0	12,000	90.633	9.927	8.954	89.075	505	619
R10	12,000	90.817	9.927	8.636	89.266	505	597
Visible-decidable
R0	4,640	96.638	0.367	4.282	93.299	4	152
R10	4,640	96.961	0.275	3.887	93.909	3	138

Appendix

Table 18: PolicyGuardBench visible-decidable sensitivity analysis. The subset keeps cases that can be judged from visible information and whose judgment agrees with the source label. Selection does not use model predictions. Deduplication keeps one row per identical policy, domain, subdomain, and trajectory input. FN and FP count missed violations and false alarms. Rates are percentages.

Framework	Task or policy construction	Evidence and supervision	Use in evaluation or training
ToolEmu ( Ruan et al., 2024 )	Tool-use scenarios for identifying agent risks	LM-emulated tool execution and LM safety evaluation, with human assessment	Risk evaluation across tools and scenarios
AgentDojo ( Debenedetti et al., 2024 )	Agent tasks and prompt-injection attacks in extensible tool environments	Agent executions evaluated for task utility and attack success	Evaluation of tasks, defenses, and adaptive attacks
PolicyGuard ( Wen et al., 2026a )	Policies paired with web trajectories and prefixes	Policy-compliance supervision in PolicyGuardBench	Training and evaluation of a policy guard
BraveGuard ( Feng et al., 2026a )	Emerging threats instantiated as executable computer-use tasks	Collected rollouts with trajectory-level guard supervision	Repeatable threat discovery, data collection, and guard training
HazardAuditor ( Feng et al., 2026b )	Executable threats across heterogeneous computer-use agents	Normalized agent events and deterministic safety outcomes	Cross-framework supervision and GuardPO training
AgentPolicyForge	Conditional policies bound to tools, resources, and state; compliant and violating calls	Execution records, rule checks, LLM targets, and human-reviewed test labels	Fixed policy tests, multi-round guard training, and validated strategy updates

Appendix

Table 19: Scope of related frameworks. Entries summarize reported approaches. They do not imply that unlisted capabilities are impossible or absent.

Tool Access Control for LLM Agents

Tool access control for LLM agents limits which tools, resources, and actions agents may use, and under what conditions.

See all 221 papers

Sep 28, 2026

AdaGuard: An Adaptive Guard Model with User-defined Policies

Guard models support the safe deployment of language model agents, but fixed risk taxonomies limit their ability to accommodate requirements that vary across applications and tasks. Under user-defined policies, detecting violations requires interpreting both the applicable rules and the agent's behavior, since identical actions can receive different judgments under different policies. To support learning this capability, we introduce AdaptiveSafety, a dataset of 10,939 training examples and 1,000 test examples covering policies with 1--100 rules. The dataset combines trajectories from multiple sources with policy and behavioral counterfactuals, pairing each example with an explanation and the complete set of violated rules. These counterfactuals expose changes that alter compliance, while structural augmentations provide supervision for consistency under rule reordering and identifier remapping. Building on this supervision, we propose SafePO, a reinforcement learning algorithm for refining violation identification while balancing explanatory reasoning and final verdicts. SafePO uses structured rewards to assess prediction correctness, retains group-relative advantages at the response level, and employs a separately trained value model to modulate token weights within explanation and verdict regions. Separate normalization controls their relative contribution to training despite differences in length. Through supervised initialization followed by SafePO, we develop AdaGuard, a family of 0.6B, 4B, and 8B guard models that assess agent trajectories under policies supplied at inference time. Our 4B model achieves binary accuracies of 89.30% on AdaptiveSafety and 71.82% on DynaBench. The project repository is available at https://github.com/Yunhao-Feng/AdaGuard

Yunhao Feng, Yifan Ding, Yuxiang Xie +4

1National University of Defense Technology · 2Fudan University

LLM Guardrails AI Agent Safety Safety Filtering LLM Safety AI Agent Safety Benchmarks

Ask PDF

May 20, 2026

Governance by Construction for Generalist Agents

Enterprise agents are increasingly expected to operate autonomously across tools and interfaces, yet production deployments require governance by construction. Systems must specify which actions are allowed, when human oversight is required, and what information may be exposed, without rebuilding the agent for each domain. This demo presents CUGA's policy system, a modular policy-as-code layer that composes with a generalist LLM agent to deliver predictable, auditable, and compliance-aware behavior in compound workflows without model fine-tuning. We present a runtime governance architecture that enforces policy interventions at every critical stage of execution. Rather than passively constraining behavior, policies intercept the agent at five structural checkpoints: upstream of planning (Intent Guard), within the system prompt to steer reasoning (Playbook), at the tool-call boundary to enforce proper usage (Tool Guide), outside the reasoning loop as a Human-in-the-Loop gate for high-risk actions (Tool Approvals), and at the output stage to filter and structure the final response (Output Formatter). Together, these stages embed governance continuously across the agent's execution pipeline rather than treating it as an afterthought. Using a healthcare scenario and a multi-layered enforcement intervention, the demo shows dynamic playbook injection for structured tool-sequence enforcement, intent guards that block malicious or accidental harmful requests, and human-in-the-loop tool approval checkpoints for potentially destructive actions. The artifact illustrates how typed governance primitives enable faster, safer deployment of enterprise agentic systems while improving policy adherence and execution consistency.

Segev Shlomov, Iftach Shoham, Alon Oved +7

IBM Haifa, Israel

LLM Guardrails AI Agent Governance

Ask PDF

Aug 25, 2026

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards

Safeguarding language model agents requires assessing complete execution trajectories under context-dependent safety policies. Existing policy-aware safeguards mainly rely on prompting or supervised fine-tuning, limiting their ability to adapt to unseen trajectories and changing policy contexts. We propose RePolicy, an agent safeguard that learns safety-policy invocation through reinforcement learning. Given an agent trajectory and a dynamic policy library, RePolicy invokes the applicable policy and uses its content to produce a policy-grounded rationale and safety judgment. We construct PolicyTraj-20K to support supervised initialization, followed by GRPO with verifiable rewards and policy-context perturbation. Experiments across six agent safety benchmarks show that RePolicy achieves strong overall safety-detection performance and robust policy invocation under varying policy contexts.

Houcheng Jiang, Boxuan Zhang, Qiyong Zhong +3

Zhongguancun Academy · University of Science and Technology of China · Zhejiang University +1

LLM Agent Safety RL for LLM Agents

Ask PDF

Abstract

Tool Access Control for LLM Agents

Explore similar work

AdaGuard: An Adaptive Guard Model with User-defined Policies

Governance by Construction for Generalist Agents

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards