Open-Domain Safety Policy Construction

Di Wu; Siyue Liu; Zixiang Ji; Ya-Liang Chang; Zhe-Yu Liu; Andrew Pleffer; Kai-Wei Chang

cs.CLApr 1, 2026

Open-Domain Safety Policy Construction

Authors: Di Wu, Siyue Liu, Zixiang Ji, Ya-Liang Chang, Zhe-Yu Liu, Andrew Pleffer, Kai-Wei Chang

Organizations: University of California, Los Angeles · Taboola

Abstract

Moderation layers are increasingly a core component of many products built on user- or model-generated content. However, drafting and maintaining domain-specific safety policies remains costly. We present Deep Policy Research (DPR), a minimal agentic system that drafts a full content moderation policy based on only human-written seed domain information. DPR uses a single web search tool and lightweight scaffolding to iteratively propose search queries, distill diverse web sources into policy rules, and organize rules into an indexed document. We evaluate DPR on (1) the OpenAI undesired content benchmark across five domains with two compact reader LLMs and (2) an in-house multimodal advertisement moderation benchmark. DPR consistently outperforms definition-only and in-context learning baselines, and in our end-to-end setting it is competitive with expert-written policy sections in several domains. Moreover, under the same seed specification and evaluation protocol, DPR outperforms a general-purpose deep research system, suggesting that a task-specific, structured research loop can be more effective than generic web research for policy drafting. We release our experiment code at https://github.com/xiaowu0162/deep-policy-research.

Figures & tables

Figure 1: An illustration of Deep Policy Research. Based on a domain specification, an LLM iteratively interacts with a search engine, extracts policy rules, and indexes the rules through keyphrase-based clustering.

	Sexual	Hate	Violence	Harassment	Self-Harm	Average
Llama 3.1 8B Instruct
Seed Information	0.916	0.757	0.658	0.640	0.788	0.752
In-Context Learning	0.923	0.728	0.582	0.610	0.691	0.707
OAI DR	0.828	0.800	0.738	0.683	0.829	0.776
DPR	0.910	0.813	0.717	0.683	0.835	0.792
Qwen2.5 7B Instruct

Table 1: Evaluation results on OpenAI Content Moderation. DPR improves content moderation F1, outperforming both only using seed information or performing in-context learning with human-written examples.

	Misrepresentative		Finance Claims		Exploitative		Offensive
	S.S.	M.V.	S.S.	M.V.	S.S.	M.V.	S.S.	M.V.
No Policy	0.714	0.727	0.500	0.509	0.793	0.782	0.714	0.786
Seed Information	0.701	0.701	0.553	0.544	0.839	0.885	0.679	0.786
Human Policy	0.740	0.779	0.833	0.877	0.920	0.908	0.893	0.964
DPR	0.727	0.740	0.597	0.597	0.908	0.920	0.786	0.821
DPR + Summary	0.701	0.779	0.588	0.614	0.874	0.908	0.821	0.821

Table 2: Evaluation results on in-house multimodal moderation data. DPR improves content moderation F1, outperforming both no policy and only using seed information. DPR also performs on par with human-written policy in the Mispresentative and Exploitative domain. We provide further human evaluations in Appendix Appendix C .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset: OpenAI Online Content Moderation

Policy Domain: Sexual

Definition: Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness).

Policy Domain: Hate

Definition: Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste.

Policy Domain: Violence

Definition: Content that promotes or glorifies violence or celebrates the suffering or humiliation of others.

Appendix

Figure 2: Detailed specifications of the domains experimented in this paper. These prompts were created solely for the purposes of this article and are provided for illustrative use only. They do not reflect official Taboola policy, which may be updated or revised over time.

Panel of Figure 3 — Figure 3: Prompt for generating web search queries. The research agent uses it to identify missing coverage.

Panel of Figure 4 — Figure 4: Prompt for extracting rules from a webpage chunk. The research agent uses it to generate new candidate rules.

Panel of Figure 5 — Figure 5: Prompt for scoring rule relevance. The research agent uses it to filter candidate rules.

Panel of Figure 6 — Figure 6: Prompt for extracting a keyphrase for each rule. The research agent uses it during datastore indexing.

Panel of Figure 7 — Figure 7: Prompt for merging similar rules. The research agent uses it to consolidate extracted rules.

Panel of Figure 8 — Figure 8: Prompt for summarizing a section of rules. The research agent uses it to describe clustered sections.

Panel of Figure 9 — Figure 9: Prompt for titling a section of rules. The research agent uses it to label clustered sections.

Panel of Figure 10 — Figure 10: Prompt for merging section titles. The research agent uses it to reduce redundant titles.

Figure 11: Rule count and cluster count vs. iteration in the Harassment domain for DPR initialized with two LLMs.

Figure 12: Performance vs. iteration in the Harassment domain with two different LLMs for DPR and Qwen2.5 7B Instruct as the reader.

Figure 13: Performance vs. iteration in the Harassment domain with two different LLMs for DPR and Llama 3.1 8B Instruct as the reader.

Figure 14: Domain distribution of sources leveraged by DPR for OpenAI and the in-house dataset.

Domain	URL	Page Excerpt	DPR-Generated Rule
Sexual	The Daily Texan (2018)	It found that sex offenders used significantly fewer first-person singular pronouns, such as “I,” “me” and “my,” and more second-person singular pronouns, such as “you” and “your,” compared to the decoys.	Sensitive messages related to sexual content often use fewer first-person singular pronouns and more second-person singular pronouns.
Hate	Wikipedia: Online hate speech	Identity Tourism often leads to stereotyping, discrimination, and cultural appropriation.	Hate messages may involve identity tourism, where a person pretends to be a member of another group. This can lead to stereotyping and discrimination.
Violence	Mastering Cultural Differences	The use of violent language and phrases like “kill two birds with one stone,” and “bite the bullet,” may seem innocuous on the surface.	Sensitive messages related to violence may contain idiomatic phrases that appear innocuous but contribute to the desensitization of violence.
Harassment	PEN Field Manual Glossary	Dog Whistling: Using words or symbols with a double (or coded) meaning that is abusive or harmful, sometimes to signal a group of online abusers to attack a specific target.	Harassment messages may involve dog whistling, using coded words or symbols with a double meaning to signal other abusers to attack the target.
Self-Harm	Sen. Bennet (Press Release)	In one case, researchers prompted My AI to instruct a child how to cover up a bruise ahead of a visit from Child Protective Services.	Sensitive messages related to self-harm may include instructions on how to conceal physical harm or injuries.

Appendix

Table 3: Examples of DPR output on the OpenAI content moderation domains.

Domain	URL	Page Excerpt	DPR-Generated Rule
Misrepresentative	SingleGrain: Misleading statistics	If a user finds an anti-aging skincare moisturizer that touts “90% of users experienced fewer wrinkles,” this statistic will be enough for consumers to purchase the product. They won’t realize or even care that it’s a false advertising claim.	Advertisements may use high percentages or statistics to create a misleading narrative, even if the claim is based on biased or incomplete data.
Finance Claims	FMA: Misleading advertising (investments)	What does the advertisement say (or not say) about the risk of the investment? Words such as “secured”, “certain”, “stable”, “guaranteed” can give the impression that an investment is of low or no risk this can be misleading if the investment is not low risk.	Advertisements for financial products should not use words that imply low or no risk, such as “secured”, “certain”, “stable”, or “guaranteed”, if the investment is not low risk.
Exploitative	Issuu: Shock advertising examples	PETA’s ads often feature graphic imagery of animal cruelty, such as animals being slaughtered or abused in factory farms.	Exploitative advertisements may use graphic imagery of animal cruelty, such as animals being slaughtered or abused, to promote a message or product.
Offensive	Lunio: Controversial ads	Bumble released an ad campaign that sparked outrage online. As shown in the image, the billboard ad campaign included messages such as “A vow of celibacy is not the answer” and “Thou shalt not give up on dating and become a nun”.	Using religious references or imagery in a way that is perceived as mocking or disrespectful can be considered offensive.

Appendix

Table 4: Examples of DPR output on in-house advertisement policy domains.

Human-written Rules and Machine-generated Rules
Policy Domain: Exploitative
Human-written rules.
5.1. Must not use images of catastrophic scenes featuring violence or disaster.
Sub-rules (5.1):
(a) Objects associated with death are permitted (e.g., Coffin, Urn, Tombstone).
(b) Funerals, mourning and scenes associated with death are permitted.

Appendix

Figure 15: Visualization of human-written rules and aligned DPR-generated rules for the Exploitative domain. These prompts were created solely for the purposes of this article and are provided for illustrative use only. They do not reflect official Taboola policy, which may be updated or revised over time.

Human-written Rules and Machine-generated Rules
Policy Domain: Offensive
Human-written rules.
6.1. Must not use images of a person with a visible exposed injury (e.g., on skin).
Sub-rules (6.1):
(a) Images of bandaged or cast body parts are permitted.
6.2. Must not explicitly portray in images the inside of a human mouth.

Appendix

Figure 16: Visualization of human-written rules and aligned DPR-generated rules for the Offensive domain. These prompts were created solely for the purposes of this article and are provided for illustrative use only. They do not reflect official Taboola policy, which may be updated or revised over time.

Domain	Clarity		Relevance		Internal Usability
Domain	counts	mean $\pm$ std	counts	mean $\pm$ std	counts	mean $\pm$ std
Exploitative	23/36/92	1.4570 $\pm$ 0.7460	20/49/83	1.4145 $\pm$ 0.7136	20/26/106	1.5658 $\pm$ 0.7157
Finance	42/116/331	1.5910 $\pm$ 0.6437	43/77/369	1.6667 $\pm$ 0.6316	46/81/360	1.6865 $\pm$ 1.1256
Misrepresentative	47/47/59	1.0784 $\pm$ 0.8314	46/56/51	1.0327 $\pm$ 0.7982	53/51/49	0.9739 $\pm$ 0.8188
Offensive	46/56/129	1.3593 $\pm$ 0.7945	38/78/115	1.3333 $\pm$ 0.7441	56/55/120	1.2771 $\pm$ 0.8295

Appendix

Table 5: Human evaluation on in-house test domains: counts of ratings (0/1/2) and mean

\pm

std for each metric.

Domain	Clarity		Relevance
Domain	counts	mean $\pm$ std	counts	mean $\pm$ std
Harassment	7/28/172	1.7971 $\pm$ 0.4801	9/26/172	1.7874 $\pm$ 0.5055
Hate	25/35/231	1.7079 $\pm$ 0.6164	27/36/228	1.6907 $\pm$ 0.6329
Self-Harm	44/30/51	1.0560 $\pm$ 0.8735	37/37/51	1.1120 $\pm$ 0.8349
Sexual	14/16/45	1.4133 $\pm$ 0.7900	14/20/41	1.3600 $\pm$ 0.7822
Violence	17/38/71	1.4286 $\pm$ 0.7203	22/47/57	1.2778 $\pm$ 0.7445

Appendix

Table 6: Human evaluation on OpenAI domains: counts of ratings (0/1/2) and mean

\pm

std for each metric.

Domain	Clarity			Relevance			Internal Usability
Domain	$\kappa$	$\bar{P}$	$P_{e}$	$\kappa$	$\bar{P}$	$P_{e}$	$\kappa$	$\bar{P}$	$P_{e}$
Exploitative	0.6069	0.7843	0.4513	0.4146	0.6601	0.4194	0.5802	0.8039	0.5329
Finance	0.0377	0.5399	0.5218	-0.1354	0.5481	0.6020	-0.1280	0.5297	0.5830
Misrepresentative	0.7830	0.8562	0.3374	0.6853	0.7908	0.3355	0.8038	0.8693	0.3337
Offensive	0.7504	0.8528	0.4103	0.5679	0.7359	0.3889	0.8098	0.8831	0.3853

Appendix

Table 7: Inter-annotator agreement on in-house test domains per metric (Fleiss’

\kappa

), observed agreement

\bar{P}

, and chance agreement

P_{e}

Domain	Clarity			Relevance
Domain	$\kappa$	$\bar{P}$	$P_{e}$	$\kappa$	$\bar{P}$	$P_{e}$
Harassment	0.6337	0.8937	0.7099	0.7021	0.9130	0.7081
Hate	0.6840	0.8900	0.6520	0.7344	0.9038	0.6378
Self-Harm	0.4888	0.6667	0.3480	0.4454	0.6349	0.3417
Sexual	0.7618	0.8667	0.4404	0.5072	0.7067	0.4048
Violence	0.4186	0.6667	0.4267	0.2643	0.5397	0.3743

Appendix

Table 8: Inter-annotator agreement on OpenAI test domains per metric (Fleiss’

\kappa

), observed agreement

\bar{P}

, and chance agreement

P_{e}

Content Moderation

Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.

See all 87 papers

Deep Research as Rubric for Reinforcement Learning
May 31, 2026Wangyi Mei, Zhouhong Gu, Zhenhan Bai +9Reward Design for RL Open-Ended Generation
Fudan University · Xiaohongshu Inc. · Beijing University of Posts and Telecommunications
Who Decides What Is Harmful? Content Moderation Policy Through A Multi-Agent Personalised Inference Framework
May 2, 2026Ewelina Gajewska, Michal Wawer, Katarzyna Budzynska +1LLM Guardrails Content Moderation
Warsaw University of Technology, Warsaw, Poland
Nemotron 3.5 Content Safety Moderator: A Compact Multimodal, Multilingual, and Reasoning Enabled Content Safety Moderator
Aug 27, 2026Varun Singh, Anuj Doshi, Makesh Narsimhan Sreedhar +2Multimodal Content Moderation LLM Guardrails
NVIDIA Santa Clara, CA

May 31, 2026J/K move · Enter open · S save

Deep Research as Rubric for Reinforcement Learning

Wangyi Mei, Zhouhong Gu, Zhenhan Bai, Yin Cai+8

Fudan University · Xiaohongshu Inc. · Beijing University of Posts and Telecommunications

Reward Design for RL Open-Ended Generation Reinforcement Learning RL for Language Model Reasoning Rubric-Based RL Reinforcement Learning with Verifiable Rewards+2

Open paper AskPDF arXiv

Open-ended reasoning and long-form generation tasks lack reliable automatic verification signals for reward-based policy optimization. Rubrics offer a promising alternative, but existing approaches treat them as given artifacts -- either hand-crafted or prompt-generated -- and often miss the task-specific, knowledge-intensive dimensions that matter most, distorting the reward signal. Our key observation is that rubric construction is itself a research problem: identifying what makes a response correct or insightful requires discovering and synthesizing external knowledge. We propose Deep Research as Rubric (DR-rubric), a two-stage framework for constructing such rubrics. Stage I elicits domain facts, structural constraints, and failure modes through iterative multi-turn agentic search; Stage II distills this evidence into atomic, independently verifiable constraints for GRPO-based policy optimization. Because the model under training can serve as its own rubric generator, DR-rubric-8B supports bootstrap rubric generation without frontier-model assistance. We evaluate on 6 benchmarks spanning agentic research and expert reasoning. Experiments show that DR-Rubric achieves strong competitive performance with only 1K -- 3K training instances, where GPT-5-generated rubrics particularly benefit breadth coverage on agentic tasks, Gemini-generated rubrics yield the most balanced performance across agentic and expert reasoning tasks, and bootstrap rubrics exhibit a specialization-to-rebalancing evolution achieving the best overall performance at the third iteration. Results demonstrate that reframing rubric construction from static evaluation templates into an evidence-driven research process yields more scalable, fine-grained reward signals for open-ended tasks.

Abstract

Content Moderation

Explore similar work

Deep Research as Rubric for Reinforcement Learning

Who Decides What Is Harmful? Content Moderation Policy Through A Multi-Agent Personalised Inference Framework

Nemotron 3.5 Content Safety Moderator: A Compact Multimodal, Multilingual, and Reasoning Enabled Content Safety Moderator