Measuring the Prevalence of Policy Violating Content with ML Assisted Sampling and LLM Labeling

Attila Dobi; Aravindh Manickavasagam; Benjamin Thompson; Xiaohan Yang; Faisal Farooq

cs.LGFeb 19, 2026

Measuring the Prevalence of Policy Violating Content with ML Assisted Sampling and LLM Labeling

Authors: Attila Dobi, Aravindh Manickavasagam, Benjamin Thompson, Xiaohan Yang, Faisal Farooq

Organizations: Pinterest, San Francisco, CA, USA · Pinterest, Richmond, VA, USA · Pinterest, Los Angeles, CA, USA · Pinterest, Murphy, TX, USA

Abstract

Content safety teams need metrics that reflect what users actually experience, not only what is reported. We study prevalence: the fraction of user views (impressions) that went to content violating a given policy on a given day. Accurate prevalence measurement is challenging because violations are often rare and human labeling is costly, making frequent, platform-representative studies slow. We present a design-based measurement system that (i) draws daily probability samples from the impression stream using ML-assisted weights to concentrate label budget on high-exposure and high-risk content while preserving unbiasedness, (ii) labels sampled items with a multimodal LLM governed by policy prompts and gold-set validation, and (iii) produces design-consistent prevalence estimates with confidence intervals and dashboard drilldowns. A key design goal is one global sample with many pivots: the same daily sample supports prevalence by surface, viewer geography, content age, and other segments through post-stratified estimation. We describe the statistical estimators, variance and confidence interval construction, label-quality monitoring, and an engineering workflow that makes the system configurable across policies.

Figures & tables

Figure 1. Illustrative prevalence trend (indexed to baseline

=100

) with intervention markers (model launch, policy update). Production dashboards add 95% CIs, absolute levels, and launch history. A line chart of indexed prevalence over twelve months with dashed vertical intervention markers labeled model launch and policy update; the line decreases after each intervention.

Symbol	Meaning
$j$	content unit (e.g., a Pin)
$C_{j}$ , $C_{jg}$	impressions of $j$ (total; in segment $g$ )
$Y_{j}\in\{0,1\}$	policy-violation label of $j$
$s_{j}$	auxiliary risk score (sampling only, never a label)
$w_{j}$ , $p_{j}$	sampling weight; normalized draw probability
$\nu,\gamma,\epsilon$	weight hyperparameters, Eq. ( 3 )

Table 1. Notation (all quantities per day; day subscript suppressed through § 6 ).

$\nu$	$\gamma$	Interpretation
0	0	Uniform random over content units
1	0	PPS by impressions (exposure-weighted)
1	1	ML-assisted: impressions $\times$ model score
$\in(0,1)$	0	Softer exposure weighting (variance control)

Table 2. Sampling regimes via parameter choices in ( 3 ). The floor

\epsilon>0

keeps every in-frame unit (including unscored, imputed content) at nonzero probability.

Figure 2. Example LLM outputs: structured label, rationale, and confidence. (Content blurred.) Three example images with JSON-like LLM outputs marking safe or unsafe with brief rationales.

Figure 3. Prevalence workflow: impression logs and auxiliary safety scores drive sampling; an LLM labeler produces policy labels; the estimator generates prevalence and CIs; results and lineage are stored for dashboards and audits. A block diagram showing impression logs and ML scores feeding sampling logic; sampled items go to an LLM labeler; labels feed an estimator; results are stored and shown in a dashboard, with a human validation loop.

Metric	Human Review	LLM
Label Throughput	Baseline ( $1\times$ )	$\sim 100\times$
Measurement Frequency	Monthly/Quarterly	Daily
Labeling Latency	Days to Weeks	$<$ 24 Hours
Cost per 1k labels	Baseline	$\sim 95\%$ reduction

Table 3. Relative Operational Comparison: Human vs. LLM

Figure 4. Simulation POC: empirical 95% CI width vs. sample size

m

for two sampling schemes: PPS (

w=\text{impressions}

) and ML-PPS (

w=\text{impression}\times\text{model\_score}

)

Policy area	Positive-rate lift (ML-PPS : PPS)
A	6.0 $\times$
B	11.4 $\times$
C	10.8 $\times$

Table 4. Production: lift in sample positive fraction from ML-assisted sampling (ML-PPS) relative to impression-only PPS at fixed label budget. Values are ratios (higher is better).

Policy area	Daily labels $m$	MDE (relative to mean)
A	50k	6.0%
B	100k	6.5%
C	10k	5.0%

Table 5. Typical minimum detectable effects (MDE), relative to each policy area’s mean. MDEs are computed for a two-sided test with

\alpha=0.05

and 80% power. Policy areas A, B, and C differ in production prevalence and ML model quality. We omit absolute prevalence and quality values. Representative configurations; actual production parameters may vary.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Population size $N$	$300{,}000$ items
Base positive rate $p$	$0.005$ (0.5%)
Impressions mixture mass $p_{\text{small}}$	$0.93$
Impressions small component	$\mathrm{Unif}\{1,\dots,10\}$
Impressions tail component	$x_{m}(1+\mathrm{Pareto}(\alpha))$ , $\alpha=1.4$ , $x_{m}=10$
Scores $s\mid y=0$	$\mathrm{Beta}(1.5,6.0)$

Appendix

Table 6. Synthetic simulation parameters used to generate Figure 4 . All parameters are hypothetical and not fit to any production dataset.

Policy area A							Policy area B
Labeler	$\Delta$ Prec. $\uparrow$	$\Delta$ Rec. $\uparrow$	$\Delta$ F1 $\uparrow$	$\Delta$ FPR $\downarrow$	$\Delta$ FNR $\downarrow$	$\Delta$ Inf. $\uparrow$	Labeler	$\Delta$ Prec. $\uparrow$	$\Delta$ Rec. $\uparrow$	$\Delta$ F1 $\uparrow$	$\Delta$ FPR $\downarrow$	$\Delta$ FNR $\downarrow$	$\Delta$ Inf. $\uparrow$
1 $\times$ BPO (baseline)	0.0	0.0	0.0	0.0	0.0	0.0	1 $\times$ BPO (baseline)	0.0	0.0	0.0	0.0	0.0	0.0
3 $\times$ BPO majority	$+3.4$	$+3.3$	$+3.4$	$-1.9$	$-3.3$	$+5.1$	3 $\times$ BPO majority	$+2.6$	$+2.8$	$+2.7$	$-1.3$	$-2.8$	$+4.2$
GPT-5 (baseline prompt)	$-8.6$	$-8.4$	$-8.5$	$+4.5$	$+8.4$	$-12.9$
GPT-5.4 (high)	$+1.7$	$-8.0$	$-3.8$	$-2.7$	$+8.0$	$-5.3$	GPT-5.4 (high)	$-0.2$	$-7.5$	$-4.5$	$-0.6$	$+7.5$	$-6.9$
GPT-5.4 (full context)	$-2.3$	$+2.0$	$+0.2$	$+2.3$	$-2.0$	$-0.3$

Appendix

Table 7. Decision-quality deltas (percentage points) versus a

1\times

single non-expert human review (BPO) baseline per policy area, scored against SME-adjudicated gold sets. Column arrows mark the better direction: higher recall is better, lower FPR is better. “(high)” denotes the high-reasoning setting; the “full context” configuration supplies the complete policy document as labeling context. Policy area A: gold set after one round of SME re-adjudication; prompts optimized to minimize FPR. Policy area B: production prompt prior to SME review.

3\times

majority rows aggregate three independent human reviews or three independent LLM calls. Informedness

=

recall

-

FPR.

Figure 5. Relative bias of LLM-measured prevalence

\theta^{(L)}

versus the true base rate

\theta

; color encodes the labeler false positive rate

f

, line style its recall

r

(solid

r=0.90

, dashed

r=0.50

). At low prevalence the metric is extremely sensitive to FPR, which enters unbounded as

f/\theta

: at

\theta=0.1\%

, even

f=0.1\%

makes about half of measured positives false detections (

+90\%

relative at

r=0.90

). Recall loss enters only as a bounded offset: each dashed curve runs parallel to its solid counterpart, shifted down by a constant

40

points. The solid horizontal line at zero marks the Rogan–Gladen corrected estimator with known

(r,f)

. Line chart of relative bias in percent versus true prevalence on a log axis for five error profiles; false-positive-rate driven bias grows sharply as prevalence decreases, dashed low-recall curves run parallel to their solid counterparts at a constant offset, and a solid zero line marks the corrected estimator.

Figure 6. Sensitivity to the score exponent

\gamma

at fixed budget

m=10{,}000

on the Appendix A population. Left: empirical 95% CI width relative to

\gamma=1

. Right: Kish

\mathrm{ESS}/m

and sample positive fraction. Aggressive tilting (

\gamma=4

) collapses ESS and widens the CI despite many positives. Two small panels. Left panel shows CI width versus gamma forming a U shape with minimum near gamma between one and two. Right panel shows effective sample size over m decreasing with gamma while the positive fraction increases.

Drift	Rel. bias	CI width (rel.)	$\mathrm{ESS}/m$	Pos. frac.
None	$-0.2\%$	$1.00$	$0.43$	$1.9\%$
Moderate	$+0.0\%$	$1.27$	$0.43$	$1.2\%$
Severe	$+0.2\%$	$1.63$	$0.49$	$0.8\%$

Appendix

Table 8. Score drift experiment (

\gamma=1

m=10{,}000

T=500

): empirical relative bias, CI width relative to the no-drift condition, Kish

\mathrm{ESS}/m

, and mean sample positive fraction.

Content Moderation

Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.

See all 87 papers

Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments
Feb 18, 2026Zehao Xu, Tony Paek, Kevin O'Sullivan +1A/b Testing
Pinterest, Inc., Toronto, ON, Canada · Pinterest, Inc., New York, NY, USA · Pinterest, Inc., San Francisco, CA, USA
Social Pressure Breaks Majority Voting in LLM Safety Panels
Aug 5, 2026Yibo Hu, Jiaming QuNormative Conformity in Language Models Safety Filtering
Illinois Institute of Technology · Amazon
Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation
May 22, 2026Konstantin Berlin, Adam SwandaAnnotator Disagreement Language Model Safety Evaluation
Cisco AI Defense

Feb 18, 2026J/K move · Enter open · S save

Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments

Zehao Xu, Tony Paek, Kevin O'Sullivan, Attila Dobi

Pinterest, Inc., Toronto, ON, Canada · Pinterest, Inc., New York, NY, USA · Pinterest, Inc., San Francisco, CA, USA

A/b Testing

Open paper AskPDF arXiv

Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate provides daily per-arm prevalence on over 20

\times

as many concurrent arms under the same labeling budget. Across roughly 300 production audits, the surrogate's 95% confidence interval contains the LLM-based reference point estimate in 92% of evaluations, and day-level delta aggregation recovers 2--5% relative shifts that no single per-arm LLM measurement can detect.

Abstract

Content Moderation

Explore similar work

Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments

Social Pressure Breaks Majority Voting in LLM Safety Panels

Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation