Content safety teams need metrics that reflect what users actually experience, not only what is reported. We study prevalence: the fraction of user views (impressions) that went to content violating a given policy on a given day. Accurate prevalence measurement is challenging because violations are often rare and human labeling is costly, making frequent, platform-representative studies slow. We present a design-based measurement system that (i) draws daily probability samples from the impression stream using ML-assisted weights to concentrate label budget on high-exposure and high-risk content while preserving unbiasedness, (ii) labels sampled items with a multimodal LLM governed by policy prompts and gold-set validation, and (iii) produces design-consistent prevalence estimates with confidence intervals and dashboard drilldowns. A key design goal is one global sample with many pivots: the same daily sample supports prevalence by surface, viewer geography, content age, and other segments through post-stratified estimation. We describe the statistical estimators, variance and confidence interval construction, label-quality monitoring, and an engineering workflow that makes the system configurable across policies.
Figures & tables
Figure 1. Illustrative prevalence trend (indexed to baseline =100 ) with intervention markers (model launch, policy update). Production dashboards add 95% CIs, absolute levels, and launch history. A line chart of indexed prevalence over twelve months with dashed vertical intervention markers labeled model launch and policy update; the line decreases after each intervention.
Symbol
Meaning
j
content unit (e.g., a Pin)
Cj , Cjg
impressions of j (total; in segment g )
Yj∈{0,1}
policy-violation label of j
sj
auxiliary risk score (sampling only, never a label)
wj , pj
sampling weight; normalized draw probability
ν,γ,ϵ
weight hyperparameters, Eq. ( 3 )
Table 1. Notation (all quantities per day; day subscript suppressed through § 6 ).
ν
γ
Interpretation
0
0
Uniform random over content units
1
0
PPS by impressions (exposure-weighted)
1
1
ML-assisted: impressions × model score
∈(0,1)
0
Softer exposure weighting (variance control)
Table 2. Sampling regimes via parameter choices in ( 3 ). The floor ϵ>0 keeps every in-frame unit (including unscored, imputed content) at nonzero probability.
Figure 2. Example LLM outputs: structured label, rationale, and confidence. (Content blurred.) Three example images with JSON-like LLM outputs marking safe or unsafe with brief rationales.
Figure 3. Prevalence workflow: impression logs and auxiliary safety scores drive sampling; an LLM labeler produces policy labels; the estimator generates prevalence and CIs; results and lineage are stored for dashboards and audits. A block diagram showing impression logs and ML scores feeding sampling logic; sampled items go to an LLM labeler; labels feed an estimator; results are stored and shown in a dashboard, with a human validation loop.
Metric
Human Review
LLM
Label Throughput
Baseline ( 1× )
∼100×
Measurement Frequency
Monthly/Quarterly
Daily
Labeling Latency
Days to Weeks
< 24 Hours
Cost per 1k labels
Baseline
∼95% reduction
Table 3. Relative Operational Comparison: Human vs. LLM
Figure 4. Simulation POC: empirical 95% CI width vs. sample size m for two sampling schemes: PPS ( w=impressions ) and ML-PPS ( w=impression×model_score )
Policy area
Positive-rate lift (ML-PPS : PPS)
A
6.0 ×
B
11.4 ×
C
10.8 ×
Table 4. Production: lift in sample positive fraction from ML-assisted sampling (ML-PPS) relative to impression-only PPS at fixed label budget. Values are ratios (higher is better).
Policy area
Daily labels m
MDE (relative to mean)
A
50k
6.0%
B
100k
6.5%
C
10k
5.0%
Table 5. Typical minimum detectable effects (MDE), relative to each policy area’s mean. MDEs are computed for a two-sided test with α=0.05 and 80% power. Policy areas A, B, and C differ in production prevalence and ML model quality. We omit absolute prevalence and quality values. Representative configurations; actual production parameters may vary.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Population size N
300,000 items
Base positive rate p
0.005 (0.5%)
Impressions mixture mass psmall
0.93
Impressions small component
Unif{1,…,10}
Impressions tail component
xm(1+Pareto(α)) , α=1.4 , xm=10
Scores s∣y=0
Beta(1.5,6.0)
Appendix
Table 6. Synthetic simulation parameters used to generate Figure 4 . All parameters are hypothetical and not fit to any production dataset.
Policy area A
Policy area B
Labeler
Δ Prec. ↑
Δ Rec. ↑
Δ F1 ↑
Δ FPR ↓
Δ FNR ↓
Δ Inf. ↑
Labeler
Δ Prec. ↑
Δ Rec. ↑
Δ F1 ↑
Δ FPR ↓
Δ FNR ↓
Δ Inf. ↑
1 × BPO (baseline)
0.0
0.0
0.0
0.0
0.0
0.0
1 × BPO (baseline)
0.0
0.0
0.0
0.0
0.0
0.0
3 × BPO majority
+3.4
+3.3
+3.4
−1.9
−3.3
+5.1
3 × BPO majority
+2.6
+2.8
+2.7
−1.3
−2.8
+4.2
GPT-5 (baseline prompt)
−8.6
−8.4
−8.5
+4.5
+8.4
−12.9
GPT-5.4 (high)
+1.7
−8.0
−3.8
−2.7
+8.0
−5.3
GPT-5.4 (high)
−0.2
−7.5
−4.5
−0.6
+7.5
−6.9
GPT-5.4 (full context)
−2.3
+2.0
+0.2
+2.3
−2.0
−0.3
Appendix
Table 7. Decision-quality deltas (percentage points) versus a 1× single non-expert human review (BPO) baseline per policy area, scored against SME-adjudicated gold sets. Column arrows mark the better direction: higher recall is better, lower FPR is better. “(high)” denotes the high-reasoning setting; the “full context” configuration supplies the complete policy document as labeling context. Policy area A: gold set after one round of SME re-adjudication; prompts optimized to minimize FPR. Policy area B: production prompt prior to SME review. 3× majority rows aggregate three independent human reviews or three independent LLM calls. Informedness = recall − FPR.
Figure 5. Relative bias of LLM-measured prevalence θ(L) versus the true base rate θ ; color encodes the labeler false positive rate f , line style its recall r (solid r=0.90 , dashed r=0.50 ). At low prevalence the metric is extremely sensitive to FPR, which enters unbounded as f/θ : at θ=0.1% , even f=0.1% makes about half of measured positives false detections ( +90% relative at r=0.90 ). Recall loss enters only as a bounded offset: each dashed curve runs parallel to its solid counterpart, shifted down by a constant 40 points. The solid horizontal line at zero marks the Rogan–Gladen corrected estimator with known (r,f) . Line chart of relative bias in percent versus true prevalence on a log axis for five error profiles; false-positive-rate driven bias grows sharply as prevalence decreases, dashed low-recall curves run parallel to their solid counterparts at a constant offset, and a solid zero line marks the corrected estimator.
Figure 6. Sensitivity to the score exponent γ at fixed budget m=10,000 on the Appendix A population. Left: empirical 95% CI width relative to γ=1 . Right: Kish ESS/m and sample positive fraction. Aggressive tilting ( γ=4 ) collapses ESS and widens the CI despite many positives. Two small panels. Left panel shows CI width versus gamma forming a U shape with minimum near gamma between one and two. Right panel shows effective sample size over m decreasing with gamma while the positive fraction increases.
Drift
Rel. bias
CI width (rel.)
ESS/m
Pos. frac.
None
−0.2%
1.00
0.43
1.9%
Moderate
+0.0%
1.27
0.43
1.2%
Severe
+0.2%
1.63
0.49
0.8%
Appendix
Table 8. Score drift experiment ( γ=1 , m=10,000 , T=500 ): empirical relative bias, CI width relative to the no-drift condition, Kish ESS/m , and mean sample positive fraction.
Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate provides daily per-arm prevalence on over 20× as many concurrent arms under the same labeling budget. Across roughly 300 production audits, the surrogate's 95% confidence interval contains the LLM-based reference point estimate in 92% of evaluations, and day-level delta aggregation recovers 2--5% relative shifts that no single per-arm LLM measurement can detect.
Zehao Xu, Tony Paek, Kevin O'Sullivan +1
Pinterest, Inc., Toronto, ON, Canada · Pinterest, Inc., New York, NY, USA · Pinterest, Inc., San Francisco, CA, USA
Large language models (LLMs) are increasingly used to detect unsafe content. A common approach is to combine judgments from a panel of models to correct individual mistakes, but this benefit may disappear when every model sees the same misleading context before voting. We study this risk in a controlled two-round experiment. Each model first judges an item alone, then judges it again after six simulated peers either assert the wrong label or abstain. We combine the final judgments by majority vote. Across six open-weight LLMs and six datasets, we find that the wrong-label peer message raises the average reviewer false-alarm rate from 56.5% under silent peers to 87.5%, and majority voting raises the panel false-alarm rate to 100%. Without an asserted label, the same panel outperforms its average member. The effect is strongly asymmetric: reviewers follow pushes toward "unsafe" far more than pushes toward "safe" (about 75% versus 17%), so the panel's false-alarm rate rises sharply while its harmful-miss rate changes little. The proprietary-model probe shows substantial variation across models. These results identify susceptibility to shared social cues as a failure mode of safety panels and provide a simple pre-deployment diagnostic.
Many automated labeling pipelines classify inputs into categories defined by a written specification, content moderation being a prominent use case. Simple category definitions are not detailed enough for labelers to produce the accurate, consistent golden labels these pipelines require. One solution is to write a prescriptive definition that settles enough real boundary cases that labelers cannot disagree with the written interpretation. In practice, definitions at that level of detail exceed what a human annotator can hold in working memory, so annotators fall back on intuition and the labels drift from the written rules, regressing on accuracy and consistency. We propose and demonstrate the efficacy of an AI-driven workflow in which AI helps write a per-category constitution that defines the label in enough detail to cover edge cases, and a frontier LLM interprets it on each input to produce the golden label more consistently and accurately than humans reading the same document. We evaluate on three content moderation categories (harassment, hate speech, non-violent crime) and show that the approach reduces cross-model inconsistency by up to 57x compared to paragraph definitions, with cross-model disagreement diagnosing specification gaps and the human responsible for high-level decisions about what each category should mean rather than individual labeling calls. For the safety evaluation, we introduce a dual-axis formulation scoring intent and content independently over the full conversation, so downstream consumers can act on either axis or both.