Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate provides daily per-arm prevalence on over 20× as many concurrent arms under the same labeling budget. Across roughly 300 production audits, the surrogate's 95% confidence interval contains the LLM-based reference point estimate in 92% of evaluations, and day-level delta aggregation recovers 2--5% relative shifts that no single per-arm LLM measurement can detect.
Figures & tables
Figure 1. Bucket level impression share shifts for categories k1 and k2 in Experiment A. The treatment arm reduces exposure primarily by shifting impressions out of higher score buckets.
Metric
LLM-based
ML score surrogate
Pk1(Control)
3.61% [3.40%, 3.82%]
3.81% [3.67%, 3.95%]
Pk1(Treatment1)
2.91% [2.72%, 3.10%]
2.78% [2.64%, 2.92%]
Δk1,Treat1−Ctrl
−0.70%
−1.03%
p -value
<0.001 ( Stats-Sig )
<0.001 ( Stats-Sig )
Pk2(Control)
1.32% [1.20%, 1.43%]
1.36% [1.20%, 1.52%]
Pk2(Treatment2)
1.01% [0.91%, 1.11%]
0.87% [0.70%, 1.04%]
Table 1. Comparison of LLM-based vs. ML score surrogate prevalence in Experiment A for categories k1 and k2 . The ranges denote 95% confidence intervals.
Metric
LLM-based
ML score surrogate
Pk1(Control)
3.44% [3.23%, 3.64%]
3.39% [3.25%, 3.53%]
Pk1(Treatment)
3.31% [3.11%, 3.51%]
3.40% [3.26%, 3.53%]
Δk1,Treat−Ctrl
−0.13%
0.01%
p -value
0.38 ( No Stats-Sig )
0.95 ( No Stats-Sig )
Pk2(Control)
1.32% [1.19%, 1.45%]
1.39% [1.22%, 1.56%]
Pk2(Treatment)
1.40% [1.26%, 1.53%]
1.35% [1.18%, 1.52%]
Table 2. Comparison of LLM-based vs. ML score surrogate prevalence for categories k1 and k2 in Experiment B.
Figure 2. Relative prevalence reduction vs. calendar day: Δ(d)/Pk1(control;d) for category k1 in Experiment C.
Figure 3. Comparison of score bucket distributions under two sampling schemes. Left: sampling with weights proportional to impressions alone produces a very low score heavy sample, leaving few examples in high score buckets. Right: sampling with weights proportional to impression × model score yields a more balanced distribution across buckets, improving the precision of bucket level prevalence estimates.
Metric
LLM-based
ML score surrogate
# of Measured Exp
Baseline
>20×
Extra Labeling Cost
Baseline
$0
Frequency
One Run Per Exp
Daily Run Per Arm
Latency
10 to 24 Hours
<10s
Table 3. Relative Operational Comparison: LLM vs. Surrogate
Figure 4. Daily k1 prevalence for the six highest-engagement countries (A–F), comparing the LLM-based estimator and the ML-score surrogate over a 30-day window; ribbons denote 95% confidence intervals.
Content safety teams need metrics that reflect what users actually experience, not only what is reported. We study prevalence: the fraction of user views (impressions) that went to content violating a given policy on a given day. Accurate prevalence measurement is challenging because violations are often rare and human labeling is costly, making frequent, platform-representative studies slow. We present a design-based measurement system that (i) draws daily probability samples from the impression stream using ML-assisted weights to concentrate label budget on high-exposure and high-risk content while preserving unbiasedness, (ii) labels sampled items with a multimodal LLM governed by policy prompts and gold-set validation, and (iii) produces design-consistent prevalence estimates with confidence intervals and dashboard drilldowns. A key design goal is one global sample with many pivots: the same daily sample supports prevalence by surface, viewer geography, content age, and other segments through post-stratified estimation. We describe the statistical estimators, variance and confidence interval construction, label-quality monitoring, and an engineering workflow that makes the system configurable across policies.
Attila Dobi, Aravindh Manickavasagam, Benjamin Thompson +2
Pinterest, San Francisco, CA, USA · Pinterest, Richmond, VA, USA · Pinterest, Los Angeles, CA, USA +1
Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost. We study when a treatment effect estimated on LLM outcomes can recover the effect for the human population of interest. Distributional equivalence between LLM and human outcomes would make any standard estimator valid but is unrealistic. We therefore develop a statistical framework that adapts surrogate endpoint theory to LLMs, showing that calibrating LLM outcomes to human outcomes identifies the average treatment effect under surrogacy and comparability conditions that are jointly weaker than distributional equivalence. We present a falsification test for surrogacy and a bound on the worst-case bias from limited overlap between the LLM and human samples. We further show that the stochasticity inherent to LLMs can weaken surrogacy for identification while also introducing bias and variance during estimation, but that using an average over multiple LLM draws per unit as the surrogate mitigates these issues. Simulations validate the results, and an empirical application to the Upworthy Research Archive dataset shows that raw LLM outputs recover only 39% of the human treatment effect while nonparametric calibration closes the gap. A central takeaway is that A/B testing on LLM responses is correct only by assumption, whereas A/B testing on humans is correct by design, and that the required assumptions are hardest to justify precisely where LLMs promise the greatest benefit. We discuss the choice of LLM, prompting, and temperature as design variables, the compounded challenge posed by long-term outcomes, and how to size human pilot studies for validation.
Joel Persson, Mårten Schultzberg, Sebastian Ankargren
Spotify USA, Inc. New York, NY, USA · Spotify AB Stockholm, Sweden
Estimating the prevalence of a category in a population using imperfect measurement devices (diagnostic tests, classifiers, or large language models) is fundamental to science, public health, and online trust and safety. Standard approaches correct for known device error rates but assume these rates remain stable across populations. We show this assumption fails under covariate shift and that multicalibration, which enforces calibration conditional on the input features rather than just on average, is sufficient for unbiased prevalence estimation under such shift. Standard calibration and quantification methods fail to provide this guarantee. Our work connects recent theoretical work on fairness to a longstanding measurement problem spanning nearly all academic disciplines. A simulation confirms that standard methods exhibit bias growing with shift magnitude, while a multicalibrated estimator maintains near-zero bias. While we focus the discussion mostly on LLMs, our theoretical results apply to any classification model. Two empirical applications -- estimating employment prevalence across U.S. states using the American Community Survey, and classifying political texts across four countries using an LLM -- demonstrate that multicalibration substantially reduces bias in practice, while highlighting that calibration data should cover the key feature dimensions along which target populations may differ.
Fridolin Linder, Thomas Leeper, Daniel Haimovich +3
Meta Platforms Inc. · The London School of Economics and Political Science