Engagement-optimized content recommendation systems exhibit a well-documented rich-get-richer dynamic that structurally disadvantages emerging creators on large-scale platforms. Evaluating debiasing interventions at scale demands faithful simulation of creator-viewer ecosystems, yet the fidelity, diversity, and validity of such simulators remain underexplored. We present a grounded user-simulation framework that models a heterogeneous population of ∼42K creators and their coupled viewer interactions over 12weeks, and use it to evaluate equitable view-allocation policies formulated as a budgeted multi-armed bandit (MAB). The simulator captures three behavioral dimensions: (i)creator diversity, modeled via a clustering approach grounded in real platform creator-level distributions; (ii)behavioral fidelity, captured through coupled creator-content state machines that represent growth trajectories, engagement accumulation, and graduation dynamics; and (iii)population-level validity, measured by calibration against observed platform metrics. We evaluate simulative-configurations that vary content-viewership target definitions, reward functions, and a small-creator reserve lever to maximize creator-content recommendation equity. A floor-protected per content-view-target with a view-parity-gap reward guarantees 90% of view targets being met by creators every week, grows small-creator organic reach ∼+33% per week, and achieves high distribution stability (set of creators with funded-views Jaccard =0.819). We analyze the Sim2Real gap and propose validation protocols, contributing evidence for when simulation-derived fairness conclusions can be trusted.
Figures & tables
Figure 1: Simulation environment for evaluating fair view-budgeting policies. The budgeted MAB allocator operates over weekly decision epochs combining a non-negotiable parity floor (Phase 1) with reward-steered discretionary allocation (Phase 2). Creator lifecycle stages, parity-target definitions, and four reward functions are varied across configurations.
Type
Definition
Interpretation
T1, large under-parity
g>0.5
Far below target ( A<0.5E )
T2, small under-parity
0<g≤0.5
Just below target
T3, small over-parity
−1≤g≤0
Just above target
T4, large over-parity
g<−1
Far above target ( A>2E )
Table 1: Weekly-Creator type definitions by VPV-parity gap.
Wk
T1
T2
T3
T4
Total
0
11,192
1,104
1,964
12,699
26,959
4
14,799
1,312
2,282
16,459
34,852
8
17,417
1,049
1,589
20,040
40,095
11
18,045
1,019
1,526
21,727
42,317
Table 2: Simulated creators per parity-gap cluster each week.
Metric
Exploration
Growth
Graduation
Active Creators
10,775
8,488
6,148
EWTR (%)
22.93
24.25
30.54
LTR (%)
3.118
2.612
3.472
1d1K Rate (%)
83.33
87.63
92.32
1d10K Rate (%)
27.44
30.01
37.71
1d100K Rate (%)
1.99
1.37
2.18
Table 3: Simulated creator tier performance metrics per week.
Method
Spearman- ρ
Funded Jac.
Top-10% Jac.
Turnover
vpv_only
0.520
0.816
0.700
0.190
parity_gap
0.645
0.819
0.729
0.144
parity_x_eng.
0.524
0.819
0.725
0.195
hybrid
0.635
0.817
0.715
0.153
Table 4: Stability comparison across reward methods under optimal configuration.
Figure 2: Sensitivity to the reach weight w in the hybrid reward. (a) Allocation stability: Top-10% Jaccard and Spearman ρ decrease as w increases, while turnover rises. (b) Budget distribution: higher w shifts over-funding from small to large creators.
Starting cluster
Growth ratio
Median VPV: wk0 → wk11
T1 large-under
170.98 ×
10→14 (breakout tail drives mean)
T2 small-under
23.03 ×
8,997→30,920
T3 small-over
11.57 ×
14,872→34,104
T4 large-over
1.46 ×
115,809→139,824
Table 5: Mean actual-VPV growth ratio (week 11 / week 0).
Figure 3: Week-over-week allocation stability across four reward functions ( ρ=0.7 , graduated target). (a) Spearman rank correlation; (b) Funded-set Jaccard; (c) Top-10% Jaccard; (d) Turnover (lower is better). The parity-gap reward rgap dominates on all metrics, with turnover declining to 0.14 by week 11.
Figure 4: Simulation-to-production alignment of D-VPV distributions. (a) Weekly allocated D-VPVs per creator cluster. (b) Organic-to-allocated VPV ratio per cluster (log scale). Both panels show consistent week-over-week trends between simulated and production environments.
Figure 5: Simulated creator VPV-trajectories for ten sampled creators across all four tiers. The key takeaway is the compounding effect of sustained VPV-funding on reach and the subsequent transition from boost-dependence to organic self-sustainability.
Industrial recommenders give new content initial views through budgeted exploration, then use early performance to decide further delivery. On many short-video platforms, exploration is the primary way new videos reach viewers. Viewer-side tests measure consumption; the published budget objectives we review omit creator response. We analyze four experiments on a major short-video platform. An eight-month creator ablation finds production exploration raises videos posted per creator by 8.55% and creators posting at least once by 7.10% relative to a minimal floor. A budget-matched reallocation raises creator participation with no detectable short-run viewer-side change. A year-long viewer ablation finds 1.74% more video views but 2.13% less view time. A delivered view creates immediate feed value, can trigger organic take-up, and can induce creator supply. Take-up and supply replenish a shared corpus, creating two measurement limits. Viewer-side A/B tests cancel the corpus effect when both arms consume the same corpus. Giving each arm its own corpus avoids cancellation, but turnover still controls the horizon. If the corpus turns over at rate w per posting cycle, a t-cycle experiment expresses at most wt of the eventual corpus effect. More users reduce noise but do not speed turnover. Before the corpus path visibly bends, data cannot distinguish a modest fast effect from an arbitrarily large slow one, so a valid confidence interval may lack a finite upper endpoint. As predicted, the three-week co-diverted experiment cannot determine the sign of the eventual corpus effect. Within the window, it identifies the direct feed effect, and an exploratory cohort analysis detects organic lift after exploration ends. The experiments establish a positive creator response, measure the gross corpus flow visible within three weeks, and show the design and duration needed to identify total value.
Yuanyuan Shen, Yiren Yan, Wenjie Li +1
Snap Inc. New York, NY, USA · Snap Inc. Palo Alto, CA, USA
Understanding the performance of large-scale recommender systems remains an underexplored challenge, especially for content creators and model developers. The raw engagement signals available to them, such as views and clicks, conflate content quality, model behavior, presentation bias, and audience reach, making it hard to attribute outcomes to the right cause. In this work, we present a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrate its effectiveness through several production deployments. The framework treats recommender-system observability as a counterfactual measurement problem: estimating what the recommender would have done, and what engagement would have followed, in the absence of a specific content item or model decision. We articulate three stakeholder-centered observability principles for content creators and model developers, and propose measurement methodologies covering bias reduction, relativity, and incrementality, applicable to both single-stage and cascading recommender systems and serving both audiences from a single measurement foundation.
Chaoran Guo, Ding Tong, Ting-Po Lee +1
Netflix Los Gatos, California, USA · San Jose, California, USA
One of the many fairness definitions pursued in recent recommender system research targets mitigating demographic information encoded in model representations. Models optimized for this definition are typically evaluated on how well demographic attributes can be classified given model representations, with the (implicit) assumption that this measure accurately reflects recommendation parity, i.e., how similar recommendations given to different users are. We challenge this assumption by comparing the amount of demographic information encoded in representations with various measures of how the recommendations differ. We propose two new approaches for measuring how well demographic information can be classified given ranked recommendations. Our results from extensive testing of multiple models on one real and multiple synthetically generated datasets indicate that optimizing for fair representations positively affects recommendation parity, but also that evaluation at the representation level is not a good proxy for measuring this effect when comparing models. We also provide extensive insight into how recommendation-level fairness metrics behave for various models by evaluating their performances on numerous generated datasets with different properties.
Bjørnar Vassøy, Benjamin Kille, Helge Langseth
Norwegian University of Science and Technology, Trondheim, Norway