Online platforms often struggle with inflated, uninformative user ratings. This issue can be mitigated by better design of the data collection mechanism, for example through changing the phrasing of user questions. Previous work optimizes a single rating scale to best distinguish the inherent qualities of items under binary feedback. In this paper, we demonstrate the power of utilizing different rating scales for different users. First, we show that a designer can always split a single rating scale into two to improve item rankings. Next, we formally characterize the optimal design of multiple scales as a series of 0-1 step functions whose threshold values are evenly spaced. Finally, we consider an adaptive model where subsequent rating scales can depend on the responses from previous users and items are ranked by their full sequence of responses, and show that adaptivity leads to exponentially greater distinguishing power. All in all, our theoretical results show that using multiple rating scales significantly improves item rankings on platforms.
Figures & tables
Figure 1: Splitting a single scale β(θ) into two scales β+(θ)=min{2β(θ),1} and β−(θ)=max{2β(θ)−1,0} increases the platform objective Wk .
Figure 2: For Kendall’s τ function, the optimal k scales for k ratings are 0-1 step functions with evenly spaced thresholds in [0,1] . The figure illustrates the optimal configuration of k=4 scales.
Figure 3: For k≥2 , using a different scale for each rating has a concrete advantage over using the same scale. The approximation for Wksingle is calculated based on the best discretized scale computed in Figure 5 .
Figure 4: The best discretized sets of ℓ scales for ℓ∈{2,3,4} are graphed alongside the best discretized single scale β . All graphs use Kendall’s τ function, k=24 ratings, and a discretization of n=50 steps. For better visualization, we connect the dots between steps instead of graphing the discontinuous step functions themselves.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The optimal single scale for Kendall’s τ function discretized to n=50 steps is graphed for all numbers k∈{1,2,4,8,16,32,64} of ratings. The legend shows the platform objective of each discretized scale. For better visualization, we connect the dots between steps instead of graphing the discontinuous step functions themselves.