cs.MANov 17, 2026

Two is better than one: optimizing binary rating scales for online platforms

Authors: Liren Shan, Alec Sun, Jingyan Wang

Organizations: University of Chicago

Abstract

Online platforms often struggle with inflated, uninformative user ratings. This issue can be mitigated by better design of the data collection mechanism, for example through changing the phrasing of user questions. Previous work optimizes a single rating scale to best distinguish the inherent qualities of items under binary feedback. In this paper, we demonstrate the power of utilizing different rating scales for different users. First, we show that a designer can always split a single rating scale into two to improve item rankings. Next, we formally characterize the optimal design of multiple scales as a series of 0-1 step functions whose threshold values are evenly spaced. Finally, we consider an adaptive model where subsequent rating scales can depend on the responses from previous users and items are ranked by their full sequence of responses, and show that adaptivity leads to exponentially greater distinguishing power. All in all, our theoretical results show that using multiple rating scales significantly improves item rankings on platforms.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hierarchical Bayesian Crowdsourcing with Item Difficulty

    May 29, 2024Seong Woo Han, Ozan Adıgüzel, Bob CarpenterInter-Rater ReliabilityLatent Variable Models

  2. Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients

    Mar 26, 2026Michael Hardy, Joshua Gilbert, Benjamin DomingueBenchmark AuditingPsychometric Validation

  3. How Reward Gain Imbalance Shapes Ranking: Rethinking Rubric-Based Reward Aggregation

    Nov 18, 2026Zhengyuan Fan, Zhonghua Wu, Tianyu Cheng +1Rubric-Based RLRubric-Based Evaluation