One of the many fairness definitions pursued in recent recommender system research targets mitigating demographic information encoded in model representations. Models optimized for this definition are typically evaluated on how well demographic attributes can be classified given model representations, with the (implicit) assumption that this measure accurately reflects recommendation parity, i.e., how similar recommendations given to different users are. We challenge this assumption by comparing the amount of demographic information encoded in representations with various measures of how the recommendations differ. We propose two new approaches for measuring how well demographic information can be classified given ranked recommendations. Our results from extensive testing of multiple models on one real and multiple synthetically generated datasets indicate that optimizing for fair representations positively affects recommendation parity, but also that evaluation at the representation level is not a good proxy for measuring this effect when comparing models. We also provide extensive insight into how recommendation-level fairness metrics behave for various models by evaluating their performances on numerous generated datasets with different properties.
Figures & tables
Figure 1: Recommendation AUC plotted against Representation AUC for synthetic datasets with different ϵ parameters. VAE and VAERel .
Figure 2: Gender and Age Recommendation AUC plotted against Representation AUC for the Movielens 1M datasets with different model parameters. VAERel and VAEAfrl* .
Figure 3: Gender and Age Recommendation AUC plotted against Representation AUC for the Movielens 1M datasets with different model parameters. VAE2adv and VAEAfrl* .
Figure 4: Recommendation AUC plotted against Representation AUC in a synthetic dataset with ϵ=0.74 parameters.
Figure 5: Recommendation metrics plotted for different models and dataset ϵ -parameters.
Figure 6: Comparison of VAE and VAERel Demographic Ratio AUC plotted for each dataset ϵ -parameters.
Figure 7: Recommendation metrics plotted for different models and number of users in dataset.
Figure 8: Recommendation metrics plotted for different models and dataset minority ratio.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Item Ratio and Kendall-Tau plotted against Representation AUC. Comparison of VAE and VAERel on synthetic datasets with different ϵ parameters.
Figure 10: Recommendation AUC plotted against Representation AUC in synthetic datasets with three different ϵ parameters.