All against the machine: the Solo score for rating skill in variable environments
Authors: David Reguera, Xavier R. Hoffmann, Irene Pérez, Pol Colomer-de-Simón, Miquel Masoliver, Xavier Guardiola, Jan Wedekind, Marián Boguñá
Organizations: Universitat de Barcelona Institute of Complex Systems (UBICS), Universitat de Barcelona, 08028 Barcelona, Spain · King Shared Services S.L, Ciutat de Granada 150, 08018 Barcelona, Spain · Dribia Data Research, S. L., Pg. de Gràcia 55, E-08007 Barcelona, Spain
We propose a distribution-free metric to rate individual skill in player-versus-environment'' settings, where participants face heterogeneous tasks without direct opponents. Such settings are common in digital platforms, games, education, finance, and the benchmark evaluation of AI agents. They combine high randomness, tasks of widely varying difficulty, and unknown heterogeneity across individuals. Our metric maps each task outcome to a bounded performance score with zero population mean and variance bounded by $1/3$, regardless of the outcome distribution, so that scores are directly comparable across tasks. Aggregating these scores over tasks yields an interpretable skill score for each individual (the Solo'' rating), and a reshuffling null model tests whether that score exceeds what chance alone would produce. We validate the approach on the progression of 2×105 players in the mobile games Candy Crush Saga and Bubble Witch 3 Saga. The metric identifies high- and low-skill players with high statistical confidence, their classification persists over hundreds of subsequent levels, and a windowed version of the score tracks changes in performance along progression.
Figures & tables
Figure 1: (a) Distribution of pass times of level 190 of Candy Crush Saga, with average pass time ⟨n⟩=5 . (b) Three different performance metrics of the same level: log (violet), golf (blue), and survival (orange), as a function of the attempts to pass.
Figure 2: Average (a) and variance (b) of the golf (blue), log (violet), and survival (orange) performance metrics in random levels exponentially distributed, as a function of the average pass time ⟨n⟩ .
Figure 3: The orange line shows the instantaneous survival performance of an arbitrary player during the first 200 levels of Candy Crush Saga. The violet and blue curves show the Solo skill score computed from Eq. ( 4 ) with lc=25 and lc=∞ , respectively.
Figure 4: Skill score distribution of simulated null players (solid) and the Gaussian approximation given by Eq. ( 7 ) (dashed) on levels 100 and 200 for Candy Crush Saga. Skill score is measured with Eq. ( 4 ) and lc=∞ .
Figure 5: Skill score distribution of null (blue) and empirical (orange) players on level 100 of Candy Crush Saga (a) and Bubble Witch 3 Saga (c). (b,d) Fraction of empirical players that are classified as good (green) and bad (red) on levels 1 to 200 of Candy Crush Saga and Bubble Witch 3 Saga.
Figure 6: (a,c) Fraction of the empirical (orange) and null (blue) players that are classified as good on level 100 of Candy Crush Saga and Bubble Witch 3 Saga and remain classified as good in the following levels. (b,d) Average score and standard deviation of empirical (orange line and shaded area) and null (blue solid and dashed lines) players that are classified as good on level 100 with score s(100)≈0.15 of Candy Crush Saga and of Bubble Witch 3 Saga.
Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China