cs.CLSep 1, 2024

Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

Authors: Jasper Dekoninck, Maximilian Baader, Martin Vechev

Organizations: Department of Computer Science, ETH Zurich, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they require large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce Polyrating, an expressive and flexible rating system based on maximum a posteriori estimation that enables a more nuanced and thorough analysis of model performance at lower costs. Polyrating can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, Polyrating can reduce the cost of human evaluations by up to 41%41\% for new models and up to 77%77\% for new tasks by leveraging existing benchmark scores. Lastly, Polyrating enables direct comparisons of ratings across different tasks, providing a comprehensive understanding of an LLMs' strengths, weaknesses, and relative performance across different applications.

Figures & tables

Figure 1: Overview of Polyrating . Given preference datasets of nn samples for kk models over various tasks, the standard approach needs to fit separate and independent ratings for each task and cannot leverage continuous features. In contrast, Polyrating fits a single linear model for all tasks and can leverage continuous features. Attribution in App. A .
Table 1: Fitted coefficients for the biases and their average influence on model ratings for both human and LLM-based evaluation. The functional form of fbiasf_{\text{bias}} used for each bias can be found in App. E . The influence is computed as Eg(αbias⋅∣fbias(g,0)−fbias(g,1)∣)\mathbb{E}_{g}(\alpha_{\text{bias}}\cdot|f_{\text{bias}}(g,0)-f_{\text{bias}}(g,1)|) .
Figure 2: Comparison between Polyrating and univariate baseline for different tasks. The xx -axis shows the number of samples of the task the rating systems are using. The logistic loss shown is normalized by subtracting the loss of the best possible rating for that task. The grey horizontal line indicates the loss of a rating system that assigns the same rating to all models.
Figure 3: Comparison between Polyrating and the univariate baseline when leveraging information from existing benchmarks. For the left and middle plot, the xx -axis shows the number of human annotations used. For the right plot, the xx -axis shows the amount of samples from the Chinese code task. The logistic loss is normalized by subtracting the loss of the best possible rating.
RankModel NameRatingEnglishChineseHardnessCode
1gpt-4o-2024-05-131297±4.41297_{\pm 4.4}−13±7.4-13_{\pm 7.4}−003±8.7-\phantom{00}3_{\pm 8.7}−13±7.5\phantom{-}13_{\pm 7.5}−15±8.0\phantom{-}15_{\pm 8.0}
2claude-3-5-sonnet-202406201286±7.11286_{\pm 7.1}−29±10.7-29_{\pm 10.7}−019±13.8-\phantom{0}19_{\pm 13.8}−14±11.0\phantom{-}14_{\pm 11.0}−44±12.6\phantom{-}44_{\pm 12.6}
12yi-large-preview1237±4.31237_{\pm 4.3}−04±7.6-\phantom{0}4_{\pm 7.6}−037±8.5\phantom{-}\phantom{0}37_{\pm 8.5}−17±7.6\phantom{-}17_{\pm 7.6}−11±8.3\phantom{-}11_{\pm 8.3}
26llama-3-70b-instruct1187±2.91187_{\pm 2.9}−67±6.0\phantom{-}67_{\pm 6.0}−051±5.4-\phantom{0}51_{\pm 5.4}−01±5.6-\phantom{0}1_{\pm 5.6}−10±6.1-10_{\pm 6.1}
56mixtral-8x7b-instruct-v0.11114±4.11114_{\pm 4.1}−25±7.1\phantom{-}25_{\pm 7.1}−037±7.3-\phantom{0}37_{\pm 7.3}−10±6.8\phantom{-}10_{\pm 6.8}−04±7.6-\phantom{0}4_{\pm 7.6}
Table 2: Several models in a multidimensional leaderboard fitted using Polyrating . The given rank of the models indicates its rank in the complete leaderboard shown in Table 7 in App. F .
RankModel NameRatingEnglishChineseHardnessCode
1gpt-4o-2024-05-131283±3.21283_{\pm 3.2}−19±5.2-19_{\pm 5.2}−052±10.4\phantom{-}\phantom{0}52_{\pm 10.4}−05±6.7\phantom{-}\phantom{0}5_{\pm 6.7}−13±7.8\phantom{-}13_{\pm 7.8}
2claude-3-5-sonnet-202406201267±4.81267_{\pm 4.8}−23±8.1-23_{\pm 8.1}−045±14.4\phantom{-}\phantom{0}45_{\pm 14.4}−11±10.1\phantom{-}11_{\pm 10.1}−35±11.3\phantom{-}35_{\pm 11.3}
10yi-large-preview1233±3.31233_{\pm 3.3}−19±5.6-19_{\pm 5.6}−084±10.8\phantom{-}\phantom{0}84_{\pm 10.8}−07±7.1\phantom{-}\phantom{0}7_{\pm 7.1}−11±8.3\phantom{-}11_{\pm 8.3}
17llama-3-70b-instruct1202±2.51202_{\pm 2.5}−22±4.1\phantom{-}22_{\pm 4.1}−032±8.0-\phantom{0}32_{\pm 8.0}−05±5.3-\phantom{0}5_{\pm 5.3}−00±6.3\phantom{-}\phantom{0}0_{\pm 6.3}
53mixtral-8x7b-instruct-v0.11114±0.01114_{\pm 0.0}−00±0.0\phantom{-}\phantom{0}0_{\pm 0.0}−000±0.0\phantom{-}\phantom{00}0_{\pm 0.0}−00±0.0\phantom{-}\phantom{0}0_{\pm 0.0}−00±0.0\phantom{-}\phantom{0}0_{\pm 0.0}
Table 3: Several models in the leaderboard fitted using a unidimensional approach where each task is fitted separately. Modifiers for the tasks are shown instead of the fitted rating to make comparison with Polyrating easier. The complete leaderboard is shown in Table 8 in App. F .
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Logistic loss for all four alternatives on the Chatbot Arena dataset for various sizes of the training set.
Figure 5: Convergence rate of the univariate method and Polyrating when evaluating model versions. The x-axis represents the number of games available in the training set associated with the subsequent versions, while the y-axis represents the loss of the univariate method and Polyrating .
BiasDescriptionFunctional Form
LengthMeasures the length of a model answer for a given question.f(g,i)=log⁡10(length(gmi(gp)))f(g,i)=\log_{10}(\text{length}(g_{m_{i}}(g_{p})))
PositionComputes the order of the model in the game.f(g,i)=⟦i=1⟧f(g,i)=\llbracket i=1\rrbracket
FormalityComputes the formality of an answer computed by a popular formality classifier [ Babakov et al., 2023 ] . 1 1 1 https://huggingface.co/s-nlp/roberta-base-formality-rankerf(g,i)=M(gmi(gp))1f(g,i)=\mathcal{M}(g_{m_{i}}(g_{p}))_{1}
SentimentComputes the sentiment of an answer computed by a popular sentiment classifier [ Camacho-collados et al., 2022 ] . 2 2 2 https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latestf(g,i)=M(gmi(gp))2f(g,i)=\mathcal{M}(g_{m_{i}}(g_{p}))_{2}
RepetitivenessComputes the repetitiveness of the answer by computing the percentage of non-unique words in the answer.f(g,i)=5⋅# of repeated words in gmi(gp)# of words in gmi(gp)f(g,i)=5\cdot\frac{\text{\# of repeated words in }g_{m_{i}}(g_{p})}{\text{\# of words in }g_{m_{i}}(g_{p})}
ReadabilityComputes the Flesch Reading Ease score [ Kincaid et al., 1975 ] of an answer.f(g,i)=min⁡(1,max⁡(0,Flesch(gmi(gp))100))f(g,i)=\min(1,\max(0,\frac{\text{Flesch}(g_{m_{i}}(g_{p}))}{100}))
Appendix
Table 4: Overview of all biases used in Section 4.1 . The table contains a description of the bias and a functional form of the bias. Scaling constant were introduced in these functional forms to ensure that all biases output values within the same order of magnitude.
TaskDescription
EnglishQuestions that are in English.
ChineseQuestions that are in Chinese.
HardnessQuestions that are considered hard by the Chatbot Arena. These are questions that are classified as being in at least six of the following seven categories: specific, requires domain knowledge, is complex, requires problem-solving, requires creative thinking, requires technical accuracy, is a real-world question.
CodeQuestions that require code to be answered.
LLMWhether the judge is a language model.
Appendix
Table 5: Overview of all tasks used in Section 4 . For each task, we use the same data as the actual Chatbot Arena [ Chiang et al., 2024b ]
Table 6: Fitted coefficients for the biases and their average influence on the ratings of the models for both human and LLM-based evaluation. The functional form of fbiasf_{\text{bias}} used for each bias can be found in App. E . The influence is computed as Eg(αbias⋅∣fbias(g,0)−fbias(g,1)∣)\mathbb{E}_{g}(\alpha_{\text{bias}}\cdot|f_{\text{bias}}(g,0)-f_{\text{bias}}(g,1)|) and indicates the average influence the bias has on the rating of models for specific games. Errors shown are 95%95\% pivot intervals computed using bootstrapping.
RankModel NameRatingEnglishChineseHardnessCode
1gpt-4o-2024-05-131297−4.2+4.51297_{-4.2}^{+4.5}−13−10.3+4.3-13_{-10.3}^{+4.3}−003−8.2+8.8-\phantom{00}3_{-8.2}^{+8.8}−13−4.8+10.0\phantom{-}13_{-4.8}^{+10.0}−15−5.7+10.4\phantom{-}15_{-5.7}^{+10.4}
2claude-3-5-sonnet-202406201286−7.4+6.71286_{-7.4}^{+6.7}−29−13.9+8.2-29_{-13.9}^{+8.2}−019−12.9+15.0-\phantom{0}19_{-12.9}^{+15.0}−14−8.5+13.6\phantom{-}14_{-8.5}^{+13.6}−44−10.6+14.9\phantom{-}44_{-10.6}^{+14.9}
3gemini-advanced-05141285−5.1+4.51285_{-5.1}^{+4.5}−26−10.7+4.5-26_{-10.7}^{+4.5}−008−9.4+9.2\phantom{-}\phantom{00}8_{-9.4}^{+9.2}−03−5.6+10.3\phantom{-}\phantom{0}3_{-5.6}^{+10.3}−02−6.9+11.2\phantom{-}\phantom{0}2_{-6.9}^{+11.2}
4gemini-1.5-pro-api-05141273−4.5+4.81273_{-4.5}^{+4.8}−20−10.5+4.8-20_{-10.5}^{+4.8}−019−8.6+10.0\phantom{-}\phantom{0}19_{-8.6}^{+10.0}−15−4.8+10.0\phantom{-}15_{-4.8}^{+10.0}−11−5.8+11.3\phantom{-}11_{-5.8}^{+11.3}
5claude-3-opus-202402291273−2.7+3.01273_{-2.7}^{+3.0}−39−9.2+2.5-39_{-9.2}^{+2.5}−001−5.1+6.0\phantom{-}\phantom{00}1_{-5.1}^{+6.0}−15−2.9+8.4\phantom{-}15_{-2.9}^{+8.4}−12−3.7+8.4\phantom{-}12_{-3.7}^{+8.4}
6bard-jan-24-gemini-pro1271−12.6+12.41271_{-12.6}^{+12.4}−48−16.9+11.7-48_{-16.9}^{+11.7}−025−25.8+25.6-\phantom{0}25_{-25.8}^{+25.6}−45−11.7+16.9-45_{-11.7}^{+16.9}−11−13.4+19.0-11_{-13.4}^{+19.0}
Appendix
Table 7: Leaderboard of human evaluation with modifiers fitted with Polyrating . Indicated deviations are 95%95\% pivot intervals determined using bootstrapping.
RankModel NameRatingEnglishChineseHardnessCode
1gpt-4o-2024-05-131283−2.8+2.71283_{-2.8}^{+2.7}−19−4.4+4.3-19_{-4.4}^{+4.3}−052−9.2+9.0\phantom{-}\phantom{0}52_{-9.2}^{+9.0}−05−5.4+5.6\phantom{-}\phantom{0}5_{-5.4}^{+5.6}−13−6.7+6.3\phantom{-}13_{-6.7}^{+6.3}
2claude-3-5-sonnet-202406201267−4.2+4.11267_{-4.2}^{+4.1}−23−6.9+6.9-23_{-6.9}^{+6.9}−045−12.1+12.1\phantom{-}\phantom{0}45_{-12.1}^{+12.1}−11−8.8+8.5\phantom{-}11_{-8.8}^{+8.5}−35−10.0+9.1\phantom{-}35_{-10.0}^{+9.1}
3gemini-advanced-05141261−2.9+2.81261_{-2.9}^{+2.8}−28−5.1+4.8-28_{-5.1}^{+4.8}−069−8.9+8.7\phantom{-}\phantom{0}69_{-8.9}^{+8.7}−08−6.1+6.2-\phantom{0}8_{-6.1}^{+6.2}−04−7.5+7.0-\phantom{0}4_{-7.5}^{+7.0}
4gemini-1.5-pro-api-05141259−2.9+2.71259_{-2.9}^{+2.7}−25−4.9+4.8-25_{-4.9}^{+4.8}−076−8.7+8.5\phantom{-}\phantom{0}76_{-8.7}^{+8.5}−05−5.8+6.1\phantom{-}\phantom{0}5_{-5.8}^{+6.1}−08−7.0+6.8\phantom{-}\phantom{0}8_{-7.0}^{+6.8}
5gpt-4-turbo-2024-04-091252−2.3+2.31252_{-2.3}^{+2.3}−12−4.0+3.7-12_{-4.0}^{+3.7}−050−7.1+7.4\phantom{-}\phantom{0}50_{-7.1}^{+7.4}−05−4.7+5.1\phantom{-}\phantom{0}5_{-4.7}^{+5.1}−14−5.8+6.0\phantom{-}14_{-5.8}^{+6.0}
6gpt-4-1106-preview1248−2.3+2.31248_{-2.3}^{+2.3}−17−3.9+3.4-17_{-3.9}^{+3.4}−052−7.9+7.5\phantom{-}\phantom{0}52_{-7.9}^{+7.5}−00−5.2+4.9\phantom{-}\phantom{0}0_{-5.2}^{+4.9}−08−5.9+5.6\phantom{-}\phantom{0}8_{-5.9}^{+5.6}
Appendix
Table 8: Leaderboard of human evaluation with modifiers when all fitted separately using a unidimensional approach. For the four tasks, modifiers are shown to indicate the deviation from the main rating to make comparison with Polyrating easier. Indicated deviations are 95%95\% confidence intervals determined using bootstrapping.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList

  1. Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews

    Apr 21, 2026Bowen Li, Haochen Ma, Yuxin Wang +5LLM EvaluationRubric-Based Evaluation

    College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China

  2. Rating the Raters: Rasch Measurement Theory for LLM Evaluation

    Jul 14, 2026Pratik S. Sachdeva, Nathan BoudolLLM EvaluationLLM-as-a-Judge

    University of California, Berkeley · Grenoble INP

  3. Accounting for Bias Enables Sustainable LLM Evaluation

    Sep 25, 2026Harshita Katoch, David Antony Selby, Gerrit Großmann +1LLM EvaluationBias in LLM-as-a-Judge

    Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany