Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation
Organizations: LTRC, IIITH
Abstract
Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentation sensitivity in Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces. Across six datasets with 9,000 multiply annotated humor, helpfulness, and summarization items, we collect 702,000 judgments under interventions that vary positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics while preserving the evaluated texts. Jev's second-position probability advantage reaches 6.11 percentage points on summaries; reversing reader endorsements changes Perplexity's selected humor text on 40.40% of pairs; and rejection framing reduces Clef's summary agreement with human preferences by 23.36 points. Rating formats also shift score levels while often preserving rankings. Across these findings, repeatability, presentation robustness, and human alignment emerge as distinct properties of an evaluator. Reliable use therefore requires validating the decisions and score interpretations on which an evaluation pipeline depends.
Figures & tables
| Task / interface | Dataset and source | What the model evaluates | Human reference |
|---|---|---|---|
| Humor / choice | UKP Humor ( Simpson et al., 2019 ) | Funniness of two short texts: puns, jokes, or proverbs/aphorisms | Majority of 5 A/B/neither votes; split cases retained |
| Humor / score | Humicroedit ( Hossain et al., 2020 ) | Funniness of a single-word headline edit, given the original headline | Mean of 5 ratings, 0–3 |
| Helpfulness / choice | HelpSteer2-Preference ( Wang et al., 2025 ) | Helpfulness of two responses to the same request | Sign of published aggregate preference strength; 5 judgments |
| Helpfulness / score | HelpSteer2 ( Wang et al., 2024b ) | Helpfulness of one response to a request | Mean of 5 helpfulness ratings, 0–4 |
| Summarization / choice | OpenAI comparisons ( Stiennon et al., 2020 ) | Quality of two summaries of the same Reddit post | Majority of 7–23 distinct workers; split cases retained |
| Summarization / score | SummEval ( Fabbri et al., 2021 ) | Relevance of one summary to its source article | Mean of all 8 relevance ratings: 3 expert, 5 crowd; 1–5 |
| Choice investigation | Runs | Conditions and controls | Main comparison |
|---|---|---|---|
| Baseline | 1 | Original question/context; two texts labeled A/B. | Human alignment and reference judgments. |
| Position and A/B identifier bias | 3 | Swap labels, positions, or both; baseline completes the factorial. | Position, identifier, and interaction effects. |
| Numeric identifier bias | 2 | Fixed text positions labeled 1/2 or 2/1. | Reversed identifier assignments. |
| Authorship attribution | 3 | AI/Human attribution, reversed assignment, unspecified-source control. | Reversal; each assignment versus control. |
| Reader endorsement | 3 | 80%/20% reader preference, reversed assignment, unspecified-feedback control. | Reversal; each assignment versus control. |
| Previous-judge feedback | 3 | Selected/not-selected cue, reversed assignment, unspecified-feedback control. | Reversal; each assignment versus control. |
| Task model | Pair alignment % | Pair repeat flip % | Score MAE | Score Spearman | Score repeat abs. delta pp |
|---|---|---|---|---|---|
| Humor / Jev | 73.38 | 3.15 | 0.256 | 0.365 | 1.09 |
| Humor / Clef | 67.45 | 3.02 | 0.200 | 0.549 | 1.42 |
| Humor / Perplexity | 76.76 | 0.08 | 0.158 | 0.379 | 0.00 |
| Helpfulness / Jev | 68.62 | 2.58 | 0.175 | 0.448 | 0.89 |
| Helpfulness / Clef | 61.85 | 3.20 | 0.184 | 0.271 | 1.06 |
| Helpfulness / Perplexity | 73.53 | 0.00 | 0.148 | 0.581 | 0.00 |
| Task contrast | Jev pp [95% CI] | Clef Flash pp [95% CI] | Perplexity pp [95% CI] |
|---|---|---|---|
| Humor: First position | -2.3 [-3.0, -1.5]* | 0.0 [-0.3, 0.3] | 4.6 [4.1, 5.1]* |
| Humor: B versus A | 3.9 [3.7, 4.1]* | -8.6 [-9.6, -7.7]* | -1.5 [-1.7, -1.3]* |
| Humor: Human versus AI | 7.0 [6.7, 7.4]* | 3.6 [3.1, 4.1]* | 9.0 [8.6, 9.4]* |
| Humor: Reader feedback | 13.5 [13.0, 14.1]* | 4.8 [4.4, 5.2]* | 27.8 [26.8, 28.8]* |
| Helpfulness: First position | -2.3 [-2.9, -1.6]* | -0.1 [-0.3, 0.0] | -4.4 [-4.8, -3.9]* |
| Helpfulness: B versus A | -1.6 [-1.8, -1.4]* | -4.7 [-5.2, -4.1]* | -0.4 [-0.6, -0.3]* |
| Task outcome | Jev | Clef Flash | Perplexity |
|---|---|---|---|
| Humor: reader reversal (%) | 22.00 | 10.13 | 40.40 |
| Humor: unchanged repeat (%) | 3.15 | 3.02 | 0.08 |
| Humor: rejection agreement (pp) | -1.2 [-3.3, 0.8] | -2.1 [-5.5, 1.3] | 0.9 [-0.9, 2.7] |
| Helpfulness: reader reversal (%) | 18.27 | 9.73 | 27.13 |
| Helpfulness: unchanged repeat (%) | 2.58 | 3.20 | 0.00 |
| Helpfulness: rejection agreement (pp) | -1.2 [-3.1, 0.7] | -15.8 [-19.9, -11.6]* | -1.1 [-2.7, 0.5] |
| Task contrast | Jev pp [95% CI] | Clef Flash pp [95% CI] | Perplexity pp [95% CI] |
|---|---|---|---|
| Humor: Reader feedback | 4.0 [3.8, 4.1]* | 7.3 [7.0, 7.7]* | 21.2 [20.8, 21.6]* |
| Humor: Previous judge | 2.5 [2.4, 2.5]* | 2.1 [1.9, 2.3]* | 19.1 [18.8, 19.4]* |
| Humor: Three levels | -2.8 [-2.9, -2.7]* | -3.5 [-3.8, -3.3]* | -6.8 [-6.9, -6.7]* |
| Humor: Seven levels | -7.6 [-7.8, -7.5]* | -5.2 [-5.4, -5.0]* | -3.7 [-4.0, -3.5]* |
| Helpfulness: Reader feedback | 16.5 [16.1, 16.8]* | 3.4 [3.3, 3.6]* | 10.3 [10.1, 10.4]* |
| Helpfulness: Previous judge | 11.8 [11.5, 12.0]* | 1.6 [1.4, 1.7]* | 9.3 [9.1, 9.4]* |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Shared state | Exact instructions |
|---|---|---|
| UKP Humor | Two standalone short texts are being compared. | Which option contains the funnier text? |
| Humicroedit | original_headline , edited_headline | How funny is ‘edited_headline‘ as an edit of ‘original_headline‘ ? |
| HelpSteer2 pairs | user_request | Which option contains the more helpful response to ‘user_request‘ ? |
| HelpSteer2 scores | user_request , response | How helpful is ‘response‘ overall for ‘user_request‘ ? |
| OpenAI pairs | source_text | Which option contains the better summary of ‘source_text‘ ? |
| SummEval | source_article , summary | How relevant is ‘summary‘ to ‘source_article‘ ? Relevance means selecting important content from the source. Penalize redundant or excess information. |
| Task model | Rejection inconsistency % [95% CI] | Strict alignment change pp [95% CI] | X-last use % |
|---|---|---|---|
| Humor / Jev | 12.67 [10.95, 14.38] | -1.22 [-3.27, 0.82] | 4.13 |
| Humor / Clef | 28.53 [26.13, 30.94] | -2.07 [-5.47, 1.33] | 2.33 |
| Humor / Perplexity | 9.20 [7.66, 10.74] | 0.94 [-0.87, 2.75] | 9.00 |
| Helpfulness / Jev | 13.53 [11.80, 15.27] | -1.17 [-3.07, 0.72] | 0.00 |
| Helpfulness / Clef | 67.73 [65.36, 70.10] | -15.76 [-19.91, -11.61]* | 0.07 |
| Helpfulness / Perplexity | 9.20 [7.74, 10.66] | -1.11 [-2.69, 0.47] | 0.40 |
| Dataset model | First-position pp [95% CI] | B-label pp [95% CI] | Interaction pp [95% CI] |
|---|---|---|---|
| UKP / Jev | -2.25 [-2.98, -1.52]* | 3.86 [3.65, 4.06]* | 0.04 [-0.24, 0.32] |
| UKP / Clef | 0.01 [-0.26, 0.29] | -8.64 [-9.62, -7.67]* | -0.05 [-0.62, 0.52] |
| UKP / Perplexity | 4.58 [4.06, 5.10]* | -1.50 [-1.72, -1.28]* | -0.02 [-0.39, 0.36] |
| HS pair / Jev | -2.27 [-2.90, -1.65]* | -1.59 [-1.79, -1.39]* | -0.24 [-0.56, 0.08] |
| HS pair / Clef | -0.12 [-0.25, 0.01] | -4.69 [-5.25, -4.13]* | 0.30 [0.05, 0.55] |
| HS pair / Perplexity | -4.35 [-4.80, -3.90]* | -0.43 [-0.57, -0.29]* | 0.17 [-0.11, 0.46] |
| Dataset model | First-position pp [95% CI] | B-label pp [95% CI] | Interaction pp [95% CI] |
|---|---|---|---|
| UKP / Jev | -0.93 [-2.60, 0.73] | 6.20 [5.24, 7.16]* | 0.93 [-0.81, 2.68] |
| UKP / Clef | -0.07 [-0.66, 0.53] | -10.53 [-12.21, -8.85]* | -0.40 [-1.64, 0.84] |
| UKP / Perplexity | 8.60 [7.11, 10.09]* | -1.73 [-2.54, -0.93]* | 0.00 [-1.48, 1.48] |
| HS pair / Jev | -4.83 [-6.35, -3.31]* | -2.10 [-2.89, -1.31]* | -0.87 [-2.29, 0.56] |
| HS pair / Clef | -0.10 [-0.78, 0.58] | -6.10 [-8.02, -4.18]* | 0.73 [-0.55, 2.02] |
| HS pair / Perplexity | -6.47 [-7.95, -4.99]* | -1.27 [-2.00, -0.53]* | 0.67 [-0.84, 2.17] |
| Dataset model | 2 - 1 probability pp [95% CI] | 2 - 1 selection pp [95% CI] | Reversal flips % |
|---|---|---|---|
| UKP / Jev | 7.44 [6.66, 8.22]* | 8.933 [6.870, 10.996] | 14.67 |
| UKP / Clef | -7.81 [-8.86, -6.76]* | -9.067 [-10.767, -7.367] | 11.47 |
| UKP / Perplexity | -6.01 [-6.57, -5.46]* | -10.267 [-11.925, -8.609] | 11.47 |
| HS pair / Jev | 2.15 [1.48, 2.82]* | 4.067 [2.348, 5.785] | 11.67 |
| HS pair / Clef | -4.46 [-5.05, -3.86]* | -4.800 [-6.846, -2.754] | 16.53 |
| HS pair / Perplexity | 3.33 [2.87, 3.78]* | 3.667 [2.124, 5.209] | 9.40 |
| Dataset model | B - A pp [95% CI] | 2 - 1 pp [95% CI] |
|---|---|---|
| Humicroedit / Jev | 0.86 [0.78, 0.95]* | 0.94 [0.86, 1.02]* |
| Humicroedit / Clef | -0.27 [-0.43, -0.10]* | 0.05 [-0.12, 0.22] |
| Humicroedit / Perplexity | 1.35 [1.26, 1.44]* | 0.01 [-0.06, 0.09] |
| HS score / Jev | -0.16 [-0.23, -0.10]* | 0.14 [0.08, 0.21]* |
| HS score / Clef | 0.45 [0.31, 0.59]* | 0.19 [0.05, 0.34] |
| HS score / Perplexity | -0.89 [-0.95, -0.83]* | -0.20 [-0.25, -0.15]* |
| Dataset model | Human - AI pp [95% CI] | Reader high - low pp [95% CI] | Prior high - low pp [95% CI] |
|---|---|---|---|
| UKP / Jev | 7.03 [6.70, 7.36]* | 13.55 [12.96, 14.13]* | 2.66 [2.41, 2.90]* |
| UKP / Clef | 3.57 [3.07, 4.06]* | 4.77 [4.36, 5.19]* | 1.85 [1.52, 2.18]* |
| UKP / Perplexity | 9.02 [8.62, 9.42]* | 27.80 [26.78, 28.81]* | 3.80 [3.58, 4.01]* |
| HS pair / Jev | 3.38 [3.11, 3.65]* | 13.71 [13.19, 14.22]* | 2.99 [2.75, 3.22]* |
| HS pair / Clef | -0.24 [-0.48, 0.00] | 4.44 [4.16, 4.72]* | 2.60 [2.37, 2.83]* |
| HS pair / Perplexity | 3.78 [3.58, 3.97]* | 19.07 [18.62, 19.51]* | 4.98 [4.79, 5.17]* |
| Dataset model | Human - AI pp [95% CI] | Reader high - low pp [95% CI] | Prior high - low pp [95% CI] |
|---|---|---|---|
| Humicroedit / Jev | -0.17 [-0.27, -0.07]* | 3.95 [3.80, 4.11]* | 2.46 [2.37, 2.54]* |
| Humicroedit / Clef | 0.49 [0.34, 0.64]* | 7.32 [6.97, 7.67]* | 2.11 [1.92, 2.31]* |
| Humicroedit / Perplexity | -1.33 [-1.48, -1.19]* | 21.17 [20.75, 21.59]* | 19.12 [18.81, 19.43]* |
| HS score / Jev | 1.06 [0.98, 1.14]* | 16.47 [16.12, 16.81]* | 11.76 [11.50, 12.01]* |
| HS score / Clef | 1.57 [1.42, 1.73]* | 3.44 [3.25, 3.64]* | 1.57 [1.40, 1.74]* |
| HS score / Perplexity | 1.05 [0.98, 1.13]* | 10.26 [10.08, 10.44]* | 9.27 [9.09, 9.44]* |
| Dataset model | First high Agreement change pp [95% CI] | Second high Agreement change pp [95% CI] |
|---|---|---|
| UKP / Jev | 0.0 [-2.2, 2.2] | -2.3 [-4.2, -0.3] |
| UKP / Clef | 1.4 [-0.2, 3.0] | 0.0 [-1.9, 1.9] |
| UKP / Perplexity | -4.6 [-7.0, -2.2]* | -6.0 [-9.2, -2.9]* |
| HS pair / Jev | -0.6 [-2.2, 1.0] | -1.3 [-2.9, 0.3] |
| HS pair / Clef | -0.2 [-1.6, 1.1] | -1.2 [-2.6, 0.3] |
| HS pair / Perplexity | -1.2 [-3.0, 0.5] | -1.8 [-3.8, 0.2] |
| Dataset model | High feedback MAE change pp [95% CI] | Low feedback MAE change pp [95% CI] |
|---|---|---|
| Humicroedit / Jev | -0.4 [-0.5, -0.3]* | -2.9 [-3.1, -2.7]* |
| Humicroedit / Clef | 0.5 [0.3, 0.7]* | -3.9 [-4.3, -3.5]* |
| Humicroedit / Perplexity | 4.8 [4.3, 5.3]* | -0.1 [-0.6, 0.4] |
| HS score / Jev | 0.1 [0.0, 0.3] | 3.8 [3.2, 4.4]* |
| HS score / Clef | 0.2 [0.0, 0.4] | -0.5 [-0.7, -0.2]* |
| HS score / Perplexity | 0.3 [0.2, 0.5]* | 0.3 [-0.1, 0.7] |
| Dataset model | Target | Target probability pp [95% CI] | Target selection pp [95% CI] | Flips % |
|---|---|---|---|---|
| UKP / Jev | first | 1.33 [1.13, 1.53]* | 2.867 [1.769, 3.964] | 4.20 |
| UKP / Jev | second | -0.49 [-0.68, -0.29]* | -0.133 [-1.039, 0.773] | 2.93 |
| UKP / Clef | first | 2.09 [1.72, 2.46]* | 1.200 [0.006, 2.394] | 5.47 |
| UKP / Clef | second | 0.35 [-0.02, 0.71] | 0.000 [-1.184, 1.184] | 5.20 |
| UKP / Perplexity | first | 2.35 [2.12, 2.58]* | 3.267 [2.249, 4.284] | 3.67 |
| UKP / Perplexity | second | 2.37 [2.13, 2.61]* | 3.733 [2.609, 4.858] | 4.80 |
| Dataset model | Score delta pp [95% CI] | Mean abs. pp | Abs. >5 pp % | Human MAE delta pp [95% CI] |
|---|---|---|---|---|
| Humicroedit / Jev | -0.10 [-0.27, 0.07] | 2.45 | 11.87 | -0.02 [-0.19, 0.14] |
| Humicroedit / Clef | -2.90 [-3.16, -2.64]* | 4.46 | 35.07 | -0.82 [-1.09, -0.54]* |
| Humicroedit / Perplexity | -1.10 [-1.26, -0.94]* | 2.59 | 12.13 | -0.19 [-0.35, -0.03] |
| HS score / Jev | 0.65 [0.51, 0.78]* | 1.92 | 5.67 | 0.23 [0.10, 0.37]* |
| HS score / Clef | 0.92 [0.69, 1.16]* | 3.33 | 20.67 | 0.51 [0.28, 0.73]* |
| HS score / Perplexity | 0.83 [0.75, 0.91]* | 1.38 | 2.00 | -0.01 [-0.10, 0.08] |
| Dataset model | X position | X use % | Mean P(X) | Conditional A delta pp [95% CI] | Non-X flip % |
|---|---|---|---|---|---|
| UKP / Jev | Last | 4.13 | 0.124 | 0.903 [0.646, 1.159] | 4.17 |
| UKP / Jev | First | 6.20 | 0.179 | 1.734 [1.429, 2.040] | 4.34 |
| UKP / Jev | Middle | 7.20 | 0.174 | 2.274 [1.974, 2.574] | 4.67 |
| UKP / Clef | Last | 2.33 | 0.111 | -3.176 [-3.708, -2.644] | 5.67 |
| UKP / Clef | First | 2.27 | 0.111 | -2.911 [-3.428, -2.395] | 5.18 |
| UKP / Clef | Middle | 2.53 | 0.112 | -2.895 [-3.367, -2.423] | 4.51 |
| Dataset model | Native reference | N | X-last use % | Mean P(X) |
|---|---|---|---|---|
| UKP / Jev | NO_MAJORITY | 250 | 2.40 | 0.122 |
| UKP / Jev | X | 187 | 22.99 | 0.267 |
| UKP / Clef | NO_MAJORITY | 250 | 1.60 | 0.105 |
| UKP / Clef | X | 187 | 10.16 | 0.171 |
| UKP / Perplexity | NO_MAJORITY | 250 | 5.60 | 0.182 |
| UKP / Perplexity | X | 187 | 47.06 | 0.431 |
| Model | Variant | Score delta pp [95% CI] | Rank | Human MAE delta pp |
|---|---|---|---|---|
| Jev | Reverse | 2.47 [2.36, 2.57]* | 0.977 | 1.95 |
| Jev | 3 levels | -2.81 [-2.94, -2.69]* | 0.963 | -2.06 |
| Jev | 7 levels | -7.63 [-7.78, -7.48]* | 0.973 | -5.43 |
| Clef | Reverse | -3.86 [-4.07, -3.66]* | 0.984 | -2.80 |
| Clef | 3 levels | -3.55 [-3.77, -3.32]* | 0.984 | -1.79 |
| Clef | 7 levels | -5.19 [-5.41, -4.96]* | 0.978 | -3.57 |
| Model | Variant | Score delta pp [95% CI] | Rank | Human MAE delta pp |
|---|---|---|---|---|
| Jev | Reverse | 0.41 [0.30, 0.51]* | 0.995 | -0.11 |
| Jev | 3 levels | 2.19 [1.97, 2.40]* | 0.990 | 0.83 |
| Jev | 7 levels | 1.36 [1.26, 1.47]* | 0.996 | -0.21 |
| Clef | Reverse | 1.16 [1.02, 1.30]* | 0.980 | 0.25 |
| Clef | 3 levels | 4.69 [4.41, 4.98]* | 0.980 | 3.04 |
| Clef | 7 levels | -1.67 [-1.80, -1.54]* | 0.984 | -0.47 |
| Model | Variant | Score delta pp [95% CI] | Rank | Human MAE delta pp |
|---|---|---|---|---|
| Jev | Reverse | 1.99 [1.88, 2.11]* | 0.991 | 0.66 |
| Jev | 3 levels | 5.87 [5.64, 6.09]* | 0.982 | 3.97 |
| Jev | 7 levels | 2.58 [2.45, 2.71]* | 0.992 | 0.92 |
| Clef | Reverse | -0.49 [-0.71, -0.26]* | 0.903 | -0.39 |
| Clef | 3 levels | 7.94 [7.63, 8.24]* | 0.908 | 4.78 |
| Clef | 7 levels | -0.72 [-0.87, -0.57]* | 0.938 | -0.13 |
| Dataset model | Slope | SD ratio | Human mid % | Judge mid % | Human P90-P10 | Judge P90-P10 |
|---|---|---|---|---|---|---|
| Humicroedit / Jev | 0.202 | 0.576 | 20.40 | 58.73 | 0.533 | 0.254 |
| Humicroedit / Clef | 0.480 | 0.894 | 20.40 | 39.73 | 0.533 | 0.452 |
| Humicroedit / Perplexity | 0.257 | 0.678 | 20.40 | 38.67 | 0.533 | 0.337 |
| HS score / Jev | 0.462 | 0.981 | 31.80 | 28.73 | 0.600 | 0.555 |
| HS score / Clef | 0.198 | 0.704 | 31.80 | 18.67 | 0.600 | 0.381 |
| HS score / Perplexity | 0.559 | 0.939 | 31.80 | 20.87 | 0.600 | 0.521 |
| Dataset model | Unadjusted | Adjusted slope pp per length SD [95% CI] |
|---|---|---|
| UKP / Jev | 0.4630 | 10.913 [9.410, 12.417] |
| UKP / Clef | 0.3220 | 8.063 [6.099, 10.027] |
| UKP / Perplexity | 0.4797 | 10.740 [9.359, 12.120] |
| HS pair / Jev | 0.0639 | 0.443 [-1.008, 1.895] |
| HS pair / Clef | 0.3811 | 8.706 [7.399, 10.013] |
| HS pair / Perplexity | 0.1217 | 1.764 [0.544, 2.984] |
| Dataset model | Disp. vs margin | Disp. vs sensitivity | Disp. vs repeat SD | Clear abs. sensitivity pp | Disputed abs. sensitivity pp |
|---|---|---|---|---|---|
| UKP / Jev | -0.1008 | 0.0635 | 0.0956 | 7.31 | 7.76 |
| UKP / Clef | 0.0040 | 0.0278 | 0.0486 | 8.29 | 8.76 |
| UKP / Perplexity | -0.0783 | 0.0971 | 0.0011 | 6.05 | 6.93 |
| HS pair / Jev | -0.2327 | 0.1601 | 0.1841 | 6.51 | 7.81 |
| HS pair / Clef | -0.1052 | 0.0602 | 0.0163 | 6.38 | 6.78 |
| HS pair / Perplexity | -0.2693 | 0.2002 | -0.0117 | 5.04 | 6.32 |
| Dataset model | Disp. vs prob. var. | Disp. vs MAE | Disp. vs repeat SD | Clear abs. sensitivity pp | Disputed abs. sensitivity pp |
|---|---|---|---|---|---|
| Humicroedit / Jev | -0.0343 | -0.4648 | 0.0044 | 2.30 | 2.23 |
| Humicroedit / Clef | 0.0669 | -0.2645 | -0.0439 | 3.52 | 3.73 |
| Humicroedit / Perplexity | 0.1034 | -0.3196 | 0.0161 | 3.36 | 3.47 |
| HS score / Jev | 0.2807 | -0.0105 | 0.1591 | 1.10 | 1.34 |
| HS score / Clef | 0.1512 | -0.0055 | 0.0358 | 2.32 | 2.55 |
| HS score / Perplexity | 0.3446 | 0.1090 | -0.0623 | 1.13 | 1.39 |
| Model | Reference | Mean | Signed error | MAE | Pearson | Spearman |
|---|---|---|---|---|---|---|
| Jev | 3:5 pooled | 0.721 | 0.062 | 0.132 | 0.382 | 0.384 |
| Jev | Experts only | 0.693 | 0.090 | 0.143 | 0.597 | 0.561 |
| Jev | Crowd only | 0.737 | 0.046 | 0.172 | 0.061 | 0.088 |
| Jev | Equal group means | 0.715 | 0.068 | 0.126 | 0.478 | 0.462 |
| Clef | 3:5 pooled | 0.721 | 0.044 | 0.106 | 0.378 | 0.368 |
| Clef | Experts only | 0.693 | 0.072 | 0.135 | 0.592 | 0.556 |
| Dataset model | Magnitude r [95% CI] | Reader/prior event N | Intersect/union N | Jaccard |
|---|---|---|---|---|
| UKP / Jev | 0.374 [0.332, 0.414]* | 330 / 74 | 67 / 337 | 0.199 |
| UKP / Clef | 0.380 [0.312, 0.444]* | 152 / 75 | 35 / 192 | 0.182 |
| UKP / Perplexity | 0.396 [0.354, 0.436]* | 606 / 74 | 71 / 609 | 0.117 |
| HS pair / Jev | 0.585 [0.544, 0.624]* | 274 / 64 | 56 / 282 | 0.199 |
| HS pair / Clef | 0.387 [0.323, 0.447]* | 146 / 106 | 56 / 196 | 0.286 |
| HS pair / Perplexity | 0.596 [0.564, 0.625]* | 407 / 98 | 97 / 408 | 0.238 |
| Model contrast | Pooled pp [95% CI] | Q | I 2 % | 2 pp 2 |
|---|---|---|---|---|
| Jev / First-position | -3.53 [-9.03, 1.97] | 60.86 | 97.2 | 4.740 |
| Jev / B-label | 0.57 [-6.60, 7.75] | 1610.19 | 99.9 | 8.342 |
| Jev / Numeric | 5.35 [-1.68, 12.37] | 117.66 | 98.0 | 7.838 |
| Jev / Human-AI | 8.02 [-4.93, 20.96] | 931.96 | 99.9 | 27.091 |
| Jev / Reader | 13.35 [12.04, 14.65]* | 5.91 | 67.1 | 0.180 |
| Jev / Prior | 1.97 [-1.70, 5.65] | 337.62 | 99.4 | 2.170 |
| Model contrast | Pooled pp [95% CI] | Q | I 2 % | 2 pp 2 |
|---|---|---|---|---|
| Jev / B-A | 0.11 [-1.54, 1.75] | 572.89 | 99.7 | 0.438 |
| Jev / Numeric | 0.15 [-1.81, 2.10] | 760.95 | 99.8 | 0.619 |
| Jev / Human-AI | 0.76 [-1.28, 2.79] | 540.03 | 99.7 | 0.667 |
| Jev / Reader | 8.69 [-8.17, 25.55] | 4322.35 | 100.0 | 46.026 |
| Jev / Prior | 6.47 [-5.40, 18.34] | 4809.01 | 100.0 | 22.811 |
| Jev / Inject | 1.35 [-3.37, 6.07] | 896.84 | 99.8 | 3.599 |
| Model UKP contrast | Dyadic pp [95% CI] | Component pp [95% CI] | Bootstrap pp [95% CI] | Omit largest pp |
|---|---|---|---|---|
| Jev / First-position | -2.250 [-2.977, -1.524] | -2.250 [-3.147, -1.354] | [-3.223, -1.456] | -2.60 |
| Jev / B-label | 3.856 [3.655, 4.058] | 3.856 [3.646, 4.066] | [3.668, 4.090] | 3.88 |
| Jev / Human vs AI | 7.029 [6.696, 7.361] | 7.029 [6.727, 7.330] | [6.725, 7.321] | 7.03 |
| Jev / Reader high vs low | 13.545 [12.956, 14.135] | 13.545 [12.897, 14.194] | [12.972, 14.223] | 13.73 |
| Clef / First-position | 0.015 [-0.258, 0.288] | 0.015 [-0.253, 0.283] | [-0.250, 0.293] | 0.06 |
| Clef / B-label | -8.643 [-9.620, -7.665] | -8.643 [-9.564, -7.721] | [-9.647, -7.752] | -8.65 |
| Check | Quantity | Alternative estimate |
|---|---|---|
| Baseline snapshot | Jev: UKP position (pp) | -2.24 |
| Five-call mean | Jev: UKP position (pp) | -2.23 |
| Score reconstruction | Jev: Helpfulness reader (pp) | 16.52 |
| Band reweighting | Jev: Humor MAE (pp) | 25.93 |
| Lower-length subset | Jev: UKP position (pp) | -2.23 [-3.01, -1.46] |
| Component variance | Jev: UKP position (pp) | -2.25 [-3.15, -1.35] |
Bias in LLM-as-a-Judge
Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.
See all 128 papers