Rubric Semantics or Numerical Priors? Auditing Scale-Direction Bias in LLM Judges
Organizations: Soongsil University
Abstract
Recent benchmark evaluations increasingly rely on Large Language Models (LLMs) as automatic judges. This paradigm assumes LLMs evaluate based on the rubric's semantic meaning rather than its numeric scale. We question this assumption by introducing semantic faithfulness: the principle that semantically equivalent rubrics must yield consistent scores. To quantify this, we propose a scale-reversal probe that inverts score anchors without altering evaluation criteria. Furthermore, we introduce floor normalization to calibrate reversal disagreement against intrinsic model variability, enabling fair cross-model comparisons. Experiments across five meta-evaluation benchmarks with seven diverse LLM judges spanning open-weight and proprietary models reveal widespread violations of semantic faithfulness, showing that raw reversal agreement alone is misleading. Finally, we demonstrate that bidirectional anchor averaging reduces dependence on any single anchor direction without compromising alignment with human judgments. Our findings emphasize that trustworthy LLM judges must reliably follow rubric semantics, not just correlate with human ratings.
Figures & tables
| SummEval | BiGGen-Bench | MT-Bench ( – ) | JudgeBench | RewardBench | |||||||||||
| Judge | |||||||||||||||
| Gemma-4-31B | 0.808 | 0.977 | 0.169 | 0.847 | 0.970 | 0.123 | 0.623 | 0.956 | 0.333 | 0.771 | 0.911 | 0.140 | 0.749 | 0.971 | 0.222 |
| gpt-oss-120B | 0.705 | 0.782 | 0.077 | 0.788 | 0.832 | 0.044 | 0.575 | 0.699 | 0.124 | 0.757 | 0.779 | 0.022 | 0.726 | 0.807 | 0.082 |
| Qwen3.5-122B | 0.778 | 0.827 | 0.049 | 0.799 | 0.863 | 0.064 | 0.515 | 0.665 | 0.150 | 0.782 | 0.797 | 0.015 | 0.701 | 0.763 | 0.063 |
| Judge | |||
| normal avg | |||
| SummEval | |||
| Gemma-4-31B | 0.963 | +0.015 | |
| gpt-oss-120B | 0.920 | +0.019 | |
| Qwen3.5-122B | 0.946 | +0.021 | |
| Qwen3.5-27B | 0.912 | +0.019 | |
| Judge | Accuracy | [95% CI] |
| normal avg | ||
| MT-Bench | ||
| Gemma-4-31B | [ , ] | |
| gpt-oss-120B | [ , ] | |
| Qwen3.5-122B | [ , ] | |
| Qwen3.5-27B | [ , ] | |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Measure | Estimate [95% CI] |
|---|---|
| +90.23 [+88.30, +92.06] | |
| +86.62 [+84.73, +88.49] | |
| +65.04 [+61.42, +68.72] | |
| +25.19 [+21.26, +29.21] | |
| +23.39 [+19.44, +27.45] |
| Benchmark | Judge | NR-NN [95% CI] | NR-RR [95% CI] |
|---|---|---|---|
| SummEval | Gemma-4-31B | ||
| SummEval | gpt-oss-120B | ||
| SummEval | Qwen3.5-122B | ||
| SummEval | Qwen3.5-27B | ||
| BiGGen-Bench | Gemma-4-31B | ||
| BiGGen-Bench | gpt-oss-120B |
| Rule | Passes | Averaged score |
|---|---|---|
| NN | Two normal | |
| NR | Mixed | |
| RR | Two reversed |
| Judge | ||
|---|---|---|
| SummEval | ||
| Gemma-4-31B | ||
| gpt-oss-120b | ||
| Qwen3.5-122B | ||
| Qwen3.5-27B | ||
| gpt-4o-mini | ||
| Judge | [95% CI] | [95% CI] |
|---|---|---|
| SummEval | ||
| Gemma-4-31B | [ , ] | [ , ] |
| gpt-oss-120b | [ , ] | [ , ] |
| Qwen3.5-122B | [ , ] | [ , ] |
| Qwen3.5-27B | [ , ] | [ , ] |
| gpt-4o-mini | [ , ] | [ , ] |
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 610 papers