cs.CLNov 19, 2026

Rubric Semantics or Numerical Priors? Auditing Scale-Direction Bias in LLM Judges

Authors: Chanwoo Kim, Doyun Kim, Minsoo Song, Sihyeon Lee, Chanjun Park

Organizations: Soongsil University

Abstract

Recent benchmark evaluations increasingly rely on Large Language Models (LLMs) as automatic judges. This paradigm assumes LLMs evaluate based on the rubric's semantic meaning rather than its numeric scale. We question this assumption by introducing semantic faithfulness: the principle that semantically equivalent rubrics must yield consistent scores. To quantify this, we propose a scale-reversal probe that inverts score anchors without altering evaluation criteria. Furthermore, we introduce floor normalization to calibrate reversal disagreement against intrinsic model variability, enabling fair cross-model comparisons. Experiments across five meta-evaluation benchmarks with seven diverse LLM judges spanning open-weight and proprietary models reveal widespread violations of semantic faithfulness, showing that raw reversal agreement alone is misleading. Finally, we demonstrate that bidirectional anchor averaging reduces dependence on any single anchor direction without compromising alignment with human judgments. Our findings emphasize that trustworthy LLM judges must reliably follow rubric semantics, not just correlate with human ratings.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

    Jan 13, 2026Yihan Hong, Huaiyuan Yao, Bolin Shen +3LLM-as-a-JudgeRubric-Based Evaluation

  2. Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

    Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-JudgeRubric-Based Evaluation