cs.CLNov 19, 2026

Same Judgement, Different Scores: Numerical Scoring Topology in LLM-as-a-Judge Evaluation

Authors: Doyun Kim, Chanwoo Kim, Jiwoo Kum, Minsoo Song, Chanjun Park

Organizations: Soongsil University

Abstract

LLMs are increasingly used as automated evaluators to reduce the cost and time required for human annotation. However, the role of scoring-scheme design in the reliability of LLM-as-a-Judge evaluation remains underexplored. To address this problem, we introduce numerical scoring topology, which characterizes the structural design of a scoring scheme in terms of its scale width, granularity, offset, symmetry, and boundary placement. Building on this definition, we conduct the first systematic study of how numerical scoring topology influences zero-shot LLM-as-a-Judge evaluation across diverse benchmarks and judge models. Our results show that numerical scoring topology systematically affects judge scores, calibration, and robustness, particularly in judgment-intensive evaluations. Shifted and narrow ranges tend to induce some of the largest score deviations across many settings. We further find that stronger and larger judge models generally achieve higher discriminative validity and greater robustness to scoring-topology changes. These findings suggest that numerical scoring topology should be treated as an integral component of LLM-as-a-Judge evaluation and highlight the need for more principled and standardized scoring protocols.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

    Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM EvaluationLLM-as-a-Judge

  2. Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation

    Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM EvaluationLLM-as-a-Judge