LLMs are increasingly used as automated evaluators to reduce the cost and time required for human annotation. However, the role of scoring-scheme design in the reliability of LLM-as-a-Judge evaluation remains underexplored. To address this problem, we introduce numerical scoring topology, which characterizes the structural design of a scoring scheme in terms of its scale width, granularity, offset, symmetry, and boundary placement. Building on this definition, we conduct the first systematic study of how numerical scoring topology influences zero-shot LLM-as-a-Judge evaluation across diverse benchmarks and judge models. Our results show that numerical scoring topology systematically affects judge scores, calibration, and robustness, particularly in judgment-intensive evaluations. Shifted and narrow ranges tend to induce some of the largest score deviations across many settings. We further find that stronger and larger judge models generally achieve higher discriminative validity and greater robustness to scoring-topology changes. These findings suggest that numerical scoring topology should be treated as an integral component of LLM-as-a-Judge evaluation and highlight the need for more principled and standardized scoring protocols.
Figures & tables
Figure 1: Overview of the scoring-topology sensitivity analysis. We keep the evaluation prompt, model output, evaluation criteria, and judge model fixed while varying only the numerical scoring topology. Scores are normalized and compared across representative benchmarks.
Scoring Scale
MATH-500
AIME
GPQA Main
LCB
Arena-Hard
IFEval
Unit-interval (0 to 1)
85.7 (-0.8)
67.4 (-3.3)
73.1 (-4.4)
84.6 (-0.5)
94.9 (-2.5)
80.2 (-3.0)
Likert-5 (0 to 5)
86.5 (+0.0)
70.7 (+0.0)
77.5 (+0.0)
85.1 (+0.0)
97.4 (+0.0)
83.2 (+0.0)
Likert-10 (0 to 10)
86.8 (+0.3)
69.7 (-1.0)
77.8 (+0.3)
82.9 (-2.2)
93.3 (-4.1)
81.4 (-1.8)
Linear-50 (0 to 50)
87.2 (+0.7)
71.3 (+0.6)
78.6 (+1.1)
84.3 (-0.8)
94.8 (-2.6)
82.5 (-0.7)
Linear-100 (0 to 100)
87.8 (+1.3)
71.0 (+0.3)
78.0 (+0.5)
84.1 (-1.0)
93.3 (-4.1)
82.5 (-0.7)
Linear-1000 (0 to 1000)
87.4 (+0.9)
70.2 (-0.5)
77.3 (-0.2)
81.3 (-3.8)
93.9 (-3.5)
80.8 (-2.4)
Table 1: Mean normalized judge scores across diverse scoring topologies. GPT-5-mini is used both to generate the benchmark responses and to evaluate them under all scoring topologies. All raw scores are normalized to a 0–100 scale for comparison. Values in parentheses denote the deviation relative to the Likert-5 baseline; red indicates an increase and blue indicates a decrease in normalized judge scores.
Scoring Scale
MATH-500
AIME
GPQA Main
LCB
Arena-Hard
IFEval
Disagreement Rate ( ∣Δ∣>10 )
Unit-interval (0 to 1)
9.0 (45)
20.8 (25)
20.8 (93)
9.7 (17)
4.8 (24)
28.1 (152)
Likert-10 (0 to 10)
4.8 (24)
10.0 (12)
6.9 (31)
8.0 (14)
6.2 (31)
20.0 (108)
Linear-50 (0 to 50)
4.8 (24)
6.7 (8)
7.6 (34)
8.6 (15)
6.4 (32)
21.4 (116)
Linear-100 (0 to 100)
5.0 (25)
9.2 (11)
7.6 (34)
5.7 (10)
6.6 (33)
24.2 (131)
Linear-1000 (0 to 1000)
5.8 (29)
8.3 (10)
7.1 (32)
10.3 (18)
6.2 (31)
20.7 (112)
Table 2: Disagreement rates across scoring topologies. GPT-5-mini generates the benchmark responses and evaluates the same responses under all scoring topologies. Values denote the percentage of instances whose normalized scores differ from the Likert-5 reference by more than δ∈{10,20} , with counts shown in parentheses. Higher rates indicate greater sensitivity to numerical scoring topology.
Benchmark
Avg >10
Avg >20
Judgment-Intensive Tasks
Arena-Hard
9.9
2.9
IFEval
26.0
13.9
WildBench
23.1
8.6
WritingBench
20.7
6.2
Group Average
19.9
7.9
Table 3: Average sensitivity to numerical scoring topology by task type using GPT-5-mini. Each value shows the disagreement rate relative to Likert-5, averaged over the twelve non-reference scoring topologies. Group averages are macro-averaged across the four benchmarks.
Factor
Scoring Scale
MATH
AIME
GPQA
Arena
LCB
IFEval
Avg >10
Avg >20
Width sweep
Baseline (0.74 to 1.48)
8.0
14.2
17.2
13.8
12.0
30.9
16.0
6.4
Width 0.74 to 2.48
9.0
7.5
17.9
8.4
10.9
27.7
13.6
6.7
Width 0.74 to 4.74
8.0
6.7
10.3
4.4
9.1
20.3
9.8
4.4
Width 0.74 to 8.74
7.4
7.5
10.3
4.6
7.4
23.8
10.2
5.1
Width 0.74 to 15.74
6.8
4.2
8.7
4.2
8.6
25.0
9.6
4.6
Round Max
1.0 to 2.0
9.2
9.2
17.0
7.6
14.9
27.2
14.2
8.8
Table 4: Ablation analysis of boundary alignment and score width using GPT-5-mini. Each value denotes the percentage of instances whose normalized score differs from the Linear-100 reference by more than 10 ( Avg>10 ) or 20 ( Avg>20 ). MATH, GPQA, and Arena refer to MATH-500, GPQA Main, and Arena-Hard, respectively. LCB denotes LiveCodeBench. The 0.74–1.48 baseline corresponds to the shifted range used in the main experiment. Bold values highlight large increases in disagreement under non-round maximum boundaries.
Scoring Range
MFS
Count
Prop.
Likert-5 (0–5)
5.0
1,801
78.9%
Non-round (3.7–4.7)
4.6
752
32.9%
Non-round (6.2–7.2)
7.0
1,335
58.5%
Table 5: Concentration of raw judge scores under different scoring ranges. MFS denotes the most frequently assigned score, and Prop. denotes its proportion across 2,284 evaluation instances.
Scoring Scale
MATH-500
AIME
GPQA Main
LCB
Arena-Hard
IFEval
Disagreement Rate ( ∣Δ∣>10 )
Unit-interval (0 to 1)
11.6 (+2.6)
20.0 (-0.8)
21.0 (+0.2)
16.6 (+6.9)
6.8 (+2.0)
32.5 (+4.4)
Likert-10 (0 to 10)
6.2 (+1.4)
9.2 (-0.8)
12.7 (+5.8)
10.9 (+2.9)
5.6 (-0.6)
27.9 (+7.9)
Linear-50 (0 to 50)
5.6 (+0.8)
8.3 (+1.7)
12.1 (+4.5)
13.1 (+4.6)
9.4 (+3.0)
26.8 (+5.4)
Linear-100 (0 to 100)
7.0 (+2.0)
8.3 (-0.8)
12.7 (+5.1)
14.3 (+8.6)
8.2 (+1.6)
28.1 (+3.9)
Linear-1000 (0 to 1000)
6.4 (+0.6)
9.2 (+0.8)
11.6 (+4.5)
14.9 (+4.6)
9.0 (+2.8)
28.3 (+7.6)
Table 6: Cross-lingual impact on evaluator calibration. Disagreement rates under Korean-language judging relative to the condition-specific Likert-5 reference. Values in parentheses denote percentage-point differences compared with English-language judging (red: increased instability, blue: decreased instability). The results indicate that Korean-language evaluation can increase sensitivity to numerical scoring topology.
Condition
MATH
LCB
Arena
IFEval
Max
EN → EN
7.5
12.0
10.2
26.3
33.9
KO → EN
1.8
13.0
9.3
29.9
31.9
EN → KO
10.6
17.6
15.6
33.9
21.9
KO → KO
4.1
14.9
14.4
39.0
23.0
EN → ZH
4.2
16.8
17.6
33.5
14.3
EN → JA
2.1
15.3
13.8
34.2
38.7
Table 7: Disagreement rates and maximum-score usage across language conditions using GPT-5-mini. The benchmark columns show ∣Δ∣>10 disagreement rates averaged across twelve non-reference topologies. Max shows maximum-score usage for the shifted range 0.74–1.48 across four benchmarks. EN → KO, EN → ZH and EN → JA denote English responses evaluated using Korean, Chinese, and Japanese judge prompts, respectively.
Judge Model
AUROC
∣Δ∣>10
∣Δ∣>20
GPT-5.4
0.934
12.7
4.9
MiniMax-M3
0.931
17.5
10.6
GPT-OSS-120B
0.930
16.7
2.5
DeepSeek-V4-Pro
0.919
13.7
12.1
GPT-5-mini
0.913
22.7
3.3
Qwen3.7-Plus
0.907
5.3
5.3
Table 8: Judge discriminative validity and sensitivity to numerical scoring topology. AUROC is computed on pooled GPQA Main and AIME, while ∣Δ∣>10 and ∣Δ∣>20 report cross-topology disagreement rates.
Judge Model
AUROC
∣Δ∣>10
∣Δ∣>20
GPT-5.4
0.934
12.7
4.9
GPT-5-mini
0.913
22.7
3.3
GPT-4o-mini
0.898
25.0
7.6
GPT-OSS-120B
0.930
16.7
2.5
GPT-OSS-20B
0.916
51.1
10.2
Llama-3.1-70B
0.888
94.0
84.0
Table 9: Within-group judge performance and topology sensitivity. AUROC is computed on pooled GPQA Main and AIME, while ∣Δ∣>10 and ∣Δ∣>20 report cross-topology disagreement rates.
Benchmark
Scoring Range
Baseline
Allowed List
Explicit Max
∣Δ∣>10
%Max
∣Δ∣>10
%Max
∣Δ∣>10
MATH-500
[6.2, 7.2]
64.8
72.4
25.4
95.4
3.2
[3.7, 4.7]
33.3
94.3
3.7
96.5
2.5
[0.74, 1.48]
8.8
95.7
2.1
97.0
2.0
Arena-Hard
[6.2, 7.2]
97.1
3.0
93.7
24.2
71.6
[3.7, 4.7]
94.5
14.1
83.9
35.2
62.8
Table 10: Effectiveness of prompt-based mitigation strategies across six benchmarks and three scoring ranges using GPT-5-mini. %Max denotes maximum-score usage, while ∣Δ∣>10 represents the disagreement rate relative to the Likert-5 reference.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Scoring Scale
WildBench
WritingBench
Unit-interval (0 to 1)
81.3 (-4.0)
86.0 (-3.9)
Likert-5 (0 to 5)
85.3 (+0.0)
89.9 (+0.0)
Likert-10 (0 to 10)
81.9 (-3.4)
84.5 (-5.4)
Linear-50 (0 to 50)
82.4 (-2.9)
87.1 (-2.8)
Linear-100 (0 to 100)
83.4 (-1.9)
85.9 (-4.1)
Linear-1000 (0 to 1000)
82.1 (-3.3)
86.3 (-3.6)
Appendix
Table 11: Performance on WildBench and WritingBench across numerical scoring topologies. GPT-5-mini is used both to generate responses and to evaluate them under all scoring topologies. All scores are normalized to a 0–100 scale. Values in parentheses denote deviations from the Likert-5 reference.
Scoring Scale
WildBench
WritingBench
∣Δ∣>10
∣Δ∣>20
∣Δ∣>10
∣Δ∣>20
Unit-interval (0 to 1)
18.8 (48)
8.6 (22)
18.7 (104)
6.7 (37)
Likert-10 (0 to 10)
13.7 (35)
3.9 (10)
9.0 (50)
1.4 (8)
Linear-50 (0 to 50)
16.8 (43)
3.5 (9)
16.4 (91)
3.1 (17)
Linear-100 (0 to 100)
17.2 (44)
4.7 (12)
14.1 (78)
3.8 (21)
Linear-1000 (0 to 1000)
18.0 (46)
9.0 (23)
15.7 (87)
4.3 (24)
Appendix
Table 12: Disagreement rates on WildBench and WritingBench across numerical scoring topologies using GPT-5-mini. Values report the percentage of instances whose normalized scores differ from the Likert-5 reference by more than 10 or 20 points, with counts shown in parentheses. Higher values indicate greater sensitivity to numerical scoring topology.
Factor
Scoring Range
MFS
Count
Prop.
Width Sweep
0.74–2.48
2.48
993
43.5%
0.74–4.74
4.74
751
32.9%
0.74–8.74
8.74
584
25.6%
0.74–15.74
15.74
959
42.0%
Round Max
1.0–2.0
2.0
1,876
82.1%
3.0–4.0
4.0
1,759
77.0%
Appendix
Table 13: Most frequently assigned raw scores across boundary-ablation settings using GPT-5-mini. MFS denotes the most frequently assigned score, and Prop. denotes the proportion of evaluations assigned that score. Results are aggregated across 2,284 instances from MATH-500, AIME, GPQA Main, Arena-Hard, LiveCodeBench, and IFEval. Bold rows indicate the two boundary configurations with the highest disagreement rates.
Figure 2: Raw-score distributions under representative stable and unstable numerical scoring ranges using GPT-5-mini. In the stable range [1.3,2.04] , the most frequent score coincides with the specified maximum, while an additional concentration at 2.00 remains close to the upper boundary after normalization. In contrast, the unstable ranges [3.7,4.7] and [6.2,7.2] concentrate on salient values below their specified maxima. In particular, 7.0 accounts for 58.5% of judgments in [6.2,7.2] but maps to only 80 on the normalized 0–100 scale. The dashed red and dotted green lines denote the specified maximum and the most frequent score, respectively.
Scoring Scale
MATH-500
LCB
Arena-Hard
IFEval
>10
>20
>10
>20
>10
>20
>10
>20
EN → KO
Unit-interval (0 to 1)
11.6
4.4
16.6
8.0
6.8
2.6
32.5
19.6
Likert-10 (0 to 10)
6.2
3.6
10.9
6.3
5.6
2.0
27.9
15.5
Linear-50 (0 to 50)
5.6
2.6
13.1
8.0
9.4
3.0
26.8
15.2
Linear-100 (0 to 100)
7.0
3.4
14.3
8.6
8.2
2.6
28.1
15.0
Appendix
Table 14: Cross-topology disagreement rates across response–evaluation language conditions using GPT-5-mini. Values report the percentage of instances whose normalized scores differ from the condition-specific Likert-5 reference by more than 10 or 20 points. Bold values indicate the highest disagreement rates within each benchmark and language condition.
Judge Model
Score Gap
MAD
GPT-5.4
75.1
3.7
MiniMax-M3
71.5
7.2
GPT-OSS-120B
82.9
8.3
DeepSeek-V4-Pro
80.8
8.3
GPT-5-mini
69.3
6.1
Qwen3.7-Plus
80.8
3.7
Appendix
Table 15: Supplementary score separation and cross-topology deviation across judge models. Score Gap denotes the difference between the mean normalized scores assigned to correct and incorrect responses under Likert-5 scoring. MAD shows the mean absolute difference between normalized Likert-5 and Shifted-Range scores. Higher Score Gap indicates greater mean score separation, whereas lower MAD indicates greater robustness to changes in numerical scoring topology.
Response Generator
Likert-5
Likert-10
Linear-100
Shifted Range
Narrow Range
Irregular Range II
MATH-500
GPT-5-mini
97.8
97.6
97.7
96.7
96.4
97.9
GPT-OSS-120B
96.0
96.0
96.0
94.4
94.9
96.0
GPT-OSS-20B
92.4
92.5
92.7
91.2
90.9
93.0
Gemma-3-12B
85.9
85.9
86.2
83.3
83.5
85.8
GPT-4o-mini
80.3
80.3
80.4
77.7
78.1
80.7
Appendix
Table 16: Mean normalized judge scores of eight response generators on MATH-500 and Arena-Hard across numerical scoring topologies. GPT-5-mini is used as the judge for all evaluations. Each value denotes the average score assigned by the judge to a response generator after converting the raw scores under each topology to a common 0–100 scale. The evaluated topologies include Likert-5 (0 to 5), Likert-10 (0 to 10), Linear-100 (0 to 100), Shifted Range (0.74 to 1.48), Narrow Range (0.3 to 0.7), and Irregular Range II ( −2.35 to 7.85). Although absolute judge scores vary across scoring topologies, the relative ordering of all eight response generators remains unchanged on both benchmarks.
Scoring Topology
MATH-500
Arena-Hard
Likert-10
−0.1
−3.4
Linear-100
+0.3
−0.7
Shifted Range
−2.7
−9.8
Narrow Range
−2.1
−9.4
Irregular Range II
+0.6
+0.1
Appendix
Table 17: Average normalized score shifts relative to Likert-5. Values show the mean score change relative to Likert-5 across the eight response generators in Table 16 . Negative values indicate lower average judge scores than Likert-5. The Shifted and Narrow Ranges produce the largest downward shifts on Arena-Hard, reducing the mean score by 9.8 and 9.4 points, respectively. Despite these shifts, the complete response-generator ranking is preserved on both benchmarks, with Kendall’s τ=1.00 and no top-1 or pairwise ranking flips.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.
Recent benchmark evaluations increasingly rely on Large Language Models (LLMs) as automatic judges. This paradigm assumes LLMs evaluate based on the rubric's semantic meaning rather than its numeric scale. We question this assumption by introducing semantic faithfulness: the principle that semantically equivalent rubrics must yield consistent scores. To quantify this, we propose a scale-reversal probe that inverts score anchors without altering evaluation criteria. Furthermore, we introduce floor normalization to calibrate reversal disagreement against intrinsic model variability, enabling fair cross-model comparisons. Experiments across five meta-evaluation benchmarks with seven diverse LLM judges spanning open-weight and proprietary models reveal widespread violations of semantic faithfulness, showing that raw reversal agreement alone is misleading. Finally, we demonstrate that bidirectional anchor averaging reduces dependence on any single anchor direction without compromising alignment with human judgments. Our findings emphasize that trustworthy LLM judges must reliably follow rubric semantics, not just correlate with human ratings.
Large Language Models (LLMs) are often used as evaluators of text quality, known as LLM-as-a-Judge, which can outperform conventional automatic evaluation metrics that rely on reference texts. However, LLM evaluators tend to generate particular scores regardless of the context of the evaluated text, which is known as scoring bias. This study proposes a novel method to mitigate this scoring bias. An LLM is instructed to randomly generate number tokens, and the latent numerical bias of the LLM is identified by measuring the deviation of the observed distribution of numbers from the uniform distribution. A definition of a downstream task, for which an LLM evaluator is used, is added to the prompts for random number generation to measure task-specific latent number bias. In the evaluation by an LLM, the token generation probabilities for a given input are rectified considering the LLM's latent number bias. Results of the experiment on four different tasks, evaluation of LLM alignment, evaluation of summarization, Semantic Textual Similarity, and Semantic Textual Relatedness, demonstrate that our proposed method outperforms the baselines, including an LLM without debiasing and previous calibration methods. In addition, it is confirmed that scoring bias varies across LLMs, tasks, and score ranges, indicating the importance of measuring latent number bias as the case may be.