LLMs are increasingly used as automated evaluators to reduce the cost and time required for human annotation. However, the role of scoring-scheme design in the reliability of LLM-as-a-Judge evaluation remains underexplored. To address this problem, we introduce numerical scoring topology, which characterizes the structural design of a scoring scheme in terms of its scale width, granularity, offset, symmetry, and boundary placement. Building on this definition, we conduct the first systematic study of how numerical scoring topology influences zero-shot LLM-as-a-Judge evaluation across diverse benchmarks and judge models. Our results show that numerical scoring topology systematically affects judge scores, calibration, and robustness, particularly in judgment-intensive evaluations. Shifted and narrow ranges tend to induce some of the largest score deviations across many settings. We further find that stronger and larger judge models generally achieve higher discriminative validity and greater robustness to scoring-topology changes. These findings suggest that numerical scoring topology should be treated as an integral component of LLM-as-a-Judge evaluation and highlight the need for more principled and standardized scoring protocols.
Figures & tables
Figure 1: Overview of the scoring-topology sensitivity analysis. We keep the evaluation prompt, model output, evaluation criteria, and judge model fixed while varying only the numerical scoring topology. Scores are normalized and compared across representative benchmarks.
Scoring Scale
MATH-500
AIME
GPQA Main
LCB
Arena-Hard
IFEval
Unit-interval (0 to 1)
85.7 (-0.8)
67.4 (-3.3)
73.1 (-4.4)
84.6 (-0.5)
94.9 (-2.5)
80.2 (-3.0)
Likert-5 (0 to 5)
86.5 (+0.0)
70.7 (+0.0)
77.5 (+0.0)
85.1 (+0.0)
97.4 (+0.0)
83.2 (+0.0)
Likert-10 (0 to 10)
86.8 (+0.3)
69.7 (-1.0)
77.8 (+0.3)
82.9 (-2.2)
93.3 (-4.1)
81.4 (-1.8)
Linear-50 (0 to 50)
87.2 (+0.7)
71.3 (+0.6)
78.6 (+1.1)
84.3 (-0.8)
94.8 (-2.6)
82.5 (-0.7)
Linear-100 (0 to 100)
87.8 (+1.3)
71.0 (+0.3)
78.0 (+0.5)
84.1 (-1.0)
93.3 (-4.1)
82.5 (-0.7)
Linear-1000 (0 to 1000)
87.4 (+0.9)
70.2 (-0.5)
77.3 (-0.2)
81.3 (-3.8)
93.9 (-3.5)
80.8 (-2.4)
Table 1: Mean normalized judge scores across diverse scoring topologies. GPT-5-mini is used both to generate the benchmark responses and to evaluate them under all scoring topologies. All raw scores are normalized to a 0–100 scale for comparison. Values in parentheses denote the deviation relative to the Likert-5 baseline; red indicates an increase and blue indicates a decrease in normalized judge scores.
Scoring Scale
MATH-500
AIME
GPQA Main
LCB
Arena-Hard
IFEval
Disagreement Rate ( ∣Δ∣>10 )
Unit-interval (0 to 1)
9.0 (45)
20.8 (25)
20.8 (93)
9.7 (17)
4.8 (24)
28.1 (152)
Likert-10 (0 to 10)
4.8 (24)
10.0 (12)
6.9 (31)
8.0 (14)
6.2 (31)
20.0 (108)
Linear-50 (0 to 50)
4.8 (24)
6.7 (8)
7.6 (34)
8.6 (15)
6.4 (32)
21.4 (116)
Linear-100 (0 to 100)
5.0 (25)
9.2 (11)
7.6 (34)
5.7 (10)
6.6 (33)
24.2 (131)
Linear-1000 (0 to 1000)
5.8 (29)
8.3 (10)
7.1 (32)
10.3 (18)
6.2 (31)
20.7 (112)
Table 2: Disagreement rates across scoring topologies. GPT-5-mini generates the benchmark responses and evaluates the same responses under all scoring topologies. Values denote the percentage of instances whose normalized scores differ from the Likert-5 reference by more than δ∈{10,20} , with counts shown in parentheses. Higher rates indicate greater sensitivity to numerical scoring topology.
Benchmark
Avg >10
Avg >20
Judgment-Intensive Tasks
Arena-Hard
9.9
2.9
IFEval
26.0
13.9
WildBench
23.1
8.6
WritingBench
20.7
6.2
Group Average
19.9
7.9
Table 3: Average sensitivity to numerical scoring topology by task type using GPT-5-mini. Each value shows the disagreement rate relative to Likert-5, averaged over the twelve non-reference scoring topologies. Group averages are macro-averaged across the four benchmarks.
Factor
Scoring Scale
MATH
AIME
GPQA
Arena
LCB
IFEval
Avg >10
Avg >20
Width sweep
Baseline (0.74 to 1.48)
8.0
14.2
17.2
13.8
12.0
30.9
16.0
6.4
Width 0.74 to 2.48
9.0
7.5
17.9
8.4
10.9
27.7
13.6
6.7
Width 0.74 to 4.74
8.0
6.7
10.3
4.4
9.1
20.3
9.8
4.4
Width 0.74 to 8.74
7.4
7.5
10.3
4.6
7.4
23.8
10.2
5.1
Width 0.74 to 15.74
6.8
4.2
8.7
4.2
8.6
25.0
9.6
4.6
Round Max
1.0 to 2.0
9.2
9.2
17.0
7.6
14.9
27.2
14.2
8.8
Table 4: Ablation analysis of boundary alignment and score width using GPT-5-mini. Each value denotes the percentage of instances whose normalized score differs from the Linear-100 reference by more than 10 ( Avg>10 ) or 20 ( Avg>20 ). MATH, GPQA, and Arena refer to MATH-500, GPQA Main, and Arena-Hard, respectively. LCB denotes LiveCodeBench. The 0.74–1.48 baseline corresponds to the shifted range used in the main experiment. Bold values highlight large increases in disagreement under non-round maximum boundaries.
Scoring Range
MFS
Count
Prop.
Likert-5 (0–5)
5.0
1,801
78.9%
Non-round (3.7–4.7)
4.6
752
32.9%
Non-round (6.2–7.2)
7.0
1,335
58.5%
Table 5: Concentration of raw judge scores under different scoring ranges. MFS denotes the most frequently assigned score, and Prop. denotes its proportion across 2,284 evaluation instances.
Scoring Scale
MATH-500
AIME
GPQA Main
LCB
Arena-Hard
IFEval
Disagreement Rate ( ∣Δ∣>10 )
Unit-interval (0 to 1)
11.6 (+2.6)
20.0 (-0.8)
21.0 (+0.2)
16.6 (+6.9)
6.8 (+2.0)
32.5 (+4.4)
Likert-10 (0 to 10)
6.2 (+1.4)
9.2 (-0.8)
12.7 (+5.8)
10.9 (+2.9)
5.6 (-0.6)
27.9 (+7.9)
Linear-50 (0 to 50)
5.6 (+0.8)
8.3 (+1.7)
12.1 (+4.5)
13.1 (+4.6)
9.4 (+3.0)
26.8 (+5.4)
Linear-100 (0 to 100)
7.0 (+2.0)
8.3 (-0.8)
12.7 (+5.1)
14.3 (+8.6)
8.2 (+1.6)
28.1 (+3.9)
Linear-1000 (0 to 1000)
6.4 (+0.6)
9.2 (+0.8)
11.6 (+4.5)
14.9 (+4.6)
9.0 (+2.8)
28.3 (+7.6)
Table 6: Cross-lingual impact on evaluator calibration. Disagreement rates under Korean-language judging relative to the condition-specific Likert-5 reference. Values in parentheses denote percentage-point differences compared with English-language judging (red: increased instability, blue: decreased instability). The results indicate that Korean-language evaluation can increase sensitivity to numerical scoring topology.
Condition
MATH
LCB
Arena
IFEval
Max
EN → EN
7.5
12.0
10.2
26.3
33.9
KO → EN
1.8
13.0
9.3
29.9
31.9
EN → KO
10.6
17.6
15.6
33.9
21.9
KO → KO
4.1
14.9
14.4
39.0
23.0
EN → ZH
4.2
16.8
17.6
33.5
14.3
EN → JA
2.1
15.3
13.8
34.2
38.7
Table 7: Disagreement rates and maximum-score usage across language conditions using GPT-5-mini. The benchmark columns show ∣Δ∣>10 disagreement rates averaged across twelve non-reference topologies. Max shows maximum-score usage for the shifted range 0.74–1.48 across four benchmarks. EN → KO, EN → ZH and EN → JA denote English responses evaluated using Korean, Chinese, and Japanese judge prompts, respectively.
Judge Model
AUROC
∣Δ∣>10
∣Δ∣>20
GPT-5.4
0.934
12.7
4.9
MiniMax-M3
0.931
17.5
10.6
GPT-OSS-120B
0.930
16.7
2.5
DeepSeek-V4-Pro
0.919
13.7
12.1
GPT-5-mini
0.913
22.7
3.3
Qwen3.7-Plus
0.907
5.3
5.3
Table 8: Judge discriminative validity and sensitivity to numerical scoring topology. AUROC is computed on pooled GPQA Main and AIME, while ∣Δ∣>10 and ∣Δ∣>20 report cross-topology disagreement rates.
Judge Model
AUROC
∣Δ∣>10
∣Δ∣>20
GPT-5.4
0.934
12.7
4.9
GPT-5-mini
0.913
22.7
3.3
GPT-4o-mini
0.898
25.0
7.6
GPT-OSS-120B
0.930
16.7
2.5
GPT-OSS-20B
0.916
51.1
10.2
Llama-3.1-70B
0.888
94.0
84.0
Table 9: Within-group judge performance and topology sensitivity. AUROC is computed on pooled GPQA Main and AIME, while ∣Δ∣>10 and ∣Δ∣>20 report cross-topology disagreement rates.
Benchmark
Scoring Range
Baseline
Allowed List
Explicit Max
∣Δ∣>10
%Max
∣Δ∣>10
%Max
∣Δ∣>10
MATH-500
[6.2, 7.2]
64.8
72.4
25.4
95.4
3.2
[3.7, 4.7]
33.3
94.3
3.7
96.5
2.5
[0.74, 1.48]
8.8
95.7
2.1
97.0
2.0
Arena-Hard
[6.2, 7.2]
97.1
3.0
93.7
24.2
71.6
[3.7, 4.7]
94.5
14.1
83.9
35.2
62.8
Table 10: Effectiveness of prompt-based mitigation strategies across six benchmarks and three scoring ranges using GPT-5-mini. %Max denotes maximum-score usage, while ∣Δ∣>10 represents the disagreement rate relative to the Likert-5 reference.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Scoring Scale
WildBench
WritingBench
Unit-interval (0 to 1)
81.3 (-4.0)
86.0 (-3.9)
Likert-5 (0 to 5)
85.3 (+0.0)
89.9 (+0.0)
Likert-10 (0 to 10)
81.9 (-3.4)
84.5 (-5.4)
Linear-50 (0 to 50)
82.4 (-2.9)
87.1 (-2.8)
Linear-100 (0 to 100)
83.4 (-1.9)
85.9 (-4.1)
Linear-1000 (0 to 1000)
82.1 (-3.3)
86.3 (-3.6)
Appendix
Table 11: Performance on WildBench and WritingBench across numerical scoring topologies. GPT-5-mini is used both to generate responses and to evaluate them under all scoring topologies. All scores are normalized to a 0–100 scale. Values in parentheses denote deviations from the Likert-5 reference.
Scoring Scale
WildBench
WritingBench
∣Δ∣>10
∣Δ∣>20
∣Δ∣>10
∣Δ∣>20
Unit-interval (0 to 1)
18.8 (48)
8.6 (22)
18.7 (104)
6.7 (37)
Likert-10 (0 to 10)
13.7 (35)
3.9 (10)
9.0 (50)
1.4 (8)
Linear-50 (0 to 50)
16.8 (43)
3.5 (9)
16.4 (91)
3.1 (17)
Linear-100 (0 to 100)
17.2 (44)
4.7 (12)
14.1 (78)
3.8 (21)
Linear-1000 (0 to 1000)
18.0 (46)
9.0 (23)
15.7 (87)
4.3 (24)
Appendix
Table 12: Disagreement rates on WildBench and WritingBench across numerical scoring topologies using GPT-5-mini. Values report the percentage of instances whose normalized scores differ from the Likert-5 reference by more than 10 or 20 points, with counts shown in parentheses. Higher values indicate greater sensitivity to numerical scoring topology.
Factor
Scoring Range
MFS
Count
Prop.
Width Sweep
0.74–2.48
2.48
993
43.5%
0.74–4.74
4.74
751
32.9%
0.74–8.74
8.74
584
25.6%
0.74–15.74
15.74
959
42.0%
Round Max
1.0–2.0
2.0
1,876
82.1%
3.0–4.0
4.0
1,759
77.0%
Appendix
Table 13: Most frequently assigned raw scores across boundary-ablation settings using GPT-5-mini. MFS denotes the most frequently assigned score, and Prop. denotes the proportion of evaluations assigned that score. Results are aggregated across 2,284 instances from MATH-500, AIME, GPQA Main, Arena-Hard, LiveCodeBench, and IFEval. Bold rows indicate the two boundary configurations with the highest disagreement rates.
Figure 2: Raw-score distributions under representative stable and unstable numerical scoring ranges using GPT-5-mini. In the stable range [1.3,2.04] , the most frequent score coincides with the specified maximum, while an additional concentration at 2.00 remains close to the upper boundary after normalization. In contrast, the unstable ranges [3.7,4.7] and [6.2,7.2] concentrate on salient values below their specified maxima. In particular, 7.0 accounts for 58.5% of judgments in [6.2,7.2] but maps to only 80 on the normalized 0–100 scale. The dashed red and dotted green lines denote the specified maximum and the most frequent score, respectively.
Scoring Scale
MATH-500
LCB
Arena-Hard
IFEval
>10
>20
>10
>20
>10
>20
>10
>20
EN → KO
Unit-interval (0 to 1)
11.6
4.4
16.6
8.0
6.8
2.6
32.5
19.6
Likert-10 (0 to 10)
6.2
3.6
10.9
6.3
5.6
2.0
27.9
15.5
Linear-50 (0 to 50)
5.6
2.6
13.1
8.0
9.4
3.0
26.8
15.2
Linear-100 (0 to 100)
7.0
3.4
14.3
8.6
8.2
2.6
28.1
15.0
Appendix
Table 14: Cross-topology disagreement rates across response–evaluation language conditions using GPT-5-mini. Values report the percentage of instances whose normalized scores differ from the condition-specific Likert-5 reference by more than 10 or 20 points. Bold values indicate the highest disagreement rates within each benchmark and language condition.
Judge Model
Score Gap
MAD
GPT-5.4
75.1
3.7
MiniMax-M3
71.5
7.2
GPT-OSS-120B
82.9
8.3
DeepSeek-V4-Pro
80.8
8.3
GPT-5-mini
69.3
6.1
Qwen3.7-Plus
80.8
3.7
Appendix
Table 15: Supplementary score separation and cross-topology deviation across judge models. Score Gap denotes the difference between the mean normalized scores assigned to correct and incorrect responses under Likert-5 scoring. MAD shows the mean absolute difference between normalized Likert-5 and Shifted-Range scores. Higher Score Gap indicates greater mean score separation, whereas lower MAD indicates greater robustness to changes in numerical scoring topology.
Response Generator
Likert-5
Likert-10
Linear-100
Shifted Range
Narrow Range
Irregular Range II
MATH-500
GPT-5-mini
97.8
97.6
97.7
96.7
96.4
97.9
GPT-OSS-120B
96.0
96.0
96.0
94.4
94.9
96.0
GPT-OSS-20B
92.4
92.5
92.7
91.2
90.9
93.0
Gemma-3-12B
85.9
85.9
86.2
83.3
83.5
85.8
GPT-4o-mini
80.3
80.3
80.4
77.7
78.1
80.7
Appendix
Table 16: Mean normalized judge scores of eight response generators on MATH-500 and Arena-Hard across numerical scoring topologies. GPT-5-mini is used as the judge for all evaluations. Each value denotes the average score assigned by the judge to a response generator after converting the raw scores under each topology to a common 0–100 scale. The evaluated topologies include Likert-5 (0 to 5), Likert-10 (0 to 10), Linear-100 (0 to 100), Shifted Range (0.74 to 1.48), Narrow Range (0.3 to 0.7), and Irregular Range II ( −2.35 to 7.85). Although absolute judge scores vary across scoring topologies, the relative ordering of all eight response generators remains unchanged on both benchmarks.
Scoring Topology
MATH-500
Arena-Hard
Likert-10
−0.1
−3.4
Linear-100
+0.3
−0.7
Shifted Range
−2.7
−9.8
Narrow Range
−2.1
−9.4
Irregular Range II
+0.6
+0.1
Appendix
Table 17: Average normalized score shifts relative to Likert-5. Values show the mean score change relative to Likert-5 across the eight response generators in Table 16 . Negative values indicate lower average judge scores than Likert-5. The Shifted and Narrow Ranges produce the largest downward shifts on Arena-Hard, reducing the mean score by 9.8 and 9.4 points, respectively. Despite these shifts, the complete response-generator ranking is preserved on both benchmarks, with Kendall’s τ=1.00 and no top-1 or pairwise ranking flips.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.