Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faster and cheaper than human evaluation. However, a judge may produce consistent scores without necessarily agreeing with human evaluators. In this work, we study this issue using two local open-weight LLM judges, LLaMA-3-8B and Qwen2.5-7B. We evaluate 300 responses generated by an instruction-tuned GPT-2 (124M) model for 100 questions covering five categories: factual knowledge, instruction following, mathematics, reasoning, and writing. Each response is scored by nine human annotators and is evaluated three times by each LLM judge using the same rubric. We compare the judge scores with the average human scores using Pearson correlation, Spearman correlation, mean absolute error (MAE), signed bias, and self-consistency. LLaMA-3-8B shows a Pearson correlation of 0.275 with human scores, while Qwen2.5-7B achieves 0.340. Their MAEs are 27.71 and 18.64, respectively. Despite this limited agreement, both judges show high self-consistency, with exact consistency rates of 97.3% for LLaMA-3-8B and 92.3% for Qwen2.5-7B. These results show that high self-consistency does not necessarily indicate high agreement with human judgments. Our findings highlight the need to evaluate both consistency and human alignment when using local LLMs as automatic judges.
Figures & tables
Study
Evaluator
Evaluation Setting
Main Focus
Zheng et al. Zheng et al. (2023)
Strong LLM judges
MT-Bench and Chatbot Arena
Human preference alignment and judge biases
Liu et al. Liu et al. (2023)
GPT-4
NLG evaluation
Human-aligned reference-free evaluation
Huang et al. Huang et al. (2024)
Fine-tuned LLM judges
Multiple evaluation dimensions
Generalization and limitations of fine-tuned judges
Zhu et al. Zhu et al. (2023)
JudgeLM
LLM response evaluation
Scalable open-source judging and judge biases
Kim et al. Kim et al. (2023)
Prometheus
Rubric-based evaluation
Fine-grained and customizable evaluation
Ye et al. Ye et al. (2024)
FLASK
Skill-based LLM evaluation
Fine-grained alignment skill assessment
Table 1: Representative studies on LLM-based evaluation and LLM-as-a-Judge.
Figure 1: Overall study framework. One hundred questions are used to generate 300 responses with an instruction-tuned GPT-2 model. The responses are independently evaluated by nine human annotators and two local LLM judges using three repeated evaluations. The resulting scores are compared using agreement, error, consistency, category, decoding, and response-length analyses.
Metric
Value
Number of annotators
9
Mean pairwise Pearson r
0.967
Minimum pairwise Pearson r
0.939
Maximum pairwise Pearson r
0.990
ICC(2,k) absolute agreement
0.996
Table 2: Agreement among the nine human annotators.
Metric
LLaMA-3-8B
Qwen2.5-7B
Pearson correlation
0.275
0.340
Pearson 95% CI
[0.145, 0.405]
[0.178, 0.491]
Spearman correlation
0.258
0.303
Spearman 95% CI
[0.145, 0.367]
[0.194, 0.405]
MAE
27.71
18.64
MAE 95% CI
[25.67, 29.89]
[16.96, 20.45]
Table 3: Overall agreement between human ratings and automated judges. Confidence intervals are 95% bootstrap percentile intervals.
Figure 2: Comparison of human average scores with automated judge scores for LLaMA-3-8B and Qwen2.5-7B. Each point represents one generated response. The plots illustrate the limited human–judge agreement and the positive scoring tendency observed for both judges.
Figure 3: Self-consistency of LLaMA-3-8B and Qwen2.5-7B across repeated evaluations. The figure reports mean item-level standard deviation and exact consistency.
Category
LLaMA-3-8B
Qwen2.5-7B
Pearson r
MAE
Pearson r
MAE
Factual knowledge
0.592
25.79
0.678
19.46
Instruction following
0.462
24.64
0.154
16.39
Mathematics
0.026
35.95
0.506
17.32
Reasoning
0.076
25.52
0.130
18.34
Writing
0.156
26.64
0.132
21.71
Table 4: Human–judge agreement across question categories.
Setting
LLaMA-3-8B
Qwen2.5-7B
Pearson r
MAE
Pearson r
MAE
Low temperature
0.165
31.97
0.314
22.06
Medium temperature
0.372
26.81
0.436
19.13
High temperature
0.264
24.33
0.184
14.74
Table 5: Human–judge agreement across GPT-2 decoding configurations.
Metric
LLaMA-3-8B
Qwen2.5-7B
Judge–length Pearson r
0.091
-0.108
Judge–length p -value
0.1172
0.06285
Raw human–judge r
0.275
0.340
Length-controlled r
0.265
0.364
Raw − partial Δr
+0.010
-0.024
Table 6: Response-length analysis and length-controlled human–judge correlations.
Figure 4: Raw and response-length-controlled Pearson correlations between human ratings and automated judge scores. Controlling for response length produces only modest changes in the observed human–judge relationships.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.