No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

Michael Krumdick; Charles Lovering; Varshini Reddy; Seth Ebner; Chris Tanner

cs.CLMar 7, 2025

No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

Authors: Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, Chris Tanner

Organizations: Kensho Technologies · MIT

Abstract

Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-Judge framework, which uses prompted LLMs to evaluate response quality, is appealing due to its scalability, low cost, and strong correlations with human stylistic preferences. However, it remains unclear how accurately these methods can assess response quality in domains where correctness matters more than style. To address this gap, we introduce the Business and Finance Fundamentals Benchmark (BFF-Bench), a dataset of 160 challenging questions and long-form responses authored by financial professionals. These experts subsequently evaluated the correctness of 1,200 responses generated by a diverse set of LLMs on both BFF-Bench and a challenging subset of MT-Bench. With this expert-annotated dataset of judgments (VERDICTS), we analyze the agreement between a suite of automated grading methods and human experts. While we observe that LLM Judges are more reliable than other grading methods, our findings reveal a clear pattern in LLM Judge performance: when not provided with a correct reference, judges show high agreement with human experts only on questions the judges were able to correctly answer themselves. We demonstrate that providing the judges with expert-written references largely mitigates this issue, highlighting the limits of using LLM-as-a-Judge without any form of human verification.

Figures & tables

Figure 1: LLM Judge agreement with human annotators for the pairwise correctness judgment task. The left plot shows agreement when using a LLM-as-a-Judge on questions the LLM is able to answer correctly, whereas the right plot shows agreement on questions it answers incorrectly. None means the LLM Judge is provided with no reference response, Self means the LLM Judge is provided an unverified reference generated by the judge model itself, and Human means the LLM Judge is provided a human-written gold (correct) reference. When judge LLMs are not provided a correct reference, their agreement with human judgments decreases substantially for responses to questions they cannot correctly answer. However, providing a correct reference recovers much of this agreement.

Panel of Figure 2 — Figure 2: Truncated example of a question from BFF-Bench. For each question, we include a human-written gold answer that contains the final answer and a complete reasoning chain. The full version of this example is shown in Figure 7 in Appendix D .

	Benchmark		VERDICTS		Grading
Dataset	Convs.	Turns	Responses	Annotations	Single	Pairwise
BFF-Bench	80	160	960	2880	808	442
(C)MT-Bench	20	40	240	720	228	162
Total	100	200	1200	3600	1036	604

Table 1: Dataset statistics for the benchmarks and the response grading dataset. The Single and Pairwise columns denote the number of single grading and pairwise grading examples after filtering for annotator consensus.

Figure 3: LLM Judge agreement (Cohen’s

\kappa

) with human annotators across the pairwise and single grading cases for the three types of reference (human-written, judge-generated, or no reference). For every judge model in every setting, we observe an increase in agreement with the human annotators when using domain expert-written references.

Table 5

	As Candidate,	Pairwise Grading			Single Grading
Judge	Judge was:	None	Self	Human	None	Self	Human
GPT-4o	Correct	$0.78_{\pm 0.02}$	$0.86_{\pm 0.02}$	$\textbf{0.92}_{\pm 0.02}$	$0.46_{\pm 0.07}$	$0.52_{\pm 0.06}$	$0.59_{\pm 0.06}$
GPT-4o	Incorrect	$0.30_{\pm 0.09}$	$0.16_{\pm 0.08}$	$\textbf{0.83}_{\pm 0.06}$	$0.16_{\pm 0.14}$	$0.13_{\pm 0.14}$	$\textbf{0.81}_{\pm 0.13}$
Llama 3.3 70b	Correct	$0.79_{\pm 0.03}$	$0.86_{\pm 0.03}$	$\textbf{0.93}_{\pm 0.02}$	$0.61_{\pm 0.08}$	$0.54_{\pm 0.08}$	$0.73_{\pm 0.07}$
Llama 3.3 70b	Incorrect	$0.39_{\pm 0.06}$	$0.35_{\pm 0.07}$	$\textbf{0.92}_{\pm 0.03}$	$0.13_{\pm 0.06}$	$0.23_{\pm 0.10}$	$\textbf{0.69}_{\pm 0.09}$
Phi 4	Correct	$0.67_{\pm 0.07}$	$0.88_{\pm 0.04}$	$0.91_{\pm 0.04}$	$0.48_{\pm 0.09}$	$0.47_{\pm 0.07}$	$\textbf{0.67}_{\pm 0.07}$

Table 4: Judge agreement with human experts (Cohen’s

\kappa

). Results in bold indicate cases in which a reference was better than the others. Results in red indicate cases in which the model was worse when it was ruled “Correct” or “Incorrect.”

Figure 4: Error analysis for the judge models conditioned on the candidate model being graded. All models are included both as candidates and judges. Itself (blue) refers to cases in which the judge model graded responses from itself (as a candidate). Others (red) refers to cases in which the judge model graded responses from other models.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Single Grading
	Overall			Judge Answered Correctly			Judge Answered Incorrectly
	None	Self	Human	None	Self	Human	None	Self	Human
GPT-4o	$0.77_{\pm 0.03}$	$0.77_{\pm 0.03}$	$\textbf{0.85}_{\pm 0.02}$	$0.83_{\pm 0.03}$	$0.83_{\pm 0.03}$	$0.86_{\pm 0.02}$	$0.34_{\pm 0.13}$	$0.30_{\pm 0.13}$	$\textbf{0.84}_{\pm 0.12}$
Llama 3.3 70b	$0.81_{\pm 0.02}$	$0.80_{\pm 0.03}$	$\textbf{0.91}_{\pm 0.02}$	$0.93_{\pm 0.02}$	$0.88_{\pm 0.02}$	$0.94_{\pm 0.02}$	$0.39_{\pm 0.07}$	$0.41_{\pm 0.09}$	$\textbf{0.76}_{\pm 0.09}$
Phi 4	$0.78_{\pm 0.02}$	$0.74_{\pm 0.03}$	$\textbf{0.88}_{\pm 0.02}$	$0.92_{\pm 0.02}$	$0.84_{\pm 0.03}$	$0.92_{\pm 0.02}$	$0.32_{\pm 0.07}$	$0.22_{\pm 0.10}$	$\textbf{0.65}_{\pm 0.10}$
Yi 1.5 34B	$0.76_{\pm 0.03}$	$0.69_{\pm 0.03}$	$\textbf{0.83}_{\pm 0.02}$	$0.87_{\pm 0.03}$	$0.80_{\pm 0.03}$	$0.88_{\pm 0.03}$	$0.46_{\pm 0.07}$	$0.37_{\pm 0.08}$	$\textbf{0.75}_{\pm 0.07}$

Appendix

Table 5: Overall results using F1 instead of Cohen’s

\kappa

. Cells in red represent cases in which the model is worse when it was wrong as a candidate (

\alpha<0.05

). Numbers in bold indicate the maximum values on that subset with statistical significance (

\alpha<0.05

	Prompt One
	Overall			Judge Answered Correctly			Judge Answered Incorrectly
	None	Self	Human	None	Self	Human	None	Self	Human
GPT-4o	$0.47_{\pm 0.05}$	$0.52_{\pm 0.05}$	$\textbf{0.68}_{\pm 0.04}$	$0.46_{\pm 0.07}$	$0.52_{\pm 0.06}$	$0.59_{\pm 0.06}$	$0.16_{\pm 0.14}$	$0.13_{\pm 0.15}$	$\textbf{0.81}_{\pm 0.13}$
Llama 3.3 70b	$0.42_{\pm 0.05}$	$0.53_{\pm 0.05}$	$\textbf{0.78}_{\pm 0.04}$	$0.62_{\pm 0.09}$	$0.54_{\pm 0.08}$	$0.74_{\pm 0.07}$	$0.13_{\pm 0.06}$	$0.23_{\pm 0.11}$	$\textbf{0.69}_{\pm 0.10}$
Phi 4	$0.27_{\pm 0.05}$	$0.47_{\pm 0.05}$	$\textbf{0.72}_{\pm 0.04}$	$0.48_{\pm 0.10}$	$0.47_{\pm 0.07}$	$\textbf{0.67}_{\pm 0.07}$	$0.03_{\pm 0.05}$	$0.03_{\pm 0.11}$	$\textbf{0.56}_{\pm 0.12}$
Yi 1.5 34B	$0.38_{\pm 0.06}$	$0.37_{\pm 0.05}$	$\textbf{0.63}_{\pm 0.05}$	$0.23_{\pm 0.11}$	$0.29_{\pm 0.07}$	$0.43_{\pm 0.09}$	$0.18_{\pm 0.10}$	$0.14_{\pm 0.10}$	$\textbf{0.65}_{\pm 0.08}$

Appendix

Table 6: Performance comparison on single grading using the two sets of prompts. Cells in red represent cases in which the model is worse when it was wrong as a candidate (

\alpha<0.05

). Numbers in bold indicate the maximum values on that subset with statistical significance (

\alpha<0.05

Panel of Figure 5 — Figure 5: Example of an incorrect reference included in MT-Bench ( 3 should be $-3$ ). The correct answer is that the area of the triangle is 3.

Figure 6: The full graph displaying the error rates for different LLM Judges and different reference types. The same trend displayed in Figure 4 holds for each model. Rand. refers to the Random reference type.

Panel of Figure 7 — Figure 7: Untruncated example from BFF-Bench.

Panel of Figure 8 — Figure 8: Untruncated example from BFF-Bench.

	Cohen’s $\kappa$
Reward Models
INF-ORM-Llama3.1-70B	$0.44_{\pm 0.07}$
Llama-3.1-Nemotron-70B-Reward	$0.37_{\pm 0.07}$
LDL-Reward-Gemma-2-27B-v0.1	$0.31_{\pm 0.08}$
Skywork-Reward-Gemma-2-27B-v0.2	$0.30_{\pm 0.07}$
QRM-Gemma-2-27B	$0.24_{\pm 0.08}$

Appendix

Table 7: Full results table for all baselines on the pairwise grading task.

Ref	Stratum	$\mathbf{T=0.0}$	$\mathbf{T=0.2}$	$\mathbf{T=0.7}$
None	Correct	$0.39_{\pm 0.07}$	$0.45_{\pm 0.07}$	$0.42_{\pm 0.07}$
None	Incorrect	$0.25_{\pm 0.15}$	$0.16_{\pm 0.14}$	$0.15_{\pm 0.14}$
Self	Correct	$0.40_{\pm 0.06}$	$0.40_{\pm 0.06}$	$0.49_{\pm 0.06}$
Self	Incorrect	$0.21_{\pm 0.16}$	$0.22_{\pm 0.16}$	$0.13_{\pm 0.14}$
Human	Correct	$0.49_{\pm 0.06}$	$0.49_{\pm 0.05}$	$0.51_{\pm 0.05}$
Human	Incorrect	$0.72_{\pm 0.17}$	$0.72_{\pm 0.16}$	$0.69_{\pm 0.17}$

Appendix

Table 8: Ablation of temperature for the GPT-4o Judge. The LLM Judge performance is not sensitive to the temperature for this model.

Config	None	Self	Human
T=0.0 (CoT)	$0.44_{\pm 0.06}$	$0.44_{\pm 0.05}$	$0.59_{\pm 0.05}$
T=0.2 (CoT)	$0.46_{\pm 0.06}$	$0.43_{\pm 0.06}$	$0.59_{\pm 0.05}$
T=0.7 (CoT)	$0.44_{\pm 0.06}$	$0.50_{\pm 0.06}$	$0.60_{\pm 0.05}$
T=0.7 (No-CoT)	$0.46_{\pm 0.06}$	$0.47_{\pm 0.06}$	$0.59_{\pm 0.05}$

Appendix

Table 9: Ablation of chain-of-thought for the GPT-4o Judge. We do not observe significant shifts in performance with the addition of chain-of-thought.

	Pairwise			Single
Stratum	None	Self	Human	None	Self	Human
Majority Correct	$0.51_{\pm 0.07}$	$0.75_{\pm 0.07}$	$0.81_{\pm 0.05}$	$0.36_{\pm 0.10}$	$0.51_{\pm 0.08}$	$0.58_{\pm 0.08}$
Majority Incorrect	$0.40_{\pm 0.11}$	$0.46_{\pm 0.14}$	$0.85_{\pm 0.09}$	$0.07_{\pm 0.06}$	$0.01_{\pm 0.12}$	$0.51_{\pm 0.16}$

Appendix

Table 10: Majority Jury performance using all five LLM Judges, conditioned on whether the majority were marked as correct or incorrect.

Single BFF-Bench
Reference	$\kappa_{\text{Correct}}$	$\kappa_{\text{Incorrect}}$	$\Delta\kappa$	95% CI
None	0.333	0.085	+0.249	[+0.19, +0.31]
Self	0.430	0.160	+0.270	[+0.20, +0.34]
Human	0.512	0.531	$-0.019$	[ $-0.08$ , +0.05]

Appendix

Table 11: Agreement between the LLM judges and human annotators, conditioned on whether the judge answered the question correctly.

\kappa_{\text{Correct}}

and

\kappa_{\text{Incorrect}}

denote Cohen’s

\kappa

computed on the subsets of questions where the judge was respectively correct or incorrect.

\Delta\kappa=\kappa_{\text{Correct}}-\kappa_{\text{Incorrect}}

. Confidence intervals are computed via bootstrap.

	As Candidate,	Pairwise Grading			Single Grading
Judge	Judge was:	None	Self	Human	None	Self	Human
Opus 4.7	Correct	$0.86_{\pm 0.04}$	$0.90_{\pm 0.04}$	$0.91_{\pm 0.04}$	$0.63_{\pm 0.06}$	$0.69_{\pm 0.06}$	$0.77_{\pm 0.05}$
Opus 4.7	Incorrect	$0.33_{\pm 0.16}$	$0.33_{\pm 0.15}$	$0.66_{\pm 0.13}$	$0.26_{\pm 0.14}$	$0.27_{\pm 0.12}$	$0.62_{\pm 0.11}$
GPT-5.4	Correct	$0.75_{\pm 0.06}$	$0.87_{\pm 0.04}$	$0.89_{\pm 0.04}$	$0.51_{\pm 0.06}$	$0.64_{\pm 0.06}$	$0.66_{\pm 0.06}$
GPT-5.4	Incorrect	$0.46_{\pm 0.11}$	$0.45_{\pm 0.13}$	$0.87_{\pm 0.08}$	$0.32_{\pm 0.11}$	$0.25_{\pm 0.12}$	$0.67_{\pm 0.09}$
Gemini 3.1	Correct	$0.78_{\pm 0.06}$	$0.80_{\pm 0.06}$	$0.83_{\pm 0.05}$	$0.55_{\pm 0.06}$	$0.62_{\pm 0.06}$	$0.69_{\pm 0.06}$

Appendix

Table 12: Recreation of Table 4 using an LLM Judge (GPT-4o with Human references) as a proxy correctness label on a series of frontier models. These results demonstrate the same overall behavior observed in the main experiment: high agreement on the questions in which they were judged to have answered correctly, large lift in performance for using the Human references on questions in which they were judged to be incorrect.

		As Candidate,	Pairwise Grading		Single Grading
Judge	Reasoning	Judge was:	None	Human	None	Human
Opus 4.7	low	Correct	$0.78_{\pm 0.06}$	$0.90_{\pm 0.04}$	$0.63_{\pm 0.06}$	$0.77_{\pm 0.05}$
	low	Incorrect	$0.39_{\pm 0.14}$	$0.66_{\pm 0.12}$	$0.33_{\pm 0.12}$	$0.65_{\pm 0.10}$
	default	Correct	$0.86_{\pm 0.04}$	$0.91_{\pm 0.04}$	$0.63_{\pm 0.06}$	$0.77_{\pm 0.05}$
	default	Incorrect	$0.33_{\pm 0.16}$	$0.66_{\pm 0.13}$	$0.26_{\pm 0.14}$	$0.62_{\pm 0.11}$
	high	Correct	$0.86_{\pm 0.05}$	$0.91_{\pm 0.04}$	$0.65_{\pm 0.06}$	$0.77_{\pm 0.05}$

Appendix

Table 13: Recreation of Table 4 using an LLM Judge (GPT-4o with Human references) as a proxy correctness label to ablate the influence of reasoning on LLM Judge performance. Our main result is preserved across reasoning levels.

Explore similar work

LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLarge Language Model Judges Human Judgment
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation
Jun 1, 2026Junjie Chen, Yuxi Dong, Haitao Li +5Llm-As-A-Judge
Department of Computer Science and Technology, Tsinghua University · University of Science and Technology Beijing
Flaws in the LLM Automation Narrative
Jun 9, 2026George Perrett, Javae Elliott, Jennifer Hill +1Large Language Model Benchmarks Narratives
New York University

Jul 14, 2026, cs.CLJ/K move · Enter open · S save

LLM Judges Can Be Too Generous When There Is No Reference Answer

Chalamalasetti Kranti, Sowmya Vajjala

Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada

Large Language Model Judges Human Judgment Open-Ended Responses Answer

Open paper AskPDF arXiv

LLM judges are increasingly being used to evaluate open-ended model responses, often in no-reference settings where a ground-truth answer is unavailable. However, can they reliably assess in such evaluation setups? We explore this question in this paper through a two stage pipeline with a) calibration experiments that assess the judge model's knowledge of the task it is evaluating, and b) sensitivity experiments that assess how the judge model's performance is impacted by the presence and positioning of the reference answer in the prompt. Across experiments covering three languages, we show that the judge models we evaluated tend to over-credit incorrect answers in the absence of a reference answer, and adding reference answer information to the prompt flips the judge model's correct/incorrect decisions by as much as 85% in some experimental settings. Comparison with a subset of human annotations shows that these reference-driven changes generally align with human judgments. Our results emphasize the need for calibrating the LLM judges with a sample with reference-aware evaluation before using them in reference-free setups reliably, and our methodology provides a blueprint for researchers and practitioners in doing such calibration of LLM judges for other tasks.

Abstract

Explore similar work

LLM Judges Can Be Too Generous When There Is No Reference Answer

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

Flaws in the LLM Automation Narrative