From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks

Zheng Zhang; Lufei Li; Xinyue Tan; Yuanhao Zeng; Ziwei Shan; Yexin Li; Kan Ren

cs.AISep 29, 2026

From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks

Authors: Zheng Zhang, Lufei Li, Xinyue Tan, Yuanhao Zeng, Ziwei Shan, Yexin Li, Kan Ren

Organizations: School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI

Abstract

LLM-as-a-Judge is increasingly used to evaluate policy responses on open-ended tasks that lack ground-truth answers. Existing work often directly converts the resulting judgments into reward signals for policy training, paying limited attention to intrinsic judgment quality and largely restricting the use of Judges to training-time supervision. We systematically investigate judgment quality and downstream utility by examining both how judgments are elicited and how they are used. For judgment elicitation, we vary the Judge protocol along three dimensions: verdict granularity, critique usage, and evaluation batching. For judgment usage, beyond policy training, we extend Judge to test-time inference through Best-of-N selection, Judge-guided revision, and beam search. We find that, (i) Surprisingly, judgment quality and downstream utility do not always align. (ii) Judge protocol design substantially affects both intrinsic judgment quality and downstream utility. (iii) Judge guidance effectively converts test-time compute into performance gains, with benefits varying across inference strategies. Our results call for a multifaceted evaluation of LLM Judges on open-ended tasks, encompassing intrinsic judgment quality, and downstream utility.

Figures & tables

Figure 1: Overview of our framework for analyzing judgment quality and downstream utility in open-ended tasks by varying Judge protocols.

Policy	Training-Time Judge	Verdict Form	Health Bench	RaR- Medicine	Research QA	RaR- Science
Qwen2.5-1.5B-Instruct	Qwen2.5-3B-Instruct	T/F	0.140 $\pm$ 0.006	0.266 $\pm$ 0.004	$0.371\pm 0.021$	$0.302\pm 0.003$
	Qwen2.5-3B-Instruct	Rating	$0.139\pm 0.007$	$0.255\pm 0.014$	0.400 $\pm$ 0.071	0.333 $\pm$ 0.019
	gpt-oss-20b	T/F	$0.158\pm 0.014$	$0.280\pm 0.005$	$0.383\pm 0.005$	$0.334\pm 0.006$
	gpt-oss-20b	Rating	0.164 $\pm$ 0.003	0.285 $\pm$ 0.006	0.402 $\pm$ 0.028	0.349 $\pm$ 0.005
Qwen3-1.7B	Qwen2.5-3B-Instruct	T/F	0.260 $\pm$ 0.005	$0.313\pm 0.008$	0.558 $\pm$ 0.006	0.516 $\pm$ 0.005
	Qwen2.5-3B-Instruct	Rating	$0.252\pm 0.008$	0.315 $\pm$ 0.004	$0.555\pm 0.013$	$0.505\pm 0.002$

Table 1: Test performance of different policy models optimized with different training-time judges and verdict forms. Values are mean

\pm

standard deviation over repeated runs. Bold indicates the higher mean within each matched setting.

Figure 2: Judgemet quality of different verdict forms on Medicine.

Policy	Training-Time Judge	Judge Output	Health Bench	RaR- Medicine	Research QA	RaR- Science
Qwen2.5-1.5B-Instruct	Qwen2.5-3B-Instruct	Critique+Verdict	$0.140\pm 0.006$	0.266 $\pm$ 0.004	$0.371\pm 0.021$	$0.302\pm 0.003$
		Only Verdict	0.149 $\pm$ 0.007	$0.257\pm 0.012$	0.398 $\pm$ 0.019	0.340 $\pm$ 0.004
		Verdict+Critique	0.153 $\pm$ 0.005	0.275 $\pm$ 0.009	0.414 $\pm$ 0.018	0.346 $\pm$ 0.002
	gpt-oss-20b	Critique+Verdict	0.158 $\pm$ 0.014	$0.280\pm 0.005$	0.383 $\pm$ 0.005	0.334 $\pm$ 0.006
		Only Verdict	$0.156\pm 0.004$	0.285 $\pm$ 0.002	0.367 $\pm$ 0.006	0.332 $\pm$ 0.004
		Verdict+Critique	0.160 $\pm$ 0.010	0.282 $\pm$ 0.008	$0.366\pm 0.002$	0.334 $\pm$ 0.002

Table 2: Test performance of RL optimization with different critique and verdict-position settings across training-time judges. Best mean per column is bold; second-best is underlined.

Policy	Training-Time Judge	Judge Output	Health Bench	RaR- Medicine	Research QA	RaR- Science
Qwen2.5-1.5B-Instruct	Qwen2.5-3B-Instruct	Independent	$0.140\pm 0.006$	0.266 $\pm$ 0.004	0.371 $\pm$ 0.021	$0.302\pm 0.003$
	Qwen2.5-3B-Instruct	Batched	0.143 $\pm$ 0.005	$0.239\pm 0.005$	$0.332\pm 0.006$	0.306 $\pm$ 0.005
	gpt-oss-20b	Independent	0.158 $\pm$ 0.014	0.280 $\pm$ 0.005	0.383 $\pm$ 0.005	$0.334\pm 0.006$
	gpt-oss-20b	Batched	$0.155\pm 0.013$	$0.270\pm 0.003$	$0.362\pm 0.008$	0.335 $\pm$ 0.007
Qwen3-1.7B	Qwen2.5-3B-Instruct	Independent	0.260 $\pm$ 0.005	$0.313\pm 0.008$	0.558 $\pm$ 0.006	0.516 $\pm$ 0.005
	Qwen2.5-3B-Instruct	Batched	$0.252\pm 0.007$	0.319 $\pm$ 0.003	$0.547\pm 0.003$	$0.494\pm 0.005$

Table 3: Test performance of Independent evaluation and Batched evaluation. Values are mean

\pm

standard deviation over repeated runs.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Policy	Training-Time Judge	Output	Medicine		Science
			Response-Level Reward Ties $\downarrow$	Uniform-Reward Groups $\downarrow$	Response-Level Reward Ties $\downarrow$	Uniform-Reward Groups $\downarrow$
Qwen2.5-1.5B- Instruct	Qwen2.5-3B-Instruct	Critique+Verdict	36.3%	7.3%	32.5%	4.7%
		Verdict Only	40.4%	9.5%	35.8%	7.7%
		Verdict+Critique	43.7%	12.6%	33.9%	5.6%
	GPT-OSS-20B	Critique+Verdict	30.8%	4.4%	32.8%	6.4%
		Verdict Only	30.4%	4.4%	30.0%	5.0%

Appendix

Table 5: Reward resolution across critique-use protocols over training steps 1–80 for the default-seed trajectory. Bold and underline indicate the lowest and second-lowest values within each policy–Judge–domain setting.

Figure 8: Reward Signal Statistics under T/F and Rating. Lower values indicate fewer tied response pairs and fewer response groups with uniform rewards.

Policy	Training-Time Judge	Protocol	Medicine		Science
			Response-Level Reward Ties $\downarrow$	Uniform-Reward Groups $\downarrow$	Response-Level Reward Ties $\downarrow$	Uniform-Reward Groups $\downarrow$
Qwen2.5-1.5B -Instruct	Qwen2.5-3B-Instruct	Independent	36.3%	7.3%	32.5%	4.7%
	Qwen2.5-3B-Instruct	Batched	39.4%	8.3%	57.0%	25.3%
	GPT-OSS-20B	Independent	30.8%	4.4%	32.8%	6.4%
	GPT-OSS-20B	Batched	31.8%	5.2%	32.7%	6.6%
Qwen3-1.7B	Qwen2.5-3B-Instruct	Independent	38.0%	8.4%	26.3%	2.6%

Appendix

Table 6: Reward resolution under Independent and Batched Evaluation over training steps 1–80 for the default-seed trajectory. Bold indicates the lower value within each policy–Judge–domain setting.

Figure 9: Complete Verdict Granularity results across both policies, training-time Judges, and domains. The three blocks compare binary T/F and 0–10 Rating in criterion-level Accuracy, TPR, and TNR; Critique–Verdict Consistency; and Repeated-Call Agreement and Decision-Rule Robustness.

Figure 10: Complete Critique Usage results across both policies, training-time Judges, and domains. The three blocks compare Critique+Verdict, Verdict Only, and Verdict+Critique in criterion-level Accuracy, TPR, and TNR; Critique–Verdict Consistency; and Repeated-Call Agreement and Decision-Rule Robustness. Verdict Only is omitted from the consistency block because it generates no critique.

Figure 11: Complete Evaluation Batching results across both policies, training-time Judges, and domains. The three blocks compare Independent single-criterion and Batched all-criteria T/F evaluation in criterion-level Accuracy, TPR, and TNR; Critique–Verdict Consistency; and Repeated-Call Agreement and Decision-Rule Robustness.

Figure 12: Change in True-Verdict Rate from Independent to Batched Evaluation across normalized rubric positions. Positive values indicate that Batched Evaluation produces more True verdicts.

Figure 13: Complete Best-of-

N

Selection and Judge-Guided Revision results across both policies and domains.

Policy	Training-Time Judge	Verdict Form	Health Bench	RaR- Medicine	Research QA	RaR- Science
Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	T/F	$0.314$	0.533	$0.575$	$0.577$
Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	Rating	0.318	$0.523$	0.599	0.585

Appendix

Table 7: Validation performance of Qwen2.5-7B-Instruct optimized with different verdict forms. Values are the highest scores across validation checkpoints. Bold indicates the higher score within the matched setting.

Policy	Training-Time Judge	Judge Output	Health Bench	RaR- Medicine	Research QA	RaR- Science
Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	Critique+Verdict	0.314	0.533	$0.575$	0.577
		Only Verdict	$0.310$	0.518	0.586	$0.576$
		Verdict+Critique	0.318	$0.498$	0.584	0.578

Appendix

Table 8: Validation performance of Qwen2.5-7B-Instruct with different critique and verdict-position settings. Values are the highest scores across validation checkpoints. Best score per column is bold; second-best is underlined.

Policy	Training-Time Judge	Judge Output	Health Bench	RaR- Medicine	Research QA	RaR- Science
Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	Single-criterion	$0.314$	0.533	$0.575$	0.577
Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct	Group-level	0.320	$0.513$	0.582	$0.574$

Appendix

Table 9: Validation performance of Qwen2.5-7B-Instruct with Independent evaluation and Batched evaluation. Values are the highest scores across validation checkpoints. Bold indicates the higher score within the matched setting.

Code	Generic guidance
VERIFY_FACTS	Check factual claims and correct unsupported or contradictory statements.
ANSWER_DIRECTLY	State the requested answer directly and keep every part relevant to the question.
COMPLETE_REASONING	Fill in missing logical, mathematical, or causal steps needed to support the conclusion.
COVER_ALL_PARTS	Address every part of the visible user request.
USE_SUPPORT	Support important claims with an appropriate explanation, derivation, or concrete evidence.
CHECK_CONSISTENCY	Make the reasoning and final conclusion internally consistent.

Appendix

Table 10: Allowlisted revision hints.

Explore similar work

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3Llm-As-A-Judge Raw Judge Outputs
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLarge Language Model Judges Human Judgment
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo Hagendorff
AI Safety Research Group, University of Stuttgart

Feb 6, 2026, cs.CLJ/K move · Enter open · S save

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Bo Yang, Lanfei Feng, Yunkui Chen, Yu Zhang+2

College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China

Llm-As-A-Judge Raw Judge Outputs Novel Modular Learning Framework

Open paper AskPDF arXiv

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.

Abstract

Explore similar work

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

LLM Judges Can Be Too Generous When There Is No Reference Answer

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models