References Improve LLM Alignment in Non-Verifiable Domains

Kejian Shi; Yixin Liu; Peifeng Wang; Alexander R. Fabbri; Shafiq Joty; Arman Cohan

cs.CLFeb 18, 2026

References Improve LLM Alignment in Non-Verifiable Domains

Authors: Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri, Shafiq Joty, Arman Cohan

Organizations: Yale University · Meta · Scale AI · Salesforce Research ·

Abstract

While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.

Figures & tables

Figure 1: Overview of our study on reference-guided LLM-as-a-Judge for LLM alignment. Conceptual plots illustrating (I) the improvement in average accuracy from reference-guided evaluation (§ 3.3 ) and (II) the reference-guided self-improvement (§ 4 ).

Panel of Figure 2 — Figure 2: A snapshot of RefEval method.

Method	Nat	Adv	MT	Ins	HREF	Avg
LLMBar-Base	83.1	61.7	74.6	70.2	72.0	72.3 (-1.4, +1.5) *
HREF-Base	84.1	54.0	76.5	70.8	77.3	72.5 (-1.5, +1.6) *
CoT	82.0	60.1	75.4	69.1	69.6	71.2 (-1.5, +1.6) *
Prepair	81.7	71.5	72.6	68.8	75.2	74.0 (-1.3, +1.4) *
Self-Ref	84.6	66.7	73.5	69.5	72.4	73.3 (-1.3, +1.3) *
Self-Metric-Ref	85.5	67.4	75.0	70.5	74.4	74.6 (-1.3, +1.4) *

Table 1: Average evaluation accuracy (%) across five datasets using 11 open-source models as judges. We report the 95% bootstrap confidence interval for the average performance. * indicates the method is significantly worse than RefEval (

p<0.05

). Dataset acronyms are: Natural (Nat), Adversarial (Adv), MTBench (MT), and InstruSum (Ins).

Method	qwen-2.5 -72b	llama-3.1 -70b	gemma-2 -27b	qwen-2.5 -14b	mistral -nemo	gemma-2 -9b	glm-4 -9b	llama-3.1 -8b	qwen-2.5 -7b	llama-3 -8b	mistral-7b -v0.3
Base	79.4	85.2	82.3	81.5	65.6	80.8	71.8	65.0	73.5	60.1	47.0
RefFree	83.4	85.6	80.1	83.3	63.7	80.8	77.3	71.8	74.5	72.3	61.2
RefEval	84.6	85.9	84.9	82.4	73.2	85.7	79.5	79.4	77.4	77.5	69.6

Table 2: Comparison of average evaluation accuracy across 11 LLM judges (averaged over five datasets). References for RefEval were generated by GPT-4o . Best performance for each model is bolded. Base denotes LLMBar-Base . Full statistical significance analysis is provided in Appendix A.1 .

	Llama-3-8B-Instruct		Qwen2.5-7B-SFT
Method	AE	AH	AE	AH
Base	25.0 (-0.2, +0.3)	27.1 (-1.8, 1.7)	14.4 (-0.1, +0.2)	23.4 (-2.1, 2.1)
ArmoRM-Base	49.2 (-0.5, +0.5)	40.4 (-2.4, 2.4)	32.6 (-0.3, +0.3)	58.6 (-2.3, 2.6)
DSV3-Distill	53.9 (-0.5, +0.6)	42.2 (-2.1, 2.3)	48.8 (-0.5, +0.5)	56.5 (-2.3, 2.1)
Below are fine-tuned from DSV3-Distill using DPO
ROUGE	56.4 (-0.6, +0.6)	52.1 (-2.2, 2.5)	50.9 (-0.5, +0.5)	67.4 (-2.0, 2.5)

Table 3: Performance comparison of training methods. Scores are for length-controlled AlpacaEval (AE) and Arena-Hard (AH).

Compared Baseline	Llama-AE	Llama-AH	Qwen-AE	Qwen-AH
DSV3-Distill	+19.2	+16.5	+21.2	+17.6
RefFree	+5.6	+4.9	+4.9	+2.3

Table 4: Absolute gain (%) of RefEval over key baselines from Table 3 .

	AlpacaEval	Arena-Hard
DeepSeek-V3	84.8	94.9
SimPO-Llama3-8B-Inst	51.6	36.2
RefEval-Llama3-8B-Inst	73.1	58.7
Qwen2.5-7B-Inst	29.9	58.0
RefEval-Qwen2.5-7B	70.0	74.1

Table 5: Performance of our self-improved models vs. strong baselines on AE and AH.

Figure 3: Comparison of reference-free and reference-guided self-improvement across task categories on AlpacaEval and Arena-Hard.

Method	AE	AH
Reference Source: DeepSeek-V3
Distill	53.9	42.2
RefFree	67.5	53.8
RefEval	73.1	58.7
Reference Source: GPT-4o-mini
Distill	28.7	40.7

Table 6: Reference-quality ablation for training on Llama-3-8B-Instruct.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Judge Model	LLMBar-Base	Ref-Free (Ours)	RefEval
qwen-2.5-72b	79.4 (-1.9, +1.8) *	83.4 (-1.9, +1.8) *	84.6 (-1.8, +1.8)
llama-3.1-70b	85.2 (-1.8, +1.7) *	85.6 (-1.7, +1.8)	85.9 (-1.7, +1.8)
gemma-2-27b	82.3 (-2.1, +2.0) *	80.1 (-1.9, +2.0) *	84.9 (-1.7, +1.8)
qwen-2.5-14b	81.5 (-1.9, +1.9) *	83.3 (-1.8, +1.9) *	82.4 (-2.2, +2.1)
mistral-nemo	65.6 (-2.2, +2.4) *	63.7 (-2.2, +2.3) *	73.2 (-2.1, +2.2)
gemma-2-9b	80.8 (-1.8, +1.9) *	80.8 (-2.2, +2.0) *	85.7 (-2.0, +2.0)

Appendix

Table 7: Detailed evaluation accuracy (%) by model with 95% Bootstrap Confidence Intervals. * indicates the method is significantly worse than RefEval (

p<0.05

). Comparison across 11 Open-Source Models. This table provides the full statistical context for Table 2 .

Method	Nat	Adv	MT	Instrusum	HREF	Avg
RefEval	0.975	0.889	0.800	0.807	0.907	0.876
RefEval-LLMBarRules	0.970	0.865	0.820	0.803	0.899	0.871
LLMBar-Base	0.975	0.845	0.798	0.818	0.907	0.869
Metric-Reference	0.970	0.862	0.805	0.793	0.909	0.868
Ref-Free (Ours)	0.960	0.854	0.788	0.810	0.928	0.868
Self-Reference	0.960	0.853	0.805	0.793	0.906	0.863

Appendix

Table 8: Full average evaluation accuracy (%) across five datasets using GPT-4o as the judge, with references generated by GPT-4o itself.

Method	Nat	Adv	MT	Instrusum	HREF	Avg
RefEval	0.868	0.749	0.767	0.745	0.827	0.791
RefMatch	0.846	0.741	0.763	0.729	0.804	0.777
HREF-Ref	0.853	0.623	0.765	0.708	0.792	0.748
RefMatch-Rules-CoT	0.866	0.733	0.757	0.704	0.749	0.762
RefEval-Rules	0.862	0.672	0.760	0.718	0.749	0.752
HREF-Base	0.841	0.540	0.765	0.708	0.773	0.725

Appendix

Table 9: Full average evaluation accuracy (%) across five datasets using 11 open-source models as judges, with references generated by GPT-4o .

Figure 4: Aggregate performance by dataset for Larger Models (

>

9B parameters, including GPT-4o variants; top panel) and Smaller Models (

\leq

9B parameters; bottom panel). RefEval demonstrates consistent improvements across most datasets for both model groups.

Method	Nat	Adv	MT	Instrusum	HREF	Avg
RefEval	0.915	0.813	0.795	0.797	0.878	0.840
Metric-Ref	0.935	0.790	0.825	0.773	0.857	0.836
Self-Ref	0.938	0.775	0.816	0.768	0.846	0.829
RefMatch	0.893	0.813	0.793	0.773	0.863	0.827
Ref-Free (Ours)	0.910	0.756	0.811	0.772	0.864	0.823
LLMBar-Ref	0.915	0.786	0.821	0.777	0.832	0.826

Appendix

Table 10: Full average evaluation accuracy (%) for larger open-source models ( Qwen-2.5-72B , Llama3.1-70B ) as judges, across five datasets. References by GPT-4o .

Method	Nat	Adv	MT	Instrusum	HREF	Avg
RefEval	0.845	0.727	0.759	0.715	0.808	0.771
RefMatch	0.817	0.701	0.751	0.701	0.759	0.746
HREF-Ref	0.828	0.580	0.741	0.684	0.759	0.718
RefMatch-Rules-CoT	0.837	0.696	0.735	0.684	0.722	0.735
HREF-Base	0.817	0.492	0.740	0.674	0.744	0.693
Prepair	0.784	0.678	0.698	0.646	0.725	0.706

Appendix

Table 11: Full average evaluation accuracy (%) for smaller open-source models (

\leq

9B parameters) as judges, across five datasets. References by GPT-4o .

Method	Average Accuracy (%) on Datasets
Method	Nat.	Adv.	MT.	Ins.	Avg.
RefEval-Vote	0.885	0.748	0.768	0.756	0.789
RefEval (Single Ref)	0.868	0.749	0.767	0.745	0.782
RefMatch-Vote	0.858	0.748	0.775	0.736	0.779
RefMatch-Rules-CoT-Vote	0.888	0.745	0.755	0.728	0.779
RefMatch (Single Ref)	0.846	0.741	0.763	0.729	0.770

Appendix

Table 12: Full average evaluation accuracy (%) with Multi-Reference Strategies using 11 open-source models as judges. “Vote” indicates Multi-Voting Aggregation. Methods without “Vote” (except Multi-Prompt) use a single GPT-4o reference unless they are inherently reference-free. Averages are over 4 datasets (Nat, Adv, MT, Ins).

Figure 5: Evaluation accuracy of 11 open-source LLM-judges using RefEval and RefMatch with single references from various frontier models, and their voted versions. Horizontal dashed lines indicate reference-free baselines. Results are averaged over five datasets.

	GPT-4o	GPT-4.1	Qwen	Llama
Ref-Free (Ours)	85.4	85.3	71.0	80.3
RefMatch	84.2	86.1	81.0	79.9
RefMatch-Oracle	85.9	88.2	82.6	83.9
RefEval	86.8	86.7	79.9	82.8
RefEval-Oracle	88.4	88.6	81.8	84.6

Appendix

Table 13: Accuracy (%) on LLMBar-Adversarial comparing standard vs. human-edited “Oracle” references. Qwen is Qwen-2.5-72B and Llama is Llama-3.1-70B.

Method	Nat	Adv	MT	Instrusum	HREF	Avg.
Average of 11 Open-Source Models
Base-point	77.8	65.0	68.4	60.8	67.2	67.8
RefEval-point	82.7	71.7	73.1	67.4	73.1	73.6
Average of GPT-4o and GPT-4.1
Base-point	90.5	84.0	74.5	71.5	79.9	80.0
RefEval-point	89.5	87.1	75.2	72.6	83.9	81.7

Appendix

Table 14: Evaluation accuracy (%) of pointwise scoring methods. Pointwise scores are used to infer pairwise preferences, which are then compared against human labels.

Name	Size	License	Description
gemma-2-9b	9b	Gemma	Gemma is a family of open models from Google ( Gemma et al., 2024 ) . These are instruct-tuned versions.
gemma-2-27b	27b	Gemma
glm-4-9b	9b	GLM-4	GLM-4-9B is an open-source version of the latest generation of pre-trained models launched by Zhipu AI ( Du et al., 2022 ) .
llama-3-8b	8b	llama 3 Community	llama 3 are the latest open models from Meta AI ( Meta AI, 2024 ) , pretrained on 15T tokens.
llama-3.1-8b	8b	llama 3.1 Community	llama 3.1 collection offers a series of multilingual models that outperform many open and closed chat models on industry benchmarks ( Llama3, 2024 ) .
llama-3.1-70b	70b	llama 3.1 Community

Appendix

Table 15: Model registry and metadata described in § 3.3 . All models are post-trained.

Judge Model	Reference Source	Method	Avg. Acc.
Llama-3.1-70B	GPT-4o (Oracle)	RefEval	0.856
	Nemo-12B	RefEval	0.850
	Tulu-2-7B	RefEval	0.834
	None	Ref-Free	0.832
Qwen-2.5-7B	GPT-4o (Oracle)	RefEval	0.735
	Nemo-12B	RefEval	0.726

Appendix

Table 16: Ablation of Reference Source Quality on Evaluation Accuracy (Avg over 4 datasets: Nat, Adv, MT, Ins).

Method	AlpacaEval	Arena-Hard
Base (Llama-3-8B-Inst)	25.0	27.1
Reference Source: DeepSeek-V3
DSV3-Distill	53.9	42.2
V3-RefFree	67.5	53.8
V3-RefEval	73.1	58.7
Reference Source: GPT-4o-mini

Appendix

Table 17: Ablation of Reference Source Quality on Training (Llama-3-8B-Instruct).

Dataset Group	Ref-Free	RefEval	Difference
LLMBar-Natural	80.98	85.31	+4.33
LLMBar-Adversarial	75.59	77.61	+2.02
MTBench	75.35	82.42	+7.07
Instrusum	76.95	79.69	+2.74
HREF	74.17	81.83	+7.66
Average	76.61	81.37	+4.76

Appendix

Table 18: Average Pairwise Inter-Judge Agreement (%) among 11 open-source models.

Configuration	# Refs	Avg. Accuracy	Gain
Single Best (Avg)	1	81.4%	-
Pairwise Ensemble (Simulated)	2	81.8%	+0.4%
Majority Vote	3	82.2%	+0.4%
Majority Vote	5	82.3%	+0.1%

Appendix

Table 19: Marginal gains from increasing reference count (Average across 11 judges).

Dataset	No. of Instances
LLMBar-Natural (Nat)	100
LLMBar-Adversarial (Adv)	319
MTBench (MT)	200
Instrusum (Ins)	411
HREF - CLS	56
HREF - CQA	73

Appendix

Table 20: Number of instances for each evaluation dataset detailed in Appendix D .

Figure 6: Distribution of different instruction types in AlpacaEval.

Figure 7: Distribution of different instruction types in Arena-Hard.

Panel of Figure 8 — Figure 8: Utility prompt – classification of instruction type introduced in § 4.3 .

Panel of Figure 9 — Figure 9: Prompt template for our Ref-Free (Ours) baseline prompting method introduced in § 3 .

Panel of Figure 10 — Figure 10: Prompt template for our RefEval prompting method introduced in § 3 .

Panel of Figure 11 — Figure 11: Prompt template for the RefMatch prompting method introduced in § 3 .

Panel of Figure 12 — Figure 12: Prompt template for the HREF-Base prompting method from Lyu et al. (2024) .

Panel of Figure 13 — Figure 13: Prompt template for the HREF-Ref prompting method from Lyu et al. (2024) .

Panel of Figure 14 — Figure 14: Prompt template for the Multi-Ref Avg prompting method. Details in § A.2

Panel of Figure 15 — Figure 15: Prompt template for the Multi-Ref MAX prompting method. Details in § A.2 .

Panel of Figure 16 — Figure 16: Prompt template for the RefMatch-Rules-CoT prompting method variant. § A.2

Panel of Figure 17 — Figure 17: Prompt template for the RefEval-Rules prompting method variant. § A.2

Panel of Figure 18 — Figure 18: Prompt for cot prompting method described in § 3

Panel of Figure 19 — Figure 19: Prompt for metric + reference prompting method described in § 3 .

Panel of Figure 20 — Figure 20: Prompt for prepair prompting method described in § 3 . This is the prompt for pointwise analysis (the first stage) within the method.

Panel of Figure 21 — Figure 21: Prompt for prepair prompting method described in § 3 . This is the pairwise evaluation stage (the second stage) within the method.

Panel of Figure 22 — Figure 22: Prompt for Self-Ref prompting method described in § 3 .

Panel of Figure 23 — Figure 23: Prompt for Base-point prompting method described in § B .

Panel of Figure 24 — Figure 24: Prompt for RefEval-point prompting method described in § B .

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Mar 12, 2026Yixin Liu, Yue Yu, DiJia Su +7LLM-as-a-Judge LLM Alignment
Meta Superintelligence Labs · Yale Univeristy
Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks
Mar 13, 2026Kun Wang, Reinhard HeckelLLM Evaluation Inference-Time Language Model Alignment
School of Computation, Information and Technology, Technical University of Munich · Munich Center for Machine Learning
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada

Mar 12, 2026J/K move · Enter open · S save

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

Yixin Liu, Yue Yu, DiJia Su, Sid Wang+6

Meta Superintelligence Labs · Yale Univeristy

LLM-as-a-Judge LLM Alignment LLM-Guided RL Reward Hacking

Open paper AskPDF arXiv

Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.

Abstract

LLM Alignment

Explore similar work

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks

LLM Judges Can Be Too Generous When There Is No Reference Answer