While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Figures & tables
Figure 1: Overview of our study on reference-guided LLM-as-a-Judge for LLM alignment. Conceptual plots illustrating (I) the improvement in average accuracy from reference-guided evaluation (§ 3.3 ) and (II) the reference-guided self-improvement (§ 4 ).
Figure 2: A snapshot of RefEval method.
Method
Nat
Adv
MT
Ins
HREF
Avg
LLMBar-Base
83.1
61.7
74.6
70.2
72.0
72.3 (-1.4, +1.5) *
HREF-Base
84.1
54.0
76.5
70.8
77.3
72.5 (-1.5, +1.6) *
CoT
82.0
60.1
75.4
69.1
69.6
71.2 (-1.5, +1.6) *
Prepair
81.7
71.5
72.6
68.8
75.2
74.0 (-1.3, +1.4) *
Self-Ref
84.6
66.7
73.5
69.5
72.4
73.3 (-1.3, +1.3) *
Self-Metric-Ref
85.5
67.4
75.0
70.5
74.4
74.6 (-1.3, +1.4) *
Table 1: Average evaluation accuracy (%) across five datasets using 11 open-source models as judges. We report the 95% bootstrap confidence interval for the average performance. * indicates the method is significantly worse than RefEval ( p<0.05 ). Dataset acronyms are: Natural (Nat), Adversarial (Adv), MTBench (MT), and InstruSum (Ins).
Method
qwen-2.5 -72b
llama-3.1 -70b
gemma-2 -27b
qwen-2.5 -14b
mistral -nemo
gemma-2 -9b
glm-4 -9b
llama-3.1 -8b
qwen-2.5 -7b
llama-3 -8b
mistral-7b -v0.3
Base
79.4
85.2
82.3
81.5
65.6
80.8
71.8
65.0
73.5
60.1
47.0
RefFree
83.4
85.6
80.1
83.3
63.7
80.8
77.3
71.8
74.5
72.3
61.2
RefEval
84.6
85.9
84.9
82.4
73.2
85.7
79.5
79.4
77.4
77.5
69.6
Table 2: Comparison of average evaluation accuracy across 11 LLM judges (averaged over five datasets). References for RefEval were generated by GPT-4o . Best performance for each model is bolded. Base denotes LLMBar-Base . Full statistical significance analysis is provided in Appendix A.1 .
Llama-3-8B-Instruct
Qwen2.5-7B-SFT
Method
AE
AH
AE
AH
Base
25.0 (-0.2, +0.3)
27.1 (-1.8, 1.7)
14.4 (-0.1, +0.2)
23.4 (-2.1, 2.1)
ArmoRM-Base
49.2 (-0.5, +0.5)
40.4 (-2.4, 2.4)
32.6 (-0.3, +0.3)
58.6 (-2.3, 2.6)
DSV3-Distill
53.9 (-0.5, +0.6)
42.2 (-2.1, 2.3)
48.8 (-0.5, +0.5)
56.5 (-2.3, 2.1)
Below are fine-tuned from DSV3-Distill using DPO
ROUGE
56.4 (-0.6, +0.6)
52.1 (-2.2, 2.5)
50.9 (-0.5, +0.5)
67.4 (-2.0, 2.5)
Table 3: Performance comparison of training methods. Scores are for length-controlled AlpacaEval (AE) and Arena-Hard (AH).
Compared Baseline
Llama-AE
Llama-AH
Qwen-AE
Qwen-AH
DSV3-Distill
+19.2
+16.5
+21.2
+17.6
RefFree
+5.6
+4.9
+4.9
+2.3
Table 4: Absolute gain (%) of RefEval over key baselines from Table 3 .
AlpacaEval
Arena-Hard
DeepSeek-V3
84.8
94.9
SimPO-Llama3-8B-Inst
51.6
36.2
RefEval-Llama3-8B-Inst
73.1
58.7
Qwen2.5-7B-Inst
29.9
58.0
RefEval-Qwen2.5-7B
70.0
74.1
Table 5: Performance of our self-improved models vs. strong baselines on AE and AH.
Figure 3: Comparison of reference-free and reference-guided self-improvement across task categories on AlpacaEval and Arena-Hard.
Method
AE
AH
Reference Source: DeepSeek-V3
Distill
53.9
42.2
RefFree
67.5
53.8
RefEval
73.1
58.7
Reference Source: GPT-4o-mini
Distill
28.7
40.7
Table 6: Reference-quality ablation for training on Llama-3-8B-Instruct.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Judge Model
LLMBar-Base
Ref-Free (Ours)
RefEval
qwen-2.5-72b
79.4 (-1.9, +1.8) *
83.4 (-1.9, +1.8) *
84.6 (-1.8, +1.8)
llama-3.1-70b
85.2 (-1.8, +1.7) *
85.6 (-1.7, +1.8)
85.9 (-1.7, +1.8)
gemma-2-27b
82.3 (-2.1, +2.0) *
80.1 (-1.9, +2.0) *
84.9 (-1.7, +1.8)
qwen-2.5-14b
81.5 (-1.9, +1.9) *
83.3 (-1.8, +1.9) *
82.4 (-2.2, +2.1)
mistral-nemo
65.6 (-2.2, +2.4) *
63.7 (-2.2, +2.3) *
73.2 (-2.1, +2.2)
gemma-2-9b
80.8 (-1.8, +1.9) *
80.8 (-2.2, +2.0) *
85.7 (-2.0, +2.0)
Appendix
Table 7: Detailed evaluation accuracy (%) by model with 95% Bootstrap Confidence Intervals. * indicates the method is significantly worse than RefEval ( p<0.05 ). Comparison across 11 Open-Source Models. This table provides the full statistical context for Table 2 .
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.975
0.889
0.800
0.807
0.907
0.876
RefEval-LLMBarRules
0.970
0.865
0.820
0.803
0.899
0.871
LLMBar-Base
0.975
0.845
0.798
0.818
0.907
0.869
Metric-Reference
0.970
0.862
0.805
0.793
0.909
0.868
Ref-Free (Ours)
0.960
0.854
0.788
0.810
0.928
0.868
Self-Reference
0.960
0.853
0.805
0.793
0.906
0.863
Appendix
Table 8: Full average evaluation accuracy (%) across five datasets using GPT-4o as the judge, with references generated by GPT-4o itself.
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.868
0.749
0.767
0.745
0.827
0.791
RefMatch
0.846
0.741
0.763
0.729
0.804
0.777
HREF-Ref
0.853
0.623
0.765
0.708
0.792
0.748
RefMatch-Rules-CoT
0.866
0.733
0.757
0.704
0.749
0.762
RefEval-Rules
0.862
0.672
0.760
0.718
0.749
0.752
HREF-Base
0.841
0.540
0.765
0.708
0.773
0.725
Appendix
Table 9: Full average evaluation accuracy (%) across five datasets using 11 open-source models as judges, with references generated by GPT-4o .
Figure 4: Aggregate performance by dataset for Larger Models ( > 9B parameters, including GPT-4o variants; top panel) and Smaller Models ( ≤ 9B parameters; bottom panel). RefEval demonstrates consistent improvements across most datasets for both model groups.
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.915
0.813
0.795
0.797
0.878
0.840
Metric-Ref
0.935
0.790
0.825
0.773
0.857
0.836
Self-Ref
0.938
0.775
0.816
0.768
0.846
0.829
RefMatch
0.893
0.813
0.793
0.773
0.863
0.827
Ref-Free (Ours)
0.910
0.756
0.811
0.772
0.864
0.823
LLMBar-Ref
0.915
0.786
0.821
0.777
0.832
0.826
Appendix
Table 10: Full average evaluation accuracy (%) for larger open-source models ( Qwen-2.5-72B , Llama3.1-70B ) as judges, across five datasets. References by GPT-4o .
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.845
0.727
0.759
0.715
0.808
0.771
RefMatch
0.817
0.701
0.751
0.701
0.759
0.746
HREF-Ref
0.828
0.580
0.741
0.684
0.759
0.718
RefMatch-Rules-CoT
0.837
0.696
0.735
0.684
0.722
0.735
HREF-Base
0.817
0.492
0.740
0.674
0.744
0.693
Prepair
0.784
0.678
0.698
0.646
0.725
0.706
Appendix
Table 11: Full average evaluation accuracy (%) for smaller open-source models ( ≤ 9B parameters) as judges, across five datasets. References by GPT-4o .
Method
Average Accuracy (%) on Datasets
Nat.
Adv.
MT.
Ins.
Avg.
RefEval-Vote
0.885
0.748
0.768
0.756
0.789
RefEval (Single Ref)
0.868
0.749
0.767
0.745
0.782
RefMatch-Vote
0.858
0.748
0.775
0.736
0.779
RefMatch-Rules-CoT-Vote
0.888
0.745
0.755
0.728
0.779
RefMatch (Single Ref)
0.846
0.741
0.763
0.729
0.770
Appendix
Table 12: Full average evaluation accuracy (%) with Multi-Reference Strategies using 11 open-source models as judges. “Vote” indicates Multi-Voting Aggregation. Methods without “Vote” (except Multi-Prompt) use a single GPT-4o reference unless they are inherently reference-free. Averages are over 4 datasets (Nat, Adv, MT, Ins).
Figure 5: Evaluation accuracy of 11 open-source LLM-judges using RefEval and RefMatch with single references from various frontier models, and their voted versions. Horizontal dashed lines indicate reference-free baselines. Results are averaged over five datasets.
GPT-4o
GPT-4.1
Qwen
Llama
Ref-Free (Ours)
85.4
85.3
71.0
80.3
RefMatch
84.2
86.1
81.0
79.9
RefMatch-Oracle
85.9
88.2
82.6
83.9
RefEval
86.8
86.7
79.9
82.8
RefEval-Oracle
88.4
88.6
81.8
84.6
Appendix
Table 13: Accuracy (%) on LLMBar-Adversarial comparing standard vs. human-edited “Oracle” references. Qwen is Qwen-2.5-72B and Llama is Llama-3.1-70B.
Method
Nat
Adv
MT
Instrusum
HREF
Avg.
Average of 11 Open-Source Models
Base-point
77.8
65.0
68.4
60.8
67.2
67.8
RefEval-point
82.7
71.7
73.1
67.4
73.1
73.6
Average of GPT-4o and GPT-4.1
Base-point
90.5
84.0
74.5
71.5
79.9
80.0
RefEval-point
89.5
87.1
75.2
72.6
83.9
81.7
Appendix
Table 14: Evaluation accuracy (%) of pointwise scoring methods. Pointwise scores are used to infer pairwise preferences, which are then compared against human labels.
Name
Size
License
Description
gemma-2-9b
9b
Gemma
Gemma is a family of open models from Google ( Gemma et al., 2024 ) . These are instruct-tuned versions.
gemma-2-27b
27b
Gemma
glm-4-9b
9b
GLM-4
GLM-4-9B is an open-source version of the latest generation of pre-trained models launched by Zhipu AI ( Du et al., 2022 ) .
llama-3-8b
8b
llama 3 Community
llama 3 are the latest open models from Meta AI ( Meta AI, 2024 ) , pretrained on 15T tokens.
llama-3.1-8b
8b
llama 3.1 Community
llama 3.1 collection offers a series of multilingual models that outperform many open and closed chat models on industry benchmarks ( Llama3, 2024 ) .
llama-3.1-70b
70b
llama 3.1 Community
Appendix
Table 15: Model registry and metadata described in § 3.3 . All models are post-trained.
Judge Model
Reference Source
Method
Avg. Acc.
Llama-3.1-70B
GPT-4o (Oracle)
RefEval
0.856
Nemo-12B
RefEval
0.850
Tulu-2-7B
RefEval
0.834
None
Ref-Free
0.832
Qwen-2.5-7B
GPT-4o (Oracle)
RefEval
0.735
Nemo-12B
RefEval
0.726
Appendix
Table 16: Ablation of Reference Source Quality on Evaluation Accuracy (Avg over 4 datasets: Nat, Adv, MT, Ins).
Method
AlpacaEval
Arena-Hard
Base (Llama-3-8B-Inst)
25.0
27.1
Reference Source: DeepSeek-V3
DSV3-Distill
53.9
42.2
V3-RefFree
67.5
53.8
V3-RefEval
73.1
58.7
Reference Source: GPT-4o-mini
Appendix
Table 17: Ablation of Reference Source Quality on Training (Llama-3-8B-Instruct).
Dataset Group
Ref-Free
RefEval
Difference
LLMBar-Natural
80.98
85.31
+4.33
LLMBar-Adversarial
75.59
77.61
+2.02
MTBench
75.35
82.42
+7.07
Instrusum
76.95
79.69
+2.74
HREF
74.17
81.83
+7.66
Average
76.61
81.37
+4.76
Appendix
Table 18: Average Pairwise Inter-Judge Agreement (%) among 11 open-source models.
Configuration
# Refs
Avg. Accuracy
Gain
Single Best (Avg)
1
81.4%
-
Pairwise Ensemble (Simulated)
2
81.8%
+0.4%
Majority Vote
3
82.2%
+0.4%
Majority Vote
5
82.3%
+0.1%
Appendix
Table 19: Marginal gains from increasing reference count (Average across 11 judges).
Dataset
No. of Instances
LLMBar-Natural (Nat)
100
LLMBar-Adversarial (Adv)
319
MTBench (MT)
200
Instrusum (Ins)
411
HREF - CLS
56
HREF - CQA
73
Appendix
Table 20: Number of instances for each evaluation dataset detailed in Appendix D .
Figure 6: Distribution of different instruction types in AlpacaEval.
Figure 7: Distribution of different instruction types in Arena-Hard.
Figure 8: Utility prompt – classification of instruction type introduced in § 4.3 .
Figure 9: Prompt template for our Ref-Free (Ours) baseline prompting method introduced in § 3 .
Figure 10: Prompt template for our RefEval prompting method introduced in § 3 .
Figure 11: Prompt template for the RefMatch prompting method introduced in § 3 .
Figure 12: Prompt template for the HREF-Base prompting method from Lyu et al. (2024) .
Figure 13: Prompt template for the HREF-Ref prompting method from Lyu et al. (2024) .
Figure 14: Prompt template for the Multi-Ref Avg prompting method. Details in § A.2
Figure 15: Prompt template for the Multi-Ref MAX prompting method. Details in § A.2 .
Figure 16: Prompt template for the RefMatch-Rules-CoT prompting method variant. § A.2
Figure 17: Prompt template for the RefEval-Rules prompting method variant. § A.2
Figure 18: Prompt for cot prompting method described in § 3
Figure 19: Prompt for metric + reference prompting method described in § 3 .
Figure 20: Prompt for prepair prompting method described in § 3 . This is the prompt for pointwise analysis (the first stage) within the method.
Figure 21: Prompt for prepair prompting method described in § 3 . This is the pairwise evaluation stage (the second stage) within the method.
Figure 22: Prompt for Self-Ref prompting method described in § 3 .
Figure 23: Prompt for Base-point prompting method described in § B .
Figure 24: Prompt for RefEval-point prompting method described in § B .
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.
Direct evaluation of LLMs on benchmarks can be misleading because comparatively strong performance may reflect task familiarity rather than capability. The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation, originally through supervised finetuning. However, suitable training data is often hard to come by, and evaluation results vary with the data chosen. In this paper, we propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test. First, RL with a single sample provides a first alignment of the model to the task format, and second, test-time RL with majority-voting reward aligns the model to the benchmark distribution. Our test-time RL alignment method aligns similarly well as SFT-based train-before test, but without requiring a task-specific training set. On a domain-specific benchmark without training data, we show that direct evaluation underestimates base models which perform substantially better once aligned, yielding a more faithful evaluation of their capabilities. Moreover, for reasoning tasks, the performance gap between fine-tuned models and their base models largely disappears after alignment, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.
Kun Wang, Reinhard Heckel
School of Computation, Information and Technology, Technical University of Munich · Munich Center for Machine Learning
LLM judges are increasingly being used to evaluate open-ended model responses, often in no-reference settings where a ground-truth answer is unavailable. However, can they reliably assess in such evaluation setups? We explore this question in this paper through a two stage pipeline with a) calibration experiments that assess the judge model's knowledge of the task it is evaluating, and b) sensitivity experiments that assess how the judge model's performance is impacted by the presence and positioning of the reference answer in the prompt. Across experiments covering three languages, we show that the judge models we evaluated tend to over-credit incorrect answers in the absence of a reference answer, and adding reference answer information to the prompt flips the judge model's correct/incorrect decisions by as much as 85% in some experimental settings. Comparison with a subset of human annotations shows that these reference-driven changes generally align with human judgments. Our results emphasize the need for calibrating the LLM judges with a sample with reference-aware evaluation before using them in reference-free setups reliably, and our methodology provides a blueprint for researchers and practitioners in doing such calibration of LLM judges for other tasks.
Chalamalasetti Kranti, Sowmya Vajjala
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada