Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.
Figures & tables
Figure 1: Illustration of our synthetic experiment setting (left). In the middle, we show that a Llama-3.1-8B policy trained with a fine-tuned Qwen3-4B reasoning judge can achieve strong performance under the gold-standard judge gpt-oss-120b’s evaluation, while the policy trained with a fine-tuned Qwen3-14B non-reasoning judge cannot and exhibits severe reward hacking. The table on the right shows results on the creative writing subset of Arena-Hard-V2. The Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge is able to achieve superior performance by learning to generate highly effective adversarial outputs.
Figure 2: Performance of various LLM-judges based on their agreement (Krippendorff’s Alpha) with the gold-standard judge, gpt-oss-120b. The LLM-judges are all based on Qwen3 models and grouped by their sizes. Both the base judges and fine-tuned judges are evaluated, and non-reasoning judges and reasoning judges are compared.
Figure 3: Performance comparison of policies trained with non-reasoning judges of different sizes under the gold-standard judge’s evaluation. The sub-figures show policies trained from different initial LLMs.
Figure 4: Performance of policies trained with non-reasoning judges of different sizes evaluated by the judges used in training .
Figure 5: Performance of policies trained with Qwen3-4B and Qwen3-8B reasoning judges . The policy performance at different training steps on the test set is shown when evaluated by the training judges (a) and by the gold-standard judge gpt-oss-120b (b).
Figure 6: Output example of the Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge after 1100 training steps.
Creative Writing
Hard Prompt
Model
Score (%)
Model
Score (%)
o3-2025-04-16
92.4
o3-2025-04-16
86.8
Qwen3-4B Reasoning-Judge + Llama-3.1-8B
89.6
o4-mini-2025-04-16-high
81.2
deepseek-r1
89.2
gemini-2.5
55.9
gemini-2.5
85.2
deepseek-r1
48.5
o4-mini-2025-04-16-high
79.8
claude-3-7-sonnet-20250219-thinking-16k
47.9
Table 1: Performance of the Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge on Arena-Hard-V2.
Figure 7: Performance of policies trained with RL-only reasoning judges . The policy performance at different training steps on the test set is shown when evaluated by the gold-standard judge (a) and by the judges used in training (b).
Training Method / Base Model
Qwen3-4B
Qwen3-8B
Distillation+RL
85.94
89.34
RL-Only
85.10
85.99
Table 2: Performance of reasoning judges trained with RL only compared to judges trained with both SFT distillation and RL. The judges’ agreements with the gold-standard judge are reported.
Figure 8: Performance of policies trained with non-reasoning judges provided with rubrics generated by the gold-standard judge. The policy performance at different training steps on the test set is shown when evaluated by the gold-standard judge (a) and by the judges used in training (b).
Judge
w/o Rubrics
w/ Rubrics
Qwen3-14B
41.73
60.90
Qwen3-14B-Fine-Tuned
87.82
89.72
Table 3: Performance comparison of non-reasoning judges prompted/fine-tuned with and without the rubrics generated by the gold-standard judge. The judges’ agreements with the gold-standard judge (Krippendorff’s Alpha) are reported.
Figure 9: Performance of policies trained with reasoning judges with low and medium reasoning efforts fine-tuned from Qwen3-8B. The policy performance at different training steps on the test set is shown when evaluated by the judges used in training (a) and by the gold-standard judge (b).
Reasoning Effort
Agreement
#Tokens
Low
79.88
43.2
Medium
85.99
200.3
High
89.34
981.6
Table 4: Performance of Qwen3-8B reasoning judges with varied reasoning efforts. We report their agreement with the gold-standard judge and the average number of tokens in their thinking traces.
Judge
Non-Reasoning
Reasoning
Qwen3-8B
79.8
85.0
Qwen3-8B-Fine-Tuned
90.0
94.8
Table 5: The accuracy of pairwise judges evaluated using the gold-standard judge’s labels as the ground-truth. The judges are based on Qwen3-8B, in both non-reasoning and reasoning modes.
Figure 10: Performance of Qwen3-8B policies trained with pairwise judges . The policy performance at different training steps on the test set is evaluated by the gold-standard judge and the judge used in training, by comparing against a baseline, GPT-4o.
Creative Writing
Hard Prompt
Model
Score (%)
Model
Score (%)
o3-2025-04-16
92.4
o3-2025-04-16
86.8
Pairwise Reasoning-Judge + Llama-3.1-8B
90.8
Pairwise Reasoning-Judge + Llama-3.1-8B
86.2
deepseek-r1
89.2
o4-mini-2025-04-16-high
81.2
Qwen3-235B-A22B
85.5
o4-mini-2025-04-16
77.4
gemini-2.5
85.2
o3-mini-2025-01-31-high
66.1
Table 6: Performance of the Llama-3.1-8B policy trained with the pairwise Qwen3-8B reasoning judge on Arena-Hard-V2.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Performance of policies trained with Qwen3-1.7B non-reasoning and reasoning judges, evaluated by (a) the training judge and (b) the gold-standard judge gpt-oss-120b.
Figure 12: Output example of the Llama-3.1-8B policy trained with the Qwen3-4B pointwise reasoning judge.
Figure 13: Performance of policies trained with Qwen3-14B non-reasoning judges under varying KL-penalty weights, evaluated on the test set by (a) the training judges and (b) the gold-standard judge gpt-oss-120b.
Figure 14: Performance of policies trained with Qwen3-4B as the reasoning judge, evaluated on the test set by (a) the gold-standard judge gpt-oss-120b and (b) Qwen3-4B.
Table 10: Full benchmark results of the “hard prompt” subset of Arena-Hard-v2 without style control.
Figure 15: Prompt Template for Pointwise LLM-Judge.
Figure 16: Prompt Template for Rubric Generation
Figure 17: Prompt Template for Pointwise LLM-Judge with Rubrics
Figure 18: Prompt Template for Pairwise LLM-Judge
Figure 19: Prompt Template used in the “hard prompt” subset of Arena-Hard.
Figure 20: Prompt Template used in the “creative writing” subset of Arena-Hard.
Figure 21: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 1).
Figure 22: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 2).
Figure 23: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 3).
Figure 24: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 4).
Figure 25: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 5).
Figure 26: Example of the judgment given by GPT-4.1 for the output of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge on Arena-Hard-V2 (Part 1).
Figure 27: Example of the judgment given by GPT-4.1 for the output of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge on Arena-Hard-V2 (Part 2). Assistant B is the Llama-3.1-8B policy, while Assistant A is the baseline system o3-mini-2025-01-31.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation. Additionally, surfacing reasoning mistakes that the model makes would enable improving the model's performance at runtime through providing feedback. Due to the difficulty of this complex task on long reasoning traces, single-model judges (even frontier models) do not do well at identifying reasoning defects. Additionally, leveraging frontier models during online training of reasoning LLMs is generally prohibited due to guardrails in terms of use. In this work, we introduce Reasoning Jury, a system that replaces the single judge with a jury of LLMs and a moderated consensus mechanism, to improve the fidelity of judgments for identifying reasoning defects. In reasoning jury, defects of a reasoning trace and their severity are surfaced through a deliberation where a moderator conducts a discussion amongst the jury where the jurors critique each other's judgments and get to modify their initial votes. The moderator derives a consensus through deliberation amongst jurors or consolidation of judgements. We show that Reasoning Jury with a jury of open-weight models (e.g., gpt-oss-120b) is able to significantly outperform frontier models (opus-4.6, sonnet-4.6, and gemini-3.1-pro) at correctly identifying reasoning defects. Besides accuracy performance improvements, the aggregated cost of the jury (initial verdicts, deliberations, consolidation, etc.) is a fraction (8 to 15%) of the cost of running frontier models in LLM-as-a-judge setup. We also show how these judgements can be leveraged to understand failure modes of reasoning LLMs on benchmarks, which allows much deeper understanding of a model's performance.
Reasoning-capable large language models (LLMs) have recently been adopted as automated judges, but their benefits and costs in LLM-as-a-Judge settings remain unclear. Through controlled comparisons between reasoning and non-reasoning judges, we show that explicit reasoning substantially improves judgment accuracy on tasks requiring structured verification (e.g., math and coding), while offering limited or even negative gains on simpler evaluations and incurring significantly higher computational cost. These findings motivate that reasoning should be used selectively rather than universally, with awareness of possible distribution shift. We propose a Robust Adaptive Cost-Efficient Routing (RACER), which dynamically selects between reasoning and non-reasoning judges under a fixed budget by formulating routing as a constrained distributionally robust optimization problem. RACER explicitly accounts for distribution shift via a KL-divergence uncertainty set, admits an efficient primal--dual algorithm, and enjoys theoretical guarantees including uniqueness of the optimal policy and linear convergence. Extensive experiments show that RACER achieves superior accuracy--cost trade-offs under distribution shift.
Wenbo Zhang, Lijinghua Zhang, Liner Xiang +1
Department of Statistics, University of California, Irvine, USA.