cs.AIMar 12, 2026
SaveExamining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Organizations: Meta Superintelligence Labs · Yale Univeristy
Abstract
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.
Figures & tables
Figure 1: Illustration of our synthetic experiment setting (left). In the middle, we show that a Llama-3.1-8B policy trained with a fine-tuned Qwen3-4B reasoning judge can achieve strong performance under the gold-standard judge gpt-oss-120b’s evaluation, while the policy trained with a fine-tuned Qwen3-14B non-reasoning judge cannot and exhibits severe reward hacking. The table on the right shows results on the creative writing subset of Arena-Hard-V2. The Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge is able to achieve superior performance by learning to generate highly effective adversarial outputs.
Figure 2: Performance of various LLM-judges based on their agreement (Krippendorff’s Alpha) with the gold-standard judge, gpt-oss-120b. The LLM-judges are all based on Qwen3 models and grouped by their sizes. Both the base judges and fine-tuned judges are evaluated, and non-reasoning judges and reasoning judges are compared.
Figure 3: Performance comparison of policies trained with non-reasoning judges of different sizes under the gold-standard judge’s evaluation. The sub-figures show policies trained from different initial LLMs.
Figure 4: Performance of policies trained with non-reasoning judges of different sizes evaluated by the judges used in training .
Figure 5: Performance of policies trained with Qwen3-4B and Qwen3-8B reasoning judges . The policy performance at different training steps on the test set is shown when evaluated by the training judges (a) and by the gold-standard judge gpt-oss-120b (b).
Figure 6: Output example of the Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge after 1100 training steps.
| Creative Writing | Hard Prompt | ||
|---|---|---|---|
| Model | Score (%) | Model | Score (%) |
| o3-2025-04-16 | 92.4 | o3-2025-04-16 | 86.8 |
| Qwen3-4B Reasoning-Judge + Llama-3.1-8B | 89.6 | o4-mini-2025-04-16-high | 81.2 |
| deepseek-r1 | 89.2 | gemini-2.5 | 55.9 |
| gemini-2.5 | 85.2 | deepseek-r1 | 48.5 |
| o4-mini-2025-04-16-high | 79.8 | claude-3-7-sonnet-20250219-thinking-16k | 47.9 |
Table 1: Performance of the Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge on Arena-Hard-V2.
Figure 7: Performance of policies trained with RL-only reasoning judges . The policy performance at different training steps on the test set is shown when evaluated by the gold-standard judge (a) and by the judges used in training (b).
| Training Method / Base Model | Qwen3-4B | Qwen3-8B |
|---|---|---|
| Distillation+RL | 85.94 | 89.34 |
| RL-Only | 85.10 | 85.99 |
Table 2: Performance of reasoning judges trained with RL only compared to judges trained with both SFT distillation and RL. The judges’ agreements with the gold-standard judge are reported.
Figure 8: Performance of policies trained with non-reasoning judges provided with rubrics generated by the gold-standard judge. The policy performance at different training steps on the test set is shown when evaluated by the gold-standard judge (a) and by the judges used in training (b).
| Judge | w/o Rubrics | w/ Rubrics |
|---|---|---|
| Qwen3-14B | 41.73 | 60.90 |
| Qwen3-14B-Fine-Tuned | 87.82 | 89.72 |
Table 3: Performance comparison of non-reasoning judges prompted/fine-tuned with and without the rubrics generated by the gold-standard judge. The judges’ agreements with the gold-standard judge (Krippendorff’s Alpha) are reported.
Figure 9: Performance of policies trained with reasoning judges with low and medium reasoning efforts fine-tuned from Qwen3-8B. The policy performance at different training steps on the test set is shown when evaluated by the judges used in training (a) and by the gold-standard judge (b).
| Reasoning Effort | Agreement | #Tokens |
|---|---|---|
| Low | 79.88 | 43.2 |
| Medium | 85.99 | 200.3 |
| High | 89.34 | 981.6 |
Table 4: Performance of Qwen3-8B reasoning judges with varied reasoning efforts. We report their agreement with the gold-standard judge and the average number of tokens in their thinking traces.
| Judge | Non-Reasoning | Reasoning |
|---|---|---|
| Qwen3-8B | 79.8 | 85.0 |
| Qwen3-8B-Fine-Tuned | 90.0 | 94.8 |
Table 5: The accuracy of pairwise judges evaluated using the gold-standard judge’s labels as the ground-truth. The judges are based on Qwen3-8B, in both non-reasoning and reasoning modes.
Figure 10: Performance of Qwen3-8B policies trained with pairwise judges . The policy performance at different training steps on the test set is evaluated by the gold-standard judge and the judge used in training, by comparing against a baseline, GPT-4o.
| Creative Writing | Hard Prompt | ||
|---|---|---|---|
| Model | Score (%) | Model | Score (%) |
| o3-2025-04-16 | 92.4 | o3-2025-04-16 | 86.8 |
| Pairwise Reasoning-Judge + Llama-3.1-8B | 90.8 | Pairwise Reasoning-Judge + Llama-3.1-8B | 86.2 |
| deepseek-r1 | 89.2 | o4-mini-2025-04-16-high | 81.2 |
| Qwen3-235B-A22B | 85.5 | o4-mini-2025-04-16 | 77.4 |
| gemini-2.5 | 85.2 | o3-mini-2025-01-31-high | 66.1 |
Table 6: Performance of the Llama-3.1-8B policy trained with the pairwise Qwen3-8B reasoning judge on Arena-Hard-V2.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Performance of policies trained with Qwen3-1.7B non-reasoning and reasoning judges, evaluated by (a) the training judge and (b) the gold-standard judge gpt-oss-120b.
Figure 12: Output example of the Llama-3.1-8B policy trained with the Qwen3-4B pointwise reasoning judge.
Figure 13: Performance of policies trained with Qwen3-14B non-reasoning judges under varying KL-penalty weights, evaluated on the test set by (a) the training judges and (b) the gold-standard judge gpt-oss-120b.
Figure 14: Performance of policies trained with Qwen3-4B as the reasoning judge, evaluated on the test set by (a) the gold-standard judge gpt-oss-120b and (b) Qwen3-4B.
| Rank | Model | Score (%) | Confidence Interval (%) |
|---|---|---|---|
| 1 | o3-2025-04-16 | 92.4 | (-1.1 / +1.0) |
| 2 | Pairwise Qwen3-8B-Reasoning Judge + Llama-3.1-8B-Instruct Policy | 90.8 | (-2.3 / +1.8) |
| 3 | Pointwise Qwen3-4B-Reasoning-Judge + Llama-3.1-8B-Instruct Policy | 89.6 | (-2.4 / +1.7) |
| 4 | deepseek-r1 | 89.2 | (-1.4 / +1.2) |
| 5 | Qwen3-235B-A22B | 85.5 | (-1.7 / +1.9) |
| 6 | gemini-2.5 | 85.2 | (-2.6 / +2.5) |
Table 7: Full benchmark results of the “creative writing” subset of Arena-Hard-V2 with style control.
| Rank | Model | Score (%) | Confidence Interval (%) |
|---|---|---|---|
| 1 | o3-2025-04-16 | 86.8 | (-0.9 / +1.3) |
| 2 | Pairwise Qwen3-8B-Reasoning Judge + Llama-3.1-8B-Instruct Policy | 86.2 | (-1.3 / +1.7) |
| 3 | o4-mini-2025-04-16-high | 81.2 | (-1.3 / +1.1) |
| 4 | o4-mini-2025-04-16 | 77.4 | (-1.4 / +1.4) |
| 5 | o3-mini-2025-01-31-high | 66.1 | (-1.8 / +2.0) |
| 6 | gpt-4.1 | 60.6 | (-2.6 / +1.8) |
Table 8: Full benchmark results of the “hard prompt” subset of Arena-Hard-v2 with style control.
| Rank | Model | Score (%) | Confidence Interval (%) |
|---|---|---|---|
| 1 | Pairwise Qwen3-8B-Reasoning Judge + Llama-3.1-8B-Instruct Policy | 99.2 | (-0.4 / +0.4) |
| 2 | o3-2025-04-16 | 89.3 | (-1.3 / +1.4) |
| 3 | gemini-2.5 | 88.8 | (-1.5 / +1.5) |
| 4 | deepseek-r1 | 85.9 | (-2.1 / +1.9) |
| 5 | Pointwise Qwen3-4B-Reasoning-Judge + Llama-3.1-8B-Instruct Policy | 85.6 | (-1.5 / +1.6) |
| 6 | Qwen3-235B-A22B | 83.4 | (-1.9 / +2.0) |
Table 9: Full benchmark results of the “creative writing” subset of Arena-Hard-V2 without style control.
| Rank | Model | Score (%) | Confidence Interval (%) |
|---|---|---|---|
| 1 | Pairwise Qwen3-8B-Reasoning Judge + Llama-3.1-8B-Instruct Policy | 97.2 | (-0.5 / +0.5) |
| 2 | o3-2025-04-16 | 80.8 | (-1.7 / +1.5) |
| 3 | gemini-2.5 | 70.1 | (-1.8 / +1.9) |
| 4 | o4-mini-2025-04-16-high | 67.9 | (-1.7 / +2.0) |
| 5 | gemini-2.5-flash | 64.9 | (-1.7 / +1.9) |
| 6 | gpt-4.1 | 61.9 | (-2.1 / +2.1) |
Table 10: Full benchmark results of the “hard prompt” subset of Arena-Hard-v2 without style control.
Figure 15: Prompt Template for Pointwise LLM-Judge.
Figure 16: Prompt Template for Rubric Generation
Figure 17: Prompt Template for Pointwise LLM-Judge with Rubrics
Figure 18: Prompt Template for Pairwise LLM-Judge
Figure 19: Prompt Template used in the “hard prompt” subset of Arena-Hard.
Figure 20: Prompt Template used in the “creative writing” subset of Arena-Hard.
Figure 21: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 1).
Figure 22: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 2).
Figure 23: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 3).
Figure 24: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 4).
Figure 25: Output example of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge (Part 5).
Figure 26: Example of the judgment given by GPT-4.1 for the output of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge on Arena-Hard-V2 (Part 1).
Figure 27: Example of the judgment given by GPT-4.1 for the output of the Llama-3.1-8B policy trained with the Qwen3-8B pairwise reasoning judge on Arena-Hard-V2 (Part 2). Assistant B is the Llama-3.1-8B policy, while Assistant A is the baseline system o3-mini-2025-01-31.
More in this topic
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 537 papers