REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

Yasi Zhang; Tianyu Chen; Mingyuan Zhou; Oscar Leong; Ying Nian Wu; Michal Lukasik

cs.LGMar 17, 2026

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

Authors: Yasi Zhang, Tianyu Chen, Mingyuan Zhou, Oscar Leong, Ying Nian Wu, Michal Lukasik

Organizations: University of California, Los Angeles · The University of Texas at Austin · Google Research

Abstract

Large language models (LLMs) are increasingly deployed as automated evaluators that assign numeric scores to model outputs, a paradigm known as LLM-as-a-Judge. However, standard Reinforcement Learning (RL) methods typically rely on binary rewards (e.g., 0-1 accuracy), thereby ignoring the ordinal structure inherent in regression tasks; for instance, they fail to recognize that predicting 4 is significantly better than predicting 1 when the ground truth is 5. Conversely, existing regression-aware approaches are often confined to Supervised Fine-Tuning (SFT), limiting their ability to explore optimal reasoning paths. To bridge this gap, we propose REAL (REgression-Aware Reinforcement Learning), a principled RL framework designed to optimize regression rewards, and also proven to be optimal for correlation metrics. A key technical challenge is that the regression objective is explicitly policy-dependent, thus invalidating standard policy gradient methods. To address this, we employ the generalized policy gradient estimator, which naturally decomposes optimization into two complementary components: (1) exploration over Chain-of-Thought (CoT) trajectory, and (2) regression-aware prediction refinement of the final score. Extensive experiments across model scales (8B to 32B) demonstrate that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods, exhibiting significantly better generalization on out-of-domain benchmarks. On Qwen3-32B specifically, we achieve gains of +8.40 Pearson and +7.20 Spearman correlation over the SFT baseline, and +18.30/+11.20 over the base model. These findings highlight the critical value of integrating regression objectives into RL exploration for accurate LLM evaluation.

Figures & tables

Figure 1: Overview of the REAL framework. REAL addresses the limitations of standard RL in LLM-as-a-Judge tasks by optimizing a policy-dependent regression reward. The framework employs a generalized policy gradient that leads to a gradient update that decomposes into two terms: (1) Exploration Over Reasoning Trajectory ; and (2) Regression-Aware Prediction Refinement . This enables principled optimization of ordinal structures that standard RL with binary rewards typically ignores. The full algorithm is in Alg. 2 .

Figure 2: Evaluation performance during RL training. Both standard RL with binary reward (i.e., $r_{\text{acc}}=\mathbf{1}(y=y^{*})$ ) and REAL with regression-aware reward (i.e., Eq. 5 ) were initialized from the SOTA SFT checkpoint, i.e., TRACT ( Chiang et al., 2025 ) . Standard RL results in suboptimal performance in correlation metrics compared to our proposed approach: REAL.

Figure 3: Performance gains on Qwen3-32B. Our method achieves an average increase of +8.40 Pearson and +7.20 Spearman correlation over the SFT baseline, and +18.30/+11.20 over the base model.

Method	Paradigm	CoT Exploration	Ordinal Awareness	Gradient
SFT	SFT	✗	✗	$\mathbb{E}_{(x,y^{})\sim\mathcal{D}}\left[\nabla_{\theta}\log\pi_{\theta}(y^{}\|x)\right]$
RAFT/TRACT	SFT	✗	✓	$\mathbb{E}_{(x,c,y^{*})\sim\mathcal{D}}\left[\nabla_{\theta}r(\theta,x,c)\right]$
Standard RL	RL	✓	✗	$\mathbb{E}_{x\sim\mathcal{D},(c,y)\sim\pi_{\theta}(\cdot\mid x)}\left[r(x,y)\nabla_{\theta}\log\pi_{\theta}(y,c\|x)\right]$
JEPO ( Tang et al., 2025 )	RL	✓	✗	$\mathbb{E}_{x\sim\mathcal{D},c\sim\pi_{\theta}(\cdot\mid x)}\left[\log\pi_{\theta}(y^{}\|x,c)\nabla_{\theta}\log\pi_{\theta}(c\|x)+\nabla_{\theta}\log\pi_{\theta}(y^{}\|x,c)\right]$
REAL (ours)	RL	✓	✓	$\mathbb{E}_{(x,y^{*})\sim\mathcal{D},c\sim\pi_{\theta}(\cdot\mid x)}\left[r(\theta,x,c)\nabla_{\theta}\log\pi_{\theta}(c\mid x)+\nabla_{\theta}r(\theta,x,c)\right]$

Table 1: Conceptual comparison of selected SFT and RL training methods on non-ordinal and ordinal targets.

r(\theta,x,c)=-(\hat{y}_{\theta}(x,c)-y^{*})^{2}+\lambda\log\pi_{\theta}(y^{*}|x,c)

represents the regression-aware reward. REAL is the first framework to combine a regression-aware reward with reinforcement learning for reasoning exploration. Note that JEPO further approximates the gradient with the multi-sample Jensen’s Bound.

Method	Training	Inference	FB Bench			FLASK			Vic. Bench			MT Bench			Average
			$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$
Baseline Reward Models
GPT-3.5-Turbo-0613	None	Standard	56.3	52.1	45.3	27.0	23.2	18.7	27.5	26.7	20.2	42.2	37.1	29.9	38.3	34.8	28.5
Prometheus-1-7B	SFT	Standard	84.7	84.9	76.7	45.7	45.7	36.5	29.3	29.5	21.6	36.7	37.1	28.5	49.1	49.3	40.8
Prometheus-1-13B	SFT	Standard	86.0	85.8	77.1	46.6	42.9	34.6	47.3	45.1	34.1	46.7	45.5	34.5	56.7	54.8	45.1
Prometheus-2-7B	SFT	Standard	84.5	84.7	76.5	51.2	49.3	40.5	48.8	48.0	41.1	51.9	48.3	39.2	59.1	57.6	49.3

Table 2: Evaluation results (

\times 100

) across benchmarks using Pearson (

r

), Spearman (

\rho

), and Kendall (

\tau

). All baseline reward model scores are taken from TRACT ( Chiang et al., 2025 ) and Prometheus ( Kim et al., 2023 ) . Results for the Mistral and Qwen models are obtained from our own runs. Underline indicates the initialization checkpoint used by REAL. We find that TRACT saturates on the Qwen3 series models, so we choose RAFT as the initialization checkpoint. Mistral2-7B’s base model failed to follow instructions and respond in the correct format, so we applied a 100-step warmup. Due to computational constraints we run only RAFT and REAL for Qwen3-32B; we include all baselines for other models.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
$N=1$	67.9	66.2	53.0	0.968	0.697
$N=4$	68.1	66.2	53.0	0.965	0.695
$N=10$	68.1	66.2	53.0	0.964	0.695

Table 3: Average-of-N results at inference. Averaging the results of

N

generations slightly improves performance, although REAL already enjoys efficiency and strong performance at

N=1

Method	Type	$r$	$\rho$	$\tau$
TRACT	SFT	63.2	62.2	49.8
Standard RL(acc)	RL	64.1	62.7	50.3
Standard RL(reg)	RL	63.4	62.6	50.1
REAL	RL	67.9	66.2	53.0

Table 4: Comparison between standard RL baselines and REAL on Mistral2-7B. Standard RL(reg) with regression-aware policy-independent rewards also fail to improve upon the SFT checkpoint.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
$K=4$	63.7	62.6	50.2	1.063	0.775
$K=8$	67.2	65.0	52.0	0.943	0.679
$K=16$	67.9	66.2	53.0	0.968	0.697

Table 5: Ablation on group size

K

. Larger values of

K

generally result in better correlation performance.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
TRACT	63.2	62.2	49.8	1.103	0.813
$\lambda=0$	66.0	64.6	51.8	1.027	0.736
$\lambda=1$	67.9	66.2	53.0	0.968	0.697
$\lambda=5$	68.1	65.8	52.7	0.955	0.719
$\lambda=10$	68.1	66.1	52.9	0.928	0.699

Table 6: Ablation on log-likelihood weight

\lambda

. Using regression-aware reward alone (

\lambda=0

) already yields excellent results, while extra log-likelihood reward further improves.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
$0<\text{acc}<1$	67.2	65.0	52.0	0.943	0.679
$0<\text{acc}\leq 1$	65.6	64.1	51.3	1.000	0.711
$0\leq\text{acc}<1$	65.3	63.4	50.9	1.001	0.728
$0\leq\text{acc}\leq 1$	65.5	63.9	51.1	0.990	0.715

Table 7: Ablation on dynamic sampling when group size

K=8

. Partially correct prompts provide the best learning signal.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
base	52.1	50.8	39.5	1.225	0.901
init. from base	58.8	56.8	45.1	1.155	0.793
$\Delta$	+6.7	+6.0	+5.6	-0.070	-0.108
SFT	63.2	62.2	49.8	1.103	0.813
init. from SFT	67.9	66.2	53.0	0.968	0.697
$\Delta$	+4.7	+4.0	+3.2	-0.135	-0.116

Table 8: Initialization from different checkpoints. Initializing from a high-quality SFT checkpoint can lead to better SOTA performance. Our algorithm works with different checkpoint types.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
Init. from Base
$\beta=0$	57.7	56.9	45.1	1.200	0.831
$\beta=0.01$	58.8	56.8	45.1	1.155	0.793
$\beta=1$	57.9	56.4	44.4	1.184	0.822
Init. from SFT
$\beta=0$	67.3	65.9	52.5	0.980	0.701

Table 9: Ablation on prediction weight

\beta

. We choose

\beta=0.01

(See Eq. 8 ) for initialization from both base and SFT checkpoints.

Setting	$r$ $\uparrow$	$\rho$ $\uparrow$	$\tau$ $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$
$T=0.8$	66.8	64.7	51.9	0.973	0.725
$T=1$	67.9	66.2	53.0	0.968	0.697

Table 10: Ablation on temperature

T

T=1

is chosen by default. The temperature parameter is applied in both the training and inference stages, for both CoT generation and score prediction.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Type	Model Size	Hardware	GPU Hours
Prometheus2	SFT	7B	8 $\times$ A100 (40GB)	800
TRACT	SFT	7B	1 $\times$ RTX A6000	100
Standard RL	RL	7B	8 $\times$ A100 (80GB)	100
REAL	RL	7B	8 $\times$ A100 (80GB)	100
REAL	RL	32B	16 $\times$ A100 (80GB)	500

Appendix

Table 11: Computation cost comparison across methods.

	Preference-Bench			HHH	MT-Bench	Auto-J
Method	Pearson $\uparrow$	Spearman $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$
Base	9.30	17.6	12.2	50.7%	64.8%	64.7%
TRACT	93.9	92.8	82.3	75.6%	75.6%	79.2%
REAL	93.4	92.8	84.1	76.9%	76.3%	79.7%

Appendix

Table 12: Results on score-difference prediction using the Preference_Collection dataset. REAL achieves better out-of-domain generalization while maintaining competitive in-domain performance.

Response Model	Judge Model	Agree w/ GPT-4.1 $\uparrow$	Agree w/ Gemini-2.5 $\uparrow$
o4-mini-2025-04-16	Base	5	1
	TRACT	71	55
	REAL	81	67
Gemini-2.5	Base	0	2
	TRACT	56	49
	REAL	81	71

Appendix

Table 13: Agreement results on ArenaHard 2.0. We report agreement counts (out of 750) with closed-source judges GPT-4.1 and Gemini-2.5. REAL consistently achieves the strongest alignment across different response-model families.

Panel of Figure 4 — Figure 4: Response length and entropy during REAL training. Response length increases, and the per-token entropy of the policy model decreases steadily.

Method	FB Bench			FLASK			Vic. Bench			MT Bench			Average
	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$	$r$	$\rho$	$\tau$
JEPO	92.5	92.9	81.6	54.3	52.6	39.6	61.7	59.7	45.9	55.7	54.9	39.8	66.3	64.8	51.8
REAL	93.2	93.4	82.5	56.0	54.1	41.1	63.3	60.2	46.3	59.3	56.9	42.2	67.9	66.2	53.0

Appendix

Table 14: Comparison with JEPO with Mistral2-7B.

Sample Question (Vicuna Bench)

Task Description:

An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.

1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.

2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.

3. The output format should look as follows: ”Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)

4. Please do not generate any other opening, closing, and explanations.

Appendix

Table 19

GPT4 Output: The response is comprehensive and offers a series of detailed and actionable steps for improving time management skills . Steps such as prioritizing tasks , using a calendar or planner, setting goals, eliminating distractions, taking breaks, learning to say no, and reviewing and adjusting schedules can be directly implemented by anyone. These steps are firmly grounded on real-world scenarios , making them practical and beneficial for individuals looking to enhance their time management skills.

REAL Output: The response provided is comprehensive and offers detailed and actionable steps to improve time management skills. It covers various aspects such as setting goals, prioritizing tasks , creating a schedule, breaking down tasks, eliminating distractions, delegating, using productivity tools, time-blocking, learning to say no, and taking breaks. The response also emphasizes the importance of self-care and mental health, which is essential in real-world scenarios.

Score: 5

Greedy decoding: 5, RAIL inference: 4.99

Appendix

Table 15: Example generation by REAL. GPT4 output serves as the reference ground truth answer in the dataset.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

Apr 15, 2026

On Cost-Effective LLM-as-a-Judge Improvement Techniques

Using a language model to score or rank candidate responses has become a scalable alternative to human evaluation in reinforcement learning from human feedback (RLHF) pipelines, benchmarking, and application layer evaluations. However, output reliability depends heavily on prompting and aggregation strategy. We present an empirical investigation of four drop-in techniques -- ensemble scoring, task-specific criteria injection, calibration context, and adaptive model escalation -- for improving LLM judge accuracy on RewardBench 2, with a unifying lens of noise control on the stochastic judge: ensembling as Monte Carlo averaging over per-call noise, criteria injection as between-response discrimination sharpening, and per-response score variance as an uncertainty signal. Ensemble scoring and task-specific criteria injection (the latter virtually cost free) together reach up to 85.8% accuracy, +13.5pp over baseline. Calibration context and adaptive model escalation also improve over baseline but are dominated by criteria + ensembling on the cost-accuracy Pareto frontier. Small models benefit disproportionately from ensembling, making high-accuracy LLM judges accessible at low cost. We show that these techniques generalise across model providers, evaluating on both OpenAI GPT and Anthropic Claude families.

Ryan Lail, Luke Markham

Composo AI.

LLM-as-a-Judge LLM Evaluation Prompt Engineering Reward Model Evaluation

Ask PDF

Jun 25, 2026

Reinforcement Learning without Ground-Truth Solutions can Improve LLMs

Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applicability to tasks where the ground-truth solution is unknown. We introduce a Ranking-induced VERifiable framework (RiVER) that trains LLMs on score-based optimization tasks without ground-truth solutions, using deterministic execution feedback as continuous-valued supervision. When applying group-relative RL to such continuous rewards, we identify two key challenges: scale dominance, where uncalibrated score magnitudes across test instances distort policy updates, and frequency dominance, where repeatedly sampled suboptimal solutions can outweigh rare but stronger candidates. RiVER addresses these challenges with calibrated reward shaping that uses instance-wise comparisons and emphasizes top-ranked solvers while retaining bounded feedback for other valid solutions. We train on 12 AtCoder Heuristic Contest tasks and evaluate on Algorithm Engineering Benchmark (ALE-Bench), LiveCodeBench, and USACO. RiVER advances Qwen3-8B and GLM-Z1-9B-0414 by 8.9% and 9.4% in ALE rating rank. More importantly, despite training exclusively on score-based tasks without any ground-truth solutions, RiVER also improves the backbones across exact-solution benchmarks such as LiveCodeBench and USACO by an absolute average improvement of 2.4% and 3.5%. By contrast, baselines trained with raw execution scores improve ALE rating but fail to transfer to exact-solution benchmarks. These results suggest that score-based optimization tasks, combined with proper reward calibration, can serve as effective training environments for general coding ability without ground-truth solutions.

Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang +6

University of California, San Diego · Snowflake AI Research

RL for Code Generation Reinforcement Learning Reinforcement Learning with Verifiable Rewards RL Post-Training Reward Design for RL

Ask PDF

May 20, 2026

Reinforcement Learning over Predictive Distributions for LLM Regression

Large language models (LLMs) have emerged as flexible regressors capable of predicting real-valued quantities from heterogeneous inputs. Yet most LLM regression objectives optimize predictions independently, often yielding poor calibration. We introduce Distribution-Aware Reward (DAR), an on-policy reinforcement learning objective that instead jointly evaluates the empirical predictive distribution formed by multiple predictions for the same input. To translate this distribution-level objective into rollout-level rewards, we assign each prediction credit based on its leave-one-out contribution to the quality of the overall predictive distribution. This encourages predictions that are well-centered and appropriately dispersed around the target. We evaluate on three regression settings: a synthetic task probing interpolation and extrapolation, and two real-world scientific tasks involving code and molecular data. Across tasks, DAR produces better-calibrated uncertainty estimates while consistently reducing prediction error and improving ranking quality over supervised fine-tuning and pointwise reinforcement learning. Together, these results highlight the benefits of distribution-aware training for LLM regression.

Jungsoo Park, Hyungjoo Chae, Ethan Mendes +4

Georgia Institute of Technology · Allen Institute for AI

Distributional RL RL for Language Models Language Model Calibration

Ask PDF

Abstract

RL for Language Models

Explore similar work

On Cost-Effective LLM-as-a-Judge Improvement Techniques

Reinforcement Learning without Ground-Truth Solutions can Improve LLMs

Reinforcement Learning over Predictive Distributions for LLM Regression