Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge

Yiyang Shen; Lifu Tu; Weiran Wang

cs.CLApr 3, 2026

Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge

Authors: Yiyang Shen, Lifu Tu, Weiran Wang

Organizations: University of Iowa · Oracle AI

Abstract

Reinforcement Learning (RL) substantially improves the reasoning capabilities of language models, but most existing RL fine-tuning approaches rely entirely on ground-truth verifiable rewards and thus labeled datasets with verifiable answers. To overcome this, we propose a RL framework for reasoning distillation that leverages continuous, LLM-based rewards. Our method employs an efficient mechanism that computes a continuous CoT reward (CCR) directly from a single-token logit of a judge LLM, evaluating the student model's reasoning trajectory. This formulation provides an effective and scalable online training signal that can be applied to large volumes of unlabeled data. We demonstrate that, when paired with a strong judge, simply using CCR achieves performance comparable to that of ground-truth or pseudo-label verifiable rewards, and even surpasses them as the amount of unlabeled data increases. Furthermore, we find that combining them in a semi-supervised setup is highly synergistic: verifiable rewards help stabilize the CCR, while CCR improves the generalizability of verifiable rewards to related tasks. We also provide a comprehensive empirical comparison of various reward sources across multiple model architectures and dataset sizes. Our results show that this semi-supervised approach consistently enhances mathematical reasoning, yielding an absolute improvement of 5-10% on multiple reasoning tasks.

Figures & tables

Dataset Train Test Supervised GSM8K 7,356 1,319 SVAMP 3,043 1,000 Unsupervised GSM8K-aug 385.5K — GSM-Plus 9,233 — Evaluation GSM-Plus — 9,233 GSM-Symbolic –main — 5,000 –p1 — 5,000 –p2 — 2,500

Dataset Train Test Supervised AIME 1983-2023 919 — Unsupervised DAPO-Math-17K 14,110 — Evaluation AIME 2024 — 30 AIME 2025 — 30 AIME 2026 — 30

Table 1: Datasets Statistics. Left : Grade School-level Math. Right : Competition-level Math.

Signal	Label	Judge	GSM8K	SVAMP
VR	✓	—	100.0	100.0
Likert		Qwen3-8B	86.5	87.5
	✓	Llama3.1-8B	72.0	80.5
		Gemma3-12B	79.5	85.5
RaR		Qwen3-8B	74.5	82.0
	✗	Llama3.1-8B	57.5	62.0

Table 2: Judge quality on Grade School-level Math.

Judge Candidate	CCR	LL	RaR	Likert
Qwen3-8B	91.0	93.6	89.7	97.4
Qwen3.5-9B	94.9	92.3	71.8	95.0
Gemma3-12B	84.6	37.2	61.5	96.2

Table 3: Judge quality on AIME.

Original GSM8K sup
Setup	GSM8K	GSM-Plus	GSM-Symbolic			SVAMP
			main	p1	p2
SFT	25.25	15.14	22.88	7.40	3.64	32.30
V	30.48	18.38	30.96	8.42	0.96	31.30
Pseudo-labeled GSM8K-aug sup
Base 1	28.89	16.78	25.00	6.48	2.16	44.60

Table 4: Results of Galactica-125M as measured by accuracy (%) on GSM8K. V = VR, C = CCR, and P = PRR. We apply equal weights (0.5) when combining two rewards, and set

\lambda=0.5

\mu=0.25

, and

\rho=0.25

for V+C+P.

GSM8K	GSM-Plus	GSM-Symbolic			SVAMP
		main	p1	p2
GSM8K sup + 5% GSM8K-aug unsup
36.54	22.85	31.96	10.24	0.96	43.10
GSM8K sup + 25% GSM8K-aug unsup
38.36	24.61	36.42	13.74	1.16	42.30
GSM8K sup + 100% GSM8K-aug unsup

Table 5: More and higher quality unsupervised data improves performance on GSM8K.

\lambda=0.5,\mu=0.25,\rho=0.25

Reported by Trung et al. (2024)
Setup	GSM8K	GSM-Plus	GSM-Symbolic			SVAMP
			main	p1	p2
V	68.9
Continuing from the above checkpoint
V	69.1	52.0	64.2	54.8	23.4	69.4
+ voting	71.7	53.3	67.4	57.9	27.0	70.8

Table 6: Results of finetuning Galactica-6.7B on GSM8K. Here

\lambda=0.5,\mu=0.5

for V+C.

Setup	GSM8K	GSM-Plus	GSM-Symbolic			SVAMP
			main	p1	p2
Original GSM8K only
SFT	22.97	14.07	23.40	8.34	2.96	28.20
V	26.91	16.35	29.90	10.54	4.16	34.30
GSM8K sup + GSM8K-aug unsup
V+C	31.01	19.11	33.86	11.54	4.72	38.50

Table 7: Results of CodeGen-350M on GSM8K.

Original SVAMP only
Setup	Galactica-125M	CodeGen-350M
SFT	37.40	35.80
V	40.10	41.40
GSM8K-aug unsup
C	44.50	$\times$
P	$\times$	$\times$

Table 8: Results on SVAMP.

\times

indicates model collapse. We apply equal weights (0.5) when combining two rewards, and set

\lambda=0.5

\mu=0.25

, and

\rho=0.25

for V+C+P.

Supervised
Setup	AIME’24	AIME’25	AIME’26
HF Ckpt	9.2	16.5	11.5
V	10.7	15.1	11.0
Unsupervised (until ’23)
C	12.0	16.9	12.7
Unsupervised (until ’23 + DAPO)

Table 9: Avg@32 results of Qwen3-0.6B-Thinking on AIME.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Performance comparison between PPO and GRPO. Learning curves for Galactica-125M trained on GSM8K. The V+C experiments use GSM8K-aug as the unsupervised dataset.

Setup	Algorithm	GSM8K	GSM-Plus	GSM-Symbolic			SVAMP
				main	p1	p2
V	PPO	30.48	18.38	30.96	8.42	0.96	31.30
V+C	PPO	35.10	22.14	34.08	12.76	3.32	40.70
V	GRPO	30.33	18.00	26.30	7.98	1.36	28.50
V+C	GRPO	37.30	23.71	35.36	11.18	2.20	45.20

Appendix

Table 10: Final accuracies obtained by PPO and GRPO for Galactica-125M. The V+C experiments use GSM8K-aug as the unsupervised dataset.

Panel of Table 11 — Table 11: GPT-5.6 Generated RaR criteria for GSM.

Panel of Table 12 — Table 12: GPT-5.6 Generated RaR criteria for AIME.

Likert
Judge	GSM8K	SVAMP
Qwen3-8B	360	204
Llama3.1-8B	1098	587
Gemma3-12B	760	456
RaR
Qwen3-8B	152	196

Appendix

Table 13: Runtime (seconds) on Grade School-level Math.

Judge Candidate	CCR	LL	RaR	Likert
Qwen3-8B	16	23	68	82
Qwen3.5-9B	34	48	117	153
Gemma3-12B	29	36	119	162

Appendix

Table 14: Runtime (seconds) on AIME.

Panel of Figure 2 — Figure 2: All three types of our active training reward on Galactica-125M in a semi-supervised setup.

Figure 3: Evaluation Accuracy of Galactica-125M on GSM8K during RL training.

Reported by Trung et al. (2024)
Coefficients			GSM8K	GSM-Plus	GSM-Symbolic			SVAMP
$\lambda$	$\mu$	$\rho$			main	p1	p2
SFT Complete			23.70	—	—	—	—	—
1.0	–	–	29.80	—	—	—	—	—
Our reproduction
SFT Warm-up			21.83	12.49	20.24	4.88	1.80	29.50

Appendix

Table 15: Galactica-125M trained on GSM8K. Results with coefficients.

$\lambda$	$\mu$	AIME’24	AIME’25	AIME’26
0.1	0.9	16.67	20.73	15.21
0.5	0.5	11.25	16.88	11.87
0.9	0.1	15.42	19.06	14.90

Appendix

Table 16: Avg@32 results of Qwen3-0.6B-thinking on AIME with semi-supervised RL.

Figure 4: Judge Mean Reward Curve

Panel of Figure 5 — Figure 5: Example completions with CCR scores only. Notice how the hacked response does not generate anything given the declaration and docstring, yet obtaining a good score.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
Mar 13, 2026Tianyang Luo, Tao Feng, Zhigang Hua +4RL for Language Model Reasoning RL Fine-Tuning
University of Illinois Urbana-Champaign · Meta
Reinforcement-aware Knowledge Distillation for LLM Reasoning
Feb 26, 2026Zhaoyang Zhang, Shuli Jiang, Yantao Shen +6Knowledge Distillation RL for Knowledge Distillation
AWS Agentic AI
Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier
Jun 15, 2026Keizo Kato, Chenhui Chu, Yugo Murawaki +1Pseudo-Labeling LLM Reasoning
Fujitsu Limited, Kawasaki, Japan · Kyoto University, Kyoto, Japan · National Institute of Informatics, Tokyo, Japan

Mar 13, 2026J/K move · Enter open · S save

MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels

Tianyang Luo, Tao Feng, Zhigang Hua, Yan Xie+3

University of Illinois Urbana-Champaign · Meta

RL for Language Model Reasoning RL Fine-Tuning Reward Modeling

Open paper AskPDF arXiv

Reinforcement learning has emerged as a powerful paradigm for improving large language model (LLM) reasoning, where rollouts are sampled from the policy and reward signals computed on those rollouts are used to update the policy. However, in data-scarce scenarios, obtaining ground-truth labels to verify rollouts at scale often requires expensive human annotation or labor-intensive expert verification. For instance, evaluating mathematical proofs demands expert review, and open-ended question answering lacks definitive ground truth. When ground-truth labels are scarce, the effectiveness of reinforcement learning fine-tuning is constrained. Inspired by the success of semi-supervised learning in propagating labels from labeled to unlabeled samples, we propose MemReward, a graph-based experience memory framework that integrates reward propagation directly into online policy optimization. MemReward stores rollouts (thinking processes and final answers) from an initial LLM policy as nodes in a heterogeneous graph connected by similarity and structural edges, over which a GNN propagates rewards from labeled to unlabeled rollouts. To train such a framework, we first warm up the GNN on labeled rollouts to predict rewards via heterogeneous aggregation over query, thinking, and answer nodes. During online RL fine-tuning, unlabeled rollouts are attached to the graph by query similarity, and the GNN predicts their rewards, yielding a hybrid reward acquisition strategy that combines ground-truth and GNN-predicted rewards. Experiments on Qwen2.5-1.5B and 3B in mathematics, question answering, and code generation demonstrate that MemReward, with ground-truth rewards on only 20% of rollouts, achieves 96.6% of Oracle performance on 1.5B and 97.3% on 3B, and closely approaches Oracle on out-of-domain tasks.

Abstract

RL for Language Model Reasoning

Explore similar work

MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels

Reinforcement-aware Knowledge Distillation for LLM Reasoning

Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier