Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning

Jinhao Pan; Bowen Wei; Yuqing Zhou; Ziwei Zhu

cs.AIOct 13, 2026

Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning

Authors: Jinhao Pan, Bowen Wei, Yuqing Zhou, Ziwei Zhu

Organizations: George Mason University

Abstract

Reference solutions and question-specific rubrics are not always available to guide the teacher in self-distillation. A large language model's own solution attempts offer an alternative, but their quality is difficult to assess in open-ended tasks without automatic correctness checks, and even correct attempts can involve uncertain token predictions. We propose Reliability--Uncertainty Self-Distillation (\method{}) to learn from these attempts. The same model serves as teacher and student: the teacher receives the question and completed attempt, and the student matches the teacher's token-level predictions using a question-only prompt. \method{} weights each attempt's distillation loss by additively combining judged reliability, estimated from a self-judge's score and confidence, with student predictive uncertainty. Across eleven reasoning benchmarks, \method{} outperforms all five baselines in average accuracy, reaching 70.47% on Qwen3-4B and 76.69% on Qwen3-8B (8.90 and 2.10 percentage points above the base models). Data and code are available at https://github.com/JP-25/RUSD.

Figures & tables

Figure 1: Overview of RUSD. 1

\pi_{\theta}

generates attempt

\mathbf{y}

for question

\mathbf{x}

. Teacher and student predict next-token distributions from the same response prefix

\mathbf{y}_{<t}

under different prompts; only the teacher prompt includes the complete attempt. The teacher generates no additional response. 2 Judge score

s

, normalized to

\bar{s}

, and confidence

c^{\mathrm{J}}

give judged reliability

r^{\mathrm{J}}

. 3 Normalized student entropy averaged over the response gives uncertainty

u^{\mathrm{S}}

. 4 A weighted sum of reliability and uncertainty gives response weight

w

. 5 The weighted distillation loss updates the shared model. Numbers match Sections 3.1 – 3.5 ; bars are schematic.

Qwen3-4B
	Science and general reasoning					Mathematical reasoning					Logic	Overall
Method	GPQA	SciB	MMLU	LiveB	OBQA	GSM8K	MATH	AMC	Minerva	Olymp	Zebra	Average
Base	41.16	35.77	78.12	46.50	91.40	93.03	81.68	62.35	29.04	46.03	72.13	61.56
SFT †	42.30	35.91	78.19	42.00	91.00	93.18	80.50	54.22	29.41	44.51	72.10	60.30
GRPO †	43.81	43.89	78.85	47.00	93.00	93.78	81.86	63.86	31.25	47.92	72.58	63.44
RCSD †	41.41	42.56	78.66	49.50	92.80	93.25	81.64	61.45	30.15	45.99	71.53	62.63

Table 1: Accuracy (Avg@4, %) on eleven reasoning benchmarks. RUSD achieves the highest macroaverage at both model sizes. Average is the unweighted mean over benchmarks; bold marks the best score in each column within a model size, including ties. † Uses supervision beyond the training questions and the model’s own outputs: dataset responses (SFT, OPSD), generated rubrics (RCSD), or an external LLM judge (GRPO). GPQA: GPQA Diamond; SciB: SciBench; LiveB: LiveBench Reasoning; OBQA: OpenBookQA; Minerva: Minerva Math; Olymp: OlympiadBench; Zebra: ZebraLogic.

Panel of Figure 2 — Figure 2: Judge assessments and student uncertainty during Qwen3-4B RUSD training with $\alpha=0.5$ . (a) Distribution of judge scores over completed responses in summaries reported at training steps 100, 200, and 300. (b) Percentage of high-confidence judgments ( $c_{i}^{\mathrm{J}}\geq 0.7$ ) within the score-4 and score-5 categories; shading shows 95% Wilson confidence intervals for these proportions. (c) Mean student uncertainty over completed score-4/5 responses, with shading spanning the 25th–75th percentiles.

Panel of Figure 3 — Figure 3: Response-group composition and low student uncertainty during Qwen3-4B RUSD training with $\alpha=0.5$ . High judge confidence means $c_{i}^{\mathrm{J}}\geq 0.7$ . (a) Shares of completed score-4/5 responses in four groups defined by judge score and confidence. (b) Percentage of responses within each high-confidence group whose student uncertainty is below the pooled mean of all completed score-4/5 responses at that reporting step. Shading shows 95% Wilson confidence intervals from group response counts. Both panels use the same response population at each reporting step (Appendix B ).

Predictors	Multiple $R$	$R^{2}$ (%)	$p$ -value
Judge score	0.46	21	$8.9\times 10^{-12}$
Judge confidence	0.62	39	$5.0\times 10^{-23}$
Score + confidence	0.77	59	$7.2\times 10^{-39}$

Table 2: Correlation of judge score and confidence with GPT scores ( gpt-5.6sol quality ratings) on 200 responses. The multiple correlation coefficient

R

is the Pearson correlation between fitted and observed GPT scores;

R^{2}

is in-sample explained variation. The

p

-values are from overall regression F-tests.

Response weighting	Average
Score only: $w_{i}=\bar{s}_{i}$	68.44
Uncertainty only: $w_{i}=u_{i}^{\mathrm{S}}$	69.93
Reliability only: $w_{i}=r_{i}^{\mathrm{J}}$	70.12
RUSD: $w_{i}=0.5r_{i}^{\mathrm{J}}+0.5u_{i}^{\mathrm{S}}$	70.47
Loss type	Average
Forward KL	70.47

Table 3: Response-weighting and loss ablations on Qwen3-4B, reporting average accuracy (Avg@4, %) across eleven benchmarks. Weighting variants use the forward-KL configuration; loss variants use the full weighting rule at

\alpha=0.5

. Jensen–Shannon divergence uses mixture coefficient 0.5.

Figure 4: Effect of the mixing coefficient

\alpha

on Qwen3-4B RUSD, measured by average accuracy across eleven benchmarks. All settings use the dynamic teacher. The endpoints,

\alpha=0

(uncertainty only) and

\alpha=1

(reliability only), match the corresponding runs in Table 3 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Student (user turn)

Question: {question}

Please reason step by step, and put your final answer within \boxed{}.

Teacher (user turn)

Question: {question}

Here is a candidate analysis that may contain useful or misleading information:

=== Candidate Analysis Begin === {attempt} === Candidate Analysis End ===

Appendix

Table 4: Student, attempt-conditioned teacher, and judge prompts used by RUSD. The judge applies a common five-level quality rubric.

Parameter	Setting
Optimizer	AdamW
Learning rate	$5\times 10^{-6}$
Epoch budget (maximum)	3
Effective batch size	32
Training steps for reported results	150
Attempts per sampled question	1

Appendix

Table 5: Key training hyperparameters for the main Qwen3-4B and Qwen3-8B RUSD runs.

Benchmark	Score only	RUSD	$\Delta$
GPQA Diamond	52.40	52.78	+0.38
SciBench	66.11	70.66	+4.55
MMLU	78.12	79.38	+1.26
LiveBench Reasoning	58.50	61.25	+2.75
OpenBookQA	91.75	92.60	+0.85
GSM8K	94.56	94.81	+0.25

Appendix

Table 6: Qwen3-4B accuracy (Avg@4, %) under score-only and RUSD weighting.

\Delta

is RUSD’s absolute gain over score-only weighting on the same 0–100 scale. RUSD improves all eleven benchmarks.

Figure 5: Average accuracy (Avg@4, %) across eleven benchmarks for Qwen3-4B RUSD with

\alpha=0.5

at six evaluated training steps from 50 to 300. The highest observed value is 70.47% at step 150. The dashed line marks the strongest Qwen3-4B baseline average (NSD, 65.43%; Table 1 ).

Question. A solid has a melting point of $90.0^{\circ}$ C. Calculate the total heat required to change 2.5 kg of this solid from $40.0^{\circ}$ C to a liquid, given that the specific heat of the solid is 390 J/kg-K and its heat of fusion is 4000 J/kg. Show your work and explain your reasoning.
Earlier response (step 30)	Later response (step 105)
To determine the total heat required to change 2.5 kg of a solid from 40.0 ∘ C to a liquid, we need to account for two distinct processes: 1. Heating the solid from its initial temperature to its melting point 2. Melting the solid at its melting point into a liquid	To calculate the total heat required to change 2.5 kg of a solid from 40.0 ∘ C to a liquid, we need to consider two distinct processes: 1. Heating the solid from its initial temperature (40.0 ∘ C) to its melting point (90.0 ∘ C). 2. Melting the solid at its melting point (90.0 ∘ C) into a liquid.
Step 1: Heating the solid to its melting point The temperature difference between the initial temperature and the melting point is: $\Delta T=90.0^{\circ}C-40.0^{\circ}C=50.0^{\circ}C$ The heat required to raise the temperature of the solid is calculated using the specific heat formula: $Q_{1}=m\cdot c\cdot\Delta T$ Where: $\bullet$ $m=2.5\,\text{kg}$ $\bullet$ $c=390\,\text{J/kg}\mathord{\cdot}\text{K}$ $\bullet$ $\Delta T=50.0\,\text{K}$ (since a change of 1 ∘ C = 1 K) $\displaystyle Q_{1}$ $\displaystyle=2.5\,\text{kg}\cdot 390\,\text{J/kg}\mathord{\cdot}\text{K}\cdot 50.0\,\text{K}$ $\displaystyle=48750\,\text{J}$	Step 1: Heat Required to Raise the Temperature of the Solid The formula for the heat required to change the temperature of a substance is: $Q_{1}=m\cdot c\cdot\Delta T$ Where: $\bullet$ $m=2.5\,\text{kg}$ (mass of the solid), $\bullet$ $c=390\,\text{J/kg-K}$ (specific heat of the solid), $\bullet$ $\Delta T=T_{\text{melt}}-T_{\text{initial}}$ $=90.0^{\circ}\text{C}-40.0^{\circ}\text{C}=50.0^{\circ}\text{C}$ . $\displaystyle Q_{1}$ $\displaystyle=2.5\,\text{kg}\times 390\,\text{J/kg-K}\times 50.0\,\text{K}$ $\displaystyle=48,750\,\text{J}$
Step 2: Melting the solid at its melting point The heat required to melt the solid is calculated using the heat of fusion formula: $Q_{2}=m\cdot L_{f}$ Where: $\bullet$ $m=2.5\,\text{kg}$ $\bullet$ $L_{f}=4000\,\text{J/kg}$ $\displaystyle Q_{2}$ $\displaystyle=2.5\,\text{kg}\cdot 4000\,\text{J/kg}$ $\displaystyle=10000\,\text{J}$	Step 2: Heat Required to Melt the Solid The formula for the heat required to melt a substance is: $Q_{2}=m\cdot L_{f}$ Where: $\bullet$ $m=2.5\,\text{kg}$ (mass of the solid), $\bullet$ $L_{f}=4000\,\text{J/kg}$ (heat of fusion). $\displaystyle Q_{2}$ $\displaystyle=2.5\,\text{kg}\times 4000\,\text{J/kg}$ $\displaystyle=10,000\,\text{J}$
Step 3: Total Heat Required $\displaystyle Q_{\text{total}}$ $\displaystyle=Q_{1}+Q_{2}$ $\displaystyle=48750\,\text{J}+10000\,\text{J}$ $\displaystyle=58750\,\text{J}$	Step 3: Total Heat Required $\displaystyle Q_{\text{total}}$ $\displaystyle=Q_{1}+Q_{2}$ $\displaystyle=48,750\,\text{J}+10,000\,\text{J}$ $\displaystyle=\boxed{58750}\,\text{J}$
Final Answer $\boxed{58750}$	Final Answer $\boxed{58750}\,\text{J}$

Appendix

Table 7: Complete saved final responses (after thinking) and training signals for a selected heat-calculation case from an auxiliary fixed-teacher run. Both answers are correct; judge confidence increases and student uncertainty decreases between steps 30 and 105. Appendix D gives the calculations and case-selection details.

	Judge score 5		Judge score 4
Step	High conf.	Low conf.	High conf.	Low conf.
0	135/314	103/334	116/137	215/351
50	156/319	165/332	109/196	204/359
100	231/358	158/322	140/259	146/310
150	335/463	107/311	151/249	135/314
200	361/482	115/290	235/373	95/218

Appendix

Table 8: Response counts underlying Figures 2 (b), 3 , and 6 . Each entry is

k/n

n

is the group size and

k

counts responses with student entropy below the pooled mean of completed score-4/5 responses in the same reported summary. High judge confidence means

c^{\mathrm{J}}\geq 0.7

; low confidence means

c^{\mathrm{J}}<0.7

. The figures show reported steps 50–300; step 0 is included here for completeness.

Figure 6: Judge-group composition among low-entropy responses during Qwen3-4B RUSD training with

\alpha=0.5

. Only completed score-4/5 responses are included; low entropy means student entropy below their pooled mean at that reporting step. Each curve shows a group’s share of this subset; the four shares sum to 100%. High judge confidence means

c_{i}^{\mathrm{J}}\geq 0.7

. Shading shows 95% Wilson confidence intervals.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 514 papers

May 20, 2026

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Can post-trained large language models (LLMs) further improve themselves using only unlabeled prompts, without external teachers or feedback from tools? We study this setting starting only from unlabeled seed questions with no ground-truth solutions, across three reasoning domains: math, science, and coding. We propose Self-Verified Distillation, a simple post-training refinement algorithm in which the model generates candidate solutions to these seed questions, filters them using prompt-based self-verification, and trains on the resulting self-curated dataset. Inspired by the UQ benchmark's use of multiple validators to screen candidate answers to hard unsolved questions, we adapt this validation-based filtering idea to self-training: the model filters its own generated solutions through a three-stage cascade of cycle-consistency, factuality, and correctness checks, accepting a solution only if it passes all stages with unanimous judge votes. We find that sampling more candidate generations and using a larger verification budget during training data construction produces higher-quality self-curated data and, in turn, better reasoning models. We then train Qwen3 models at multiple scales with Self-Verified Distillation and obtain gains across all three domains. For Qwen3-4B, our method improves aggregate held-out pass@1 by +16.7 points in math (AIME26 and HMMT), +11.1 points in science (GPQA Diamond and HLE), and +8.3 points in coding (LCBv5 and LCBv6), with gains also extending to 0.6B and 8B models. Compared to our test-time-only baseline (UQ-TTC), which improves performance by spending extra compute at inference time, Self-Verified Distillation achieves better performance in most settings while requiring only a single inference call at test time.

Tony Lee, Percy Liang

Stanford University

Language Model Self-Verification Self-Training for Language Models Language Model Self-Assessment Language Model Distillation Synthetic Data Generation for Language Models Language Model Self-Improvement+1

Ask PDF

Oct 13, 2026

Does self-distillation improve reasoning?

Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning

Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5

Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Self-Distillation On-Policy Self-Distillation Knowledge Distillation RL for Language Model Reasoning

Ask PDF

May 13, 2026

Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning

On-policy self-distillation trains a reasoning model on its own rollouts while a teacher, often the same model conditioned on privileged context, provides dense token-level supervision. Existing objectives typically weight the teacher's token-level signal uniformly across a chain-of-thought sequence, despite substantial variation in the entropy of the teacher's predictive distribution. We propose EGRSD (Entropy-Guided Reinforced Self-Distillation), which unifies token-level updates through three signals: a reward-grounded direction, a teacher-student likelihood-ratio magnitude, and the proposed teacher-entropy confidence gate that down-weights high-entropy token positions while maintaining a nonzero lower bound on every token weight. We further introduce CL-EGRSD, a causal-lookahead variant that distinguishes sustained high-entropy spans from transient high-entropy positions whose following context rapidly becomes low entropy. Experiments with Qwen3-4B and Qwen3-8B in thinking mode show that EGRSD and CL-EGRSD advance the accuracy-length frontier among the compared trainable methods.

Junlong Ke, Zichen Wen, Weijia Li +2

2Tsinghua University · 1Shanghai Jiao Tong University · 3Shanghai AI Laboratory

On-Policy Self-Distillation RL for Language Model Reasoning Uncertainty-Aware Knowledge Distillation Knowledge Distillation LLM Reasoning

Ask PDF

Abstract

Language Model Distillation

Explore similar work

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Does self-distillation improve reasoning?

Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning