Learning from Unverified Peers via Shared Residual Distillation

Feiyang Chen; Runze Ma; Zichen Wang; Jiong Lou; Zihan Wang; Shuangjia Zheng; Jie Li

cs.AIOct 15, 2026

Learning from Unverified Peers via Shared Residual Distillation

Authors: Feiyang Chen, Runze Ma, Zichen Wang, Jiong Lou, Zihan Wang, Shuangjia Zheng, Jie Li

Organizations: Shanghai Jiao Tong University

Abstract

A language model can generate several solutions to the same problem, some leading to correct answers and others to incorrect ones. Many post-training methods turn these trajectories into learning signals using supervision from reference answers, verifiable feedback, or external teachers. Without such supervision, learning from the model's own generations becomes less straightforward, as their correctness cannot be reliably assessed. An incorrect solution may nevertheless contain useful intermediate reasoning, so its value as supervision need not be determined solely by its final answer. Can a model improve by learning from its unverified peer solutions without first determining which are correct? We propose Shared Residual Distillation (SRD), an on-policy self-distillation method that learns from the model's own unverified peer trajectories. SRD compares how peers change the prediction distribution relative to a control, retaining adjustments that agree in direction and are conservative in magnitude. This yields supervision without treating any complete solution as correct, and the trained model requires no peers at inference time. Starting from the corresponding base models, SRD raises mean AIME24--26 avg@12 from 11.02 to 16.20 on Qwen3-1.7B, from 19.07 to 45.46 on Qwen3-4B, and from 21.39 to 54.44 on Qwen3-8B. Controlled ablations support the value of the shared-residual construction, and we observe no degradation relative to the base models on two general-capability benchmarks.

Figures & tables

Figure 1: Intuition behind SRD. Sampled solutions may contain useful steps even when their answers are unverified. When peer effects disagree, SRD makes no shared adjustment; when they agree, it retains a conservative adjustment without deciding which peer is correct.

Method	Samples	Signal	Teacher	Labels
RLVR (GRPO)	✓ On-policy	✗ Sparse	–	Gold
External OPD	✓ On-policy	✓ Dense	✗ External	None
Reference OPSD	✓ On-policy	✓ Dense	✓ Self	Gold
TTRL	✓ On-policy	✗ Sparse	–	Pseudo
SRD (ours)	✓ On-policy	✓ Dense	✓ Self	None

Table 1: Post-training supervision. Signal is token-level (dense) or outcome-level (sparse); Labels denotes answer-level supervision. –: no distillation teacher.

Figure 2: Overview of SRD. Phase 1: Peer rollout. The student samples solutions for each problem, split into peer contexts and target trajectories. Phase 2: Shared residual distillation. A frozen teacher compares peer-conditioned predictions with a cross-problem control and retains conservative shared adjustments. These adjustments modify the frozen model’s question-only base distribution to form targets for student updates.

Qwen3-1.7B
	Mathematics (ID)						General (OOD)
Method	AIME24	AIME25	AIME26	HMMT25	HMMT26	MATH-500	IFEval	MMLU-Pro
Base	13.33	8.61	11.11	5.97	8.08	74.00	66.91	43.87
GRPO	15.28	10.83	8.89	5.69	10.86	75.80	67.84	44.68
TTRL †	14.44	11.67	10.28	3.89	8.59	73.80	68.58	44.51
OPSD ‡	14.17	11.39	10.56	5.97	8.33	72.60	67.10	44.17

Table 2: Main results across three Qwen3 model sizes (%; higher is better). AIME and HMMT report avg@12; MATH-500 reports pass@1. Bold marks the best result per model and benchmark.

\dagger

: TTRL trained on the common training split;

\ddagger

: OPSD with two target trajectories per problem.

Figure 3: AIME accuracy, response length, and training dynamics. (a) Mean AIME24–26 avg@12 (bars) and response length (line) on Qwen3-4B. (b) Mean AIME24–26 avg@12 throughout training for OPSD (dashed) and SRD (solid) at three model sizes.

Variant	AIME24	AIME25	AIME26	Average
Target construction
Mean residual	36.67	29.17	36.11	33.98
Random residual mask (matched sparsity)	41.67	32.50	41.67	38.61
Signed-mask mean	38.89	32.78	40.83	37.50
Shared residual ( $q_{0}$ reference)	40.28	40.56	42.22	41.02
Full SRD (ours)	47.78	41.11	47.50	45.46

Table 3: Qwen3-4B target-construction ablations with clipping at the main-table checkpoint (avg@12, %). Average is the mean over AIME24–26.

Qwen3-1.7B
Method	AIME24		AIME25		AIME26		Average
	avg@12	pass@12	avg@12	pass@12	avg@12	pass@12	avg@12	pass@12
OPSD ‡	11.39	30.00	11.11	23.33	9.17	23.33	10.56	25.55
SRD (ours)	12.78	43.33	11.67	30.00	11.67	20.00	12.04	31.11
Qwen3-4B
OPSD ‡	23.61	43.33	19.72	30.00	16.11	36.67	19.81	36.67

Table 4: OPSD and SRD without clipping (%). Average is the mean over AIME24–26; bold marks column-wise best results within each model.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	OPSD ‡	SRD
Problems per update	32	32
Rollouts per problem	2	4
Target trajectories per update	64	64
Maximum rollout length	1,024	1,024
Maximum scored target prefix	1,024	1,024
Maximum training context	20,000	20,000

Appendix

Table 5: Training configuration for Qwen3-4B distillation. OPSD ‡ matches SRD’s number of scored target trajectories without generating unused peer contexts. Both methods use a 100-step schedule.

Setting	GRPO	TTRL †
Supervision	Verified reward	Majority pseudo-reward
Sampling	On-policy	On-policy
Problems per update	32	32
Responses per problem	8	8
Trajectories per update	256	256
Maximum response length	8192	8192

Appendix

Table 6: Controlled configurations for the GRPO and TTRL baselines, which differ only in their reward labels.

Qwen3-1.7B
Method	AIME24	AIME25	AIME26	HMMT25	HMMT26
Base	33.33	23.33	26.67	16.67	27.27
GRPO	40.00	23.33	20.00	18.33	24.24
OPSD ‡	33.33	26.67	26.67	21.67	24.24
TTRL †	36.67	33.33	20.00	11.67	24.24
SRD (ours)	46.67	40.00	33.33	20.00	30.30

Appendix

Table 7: Pass@12 on AIME and HMMT (%; higher is better). The best score within each model–benchmark block is bold, including ties. Method annotations follow Table 2 .

Variant	AIME24	AIME25	AIME26	Average
Additional target controls
Prompt-only teacher	7.22	4.17	8.33	6.57
Two-peer probability mean	40.56	29.72	37.50	35.93

Appendix

Table 8: Additional Qwen3-4B target-construction ablations. Metrics follow Table 3 ; Average is the arithmetic mean over AIME24–26.

Figure 4: Sampling sensitivity and peer-count analysis on Qwen3-4B. Panels (a)–(b) report mean AIME24–26 avg@12 throughout training while varying rollout group size and rollout length. Panel (c) jointly reports performance at the main-table checkpoint and the training-average residual zeroing rate induced by $M\in\{2,4,6\}$ peers, with two scored targets in every setting.

Evaluation at the main-table checkpoint
Peer pair	AIME24	AIME25	AIME26	Average
Both correct	40.00	38.61	38.61	39.07
One correct, one incorrect	45.83	37.78	38.89	40.83
Both incorrect	49.44	37.22	43.61	43.43

Appendix

Table 9: Peer-correctness composition analysis on Qwen3-4B at the main-table checkpoint. Every condition uses 4,096-token rollouts,

M=2

peers, and 1,600 accepted training groups. Reference answers are used only to determine pair composition. All AIME scores use avg@12, and Average is their arithmetic mean.

Model	$\bm{K}$	Targets/update	AIME24	AIME25	AIME26
Qwen3-1.7B	1	32	13.33	8.33	12.50
	2	64	14.17	11.39	10.56
Qwen3-4B	1	32	23.61	24.44	24.44
	2	64	22.50	26.39	25.00
Qwen3-8B	1	32	42.22	32.22	33.89
	2	64	40.56	31.94	27.50

Appendix

Table 10: OPSD target-batch scaling across model sizes (avg@12, %; higher is better).

K=1

is the original configuration and

K=2

is the target-count-matched variant used in the main comparison.

Self-Training for Language Models

Self-training for language models uses their own generations, feedback, and interaction traces as supervision to improve capabilities with less labeled data.

See all 84 papers

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
May 20, 2026Tony Lee, Percy LiangLanguage Model Self-Verification Self-Training for Language Models
Stanford University
On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3On-Policy Self-Distillation Language Model Self-Improvement
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning
Oct 13, 2026Jinhao Pan, Bowen Wei, Yuqing Zhou +1Language Model Distillation LLM Reasoning
George Mason University

May 20, 2026J/K move · Enter open · S save

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Tony Lee, Percy Liang

Stanford University

Language Model Self-Verification Self-Training for Language Models Language Model Self-Assessment Language Model Distillation Synthetic Data Generation for Language Models Language Model Self-Improvement+1

Open paper AskPDF arXiv

Can post-trained large language models (LLMs) further improve themselves using only unlabeled prompts, without external teachers or feedback from tools? We study this setting starting only from unlabeled seed questions with no ground-truth solutions, across three reasoning domains: math, science, and coding. We propose Self-Verified Distillation, a simple post-training refinement algorithm in which the model generates candidate solutions to these seed questions, filters them using prompt-based self-verification, and trains on the resulting self-curated dataset. Inspired by the UQ benchmark's use of multiple validators to screen candidate answers to hard unsolved questions, we adapt this validation-based filtering idea to self-training: the model filters its own generated solutions through a three-stage cascade of cycle-consistency, factuality, and correctness checks, accepting a solution only if it passes all stages with unanimous judge votes. We find that sampling more candidate generations and using a larger verification budget during training data construction produces higher-quality self-curated data and, in turn, better reasoning models. We then train Qwen3 models at multiple scales with Self-Verified Distillation and obtain gains across all three domains. For Qwen3-4B, our method improves aggregate held-out pass@1 by +16.7 points in math (AIME26 and HMMT), +11.1 points in science (GPQA Diamond and HLE), and +8.3 points in coding (LCBv5 and LCBv6), with gains also extending to 0.6B and 8B models. Compared to our test-time-only baseline (UQ-TTC), which improves performance by spending extra compute at inference time, Self-Verified Distillation achieves better performance in most settings while requiring only a single inference call at test time.

Abstract

Self-Training for Language Models

Explore similar work

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

On-Policy Self-Distillation without Any Supervision

Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning