cs.CLOct 8, 2026

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards

Authors: Xiaobing Chen, Zhiqi Pang

Organizations: Harbin Engineering University · Tencent · Harbin Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have become two main paradigms for post-training reasoning models. RLVR gives each response a single outcome label, leaving the steps inside it without separate credit. OPD provides token-level guidance at student-visited prefixes, but its pointwise signal does not directly reflect the pattern of teacher--student disagreement across the vocabulary. Dense, unbounded log-ratio supervision can amplify the teacher's influence, yet a strong solver is not necessarily a suitable guide when the student's solution paths depart from the teacher's. We propose Residual Advantage (\RA{}), which treats the teacher--student probability residual as a bounded one-step reward, subtracts the corresponding state value under the student policy to form a standard advantage, and centers the result within each response before adding it to the verifier advantage. The guidance term has zero mean within each response, so the verifier advantage remains the response's mean label and the teacher only redistributes credit among the steps within it. \CoRA{} further updates a teacher LoRA with verifier advantages on the same scored student batch and uses the updated teacher in the next iteration's residual, adapting guidance to the student's attempts. With Qwen3-1.7B-Base and Qwen3-4B-Base students and a Qwen3-8B teacher, \RA{} combined with GRPO or REINFORCE++ improves the underlying sequence-advantage algorithm in all 24 comparisons on three mathematical benchmarks, raising macro Avg@8 by 1.7--3.6 points and Pass@8 by 3.9--6.3 points. Both combinations surpass teacher-only OPD, and \CoRA{} adds a further 1.0--1.5 Avg@8 points.

Figures & tables

Figure 1: Local teacher guidance redistributes the verifier’s response-level credit. (a) At one prefix, the reward Qd(s,v)=qs(v)−ps(v)Q_{d}(s,v)=q_{s}(v)-p_{s}(v) over the student’s likeliest tokens, its value Vdp(s)V_{d}^{p}(s) under the student policy, and the advantage Asd(a)A_{s}^{d}(a) of the sampled token, here the student’s third choice and the teacher’s first. (b) Labels along one response, with blue and coral stems for positive and negative guidance terms δt\delta_{t} ; each label is AVA^{V} plus a centered guidance term, so the labels’ weighted mean is AVA^{V} .
Figure 2: One iteration of RA and Co- RA . Three schematic responses, only the middle one correct, receive verifier advantages AiVA_{i}^{V} . RA adds a centered guidance term to every token label. Token strips show one cell per position, blue for At>0A_{t}>0 and rose for At<0A_{t}<0 , darker for larger ∣At∣|A_{t}| ; the last cell is the response mean, which centering pins to AiVA_{i}^{V} . Insets show (a) the student value baseline at one prefix and (b) response centering. In Co- RA , a LoRA adapter on the frozen teacher is trained with the verifier advantages of the same scored batch and supplies the next iteration’s residual.
MethodMATH-500AIME24AIME25Macro
Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
Qwen3-1.7B-Base36.8376.672.0810.001.538.8913.4831.85
OPD (teacher only)67.7885.008.6123.335.9715.5627.4541.30
REINFORCE++65.7683.877.7820.003.4714.4425.6739.44
 
+ RA69.6985.6013.0630.005.0020.0029.2545.20
Table 1: Residual guidance improves both RLVR estimators on every benchmark and metric, and the adapted teacher raises accuracy further. Three-seed means in %. Student rows are untrained, rows marked + add RA or Co- RA to the estimator above, bold marks the best per column and student, and the last row is the Qwen3-8B teacher; per-seed scores and SD are in Appendix D .
Variant (weight)Macro Avg@8 at weightMacro Pass@8 at weight
5102051020
Full RA ( λ\lambda )28.8429.2529.2944.9445.2045.29
w/o student value baseline ( γx\gamma_{x} )26.9627.1626.5341.9341.6240.76
w/o response centering ( λ\lambda )28.3728.7128.6644.4644.7444.67
Raw residual (w/o both, γraw\gamma_{\rm raw} )27.4827.5226.9642.6342.4340.96
Direct L2L_{2} loss ( α\alpha )27.8328.0227.7143.3943.4442.96
Table 2: Every residual variant improves on verifier-only training at every tested weight, and full RA stays above every control at every matched weight. Qwen3-1.7B-Base with REINFORCE++; three-seed macro means in %. The weight is λ\lambda , γx\gamma_{x} , γraw\gamma_{\rm raw} , or α\alpha as marked, with 1010 the reference weight; per-seed values and sample SD are in Table 23 .
VariantRewardBaselinesWeightAvg@8Pass@8
Full RAq−pq{-}pboth1029.25 ±\pm 0.2545.20 ±\pm 1.81
Clipped centered log ratio ( c=5c{=}5 )log⁡(q/p)\log(q/p)both0.6†0.6^{\dagger}28.04 ±\pm 0.2843.50 ±\pm 0.59
Centered log ratiolog⁡(q/p)\log(q/p)both0.6†0.6^{\dagger}27.85 ±\pm 0.3743.07 ±\pm 0.21
Raw residual (w/o both)q−pq{-}pneither1027.52 ±\pm 0.4342.43 ±\pm 0.79
Raw log ratiolog⁡(q/p)\log(q/p)neither0.2†0.2^{\dagger}27.28 ±\pm 0.3742.50 ±\pm 0.68
Verifier only (REINFORCE++)–––25.67 ±\pm 0.6639.44 ±\pm 1.65
Table 3: The probability residual with both baselines is the best construction, and no log-ratio variant reaches it at any swept weight. Three-seed macro means in % with SD; weights marked † were chosen on the validation split (Appendix B.5 ), and Tables 23 and 24 give all swept weights.
 Effect of the teacher-peak branch
 Sampled OPDREINFORCE++RA +REINFORCE++
 no weightλ=5\lambda{=}5λ=10\lambda{=}10λ=20\lambda{=}20λ=5\lambda{=}5λ=10\lambda{=}10λ=20\lambda{=}20
Δ\Delta macro Avg@8+0.30+0.30−0.38-0.38−0.88-0.88−1.76-1.76−2.05-2.05−2.07-2.07−2.16-2.16
Δ\Delta macro Pass@8+0.64+0.64−0.46-0.46−0.86-0.86−2.11-2.11−3.91-3.91−3.41-3.41−3.89-3.89
Table 4: The same teacher-peak branch slightly helps sampled OPD but lowers both students trained with the verifier. Change in three-seed macro score when the branch is added, Qwen3-1.7B-Base, in points. The weight λ\lambda multiplies the branch’s teacher part and, with RA , also the sampled guidance. Absolute, benchmark, and per-seed scores are in Appendix D.3 .
Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

RoleQuantityValue
InputStudent pp(0.1,0.3,0.6)(0.1,0.3,0.6)
InputTeacher qq(0.2,0.1,0.7)(0.2,0.1,0.7)
InputSampled action aa11
OutputLocal advantage Ad(a)A^{d}(a)0.090.09
OutputExpected update JpdJ_{p}d(0.009,−0.063,0.054)(0.009,-0.063,0.054)
OutputSampled update gag_{a}(0.081,−0.027,−0.054)(0.081,-0.027,-0.054)
Appendix
Table 5: A sampled update that opposes the expected update. The probability vectors and sampled action are inputs; the remaining entries follow by substitution.
Figure 3: The same sampled reward can yield local advantages of opposite signs. Both teachers assign the sampled token probability 0.280.28 against the student’s 0.300.30 , giving the same log ratio ( −0.069-0.069 ) and residual ( −0.02-0.02 ). Their residual vectors have the same squared norm, 0.03980.0398 , but the student values differ. The resulting Ad(2)A^{d}(2) is +0.021+0.021 for teacher I and −0.063-0.063 for teacher II.
QuantityTeacher ITeacher II
Student pp(0.20,0.30,0.50)(0.20,0.30,0.50)(0.20,0.30,0.50)(0.20,0.30,0.50)
Teacher qq(0.35,0.28,0.37)(0.35,0.28,0.37)(0.07,0.28,0.65)(0.07,0.28,0.65)
Residual d=q−pd=q-p(+0.150,−0.020,−0.130)(+0.150,-0.020,-0.130)(−0.130,−0.020,+0.150)(-0.130,-0.020,+0.150)
Sampled action aa22
Sampled residual d(a)d(a)-0.020-0.020
Squared residual norm ∥d∥22\|d\|_{2}^{2}0.03980.0398
Appendix
Table 6: The two teachers of Figure 3 . The student distribution and sampled action are shared. The last row pairs the two teachers as positions of one response with equal position weights and unit guidance weight.
RoleQuantityValue
InputStudent pp(0.99,0.01)(0.99,0.01)
InputTeacher qq(0.01,0.99)(0.01,0.99)
OutputCommon-action update(−0.000196,0.000196)(-0.000196,0.000196)
OutputExpected update(−0.019404,0.019404)(-0.019404,0.019404)
Appendix
Table 7: A confident student. The sampled and expected student-centered logit updates are computed from the stipulated probability vectors.
TeacherExpected logit update JpdJ_{p}dFirst-position guidance term with an agreement position
I(0.0382,0.0063,−0.0445)(0.0382,0.0063,-0.0445)0.0105λ0.0105\lambda
II(−0.0346,−0.0189,0.0535)(-0.0346,-0.0189,0.0535)−0.0315λ-0.0315\lambda
Appendix
Table 8: Further quantities for the two teachers. The teachers are those of Figure 3 . The final column pairs each teacher’s position with an agreement position at equal weights c=(1/2,1/2)c=(1/2,1/2) , keeping the symbolic weight λ\lambda .
SettingConfiguration
StudentsQwen3-1.7B-Base and Qwen3-4B-Base
Teacher backbone / inference modeQwen3-8B (post-trained) / non-thinking
Training data / prompt countDAPO-Math-17K / 17,398
Validation prompts drawn from the training file / problems used256 / 64
Validation samples per problem / temperature8 / 1.0
Training epochs / prompts per rollout batch2 / 128
Appendix
Table 9: Training and evaluation configuration. Lengths count response tokens. The correct-only linear length penalty applies to verifier runs; all runs share the response cap. Evaluation uses unshaped answer correctness.
StudentComparison (macro, pp)Δ\Delta Avg@8Δ\Delta Pass@8
1.7BRA +REINFORCE++ −- REINFORCE+++3.58+5.76
1.7BCo- RA +REINFORCE++ −- RA +REINFORCE+++1.45+1.18
1.7BRA +REINFORCE++ −- OPD+1.80+3.90
1.7BCo- RA +REINFORCE++ −- OPD+3.25+5.08
1.7BRA +REINFORCE++ with branch −- RA +REINFORCE++-2.07-3.41
1.7BRA +GRPO −- GRPO+2.37+4.35
Appendix
Table 10: Macro differences among runs. Differences of three-seed macro means from Tables 1 , 19 , and 4 , in percentage points. Each mean and difference is computed from raw counts before rounding; subtracting the displayed macro columns can differ by 0.010.01 .
SettingConfiguration
Aligned model-output width / tokenizer nominal size151936151936 / 151669151669
Probability, baseline, and label accumulationAt least float32
Actor reduction / additional label whiteningToken mean / none
PPO clipping width / epochs per rollout batch0.20.2 / 11
REINFORCE++ optimizer steps per rollout batch11
Student minibatch size256256 responses
Appendix
Table 11: Probability, actor, and reporting settings. These settings supplement Table 9 .
SettingConfiguration
GRPO group size / standard deviation88 responses / sample standard deviation
GRPO denominator stabilizer10−610^{-6}
REINFORCE++ discount / variance floor11 / 10−810^{-8}
REINFORCE++ normalization measureResponse tokens
Appendix
Table 12: Verifier advantage normalization settings. Normalization follows reward shaping, with no reward-side KL.
Operation at iteration kkRACo- RA
Teacher distribution qtq_{t}Fixed q0,tq_{0,t}Current qϕk,tq_{\phi_{k},t}
Local reward fielddt=q0,t−pk,td_{t}=q_{0,t}-p_{k,t}dt=qϕk,t−pk,td_{t}=q_{\phi_{k},t}-p_{k,t}
Student value baseline⟨pk,t,dt⟩\langle p_{k,t},d_{t}\rangle⟨pk,t,dt⟩\langle p_{k,t},d_{t}\rangle
Sampled local advantageAtd=dt(at)−⟨pk,t,dt⟩A_{t}^{d}=d_{t}(a_{t})-\langle p_{k,t},d_{t}\rangle
Centered guidance termλ(Atd−A‾d)\lambda(A_{t}^{d}-\overline{A}^{d})λ(Atd−A‾d)\lambda(A_{t}^{d}-\overline{A}^{d})
Appendix
Table 13: Residual guidance with a fixed or adapted teacher. Each method uses one teacher distribution, the student value baseline, and response centering. The weight is λ=10\lambda=10 in both reference configurations.
SettingConfiguration
Adapter rank / scaling1616 / 3232
Dropout / initial effective update00 / 00
Target matricesAttention and MLP projections
Separate optimizer / learning rateAdamW / 10−510^{-5}
Optimizer betas(0.9,0.999)(0.9,0.999)
Weight decay / gradient norm limit00 / 11
Appendix
Table 14: Adapter configuration and update schedule.
Figure 4: Training accuracy of Qwen3-4B-Base on DAPO-Math-17K. (A) Full two-epoch response budgets: 276,480 generations for GRPO runs and 34,560 for the one-response runs. (B) The first 34,560 generations, the shaded region in (A), show the early trajectories at the same response budget. Curves show logged training accuracy; Co- RA is omitted.
Figure 5: Training diagnostics for the 4B student with RA +REINFORCE++. The guidance weight is fixed at λ=10\lambda=10 , and the horizontal axis counts rollout batches. (a) Recorded mean absolute guidance term ∣δt∣|\delta_{t}| , averaged over the tokens of each batch. (b) Mean response length separated by verifier correctness. The curves are from a single training seed.
Tokens withShare of tokensShare of RA guidanceShare of centered log-ratio guidance
p(at)≥0.9p(a_{t})\geq 0.960.8%0.4% ( 0.01×0.01\times )0.6% ( 0.01×0.01\times )
0.5≤p(at)<0.90.5\leq p(a_{t})<0.911.9%17.8% ( 1.5×1.5\times )7.1% ( 0.6×0.6\times )
0.1≤p(at)<0.50.1\leq p(a_{t})<0.511.8%58.9% ( 5.0×5.0\times )31.6% ( 2.7×2.7\times )
p(at)<0.1p(a_{t})<0.115.5%22.9% ( 1.5×1.5\times )60.7% ( 3.9×3.9\times )
q(at)<10−4q(a_{t})<10^{-4}3.8%4.1% ( 1.1×1.1\times )50.1% ( 13×13\times )
Appendix
Table 15: Where the guidance lands. Share of tokens and of the summed absolute guidance term per bucket of the student’s probability of the sampled token, for full RA and the centered log-ratio control, both computed offline on the same 256 responses of the untrained 1.7B student, with the ratio of the two shares in parentheses. The last row overlaps the others.
ResponsesnnλA‾d\lambda\overline{A}^{d}Raw residual
  MeanSDMeanSD
All256+0.03+0.030.300.30−0.32-0.320.730.73
M<1,000M<1{,}000216+0.04+0.040.320.32−0.31-0.310.780.78
1,000≤M≤4,0001{,}000\leq M\leq 4{,}00034+0.00+0.000.030.03−0.38-0.380.290.29
M>4,000M>4{,}0006+0.00+0.000.010.01−0.05-0.050.240.24
Appendix
Table 16: Response mean of the guidance without centering. Mean and SD across responses of λA‾d\lambda\overline{A}^{d} (local advantage, with the student value baseline) and of λ dt(at)‾\lambda\,\overline{d_{t}(a_{t})} (raw residual), on the 256 offline responses of Table 15 , with λ=10\lambda=10 ; MM is the number of tokens and nn the number of responses in each group.
StatisticFull RAUncenteredDirect L2L_{2}
Cosine of sampled and analytic gTg_{T}0.9940.9921.000
Conditional noise of gTg_{T}0.851.080
cos⁡(gT,gV)\cos(g_{T},g_{V}) , sampled tokens+0.18+0.18+0.13+0.130.000.00
cos⁡(gT,gV)\cos(g_{T},g_{V}) , resampled tokens+0.02+0.020.000.00–
Outcome-compatible teacher mass64%60%49%
Batches with cos⁡(gT,gV)<0\cos(g_{T},g_{V})<026%33%50%
Appendix
Table 17: The sampled label reproduces the direct loss’s expected teacher gradient but shares the update with the verifier. Qwen3-1.7B-Base with REINFORCE++ at weight 1010 ; aggregates over three seeds.
MethodAvg@8 Pass@8
 S1S2S3Mean ±\pm SD S1S2S3Mean ±\pm SD
Qwen3-1.7B-Base
OPD27.6127.1727.5827.45 ±\pm 0.25 41.6040.6941.6041.30 ±\pm 0.53
REINFORCE++26.1224.9125.9725.67 ±\pm 0.66 41.2037.9339.1839.44 ±\pm 1.65
RA +REINFORCE++29.3628.9629.4329.25 ±\pm 0.25 46.1143.1146.3845.20 ±\pm 1.81
Co- RA +REINFORCE++30.1330.6531.3330.70 ±\pm 0.61 44.2947.3647.4946.38 ±\pm 1.81
Appendix
Table 18: Macro performance across three training seeds. Scores in %; ±\pm denotes sample SD across the three run-level macro scores. The middle block lists the 1.7B REINFORCE++ controls of Section 4 , whose reference is the RA +REINFORCE++ run of the first block; log-ratio weights are in parentheses.
VariantMATH-500AIME24AIME25Macro
 Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
Verifier only (REINFORCE++)65.7683.877.7820.003.4714.4425.6739.44
Teacher only (OPD)67.7885.008.6123.335.9715.5627.4541.30
Full RA ( λ=10\lambda{=}10 )69.6985.6013.0630.005.0020.0029.2545.20
Probability residual, baselines removed
w/o student value baseline67.1784.879.7224.444.5815.5627.1641.62
Appendix
Table 19: Benchmark-level results for the control runs of Section 4 . Qwen3-1.7B-Base with REINFORCE++; three-seed means in %. The macro columns repeat Tables 2 and 3 ; sample SD are reported in Table 18 .
ControlSeedMATH-500AIME24AIME25Macro
 S166.65 / 84.8010.83 / 23.334.17 / 16.6727.22 / 41.60
 S267.73 / 86.0010.42 / 23.334.17 / 20.0027.44 / 43.11
w/o student value baselineS367.13 / 83.807.92 / 26.675.42 / 10.0026.82 / 40.16
 S168.48 / 85.0013.33 / 26.674.58 / 16.6728.80 / 42.78
 S269.50 / 85.6011.25 / 30.004.17 / 20.0028.31 / 45.20
w/o response centeringS369.13 / 85.4012.50 / 30.005.42 / 23.3329.01 / 46.24
Appendix
Table 20: Per-seed benchmark results for seven controls. Qwen3-1.7B-Base with REINFORCE++; entries are Avg@8 / Pass@8 in %. Weights: no student value baseline γx=10\gamma_{x}=10 , no response centering λ=10\lambda=10 , raw residual γraw=10\gamma_{\rm raw}=10 , clipped ( c=5c=5 ) and centered log ratio γlog⁡=0.6\gamma_{\log}=0.6 , raw log ratio β=0.2\beta=0.2 , and direct L2L_{2} α=10\alpha=10 .
Base method + branch MATH-500AIME24AIME25Macro
λ\lambdaAvg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
Sampled OPD–63.2383.277.5020.004.5814.4425.10 ±\pm 1.0039.24 ±\pm 2.53
REINFORCE++565.5883.606.9418.893.3314.4425.29 ±\pm 1.1038.98 ±\pm 2.22
REINFORCE++1065.0682.406.8121.112.5012.2224.79 ±\pm 1.1738.58 ±\pm 3.02
REINFORCE++2064.3682.004.8616.672.5013.3323.91 ±\pm 1.9137.33 ±\pm 3.63
RA +REINFORCE++566.3584.209.3124.444.7214.4426.79 ±\pm 1.2241.03 ±\pm 2.07
Appendix
Table 21: Benchmark-level results for the teacher-peak branch. Qwen3-1.7B-Base; three-seed means in %, with sample SD for the macro scores.
Base method + branchλ\lambdaS1S2S3
Sampled OPD–25.63 / 41.0723.94 / 36.3625.74 / 40.29
REINFORCE++525.93 / 40.1624.02 / 36.4225.91 / 40.36
REINFORCE++1024.78 / 39.3623.62 / 35.2425.96 / 41.13
REINFORCE++2024.39 / 37.4025.53 / 40.9321.80 / 33.67
RA +REINFORCE++527.43 / 42.7125.39 / 38.7127.56 / 41.67
RA +REINFORCE++1028.19 / 44.1626.20 / 38.8427.14 / 42.38
Appendix
Table 22: Per-seed macro scores for the teacher-peak branch. Entries are macro Avg@8 / Pass@8 in %.
VariantWeightS1S2S3Avg@8Pass@8
 528.62 / 45.2729.00 / 44.4228.89 / 45.1328.84 ±\pm 0.2044.94 ±\pm 0.45
 1029.36 / 46.1128.96 / 43.1129.43 / 46.3829.25 ±\pm 0.2545.20 ±\pm 1.81
Full RA2029.67 / 45.3330.75 / 48.0927.45 / 42.4429.29 ±\pm 1.6845.29 ±\pm 2.82
 527.70 / 42.8429.19 / 45.4028.22 / 45.1328.37 ±\pm 0.7644.46 ±\pm 1.40
 1028.80 / 42.7828.31 / 45.2029.01 / 46.2428.71 ±\pm 0.3644.74 ±\pm 1.78
w/o response centering2029.46 / 46.3830.33 / 46.5826.18 / 41.0728.66 ±\pm 2.1944.67 ±\pm 3.13
Appendix
Table 23: Weight sweeps with the probability residual. Qwen3-1.7B-Base with REINFORCE++; S1–S3 entries are macro Avg@8 / Pass@8 in %. The shaded row is the reference configuration of Table 1 , and † marks the direct- L2L_{2} weight selected on the validation split and reported in Table 3 . The weight is λ\lambda for full RA and the uncentered control, γx\gamma_{x} and γraw\gamma_{\rm raw} for the two controls without the student value baseline, and α\alpha for direct L2L_{2} .
VariantWeightS1S2S3Avg@8Pass@8
 0.0526.70 / 41.4026.36 / 41.7326.71 / 41.6726.59 ±\pm 0.2041.60 ±\pm 0.18
 0.2†0.2^{\dagger}27.51 / 42.7126.85 / 41.7327.47 / 43.0427.28 ±\pm 0.3742.50 ±\pm 0.68
 0.627.02 / 42.0426.53 / 41.5327.04 / 41.3326.86 ±\pm 0.2941.64 ±\pm 0.37
 1.225.27 / 38.7825.05 / 39.1825.25 / 38.9825.19 ±\pm 0.1238.98 ±\pm 0.20
Raw log ratio ( β\beta )224.79 / 37.4023.71 / 37.8024.55 / 37.6724.35 ±\pm 0.5737.62 ±\pm 0.20
 0.0526.61 / 41.4726.30 / 40.2926.59 / 40.9626.50 ±\pm 0.1740.90 ±\pm 0.59
Appendix
Table 24: Weight sweeps with the log ratio. Qwen3-1.7B-Base with REINFORCE++; S1–S3 entries are macro Avg@8 / Pass@8 in %. For each control, † marks the weight selected on the validation split (Appendix B.5 ) and reported in Table 3 .
VariantWeightSeedMATH-500AIME24AIME25Macro
  S165.45 / 83.608.75 / 20.004.17 / 20.0026.12 / 41.20
  S265.58 / 83.807.92 / 20.001.25 / 10.0024.91 / 37.93
Verifier only–S366.25 / 84.206.67 / 20.005.00 / 13.3325.97 / 39.18
  S168.35 / 85.8012.50 / 30.005.00 / 20.0028.62 / 45.27
  S269.50 / 86.6011.67 / 26.675.83 / 20.0029.00 / 44.42
Full RA5S368.35 / 85.4012.92 / 30.005.42 / 20.0028.89 / 45.13
Appendix
Table 25: Per-seed benchmark results for the residual weight sweeps. Qwen3-1.7B-Base with REINFORCE++; entries are Avg@8 / Pass@8 in %. The shaded rows are the reference runs of Table 1 . The weight is λ\lambda for full RA and the uncentered control, γx\gamma_{x} for the control without the student value baseline, and γraw\gamma_{\rm raw} for the raw residual. Macro values average the three benchmark percentages within each run.
RunMATH-500AIME24AIME25Macro
Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
OPD (two epochs)67.7885.008.6123.335.9715.5627.45 ±\pm 0.2541.30 ±\pm 0.53
REINFORCE++65.7683.877.7820.003.4714.4425.67 ±\pm 0.6639.44 ±\pm 1.65
OPD, then REINFORCE++68.6585.279.7226.676.8118.8928.39 ±\pm 0.5743.61 ±\pm 1.88
REINFORCE++ + reverse KL66.2184.278.3321.113.8914.4426.14 ±\pm 0.9739.94 ±\pm 1.97
RA +REINFORCE++69.6985.6013.0630.005.0020.0029.25 ±\pm 0.2545.20 ±\pm 1.81
Appendix
Table 26: Sequential and joint combinations of OPD and REINFORCE++ against RA +REINFORCE++ at a matched response budget. Qwen3-1.7B-Base; three-seed means in %, with sample SD for the macro scores and the per-seed scores of the two combinations below; the reverse-KL weight is β=0.2\beta=0.2 . The other rows repeat Table 1 ; bold marks the best mean per column.
RunMATH-500AIME24AIME25Macro
Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
OPD, full vocabulary67.7885.008.6123.335.9715.5627.45 ±\pm 0.2541.30 ±\pm 0.53
OPD, sampled token62.3182.477.6420.004.4413.3324.80 ±\pm 0.7938.60 ±\pm 2.42
REINFORCE++65.7683.877.7820.003.4714.4425.67 ±\pm 0.6639.44 ±\pm 1.65
RA +REINFORCE++69.6985.6013.0630.005.0020.0029.25 ±\pm 0.2545.20 ±\pm 1.81
OPD, sampled token, by seed
Appendix
Table 27: Full-vocabulary and sampled OPD. Qwen3-1.7B-Base; three-seed means in %, with sample SD for the macro scores and the per-seed scores of the sampled run below. The other rows repeat Table 1 .
MethodK=8K{=}8K=16K{=}16K=32K{=}32K=64K{=}64K=128K{=}128K=256K{=}256K=512K{=}512
Qwen3-1.7B-Base (untrained)31.8536.0739.7341.7946.1247.8748.95
increment per doubling–+4.22+3.66+2.06+4.33+1.75+1.08
REINFORCE++39.4443.1646.0349.5752.3054.2155.50
increment per doubling–+3.72+2.87+3.54+2.73+1.91+1.29
OPD (teacher only)41.3046.3550.3753.5055.1057.7959.12
increment per doubling–+5.05+4.02+3.13+1.60+2.69+1.33
Appendix
Table 28: Macro Pass@K up to K=512K=512 , Qwen3-1.7B-Base with REINFORCE++. Three-seed means in %, with the increment from the previous KK in gray.
Figure 6: The margin over the untrained student does not close up to Pass@512. Macro Pass@K of Table 28 and each trained run’s margin over the untrained student, in points.
ModelMATH-500AIME24AIME25Macro
Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
1.7B-Base36.83 ±\pm 0.1776.67 ±\pm 1.102.08 ±\pm 1.1010.00 ±\pm 3.331.53 ±\pm 0.648.89 ±\pm 1.9213.48 ±\pm 0.5631.85 ±\pm 1.00
4B-Base51.79 ±\pm 0.3089.60 ±\pm 0.407.22 ±\pm 0.6422.22 ±\pm 1.924.72 ±\pm 0.2420.00 ±\pm 3.3321.25 ±\pm 0.1343.94 ±\pm 1.17
8B teacher84.23 ±\pm 0.3395.53 ±\pm 0.7026.25 ±\pm 1.2543.33 ±\pm 0.0021.39 ±\pm 2.2938.89 ±\pm 6.9443.95 ±\pm 1.2359.25 ±\pm 2.47
Appendix
Table 29: Untrained students and the teacher under the evaluation protocol. Qwen3-1.7B-Base and Qwen3-4B-Base before any training, and the Qwen3-8B teacher in non-thinking mode. Scores in %; ±\pm is the sample SD across three evaluation runs.
TeacherMATH-500AIME24AIME25Macro
Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
Frozen q0q_{0}84.2395.5326.2543.3321.3938.8943.95 ±\pm 1.2359.25 ±\pm 2.47
Adapter, 1.7B + REINFORCE++82.7594.2724.0341.1119.3135.5642.03 ±\pm 0.7756.98 ±\pm 2.39
Adapter, 1.7B + GRPO82.5693.9323.6138.8918.7534.4441.64 ±\pm 0.6555.76 ±\pm 2.08
Adapter, 4B + REINFORCE++82.9794.5326.3944.4420.1436.6743.16 ±\pm 0.5358.55 ±\pm 2.46
Adapter, 4B + GRPO82.9094.4024.4440.0021.5340.0042.96 ±\pm 0.8058.13 ±\pm 2.32
Appendix
Table 30: The adapted teacher is a worse solver than the frozen teacher. Qwen3-8B in non-thinking mode with the Co- RA adapter of each setting, under the evaluation protocol; scores in %. The frozen row repeats Table 29 , where ±\pm is the SD across three evaluation runs; in the adapted rows ±\pm is the sample SD across the three seeds’ adapters.
Adapter trained withS1S2S3
Qwen3-1.7B + REINFORCE++42.29 / 57.0941.16 / 54.5342.63 / 59.31
Qwen3-1.7B + GRPO42.01 / 56.8940.89 / 53.3642.02 / 57.02
Qwen3-4B + REINFORCE++43.44 / 59.3842.55 / 55.7843.50 / 60.49
Qwen3-4B + GRPO43.08 / 58.2042.10 / 55.7843.69 / 60.42
Appendix
Table 31: Per-seed macro results of the adapted teachers. Macro Avg@8 / Pass@8 in % of the adapter from each Co- RA training seed.

Explore similar work

CardsList

  1. Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

    Sep 3, 2026Boyan Li, Bingsen Chen, Chenghao Yang +3RL for Language Model ReasoningReinforcement Learning with Verifiable Rewards

    University of Alberta · New York University · NYU Shanghai +3

  2. When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

    Sep 23, 2026Jie Zhang, Jingxiao Yang, Zhehao Huang +2RL for Language Model ReasoningGroup Relative Policy Optimization

    Shanghai Jiao Tong University · Zhejiang University

  3. On-policy Distillation with Verifiable Reward

    Aug 25, 2026Wenze Lin, Jiale Zhao, Xitai Jiang +5RL for Language Model ReasoningReinforcement Learning with Verifiable Rewards

    LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2