Improving the reasoning capabilities of vision-language models (VLMs) often relies on post-training with supervision from stronger models. Two common strategies are supervised fine-tuning (SFT), which trains on teacher-generated demonstrations, and on-policy distillation (OPD), which provides teacher feedback on trajectories generated by the student itself. While OPD can be more data-efficient by supervising student-generated trajectories, it is unclear how the teacher's supervision at each state affects the student's learned behavior. To study this question, we conduct a controlled comparison of OPD and filtered SFT across data budgets and tasks where the teacher exhibits different levels of reliability, and analyze how these differences shape student accuracy and completion behavior. Across the two tasks, we find that OPD is effective only when the teacher remains reliable on student-generated trajectories. On ChartQA, where the teacher completes 99% of its solutions, OPD trained on only 10% of the data comes within 1.4 accuracy points of full-data SFT. In contrast, on Geometry3K, where the teacher fails to complete 22% of its solutions, OPD underperforms SFT by 7.9 points, with most of the gap explained by incomplete responses. More strikingly, starting OPD from an SFT student reduces its completion rate from 76% to 62%, effectively erasing the completion behavior learned through SFT. Filtering teacher demonstrations for completeness alone recovers much of SFT's advantage. These results show that on-policy supervision can transfer not only teacher knowledge but also persistent failure modes on student-visited states. Code is available at https://github.com/ffy208/opd-inherits-teacher-vlm.
Figures & tables
Figure 1: Accuracy against training questions at a fixed number of optimizer updates per method (mean over four seeds) on ChartQA (a), where the teacher concludes 99% of its solutions, and Geometry3K (b), where it concludes 78%; (c) share of Geometry3K test responses with an answer line. Shaded bands: 95% question-clustered bootstrap intervals; grey dashed: zero-shot teacher; grey dotted: zero-shot student. (d) Rollout termination rate during OPD on Geometry3K at 900 problems (five-step windows, mean and ±1 s.d. over four seeds) from the base and from the filtered-SFT student, which starts at 0.79 and is in plain OPD’s band within 40 steps (shaded).
Accuracy
OPD − SFT
Questions
SFT
OPD
paired, clustered 95%
t over seeds
100 (3%)
0.793 (.022)
0.811 (.009)
+0.018 [ −0.001 , +0.038 ]
[ −0.006 , +0.043 ]
300 (10%)
0.799 (.015)
0.826 (.007)
+0.027 [ +0.008 , +0.047 ]
[ −0.008 , +0.062 ]
900 (30%)
0.808 (.011)
0.823 (.004)
+0.015 [ −0.004 , +0.035 ]
[ +0.003 , +0.027 ]
3000 (100%)
0.840 (.007)
0.825 (.008)
−0.015 [ −0.032 , +0.002 ]
[ −0.033 , +0.003 ]
100, SFT 38 steps
0.797 (.012)
0.811 (.009)
+0.015 [ −0.005 , +0.035 ]
[ +0.001 , +0.028 ]
Table 1: ChartQA: accuracy on 500 held-out questions, mean over four seeds (across-seed sd in brackets; every mean has a question-clustered 95% interval of ±0.03 ), with the paired OPD minus SFT difference, its clustered interval, and the t interval over the four per-seed differences. The last two rows use the SFT length that scored best in Appendix K . Bold: the higher accuracy where the clustered interval excludes zero, and intervals that exclude zero. Zero-shot student 0.668 , teacher 0.844 .
Figure 2: Where the teacher’s per-token feedback lands on ChartQA. KL concentration is a token role’s share of the batch’s reverse KL divided by its share of tokens (dotted line: 1 ); it measures where the divergence concentrates, not the size of the gradient. (a) Over training, nine OPD runs with a complete per-role log (100 questions: four seeds; 300: seeds 1–3; 900: seeds 1–2), 10-step windows, mean and ±1 s.d. over runs. (b) One model per condition (zero-shot student; 300-question seed-1 OPD student) on 100 test questions, 11,625 and 12,842 tokens (Appendix C ).
Accuracy
OPD − SFT
Completion
Acc. ∣ compl.
Problems
SFT
OPD
paired, clustered 95%
t over seeds
SFT
OPD
SFT
OPD
100
0.423
0.404
−0.019 [ −0.045 , +0.007 ]
[ −0.059 , +0.021 ]
0.78
0.64
0.52 – 0.57
0.61 – 0.69
300
0.420
0.419
−0.001 [ −0.024 , +0.022 ]
[ −0.020 , +0.019 ]
0.73
0.62
0.55 – 0.59
0.65 – 0.70
900
0.481
0.402
−0.079 [ −0.105 , −0.053 ]
[ −0.110 , −0.048 ]
0.76
0.63
0.62 – 0.67
0.63 – 0.65
2101
0.478
0.411
−0.067 [ −0.092 , −0.043 ]
[ −0.094 , −0.040 ]
0.76
0.64
0.62 – 0.64
0.61 – 0.66
Table 2: Geometry3K: accuracy on 500 held-out problems, four seeds at every budget (means; across-seed sd between 0.006 and 0.025 , per seed in Appendix M ; every mean has a question-clustered 95% interval of ±0.035 to ±0.038 ), the paired OPD minus SFT difference with its clustered and seed-level t intervals, “Completion”, the fraction of responses with an answer line (every response without one ran to the 1,536-token budget; Appendix D ), and “Acc. ∣ compl.”, accuracy among those (range over seeds). Bold as in Table 1 . Zero-shot student 0.366 (completion 0.65 ), teacher 0.570 (completion 0.78 ).
Figure 3: Geometry3K, 900 problems, four seeds per row. (a) Paired accuracy difference of the three SFT data sets (filtered, the reference; unfiltered; complete-only, defined in the text) and of plain OPD and its five variants against the filtered SFT; bars: 95% question-clustered bootstrap intervals with the four models per method as fixed (seed-level t intervals in Table 12 , Appendix F ). (b) Share of test responses with an answer line; dashed: zero-shot teacher; dotted: zero-shot student. (c) OPD minus SFT accuracy at each budget, in points of the 500 test problems, split by which method concluded the problem (a problem only one method concludes counts against the other); black tick: total. From the per-problem records; printed totals rounded to one decimal (Table 10 ).
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Student / teacher
Qwen3-VL-2B-Instruct / Qwen3-VL-8B-Instruct (bf16); Qwen3-VL-32B-Instruct in Section 4.4
LoRA
rank 64, α=128 , language-model q/k/v/o and MLP projections, vision encoder and projector frozen
Data
500 test questions sampled with seed 42 from the official test split (human-written ChartQA questions; the 601 Geometry3K test problems); training budgets are nested prefixes of a seeded shuffle of the training split; images resized to a longest side of 768 px (300 to 600 visual tokens)
SFT
302 steps, batch 4 × accumulation 4, lr 10−4 , cosine, max sequence length 2,048; loss on the response tokens only; an end-of-turn token is appended to every demonstration before the sequence is cut to 2,048 tokens (the prompt is never cut)
SFT data
one teacher sample per question at T=0.7 , kept if its answer is correct (2,402 of 3,000 on ChartQA; 1,163 of 2,101 on Geometry3K, of whose samples 78% contain an answer line); the unfiltered control keeps every first sample; the complete-only control keeps the first sample when it contains an answer line, correct or not (1,645 of 2,101)
OPD
150 steps, 16 rollouts per step, micro-batch 4, lr 5⋅10−5 , 20 warmup steps, cosine, AdamW ( Loshchilov and Hutter, 2019 ) without weight decay, grad clip 1.0
Rollouts
<|im_end|> is the end-of-turn token; a rollout ends at its first occurrence or at the budget of 512 new tokens (charts) / 1,536 (geometry); T=1.0 , top- k off
Appendix
Table 3: Hyperparameters, data construction, and evaluation settings.
Figure 4: Feedback dynamics on ChartQA over the nine OPD runs with a complete per-role training log (100 questions: four seeds; 300: seeds 1–3; 900: seeds 1–2; the 300-question seed-42 run predates per-role logging and the other logs stop at an earlier checkpoint on the Hub; windows of 10 steps, shaded: ±1 s.d. over runs). (a) KL concentration per token role (share of the batch’s KL divided by share of tokens; dotted line: 1 ); (b) mean per-token KL; (c) rollout termination rate.
Figure 5: As Figure 4 , for the eleven OPD runs on Geometry3K with a complete per-role log (300 and 900 problems: four seeds each; 100 problems: seeds 1 and 42; 2,101 problems: seed 42; the Hub copies of the other 100- and 2,101-problem logs stop at an earlier checkpoint). The answer-line concentration falls from 0.45 to 0.28 ; numeric values stay between 0.48 and 0.60 ; the rollout termination rate (c) stays between 0.47 and 0.61 over 150 steps.
Figure 6: Per-token reverse KL between teacher and student on one ChartQA test question (rollouts at temperature 1.0), for the zero-shot student (top) and the OPD student trained on 300 questions (bottom), same colour scale; the underline gives the token’s role. Before training the opening of the answer line carries a large divergence; after training it carries almost none and the residual disagreement is diffuse (here largest on a wording choice). Numbers are tokenised digit by digit and the disagreement on a number sits on its first digit.
Figure 7: Share of tokens against share of the teacher’s KL mass per role over the rollouts on 100 ChartQA test questions (11,625 and 12,842 tokens), for the zero-shot student (a) and the OPD-300 student (b, seed 1); the number above each bar is the concentration (KL share divided by token share). Mean per-token KL falls from 0.47 to 0.23 .
Truncated outputs, prefix cut at
Concluded outputs
Budget
Student ( n )
256 tokens
512 tokens
1,024 tokens
truncation point
256 tokens
Rest of budget
Zero-shot (176)
0.35 [ 0.28 , 0.42 ]
0.13 [ 0.09 , 0.18 ]
0.01 [ 0.00 , 0.02 ]
0.01 [ 0.00 , 0.02 ]
0.91
SFT (241)
0.32 [ 0.25 , 0.38 ]
0.24 [ 0.19 , 0.30 ]
0.07 [ 0.04 , 0.11 ]
0.11 [ 0.07 , 0.15 ]
0.87
OPD (369)
0.34 [ 0.29 , 0.39 ]
0.17 [ 0.13 , 0.21 ]
0.06 [ 0.04 , 0.09 ]
0.03 [ 0.01 , 0.05 ]
0.90
Fixed 512 tokens
Zero-shot (176)
0.21 [ 0.15 , 0.27 ]
0.10 [ 0.06 , 0.14 ]
0.01 [ 0.00 , 0.02 ]
0.01 [ 0.00 , 0.02 ]
0.78
SFT (241)
0.14 [ 0.10 , 0.19 ]
0.12 [ 0.08 , 0.16 ]
0.08 [ 0.04 , 0.12 ]
0.13 [ 0.09 , 0.18 ]
0.77
Appendix
Table 4: Continuation probe on Geometry3K: share of the teacher’s greedy continuations that reach an answer line, from prefixes of a student’s truncated outputs (cut at 256, 512, and 1,024 tokens and at the truncation point) and, for comparison, from 256-token prefixes of its concluded outputs; the rest of the 1,536-token budget is 1,280, 1,024, 512, and 512 tokens. Seeds 1 and 42 of the 900-problem students pooled; n : truncated outputs in parentheses, and 203, 435, and 320 concluded outputs of the zero-shot, SFT, and OPD students; brackets: 95% question-clustered bootstrap intervals.
Teacher prompt
Truncated, 256
Truncated, 512
Concluded, 256
Original
0.18 [ 0.10 , 0.25 ] / 0.08
0.03 / 0.00
0.78 / 0.49 (138)
With completion guidance
0.19 [ 0.11 , 0.26 ] / 0.07
0.05 / 0.01
0.80 / 0.49 (138)
Length-matched paraphrase
0.18 [ 0.10 , 0.25 ] / 0.05
0.03 / 0.00
0.81 / 0.50 (138)
With the gold answer
0.07 [ 0.02 , 0.12 ] / 0.07
0.01 / 0.01
0.54 / 0.54 (138)
Appendix
Table 5: Fixed-budget continuation (512 tokens) by the teacher under four prompts, from prefixes of the zero-shot student’s outputs on the first 300 training problems of Geometry3K: completion / accuracy on the 91 truncated outputs (brackets: 95% bootstrap interval of the completion rate over outputs, one per problem) and on the outputs that concluded ( n in parentheses; concluded outputs shorter than a cut are not continued). Paired differences between prompts are in the text.
Concluded outputs
Truncated outputs
Student
n
EOS at stop (share >0.5 )
answer at onset
n
max EOS
max answer
Zero-shot
324
0.98 ( 0.99 )
0.63
176
<0.001
0.12
SFT
380
0.98 ( 0.99 )
0.77
120
<0.001
0.13
OPD
314
0.98 ( 0.98 )
0.76
186
<0.001
0.13
OPD-sftinit
311
0.98 ( 0.98 )
0.82
189
<0.001
0.12
Appendix
Table 6: Teacher-forced probe on Geometry3K test outputs (mean over seeds; n per 500). For outputs that conclude: the teacher’s end-of-turn probability at the position where the student stopped (“EOS at stop”), the share of outputs on which it exceeds 0.5 , and its probability mass on answer-opening tokens (a proxy, see text) where the student’s answer line begins (“answer at onset”). For outputs that ran to the budget: the maximum end-of-turn probability over the positions of an output, averaged over outputs (no single position exceeds 10−3 ), and the mean over outputs of the maximum answer-opening probability (above 0.5 on 10 to 12% of outputs).
Student
Its output ( n per cut)
256 tokens
512 tokens
1,024 tokens
Truncation point
Zero-shot
truncated (176)
0.35 / 0.19
0.13 / 0.09
0.01 / 0.01
0.01 / 0.01
concluded (203 / 45 / 4)
0.91 / 0.53
0.80 / 0.27
0.50 / 0.25
SFT
truncated (241)
0.32 / 0.18
0.24 / 0.15
0.07 / 0.05
0.11 / 0.08
concluded (435 / 163 / 63)
0.87 / 0.57
0.70 / 0.44
0.62 / 0.41
OPD
truncated (369)
0.34 / 0.21
0.17 / 0.10
0.06 / 0.04
0.03 / 0.02
concluded (320 / 108 / 23)
0.90 / 0.58
0.79 / 0.44
0.57 / 0.39
Appendix
Table 7: Continuation probe on Geometry3K (900-problem SFT and OPD students, seeds 1 and 42 pooled; the zero-shot student): completion / accuracy of the teacher’s continuation from a prefix of the student’s output, split by whether the student’s own output concluded, with the number of outputs continued at each cut (concluded outputs shorter than a cut, and prefixes that already contain an answer line, are not continued). The truncated rows are those of the upper block of Table 4 .
Answer line
No answer line
Model
Budget
stops
at budget
at budget
stops
Student, zero-shot
1,536
324
0
175
1
Teacher, zero-shot
1,536
391
0
109
0
Student, zero-shot
4,096
325
0
174
1
Teacher, zero-shot
4,096
422
0
78
0
SFT, 100
1,536
387.5
1.2
109.2
2.0
Appendix
Table 8: Geometry3K, mean counts per 500 test responses over four seeds (single models for the zero-shot rows). “Stops” means not truncated by the re-tokenisation criterion; “at budget” means truncated.
Student
Overall
Teacher concludes
Teacher does not
Zero-shot
0.648
0.731
0.349
SFT
0.760
0.848
0.445
OPD
0.628
0.748
0.197
OPD-term
0.638
0.748
0.245
OPD-fwd
0.605
0.721
0.186
OPD-sftinit
0.623
0.749
0.167
Appendix
Table 9: Geometry3K, 900 problems: each student’s completion rate on the 391 test problems the greedy teacher concludes within 1,536 tokens and on the 109 it does not (mean over four seeds; the zero-shot student is one model). At 4,096 tokens the teacher concludes 422 problems.
Comparison
Both conclude ( n )
Only first ( n )
Only second ( n )
Total
OPD − SFT, 100
+0.6 (287)
+3.5 (31)
−6.0 (102)
−1.9
OPD − SFT, 300
+2.4 (278)
+3.4 (30)
−5.9 (86)
−0.1
OPD − SFT, 900
−1.4 (285)
+2.2 (29)
−8.8 (95)
−7.9
OPD − SFT, 2101
−2.3 (291)
+2.5 (30)
−6.8 (88)
−6.7
OPD − SFT, 900 (4,096 tok.)
−2.5 (318)
+1.1 (13)
−9.4 (132)
−10.8
OPD-sftinit − SFT, 900
−0.3 (287)
+2.2 (24)
−7.8 (93)
−5.9
Appendix
Table 10: Geometry3K: decomposition of the paired accuracy difference (percentage points of the 500 test problems, mean over four seeds) by which method concluded a problem. A problem only one method concludes counts against the other, which scores it wrong; n is the mean number of problems in each group. The three components and the total are rounded independently, so the sum of the components may differ from the total by 0.1 .
1,536 tokens
4,096 tokens
Model
Accuracy
Completion
Accuracy
Completion
Student, zero-shot
0.366
0.65
0.366
0.65
Teacher, zero-shot
0.570
0.78
0.590
0.84
SFT, four seeds
0.482 [0.446, 0.520]
0.76
0.522 [0.486, 0.559]
0.90
OPD, four seeds
0.406 [0.369, 0.445]
0.63
0.414 [0.378, 0.452]
0.66
OPD − SFT (paired)
−0.076 [ −0.102 , −0.051 ]
−0.108 [ −0.135 , −0.082 ]
Appendix
Table 11: Geometry3K, 900 problems, the same four models per method scored at two generation budgets (means with question-clustered intervals). The SFT minus OPD gap grows with the budget because OPD students do not finish when given more room.
vs. filtered SFT
Variant
Accuracy
Completion
paired, clustered 95%
t over seeds
SFT
filtered
0.481 (.022)
0.76
unfiltered
0.445 (.013)
0.69
−0.036 [ −0.062 , −0.012 ]
[ −0.091 , +0.018 ]
complete-only
0.473 (.012)
0.74
−0.008 [ −0.032 , +0.016 ]
[ −0.037 , +0.021 ]
OPD
plain
0.402 (.006)
0.63
−0.079 [ −0.105 , −0.053 ]
[ −0.110 , −0.048 ]
term
0.404 (.007)
0.64
−0.077 [ −0.101 , −0.052 ]
[ −0.112 , −0.042 ]
Appendix
Table 12: Geometry3K, 900 problems, four seeds each (300-problem rows in Appendix F ): the three SFT data sets (filtered: correct teacher samples, the reference; unfiltered: every teacher sample; complete-only: the samples that contain an answer line, correct or not) and plain OPD with its five variants (defined in Section 4.3 and Appendix A ). Paired differences against the filtered SFT, with clustered and seed-level t intervals; bold: intervals that exclude zero.
vs. filtered SFT
Variant
Accuracy
Completion
paired, clustered 95%
t over seeds
SFT
filtered
0.420 (.013)
0.73
unfiltered
0.402 (.016)
0.62
−0.018 [ −0.042 , +0.007 ]
[ −0.064 , +0.028 ]
complete-only
0.432 (.011)
0.72
+0.012 [ −0.010 , +0.036 ]
[ −0.021 , +0.046 ]
OPD
plain
0.419 (.019)
0.62
−0.001 [ −0.024 , +0.022 ]
[ −0.020 , +0.019 ]
term
0.401 (.009)
0.62
−0.019 [ −0.040 , +0.003 ]
[ −0.039 , +0.001 ]
Appendix
Table 13: Geometry3K, 300 problems, four seeds each: the three SFT data sets, plain OPD, and OPD-term, as in Table 12 . No interval excludes zero.
CharXiv
ChartQAPro
Questions
SFT / OPD
OPD − SFT
SFT / OPD
OPD − SFT
100
0.258 / 0.288
+0.030 [ +0.005 , +0.054 ]
0.301 / 0.306
+0.005 [ −0.016 , +0.025 ]
300
0.278 / 0.285
+0.007 [ −0.016 , +0.030 ]
0.324 / 0.317
−0.007 [ −0.027 , +0.015 ]
900
0.286 / 0.290
+0.003 [ −0.020 , +0.027 ]
0.317 / 0.322
+0.005 [ −0.017 , +0.026 ]
3000
0.289 / 0.287
−0.001 [ −0.024 , +0.022 ]
0.326 / 0.321
−0.005 [ −0.025 , +0.014 ]
Appendix
Table 14: Out-of-distribution accuracy of ChartQA-trained models (four seeds at every budget) on CharXiv (461 numeric questions) and ChartQAPro (500 factoid questions), with paired OPD minus SFT and its question-clustered interval. Zero-shot student 0.258 / 0.276 , teacher 0.412 / 0.406 .
Comparison
Per seed
Mean
Questions
Two-way
Seeds
ChartQA 100, OPD − SFT
+3.2/+1.2/+3.0/+0.0
+1.8
[ −0.1 , +3.8 ]
[ −0.8 , +4.5 ]
[ −0.6 , +4.3 ]
ChartQA 300, OPD − SFT
+0.6/+2.0/+2.4/+5.8
+2.7
[ +0.8 , +4.7 ]
[ −0.1 , +5.7 ]
[ −0.8 , +6.2 ]
ChartQA 900, OPD − SFT
+1.4/+0.8/+2.6/+1.2
+1.5
[ −0.4 , +3.5 ]
[ −0.7 , +3.9 ]
[ +0.3 , +2.7 ]
ChartQA 3000, OPD − SFT
−1.2/−3.2/−0.6/−1.0
−1.5
[ −3.2 , +0.2 ]
[ −3.8 , +0.7 ]
[ −3.3 , +0.3 ]
ChartQA 100, OPD − SFT 38 steps
+2.6/+1.6/+1.0/+0.6
+1.5
[ −0.5 , +3.5 ]
[ −0.9 , +3.8 ]
[ +0.1 , +2.8 ]
ChartQA 300, OPD − SFT 38 steps
+1.2/+0.6/+0.6/+1.8
+1.1
[ −0.6 , +2.8 ]
[ −0.9 , +3.0 ]
[ +0.1 , +2.0 ]
Appendix
Table 15: ChartQA and out-of-distribution comparisons: paired differences in percentage points (seeds 1/2/3/42) and their intervals under three conventions.
Comparison
Per seed
Mean
Questions
Two-way
Seeds
Geometry3K 100, OPD − SFT
+0.0/+0.2/−2.6/−5.2
−1.9
[ −4.5 , +0.7 ]
[ −5.6 , +1.8 ]
[ −5.9 , +2.1 ]
Geometry3K 300, OPD − SFT
+0.8/−1.8/+0.8/+0.0
−0.1
[ −2.4 , +2.2 ]
[ −3.0 , +2.9 ]
[ −2.0 , +1.9 ]
Geometry3K 900, OPD − SFT
−6.6/−6.4/−10.6/−8.0
−7.9
[ −10.5 , −5.3 ]
[ −11.4 , −4.4 ]
[ −11.0 , −4.8 ]
Geometry3K 2101, OPD − SFT
−5.2/−9.0/−5.6/−7.0
−6.7
[ −9.2 , −4.3 ]
[ −10.0 , −3.5 ]
[ −9.4 , −4.0 ]
Geometry3K 900 (4,096 tok.), OPD − SFT
−11.2/−10.8/−9.2/−12.2
−10.8
[ −13.5 , −8.2 ]
[ −14.1 , −7.5 ]
[ −12.8 , −8.9 ]
Geometry3K 300, unfiltered − filtered SFT
−5.2/−1.8/+1.8/−2.0
−1.8
[ −4.2 , +0.7 ]
[ −5.6 , +2.1 ]
[ −6.4 , +2.8 ]
Appendix
Table 16: Geometry3K comparisons, as in Table 15 .
Tolerance
Problems
SFT
OPD
OPD − SFT
5%
100
0.423
0.404
−1.9 [ −4.5 , +0.7 ]
300
0.420
0.419
−0.1 [ −2.4 , +2.2 ]
900
0.481
0.402
−7.9 [ −10.5 , −5.3 ]
2101
0.478
0.411
−6.7 [ −9.2 , −4.3 ]
1%
100
0.407
0.395
−1.2 [ −3.6 , +1.3 ]
300
0.408
0.406
−0.2 [ −2.5 , +2.1 ]
Appendix
Table 17: Geometry3K accuracy (mean over four seeds) at three numeric tolerances, with the paired OPD minus SFT difference in points and its question-clustered interval. Zero-shot student / teacher: 0.366 / 0.570 at 5%, 0.354 / 0.552 at 1%, 0.330 / 0.534 at 0.1%.
Questions
Variant
Accuracy
vs. SFT (paired)
vs. OPD (paired)
100
OPD-term
0.814 (.006)
+0.021 [ +0.001 , +0.042 ]
+0.002 [ −0.011 , +0.015 ]
300
OPD-term
0.823 (.011)
+0.024 [ +0.004 , +0.042 ]
−0.004 [ −0.016 , +0.009 ]
300
OPD-fwd
0.828 (.005)
+0.029 [ +0.009 , +0.048 ]
+0.001 [ −0.011 , +0.015 ]
300
OPD-sftinit
0.822 (.005)
+0.023 [ +0.008 , +0.038 ]
−0.004 [ −0.021 , +0.012 ]
300
OPD-seqmean
0.825 (.007)
+0.025 [ +0.007 , +0.044 ]
−0.001 [ −0.014 , +0.011 ]
Appendix
Table 18: OPD variants on ChartQA (four seeds each, across-seed sd in brackets): no variant differs from plain OPD. Reference points: SFT 0.793 / 0.799 and OPD 0.811 / 0.826 at 100 / 300 questions. Intervals are question-clustered.
100 questions
300 questions
SFT steps
Accuracy
OPD − SFT
Accuracy
OPD − SFT
19
0.791 (.007)
+0.021 [ +0.001 , +0.039 ]
0.802 (.009)
+0.024 [ +0.008 , +0.040 ]
38
0.797 (.012)
+0.015 [ −0.005 , +0.035 ]
0.816 (.005)
+0.011 [ −0.006 , +0.028 ]
75
0.788 (.005)
+0.024 [ +0.003 , +0.044 ]
0.811 (.005)
+0.016 [ −0.002 , +0.034 ]
150
0.788 (.009)
+0.024 [ +0.003 , +0.043 ]
0.808 (.004)
+0.018 [ −0.001 , +0.039 ]
302 (protocol)
0.793 (.022)
+0.018 [ −0.001 , +0.038 ]
0.799 (.015)
+0.027 [ +0.008 , +0.047 ]
Appendix
Table 19: SFT trained for fewer steps than the protocol’s 302, four seeds each (across-seed sd in brackets), and the paired OPD minus SFT difference against each length (question-clustered intervals). 150 steps of 16 sequences equals OPD’s number of training sequences. The best length was selected on the test set.
Task
Method
100
300
900
3000 / 2101
Slope (pts / decade)
R2
ChartQA
SFT
0.793
0.799
0.808
0.840
+3.1 [ +1.3 , +4.8 ]
0.87
ChartQA
OPD
0.811
0.826
0.823
0.825
+0.8 [ −0.9 , +2.4 ]
0.51
Geometry3K
SFT
0.423
0.420
0.481
0.478
+5.1 [ +2.7 , +7.6 ]
0.78
Geometry3K
OPD
0.404
0.419
0.402
0.411
+0.0 [ −2.7 , +2.8 ]
0.00
Appendix
Table 20: Accuracy (mean over four seeds) at the four budgets and the slope of a per-seed least-squares fit of accuracy on log10 (budget), in points per decade, with the t interval over the four seeds and the R2 of the fit to the seed means.
Seed
Task
Budget
Method
1
2
3
42
Mean
ChartQA
100 questions
SFT
0.778
0.808
0.770
0.816
0.793
OPD
0.810
0.820
0.800
0.816
0.811
300 questions
SFT
0.814
0.806
0.800
0.778
0.799
OPD
0.820
0.826
0.824
0.836
0.826
OPD, 32B teacher
0.814
0.812
0.822
0.800
0.812
Appendix
Table 21: Per-seed accuracy on the 500 test questions (seeds 1, 2, 3, and 42) and the four-seed mean.
On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.
Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1
Post-training enables vision-language models (VLMs) to understand human instructions and perform various downstream tasks. Current post-training methods usually rely on human-annotated data, distillation from external models, reinforcement learning with human feedback, or verifiable answers. This limits their ability to improve without external supervision. To tackle this, we propose NOPD (Noisy Student On-Policy Self-Distillation), a simple yet effective self-distillation approach that improves VLMs without any external models or ground-truth answers. Our key insight is that prediction discrepancies between clean and corrupted inputs naturally induce a self-supervision signal. In NOPD, the model learns from corrupted inputs while using its own predictions under clean inputs as token-level supervision. We show the effectiveness of NOPD on five visual reasoning tasks; it can match and even outperform reinforcement learning approaches or distillation from external models. Notably, when trained with 2.1K samples from Geometry3K, NOPD improves Qwen2.5-VL-7B by 20 points on its validation set. It also shows generalization on out-of-distribution test sets and achieves 7.4 point gains on MathVista. Furthermore, we demonstrate that NOPD is a general approach to enhance VLMs, achieving improvements across three models on 12 benchmarks.
Shuai Wang, Daoan Zhang, Zhe Tang +2
The Hong Kong University of Science and Technology (Guangzhou) · University of Rochester · Zhejiang University of Technology +1
On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories. This aligns the distillation states with the student's own generation distribution. However, it still assumes that the complete teacher distribution is an appropriate target across student capacities. In vision--language reasoning, teacher corrections can depend on visual distinctions that a compact student cannot represent. Our target-scaling study shows that, as the target approaches the complete teacher distribution, the student realizes less of the prescribed shift and obtains worse downstream performance. We therefore propose Fisher-Projected On-Policy Distillation (FP-OPD), which distills only locally realizable teacher corrections. FP-OPD uses continuous visual perturbations to estimate the student's local visual tangent space and projects the centered teacher--student log-probability gap onto this space under the student's Fisher metric. The resulting capacity-aware target is optimized with full-vocabulary reverse KL on student trajectories, retaining the standard OPD framework. In 8B-to-2B distillation, FP-OPD improves all seven evaluated multimodal benchmarks. It raises the average score by 2.77 points over the pretrained student and by 1.60 points over standard OPD. These results demonstrate that locally realizable teacher corrections provide a more effective target for distilling compact vision--language models.
Leyan Xue, Feng Xiong, Mingjun Ma +1
College of Intelligence and Computing Tianjin University Tianjin, China