Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning

Wen Ye; Ailin Deng; Zirui Wang; Shixing Yu; Han Zhang; Tian Yun; Nikhil Mehta; Tarun Kalluri

cs.LGOct 21, 2026

Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning

Authors: Wen Ye, Ailin Deng, Zirui Wang, Shixing Yu, Han Zhang, Tian Yun, Nikhil Mehta, Tarun Kalluri,

Organizations: Meta Superintelligence Labs · University of Southern California · University of California, Berkeley · Cornell University ·

Abstract

Visual mathematical reasoning can fail before reasoning begins: when a vision-language model (VLM) misreads a diagram, subsequent reasoning proceeds from false premises. This perceptual bottleneck is especially pronounced in smaller models, while accurate textual descriptions can substantially narrow the gap to larger models. Because obtaining such descriptions at inference requires additional annotation or model calls, we propose minimal-edit corrective on-policy distillation. Using reference figure descriptions and verified reference solutions as training-only information, a teacher scores incorrect student rollouts under a minimal-edit objective, and only teacher-disfavored tokens contribute to the corrective loss. Correct rollouts receive task-reward reinforcement learning. This preserves on-policy supervision while avoiding full-response imitation. Across model scales, the method requires no descriptions at test time and meets or exceeds the corresponding base model supplied with a reference figure description at inference. It also improves perception-oriented performance, particularly low-level geometric perception, and transfers to external mathematics and perception benchmarks. Because the minimal-edit teacher evaluates corrections rather than generating its own solution, the method remains effective with a frozen same-size teacher, although its gains are smaller than with a larger teacher on most metrics.

Figures & tables

Figure 1: Overview of minimal-edit corrective on-policy distillation. A rule-based verifier routes correct rollouts to task reward; incorrect rollouts are scored by the minimal-edit teacher, and only teacher-disfavored tokens are used to update the student. Dashed arrows indicate loss calculation.

Figure 2: Geometry3K performance with image-only input or an added reference figure description. Descriptions narrow the size gap at every scale, identifying perception as a major bottleneck.

Figure 3: Token-level

k_{1}

estimates for the incorrect response to the question in Figure 1 , under privileged (top) and minimal-edit (bottom) conditioning. Red tokens (

k_{t}>0

are tokens that teacher disfavors relative to student) are retained and green tokens are discarded. Minimal-edit conditioning more precisely targets the leg–hypotenuse error, yielding a more interpretable corrective signal.

\rectanglecolor metabg13-114-7	Geometry3K		MathVerse	Fig-desc	Geoperception
Method	pass@1	pass@8	(ALL)	recall	pass@1	pass@4
Qwen3-VL-2B (base)	$0.321$	$0.540$	$37.8$	$0.567$	$0.179$	$0.311$
SFT (rejection sampling)	$0.392$	$0.675$	$35.4$	$0.605$	$0.244$	$0.433$
SFT (describe-then-solve)	$0.475$	$\underline{0.791}$	$37.9$	$0.670$	$0.362$	$0.580$
SFT (describe-figure)	$0.214$	$0.598$	$24.3$	$\underline{0.770}$	$0.359$	$0.617$
SFT (mix: rejection + describe)	$0.406$	$0.688$	$33.5$	$\bm{0.784}$	$0.289$	$0.473$

Table 1: Main results for the Qwen3-VL-2B student. Per column, the best result is

\bm{bold}

and the second best is underlined ; our minimal-edit rows are shaded.

\rectanglecolor metabg13-114-7	Geometry3K		MathVerse	Fig-desc	Geoperception
Method	pass@1	pass@8	(ALL)	recall	pass@1	pass@4
Qwen3-VL-8B (base)	$0.491$	$0.718$	$51.1$	$0.687$	$0.437$	$0.551$
SFT (rejection sampling)	$0.599$	$0.840$	$47.8$	$0.781$	$0.424$	$0.574$
SFT (describe-then-solve)	$0.691$	$\bm{0.911}$	$43.5$	$0.798$	$0.500$	$\bm{0.686}$
SFT (describe-figure)	$0.489$	$0.841$	$42.1$	$0.839$	$0.461$	$0.669$
SFT (mix: rejection + describe)	$0.624$	$0.849$	$50.9$	$\underline{0.846}$	$0.484$	$0.622$

Table 2: Main results for the Qwen3-VL-8B student. Per column, the best result is

\bm{bold}

and the second best is underlined ; our minimal-edit rows are shaded.

\rectanglecolor metabg9-19-8		Geometry3K		MathVerse	Fig-desc	Geoperception
Model	Train	pass@1	pass@8	(ALL)	recall	pass@1	pass@4
Qwen2.5-VL-7B	–	$0.373$	$0.683$	$45.1$	$0.431$	$0.330$	$\underline{0.575}$
Perception-R1-7B	1.4K	$0.504$	$0.690$	$44.9$	$\bm{0.541}$	$\bm{0.420}$	$0.570$
NoisyRollout-7B	2.1K+6.4K	$\underline{0.522}$	$\underline{0.709}$	$47.9$	$\underline{0.500}$	$\underline{0.410}$	$0.556$
PAPO-7B	38.8K	$0.407$	$0.640$	$\underline{48.9}$	$0.470$	$0.351$	$0.540$
DeepEyes-7B	47K	$0.393$	$0.693$	$46.1$	$0.447$	$0.363$	$\bm{0.588}$

Table 3: Published Qwen2.5-VL-7B checkpoints and our method on the same backbone. Metrics follow Table 1 ; per column the best result is

\bm{bold}

and the second-best value or tie is underlined .

Figure 4: Training dynamics for Qwen3-VL-2B component ablations: policy entropy (left), mean response length (center), and distillation loss (right). Only the full method remains stable.

\rectanglecolor metabg6-16-4	ME Teacher	Reward	Mask
privileged OPD	✗	✗	✗
+ ME teacher	✓	✗	✗
+ task reward	✓	✓	✗
+ random 2% mask	✓	✓	$\mathrm{R}$
full method	✓	✓	✓
$-$ task reward	✓	✗	✓

Table 4: Component configurations used in the ablation study.

\rectanglecolor metabg5-15-7	Geometry3K		MathVerse	Fig-desc	Geoperception
$\lambda$	pass@1	pass@8	(ALL)	recall	pass@1	pass@4
$0.001$	$0.442$	$0.736$	$42.5$	$0.584$	$0.373$	$0.550$
$0.005$	$0.432$	$0.721$	$39.0$	$0.589$	$0.374$	$0.569$
$0.01$	$\bm{0.554}$	$\bm{0.794}$	$\bm{45.8}$	$\bm{0.633}$	$\bm{0.455}$	$\bm{0.635}$
$0.05$	$0.407$	$0.697$	$42.3$	$0.609$	$0.396$	$0.594$
$0.1$	$0.366$	$0.655$	$41.1$	${0.610}$	$0.357$	$0.562$

Table 5: Corrective-weight sweep and reference-solution ablation for the Qwen3-VL-2B student with a Qwen3-VL-8B teacher. Performance peaks at an intermediate weight and declines without the reference solution. Per column, the best result is

\bm{bold}

and the second best is underlined ; the configuration used in our main experiments is shaded.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	8B student / 32B teacher	2B student / 8B teacher
Mean response length $R$	736.62	783.46
Mean student-prompt length $P_{s}$	209.32	209.35
Student sequence $D=P_{s}+R$	945.94	992.81
Incorrect-rollout fraction $w$	0.275	0.456
Parameter ratio $r$	4.00	4.00
Fixed edit-prompt length $P_{t}$	1369.2	1316.6

Appendix

Table 6: Inputs to the computational-cost estimate. Sequence lengths are measured in tokens.

Configuration	$D$	$P_{t}+2R$	$w^{*}$	$C_{\mathrm{naive}}$	$C_{\mathrm{edit}}$	$C_{\mathrm{edit}}/C_{\mathrm{naive}}$
8B student / 32B teacher	945.94	2842.44	0.333	$19.56\,KND$	$18.17\,KND$	0.929
2B student / 8B teacher	992.81	2883.52	0.344	$19.58\,KND$	$22.17\,KND$	1.133

Appendix

Table 7: Estimated per-step compute of minimal-edit distillation relative to naive OPD.

Figure 5: Training dynamics of the full minimal-edit run. Corrective-mask keep rate on incorrect rollouts (left) and correct-rollout fraction (right). The mask stays near 2% as accuracy rises, shifting training toward task reward without diffusing corrective supervision.

Figure 6: Qualitative examples of reference figure descriptions used for the figure-description recall evaluation. Each row pairs a Geometry3K diagram with its reference figure description, whose objects, labels, measures, and geometric relations are decomposed into atomic facts by the Gemini 3.1 Pro judge.

Method	2B student	8B student
Base	$0.567\pm 0.007$	$0.687\pm 0.002$
SFT (rejection sampling)	$0.605\pm 0.009$	$0.781\pm 0.008$
SFT (describe-then-solve)	$0.670\pm 0.010$	$0.798\pm 0.011$
SFT (describe-figure)	$\underline{0.770\pm 0.005}$	$0.839\pm 0.007$
SFT (mix: rejection + describe)	$\bm{0.784\pm 0.006}$	$\underline{0.846\pm 0.003}$
GRPO (plain prompt)	$0.552\pm 0.009$	$0.676\pm 0.007$

Appendix

Table 8: Figure-description recall for Qwen3-VL-2B and Qwen3-VL-8B students. Entries are mean

\pm

standard deviation across three independently sampled descriptions per diagram; the main tables report the corresponding means. Per column, the best result is

\bm{bold}

and the second best is underlined .

Model	Recall
Qwen2.5-VL-7B	$0.431\pm 0.007$
Perception-R1-7B	$\bm{0.541\pm 0.002}$
NoisyRollout-7B	$\underline{0.500\pm 0.014}$
PAPO-7B	$0.470\pm 0.005$
DeepEyes-7B	$0.447\pm 0.003$
Vision-R1-7B	$0.486\pm 0.007$

Appendix

Table 9: Figure-description recall on the Qwen2.5-VL-7B backbone. Entries are mean

\pm

standard deviation across three independently sampled descriptions per diagram; the main table reports the corresponding means. The best result is

\bm{bold}

and the second best is underlined .

Method	Angle	Line	OnLine	Equal	OnCirc	Parallel	Perp	Avg.
Base	$0.129$	$0.158$	$0.093$	$0.191$	$0.405$	$0.128$	$0.147$	$0.179$
SFT (rejection sampling)	$0.165$	$0.231$	$0.088$	$0.254$	$0.421$	$0.360$	$0.188$	$0.244$
SFT (describe-then-solve)	$0.346$	$0.423$	$0.241$	$0.277$	$0.450$	$0.511$	$0.284$	$0.362$
SFT (describe-figure)	$0.341$	$0.307$	$0.207$	$\bm{0.442}$	$0.355$	$\underline{0.608}$	$0.253$	$0.359$
SFT (mix)	$0.172$	$0.248$	$0.168$	$0.276$	$0.413$	$0.527$	$0.217$	$0.289$
GRPO (plain prompt)	$0.262$	$0.290$	$0.217$	$0.243$	$0.392$	$0.287$	$0.262$	$0.279$

Appendix

Table 10: Geoperception pass@1 by predicate for Qwen3-VL-2B students. Per column, the best result is

\bm{bold}

and the second best is underlined .

Method	Angle	Line	OnLine	Equal	OnCirc	Parallel	Perp	Avg.
Base	$0.349$	$0.454$	$0.004$	$0.477$	$0.755$	$0.615$	$0.406$	$0.437$
SFT (rejection sampling)	$0.345$	$0.408$	$0.033$	$0.448$	$0.693$	$0.645$	$0.398$	$0.424$
SFT (describe-then-solve)	$0.468$	$0.595$	$\underline{0.128}$	$0.459$	$0.702$	$0.720$	$0.428$	$0.500$
SFT (describe-figure)	$0.356$	$0.418$	$\bm{0.137}$	$0.547$	$0.750$	$0.712$	$0.309$	$0.461$
SFT (mix)	$0.376$	$0.466$	$0.110$	$0.467$	$0.767$	$0.761$	$0.445$	$0.484$
GRPO (plain prompt)	$\bm{0.634}$	$\bm{0.754}$	$0.012$	$0.546$	$0.733$	$0.590$	$0.443$	$0.530$

Appendix

Table 11: Geoperception pass@1 by predicate for Qwen3-VL-8B students. Per column, the best result is

\bm{bold}

and the second best is underlined .

Model	Angle	Line	OnLine	Equal	OnCirc	Parallel	Perp	Avg.
Qwen2.5-VL-7B	$0.382$	$0.475$	$0.178$	$0.258$	$0.404$	$0.365$	$0.247$	$0.330$
Perception-R1-7B	$\underline{0.526}$	$\bm{0.600}$	$0.170$	$\underline{0.293}$	$\bm{0.568}$	$\bm{0.536}$	$0.248$	$\bm{0.420}$
NoisyRollout-7B	$0.490$	$\underline{0.589}$	$\underline{0.218}$	$0.273$	$\underline{0.507}$	$0.506$	$\underline{0.287}$	$\underline{0.410}$
PAPO-7B	$0.453$	$0.571$	$0.162$	$0.249$	$0.465$	$0.334$	$0.223$	$0.351$
DeepEyes-7B	$0.442$	$0.569$	$0.155$	$0.233$	$0.478$	$0.402$	$0.265$	$0.363$
Vision-R1-7B	$\bm{0.534}$	$0.573$	$0.132$	$\bm{0.309}$	$0.488$	$0.426$	$0.279$	$0.392$

Appendix

Table 12: Geoperception pass@1 by predicate on the Qwen2.5-VL-7B backbone. Per column, the best result is

\bm{bold}

and the second best is underlined .

Figure 7: Hypothetical minimal-edit rollout for the leg–hypotenuse example in Figure 3 . Red strikethrough marks text removed from the student response, and green marks inserted text. The edit identifies 32 as the hypotenuse, revises the dependent reasoning, and changes the answer from $32\sqrt{3}$ to 16 while preserving text that remains applicable. This rollout is generated only for visualization; training does not decode an edited response and instead scores the original student tokens under the minimal-edit prompt.

Figure 8: Edit conditioning concentrates corrective-token selection near a numerical error. Token-level

k_{1}

estimates for an incorrect solution that selects the correct relation,

AB=56\cos(42^{\circ})

, but evaluates it incorrectly as 41.2 rather than 41.6. The conventional privileged teacher assigns a positive

k_{1}

estimate to many otherwise acceptable wording choices (top), whereas minimal-edit conditioning assigns high probability to most of the student’s derivation and concentrates the corrective signal near the erroneous calculation (bottom). Colors follow Figure 3 .

Figure 9: Hypothetical minimal-edit rollout for the arithmetic-error example in Figure 8 . Red strikethrough marks removed text, and green marks inserted text. The edit preserves $AB=56\cos(42^{\circ})$ , corrects the product from 41.2036 to 41.6136, and changes the rounded answer from 41.2 to 41.6 feet. This rollout is generated only for visualization; the training procedure scores the fixed student response without decoding this revision.

Figure 10: Edit conditioning concentrates corrective-token selection near a diameter–radius error. The student misreads the labeled diameter of 10 cm as the radius and consequently computes the arc length using

r=10

. The conventional privileged teacher assigns positive

k_{1}

estimates throughout the response (top), whereas the minimal-edit teacher concentrates the corrective signal on the tokens that identify 10 cm as the radius (bottom). Colors follow Figure 3 .

Figure 11: Hypothetical minimal-edit rollout for the diameter–radius example in Figure 10 . Red strikethrough marks removed text, and green marks inserted text. The edit recognizes 10 cm as the diameter, sets $r=5$ , and updates the arc length from $65\pi/9\approx 22.69$ to $65\pi/18\approx 11.34$ . This rollout is generated only for visualization; during training, the teacher generates no revised response and only scores the original student tokens.

Reasoning for VLMs

See all 55 papers

Visual-Advantage On-Policy Distillation for Vision-Language Models
May 21, 2026Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8Vision-Language Models VLM Distillation
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1
The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors
Mar 1, 2026Li Lucy, Albert Zhang, Nathan Anderson +2AI in Education VLM Evaluation
University of Washington · Insource Services · Worcester Polytechnic Institute +1
On-Policy Visual Evidence Distillation
Sep 29, 2026Shaohang Wei, Feifan Song, Guangyue Peng +9VLM Distillation Visually Grounded Reasoning
Peking University · CUHK · Nanjing University +1

May 21, 2026J/K move · Enter open · S save

Visual-Advantage On-Policy Distillation for Vision-Language Models

Ruiqi Liu, Xiaolei Lv, Gengsheng Li, Ximo Zhu+7

Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello+1

Vision-Language Models VLM Distillation VLM Reasoning Visually Grounded Reasoning On-Policy Distillation Teacher-Student Learning+1

Open paper AskPDF arXiv

On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.

Abstract

Reasoning for VLMs

Explore similar work

Visual-Advantage On-Policy Distillation for Vision-Language Models

The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors

On-Policy Visual Evidence Distillation