Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Authors: Wen Ye, Ailin Deng, Zirui Wang, Shixing Yu, Han Zhang, Tian Yun, Nikhil Mehta, Tarun Kalluri, Xuefei Cao, Xuezhe Ma, Hengduo Li
Organizations: Meta Superintelligence Labs · University of Southern California · University of California, Berkeley · Cornell University · University of California, Los Angeles
Visual mathematical reasoning can fail before reasoning begins: when a vision-language model (VLM) misreads a diagram, subsequent reasoning proceeds from false premises. This perceptual bottleneck is especially pronounced in smaller models, while accurate textual descriptions can substantially narrow the gap to larger models. Because obtaining such descriptions at inference requires additional annotation or model calls, we propose minimal-edit corrective on-policy distillation. Using reference figure descriptions and verified reference solutions as training-only information, a teacher scores incorrect student rollouts under a minimal-edit objective, and only teacher-disfavored tokens contribute to the corrective loss. Correct rollouts receive task-reward reinforcement learning. This preserves on-policy supervision while avoiding full-response imitation. Across model scales, the method requires no descriptions at test time and meets or exceeds the corresponding base model supplied with a reference figure description at inference. It also improves perception-oriented performance, particularly low-level geometric perception, and transfers to external mathematics and perception benchmarks. Because the minimal-edit teacher evaluates corrections rather than generating its own solution, the method remains effective with a frozen same-size teacher, although its gains are smaller than with a larger teacher on most metrics.
Figures & tables
Figure 1: Overview of minimal-edit corrective on-policy distillation. A rule-based verifier routes correct rollouts to task reward; incorrect rollouts are scored by the minimal-edit teacher, and only teacher-disfavored tokens are used to update the student. Dashed arrows indicate loss calculation.
Figure 2: Geometry3K performance with image-only input or an added reference figure description. Descriptions narrow the size gap at every scale, identifying perception as a major bottleneck.
Figure 3: Token-level k1 estimates for the incorrect response to the question in Figure 1 , under privileged (top) and minimal-edit (bottom) conditioning. Red tokens ( kt>0 are tokens that teacher disfavors relative to student) are retained and green tokens are discarded. Minimal-edit conditioning more precisely targets the leg–hypotenuse error, yielding a more interpretable corrective signal.
\rectanglecolor metabg13-114-7
Geometry3K
MathVerse
Fig-desc
Geoperception
Method
pass@1
pass@8
(ALL)
recall
pass@1
pass@4
Qwen3-VL-2B (base)
0.321
0.540
37.8
0.567
0.179
0.311
SFT (rejection sampling)
0.392
0.675
35.4
0.605
0.244
0.433
SFT (describe-then-solve)
0.475
0.791
37.9
0.670
0.362
0.580
SFT (describe-figure)
0.214
0.598
24.3
0.770
0.359
0.617
SFT (mix: rejection + describe)
0.406
0.688
33.5
0.784
0.289
0.473
Table 1: Main results for the Qwen3-VL-2B student. Per column, the best result is bold and the second best is underlined ; our minimal-edit rows are shaded.
\rectanglecolor metabg13-114-7
Geometry3K
MathVerse
Fig-desc
Geoperception
Method
pass@1
pass@8
(ALL)
recall
pass@1
pass@4
Qwen3-VL-8B (base)
0.491
0.718
51.1
0.687
0.437
0.551
SFT (rejection sampling)
0.599
0.840
47.8
0.781
0.424
0.574
SFT (describe-then-solve)
0.691
0.911
43.5
0.798
0.500
0.686
SFT (describe-figure)
0.489
0.841
42.1
0.839
0.461
0.669
SFT (mix: rejection + describe)
0.624
0.849
50.9
0.846
0.484
0.622
Table 2: Main results for the Qwen3-VL-8B student. Per column, the best result is bold and the second best is underlined ; our minimal-edit rows are shaded.
\rectanglecolor metabg9-19-8
Geometry3K
MathVerse
Fig-desc
Geoperception
Model
Train
pass@1
pass@8
(ALL)
recall
pass@1
pass@4
Qwen2.5-VL-7B
–
0.373
0.683
45.1
0.431
0.330
0.575
Perception-R1-7B
1.4K
0.504
0.690
44.9
0.541
0.420
0.570
NoisyRollout-7B
2.1K+6.4K
0.522
0.709
47.9
0.500
0.410
0.556
PAPO-7B
38.8K
0.407
0.640
48.9
0.470
0.351
0.540
DeepEyes-7B
47K
0.393
0.693
46.1
0.447
0.363
0.588
Table 3: Published Qwen2.5-VL-7B checkpoints and our method on the same backbone. Metrics follow Table 1 ; per column the best result is bold and the second-best value or tie is underlined .
Figure 4: Training dynamics for Qwen3-VL-2B component ablations: policy entropy (left), mean response length (center), and distillation loss (right). Only the full method remains stable.
\rectanglecolor metabg6-16-4
ME Teacher
Reward
Mask
privileged OPD
✗
✗
✗
+ ME teacher
✓
✗
✗
+ task reward
✓
✓
✗
+ random 2% mask
✓
✓
R
full method
✓
✓
✓
− task reward
✓
✗
✓
Table 4: Component configurations used in the ablation study.
\rectanglecolor metabg5-15-7
Geometry3K
MathVerse
Fig-desc
Geoperception
λ
pass@1
pass@8
(ALL)
recall
pass@1
pass@4
0.001
0.442
0.736
42.5
0.584
0.373
0.550
0.005
0.432
0.721
39.0
0.589
0.374
0.569
0.01
0.554
0.794
45.8
0.633
0.455
0.635
0.05
0.407
0.697
42.3
0.609
0.396
0.594
0.1
0.366
0.655
41.1
0.610
0.357
0.562
Table 5: Corrective-weight sweep and reference-solution ablation for the Qwen3-VL-2B student with a Qwen3-VL-8B teacher. Performance peaks at an intermediate weight and declines without the reference solution. Per column, the best result is bold and the second best is underlined ; the configuration used in our main experiments is shaded.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
8B student / 32B teacher
2B student / 8B teacher
Mean response length R
736.62
783.46
Mean student-prompt length Ps
209.32
209.35
Student sequence D=Ps+R
945.94
992.81
Incorrect-rollout fraction w
0.275
0.456
Parameter ratio r
4.00
4.00
Fixed edit-prompt length Pt
1369.2
1316.6
Appendix
Table 6: Inputs to the computational-cost estimate. Sequence lengths are measured in tokens.
Configuration
D
Pt+2R
w∗
Cnaive
Cedit
Cedit/Cnaive
8B student / 32B teacher
945.94
2842.44
0.333
19.56KND
18.17KND
0.929
2B student / 8B teacher
992.81
2883.52
0.344
19.58KND
22.17KND
1.133
Appendix
Table 7: Estimated per-step compute of minimal-edit distillation relative to naive OPD.
Figure 5: Training dynamics of the full minimal-edit run. Corrective-mask keep rate on incorrect rollouts (left) and correct-rollout fraction (right). The mask stays near 2% as accuracy rises, shifting training toward task reward without diffusing corrective supervision.
Figure 6: Qualitative examples of reference figure descriptions used for the figure-description recall evaluation. Each row pairs a Geometry3K diagram with its reference figure description, whose objects, labels, measures, and geometric relations are decomposed into atomic facts by the Gemini 3.1 Pro judge.
Method
2B student
8B student
Base
0.567±0.007
0.687±0.002
SFT (rejection sampling)
0.605±0.009
0.781±0.008
SFT (describe-then-solve)
0.670±0.010
0.798±0.011
SFT (describe-figure)
0.770±0.005
0.839±0.007
SFT (mix: rejection + describe)
0.784±0.006
0.846±0.003
GRPO (plain prompt)
0.552±0.009
0.676±0.007
Appendix
Table 8: Figure-description recall for Qwen3-VL-2B and Qwen3-VL-8B students. Entries are mean ± standard deviation across three independently sampled descriptions per diagram; the main tables report the corresponding means. Per column, the best result is bold and the second best is underlined .
Model
Recall
Qwen2.5-VL-7B
0.431±0.007
Perception-R1-7B
0.541±0.002
NoisyRollout-7B
0.500±0.014
PAPO-7B
0.470±0.005
DeepEyes-7B
0.447±0.003
Vision-R1-7B
0.486±0.007
Appendix
Table 9: Figure-description recall on the Qwen2.5-VL-7B backbone. Entries are mean ± standard deviation across three independently sampled descriptions per diagram; the main table reports the corresponding means. The best result is bold and the second best is underlined .
Method
Angle
Line
OnLine
Equal
OnCirc
Parallel
Perp
Avg.
Base
0.129
0.158
0.093
0.191
0.405
0.128
0.147
0.179
SFT (rejection sampling)
0.165
0.231
0.088
0.254
0.421
0.360
0.188
0.244
SFT (describe-then-solve)
0.346
0.423
0.241
0.277
0.450
0.511
0.284
0.362
SFT (describe-figure)
0.341
0.307
0.207
0.442
0.355
0.608
0.253
0.359
SFT (mix)
0.172
0.248
0.168
0.276
0.413
0.527
0.217
0.289
GRPO (plain prompt)
0.262
0.290
0.217
0.243
0.392
0.287
0.262
0.279
Appendix
Table 10: Geoperception pass@1 by predicate for Qwen3-VL-2B students. Per column, the best result is bold and the second best is underlined .
Method
Angle
Line
OnLine
Equal
OnCirc
Parallel
Perp
Avg.
Base
0.349
0.454
0.004
0.477
0.755
0.615
0.406
0.437
SFT (rejection sampling)
0.345
0.408
0.033
0.448
0.693
0.645
0.398
0.424
SFT (describe-then-solve)
0.468
0.595
0.128
0.459
0.702
0.720
0.428
0.500
SFT (describe-figure)
0.356
0.418
0.137
0.547
0.750
0.712
0.309
0.461
SFT (mix)
0.376
0.466
0.110
0.467
0.767
0.761
0.445
0.484
GRPO (plain prompt)
0.634
0.754
0.012
0.546
0.733
0.590
0.443
0.530
Appendix
Table 11: Geoperception pass@1 by predicate for Qwen3-VL-8B students. Per column, the best result is bold and the second best is underlined .
Model
Angle
Line
OnLine
Equal
OnCirc
Parallel
Perp
Avg.
Qwen2.5-VL-7B
0.382
0.475
0.178
0.258
0.404
0.365
0.247
0.330
Perception-R1-7B
0.526
0.600
0.170
0.293
0.568
0.536
0.248
0.420
NoisyRollout-7B
0.490
0.589
0.218
0.273
0.507
0.506
0.287
0.410
PAPO-7B
0.453
0.571
0.162
0.249
0.465
0.334
0.223
0.351
DeepEyes-7B
0.442
0.569
0.155
0.233
0.478
0.402
0.265
0.363
Vision-R1-7B
0.534
0.573
0.132
0.309
0.488
0.426
0.279
0.392
Appendix
Table 12: Geoperception pass@1 by predicate on the Qwen2.5-VL-7B backbone. Per column, the best result is bold and the second best is underlined .
Figure 7: Hypothetical minimal-edit rollout for the leg–hypotenuse example in Figure 3 . Red strikethrough marks text removed from the student response, and green marks inserted text. The edit identifies 32 as the hypotenuse, revises the dependent reasoning, and changes the answer from 323 to 16 while preserving text that remains applicable. This rollout is generated only for visualization; training does not decode an edited response and instead scores the original student tokens under the minimal-edit prompt.
Figure 8: Edit conditioning concentrates corrective-token selection near a numerical error. Token-level k1 estimates for an incorrect solution that selects the correct relation, AB=56cos(42∘) , but evaluates it incorrectly as 41.2 rather than 41.6. The conventional privileged teacher assigns a positive k1 estimate to many otherwise acceptable wording choices (top), whereas minimal-edit conditioning assigns high probability to most of the student’s derivation and concentrates the corrective signal near the erroneous calculation (bottom). Colors follow Figure 3 .
Figure 9: Hypothetical minimal-edit rollout for the arithmetic-error example in Figure 8 . Red strikethrough marks removed text, and green marks inserted text. The edit preserves AB=56cos(42∘) , corrects the product from 41.2036 to 41.6136, and changes the rounded answer from 41.2 to 41.6 feet. This rollout is generated only for visualization; the training procedure scores the fixed student response without decoding this revision.
Figure 10: Edit conditioning concentrates corrective-token selection near a diameter–radius error. The student misreads the labeled diameter of 10 cm as the radius and consequently computes the arc length using r=10 . The conventional privileged teacher assigns positive k1 estimates throughout the response (top), whereas the minimal-edit teacher concentrates the corrective signal on the tokens that identify 10 cm as the radius (bottom). Colors follow Figure 3 .
Figure 11: Hypothetical minimal-edit rollout for the diameter–radius example in Figure 10 . Red strikethrough marks removed text, and green marks inserted text. The edit recognizes 10 cm as the diameter, sets r=5 , and updates the arc length from 65π/9≈22.69 to 65π/18≈11.34 . This rollout is generated only for visualization; during training, the teacher generates no revised response and only scores the original student tokens.
On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.
Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1
Effective mathematics education requires identifying and responding to students' mistakes. For AI to support pedagogical applications, models must perform well across different levels of student proficiency. Our work provides an extensive, year-long snapshot of how 11 vision-language models (VLMs) perform on DrawEduMath, a QA benchmark involving real students' handwritten, hand-drawn responses to math problems. We find that models' weaknesses concentrate on a core component of math education: student error. All evaluated VLMs underperform when describing work from students who require more pedagogical help, and across all QA, they struggle the most on questions related to assessing student error. Thus, while VLMs may be optimized to be math problem solving experts, our results suggest that they require alternative development incentives to adequately support educational use cases.
Li Lucy, Albert Zhang, Nathan Anderson +2
University of Washington · Insource Services · Worcester Polytechnic Institute +1
Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition (Acquire), reading (Read), or answer grounding (Ground) can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce Reflection on Visual Evidence (ReVuE), an on-policy distillation method for visual agents. ReVuE compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the Acquire, Read, and Ground stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, ReVuE outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. ReVuE also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy. Code is available at https://github.com/sylvain-wei/ReVuE