Prior-as-Skill: Remember Outside and Read Inside with On-Policy Self-Distillation for Self-Evolving VLM Agents
Organizations: Newcastle University · University of Manchester · University of Melbourne · University of Nottingham ·
Abstract
Self-evolving visual agents keep what they learn as skills: key frames with step notes on a task's steps and their order. On-policy self-distillation (OPSD) is now a popular way to train such skills into a vision-language model that matches, on its own answers, a teacher that also reads the written order. At deployment, frames often go missing or show an unusual order, as when a person or a website reorders the steps. Existing internalization methods keep any order they store in the frame-reading weights: trained on full frames, the model guesses when they are missing, and trained with frames withheld, it stores the order, so memory can override what it sees. We propose PriorSplit, which fills in missing frames from an order kept outside the model, with no training beyond OPSD. It counts which of two neighboring steps comes first in training episodes, and the model checks if both asked steps are in view: if so, it answers from the frames, and otherwise the counts answer. Extensive experiments on synthetic skills with five models and on step-tagged order questions in new episodes of seen web, cooking and egocentric tasks demonstrate that, with frames missing, PriorSplit outperforms every compared method at OPSD's training cost. With Qwen3-VL-4B on synthetic skills, it also stays close to OPSD on unusual orders, where methods that gain most by storing the order fall behind.
Figures & tables
| Method | Short | Conflict | ||
|---|---|---|---|---|
| OPSD | 51.1 | 0.0 | 98.3 | 0.0 |
| Subsequence OPSD | 56.0 | 4.9 | 95.6 | 2.7 |
| All-pair subsequence OPSD (same steps) | 61.1 | 10.3 | 91.9 | 6.5 |
| All-pair subsequence OPSD (5 steps) | 66.0 | 15.3 | 90.2 | 8.1 |
| Prior distillation | 69.1 | 17.9 | 83.8 | 14.5 |
| Short side | Full context | |||||||
| Method | Short | 0 frames | Prefix 2 | Stride 2 | Interval | Conflict | Long8 | Long32 |
| Evidence-only | ||||||||
| OPSD [ 57 ] | 51.1 2.6 | 45.8 | 57.7 | 49.9 | 98.8 | 98.3 0.6 | 97.2 | 80.3 |
| Shared-weight | ||||||||
| SFT on gold answers | 56.8 1.2 | 54.2 | 61.8 | 54.6 | 98.5 | 98.7 0.7 | 97.3 | 82.8 |
| SFT on teacher answers | 57.2 1.3 | 54.1 | 62.1 | 55.4 | 96.5 | 95.0 2.3 | 95.2 | 76.9 |
| Short-side gain over | |||||
| Setting | Reader | OPSD | Strongest | Conflict | Miss |
| Mind2Web | |||||
| Order | 4B | 13.3 | 10.4 | 1.7 | 3.6 |
| Order | 8B | 13.5 | 11.9 | 1.1 | 1.9 |
| HoloAssist | |||||
| Order | 4B | 29.0 | 3.9 | 3.0 | 3.1 |
| Variant | Short | Interval | Conflict |
|---|---|---|---|
| PriorSplit (switch) | 73.5 | 98.6 | 96.5 |
| Oracle switch | 74.0 | 98.8 | 98.3 |
| Reader only (OPSD) | 51.1 | 98.8 | 98.3 |
| Prior only | 71.3 | 71.3 | 1.5 |
| Reader plus prior | 54.7 | 98.7 | 97.7 |
| Reader ratio plus prior, checked | 73.4 | 98.3 | 94.8 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Check | Kind | Within |
| Theorem 1 | ||
| Conflict accuracy at the end of the band, all and informative cases | prediction | , |
| Theorem 2 (a) | ||
| Lemma S5 recomposes every accuracy and conflict loss, a check of the data | identity | , |
| Conflict loss of Lemma S5 with every quantity from development | prediction | |
| Theorem 2 (b) | ||
| Episodes | Test questions | |||||||
| Question | Skills | Train | Dev | Test | All | Short | Conflict | Usual |
| Synthetic skills | ||||||||
| Order | 9 | 1600 | 300 | 1400 | 3200 | 1800 | 173 | 74.1 |
| Mind2Web | ||||||||
| Order | 97 | 1600 | 300 | 1400 | 3200 | 1800 | 154 | 75.3 |
| Next step | 97 | 1600 | 500 | 700 | 1600 | 900 | 181 | 43.8 |
| Setting | Question A (named unit) | Question B (next step) |
|---|---|---|
| Synthetic skills | Skill: name . Is an image labelled word among the images above? | none |
| Mind2Web, CaptainCook4D, HoloAssist (tag banner) | Skill: name . Is a step tagged tag among the steps above? | Skill: name . Is the step tagged tag followed by another step among the steps above? |
| Tag-free Mind2Web | Skill: name . Is a screenshot of the step that performs ‘ text ’ among the images above? | Skill: name . Is the step that performs ‘ text ’ followed by another step among the images above? |
| Mind2Web | Qwen2.5-VL-7B | Qwen3-VL-8B | Idefics3-8B | InternVL3.5-8B | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Short | Interval | Conflict | Short | Interval | Conflict | Short | Interval | Conflict | Short | Interval | Conflict | Short | Interval | Conflict |
| OPSD [ 57 ] | 58.6 | 99.7 | 99.4 | 53.2 | 99.3 | 98.5 | 52.2 | 98.2 | 98.5 | 54.5 | 97.8 | 98.1 | 51.8 | 99.6 | 99.0 |
| Subsequence OPSD | 54.3 | 98.4 | 97.0 | 54.9 | 98.1 | 97.1 | 55.8 | 95.6 | 94.2 | 55.3 | 91.3 | 87.3 | 56.7 | 99.9 | 99.8 |
| NCA [ 47 ] | 55.1 | 94.3 | 90.9 | 54.5 | 92.3 | 87.1 | 56.0 | 85.6 | 75.9 | 56.6 | 80.3 | 70.1 | 55.1 | 95.4 | 91.1 |
| Prior distillation | 61.4 | 99.4 | 98.9 | 69.0 | 99.8 | 99.6 | 67.4 | 94.1 | 87.1 | 69.7 | 99.2 | 97.9 | |||
| All-pair subsequence OPSD (5 steps) | 61.0 | 98.8 | 97.4 | 64.5 | 98.6 | 96.1 | 64.4 | 97.0 | 93.1 | 61.9 | 91.4 | 81.5 | 65.4 | 99.9 | 99.8 |
| Method | Short | Interval | Conflict |
|---|---|---|---|
| OPSD | 29.1 | 72.0 | 67.0 |
| Subsequence OPSD | 26.1 | 63.4 | 59.3 |
| NCA | 24.9 | 59.2 | 54.1 |
| Prior distillation | 26.9 | 66.6 | 60.4 |
| PriorSplit | 50.4 | 71.1 | 65.4 |
| Variant | Short | 0 frames | Prefix 2 | Stride 2 | Interval | Conflict | Long8 | Long32 |
|---|---|---|---|---|---|---|---|---|
| PriorSplit (switch) | 73.5 | 71.3 | 78.5 | 70.6 | 98.6 | 96.5 | 97.4 | 83.0 |
| Oracle switch | 74.0 | 71.3 | 79.3 | 71.3 | 98.8 | 98.3 | 97.2 | 80.3 |
| Reader only (OPSD) | 51.1 | 45.8 | 57.7 | 49.9 | 98.8 | 98.3 | 97.2 | 80.3 |
| Prior only | 71.3 | 71.3 | 71.3 | 71.3 | 71.3 | 1.5 | 81.0 | 85.0 |
| Reader plus prior | 54.7 | 50.4 | 61.2 | 52.4 | 98.7 | 97.7 | 97.6 | 82.5 |
| Reader ratio plus prior, checked | 73.4 | 71.3 | 78.4 | 70.6 | 98.3 | 94.8 | 96.0 | 80.8 |
| Setting | Miss | False yes | Miss conflict | Short self | Short true | Interval | Conflict |
|---|---|---|---|---|---|---|---|
| Qwen3-VL-4B | 5.3 | 2.1 | 2.3 | 22.3 | 22.8 | 0.2 | 1.7 |
| Mind2Web | 3.6 | 0.5 | 2.2 | 13.3 | 13.8 | 0.6 | 1.7 |
| Qwen2.5-VL-7B | 2.3 | 0.1 | 2.3 | 20.8 | 20.7 | 0.7 | 2.3 |
| Qwen3-VL-8B | 2.6 | 3.6 | 1.3 | 20.9 | 21.7 | 0.1 | 1.2 |
| Idefics3-8B | 25.0 | 0.1 | 16.2 | 19.4 | 19.5 | 4.3 | 16.0 |
| InternVL3.5-8B | 0.0 | 0.1 | 0.0 | 22.2 | 22.2 | 0.0 | 0.0 |
| 200 steps | 5 steps | Counts in prompt | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Cell | Questions | OPSD | Subsequence OPSD | Prior distillation | Subsequence OPSD | Prior distillation | PriorSplit | OPSD reader | Base model | Oracle reference | Gain over OPSD |
| , | 200 | 51.2 | 51.8 | 53.3 | 50.0 | 50.5 | 78.0 | 78.5 | 77.0 | 78.5 | 26.8 [18.7, 34.7] |
| , | 200 | 98.2 | 91.3 | 89.2 | 91.7 | 90.2 | 98.2 | 97.7 | 80.0 | 100.0 | 0.0 [ 1.0, 1.0] |
| , | 100 | 79.0 | 72.3 | 62.7 | 72.3 | 60.0 | 79.3 | 81.0 | 76.0 | 100.0 | 0.3 [ 2.0, 2.7] |
| Method | Short | 0 frames | Interval | Conflict | Held-out, 0 frames |
|---|---|---|---|---|---|
| OPSD | 51.1 | 45.8 | 98.8 | 98.3 | 51.2 |
| Prior only (counts) | 71.3 | 71.3 | 71.3 | 1.5 | 78.0 |
| Learned prior only | 70.6 | 70.6 | 70.6 | 8.3 | 48.7 |
| PriorSplit | 73.5 | 71.3 | 98.6 | 96.5 | 78.0 |
| PriorSplit, learned prior | 72.8 | 70.6 | 98.6 | 96.5 | 48.7 |
| Learned minus counted prior | 0.7 [ 1.9, 0.5] | 0.7 [ 1.9, 0.5] | 0.7 [ 1.9, 0.5] | 6.7 [4.4, 9.2] | 29.3 [ 37.7, 20.8] |
| Method | Short | Interval | Conflict | Long8 | Long32 | Held-out, 0 frames |
|---|---|---|---|---|---|---|
| OPSD | 51.1 | 98.8 | 98.3 | 97.2 | 80.3 | 51.2 |
| Subsequence OPSD | 56.0 | 96.2 | 95.6 | 94.6 | 76.4 | 51.8 |
| Subsequence OPSD (5 steps) | 63.5 | 94.4 | 89.0 | 93.2 | 83.3 | 50.0 |
| Prior distillation | 69.1 | 92.7 | 83.8 | 91.3 | 75.4 | 53.3 |
| Prior distillation (5 steps) | 72.6 | 93.8 | 85.4 | 89.8 | 75.9 | 50.5 |
| PriorSplit | 73.5 | 98.6 | 96.5 | 97.4 | 83.0 | 78.0 |
| Method | Short | 0 frames | Prefix 2 | Stride 2 | Interval | Conflict |
|---|---|---|---|---|---|---|
| Held-out trajectories of seen websites (35 trajectories) | ||||||
| OPSD | 61.9 | 59.0 | 69.3 | 57.4 | 94.0 | 88.6 |
| Prior distillation | 59.2 | 58.2 | 62.9 | 56.3 | 92.8 | 87.0 |
| PriorSplit | 73.4 | 71.2 | 78.4 | 70.5 | 94.0 | 88.6 |
| Oracle switch | 73.8 | 71.2 | 79.2 | 71.2 | 94.0 | 88.6 |
| Held-out websites (25 trajectories) | ||||||
| Short accuracy | PriorSplit minus the strongest method | |||||
| Setting | Strongest method | Strongest | PriorSplit | Short | Interval | Conflict |
| Qwen3-VL-4B | Prior distillation | 69.1 | 73.5 | 4.4 [2.4, 6.4] | 5.9 [4.6, 7.3] | 12.7 [9.4, 16.2] |
| Qwen2.5-VL-7B | Prior distillation | 69.0 | 74.0 | 5.0 [3.2, 6.9] | 1.2 [ 2.1, 0.4] | 3.5 [ 6.2, 1.2] |
| Qwen3-VL-8B | Prior distillation | 67.4 | 73.1 | 5.7 [3.5, 7.9] | 4.2 [2.9, 5.6] | 10.2 [6.9, 13.7] |
| Idefics3-8B | PAD | 58.4 | 73.9 | 15.6 [11.6, 19.4] | 4.7 [ 5.9, 3.6] | 17.0 [ 19.8, 14.1] |
| InternVL3.5-8B | Prior distillation | 69.7 | 74.0 | 4.2 [2.5, 6.1] | 0.4 [ 0.1, 0.8] | 1.2 [0.0, 2.5] |
| Method | ||||||||||
| OPSD | 44.8 | 50.7 | 53.8 | 99.8 | 98.4 | 96.4 | 92.1 | 87.8 | 84.7 | 80.0 |
| Subsequence OPSD | 53.7 | 52.9 | 56.3 | 98.4 | 96.4 | 94.9 | 92.0 | 87.1 | 82.6 | 76.1 |
| Prior distillation | 69.4 | 69.2 | 69.9 | 96.1 | 93.3 | 89.8 | 84.9 | 80.1 | 78.3 | 75.4 |
| Subsequence OPSD (5 steps) | 61.1 | 61.2 | 62.2 | 95.7 | 94.1 | 93.1 | 91.0 | 88.2 | 81.9 | 79.2 |
| Prior distillation (5 steps) | 72.2 | 72.2 | 73.3 | 96.8 | 93.9 | 89.1 | 82.8 | 78.0 | 76.6 | 73.1 |
| PriorSplit | 73.7 | 72.4 | 72.7 | 99.7 | 98.7 | 96.1 | 94.2 | 89.3 | 87.0 | 82.7 |
| Development | Test | ||||
| Check | Miss | False yes | To prior | Within | Conflict |
| Tag-free Mind2Web order | |||||
| Own check | 11.5 | 29.9 | 0 | 81.7 | |
| Probe | 0.2 | 0.9 | 3 | 89.5 | |
| Oracle switch | 89.5 | ||||
| HoloAssist order, 4B | |||||
| Online | Departures | ||||
|---|---|---|---|---|---|
| Reader | PriorSplit | Threshold | Oracle | PriorSplit | Threshold |
| Qwen3-VL-4B | |||||
| Qwen3-VL-8B | |||||
Memory-Augmented VLMs
Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.
See all 173 papers