Prior-as-Skill: Remember Outside and Read Inside with On-Policy Self-Distillation for Self-Evolving VLM Agents
Authors: Yan Lin, Jingyu Sun, Yifei Qian, Zhengtao Yao, Yuxia Wang, Daizong Liu, Chun Pong Lau, Zhongliang Guo, Stephen McGough
Organizations: Newcastle University · University of Manchester · University of Melbourne · University of Nottingham · University of Southern California · INSAIT, Sofia University “St. Kliment Ohridski” · Wuhan University · City University of Hong Kong · University of Aberdeen
Self-evolving visual agents keep what they learn as skills: key frames with step notes on a task's steps and their order. On-policy self-distillation (OPSD) is now a popular way to train such skills into a vision-language model that matches, on its own answers, a teacher that also reads the written order. At deployment, frames often go missing or show an unusual order, as when a person or a website reorders the steps. Existing internalization methods keep any order they store in the frame-reading weights: trained on full frames, the model guesses when they are missing, and trained with frames withheld, it stores the order, so memory can override what it sees. We propose PriorSplit, which fills in missing frames from an order kept outside the model, with no training beyond OPSD. It counts which of two neighboring steps comes first in training episodes, and the model checks if both asked steps are in view: if so, it answers from the frames, and otherwise the counts answer. Extensive experiments on synthetic skills with five models and on step-tagged order questions in new episodes of seen web, cooking and egocentric tasks demonstrate that, with frames missing, PriorSplit outperforms every compared method at OPSD's training cost. With Qwen3-VL-4B on synthetic skills, it also stays close to OPSD on unusual orders, where methods that gain most by storing the order fall behind.
Figures & tables
Figure 2: Two ways frames fail. PriorSplit lets the counts decide in (a) and the frames in (b). Steps X, Y and U. (a) Frames missing: an evidence-only reader guesses. (b) Frames contradict: a shared-weight reader follows its stored order (prior bleeding).
Figure 3: Few frames and long windows. PriorSplit stays within 1.3 points of the better of OPSD and prior only (the counts alone) at every m and is above both from m=12 . Accuracy (%) by shown frames m of a K -frame window, seed mean and range.
Method
Short
Δ
Conflict
Δ
OPSD
51.1
0.0
98.3
0.0
Subsequence OPSD
56.0
4.9
95.6
− 2.7
All-pair subsequence OPSD (same steps)
61.1
10.3
91.9
− 6.5
All-pair subsequence OPSD (5 × steps)
66.0
15.3
90.2
− 8.1
Prior distillation
69.1
17.9
83.8
− 14.5
Table 1: Shared-weight students. Each shared-weight student gains on the short side and loses on conflict items, more as it gains more. Accuracy (%), Qwen3-VL-4B, and Δ to OPSD on the same seeds and questions. Italic rows: questions neither all-pair variant trained on, where OPSD has 50.7 short and 98.4 conflict.
Figure 4: Where the order is stored. Prior distillation moves its frame-free scores to the canonical side, and 16.2% of its conflict scores cross zero against 1.7% for OPSD. Log-odds z on frame-free (light) and conflict items (solid), Qwen3-VL-4B.
Figure 5: PriorSplit. The counted prior answers when the check finds a named step out of view, and the reader answers otherwise. Offline: OPSD trains the reader E and privileged orders give pˉs . Test time: E (or the optional probe) checks the named steps and g picks E or pˉs .
Figure 6: Missing and contradicting real frames. PriorSplit is right in every row and the strongest compared method in none. Frames the reader saw, in order, with their banner words. Qwen3-VL-4B readers, seed 0. Rows 1 and 2 lack a frame of a named step, and row 3 shows an unusual order. Strongest: strongest compared method. Each row is picked to favor PriorSplit. Multimodal-Mind2Web [ 9 , 59 ] (OpenRAIL), HoloAssist [ 46 ] (CDLA-Permissive-2.0) and CaptainCook4D [ 27 ] (Apache 2.0) frames.
Short side
Full context
Method
Short
0 frames
Prefix 2
Stride 2
Interval
Conflict
Long8
Long32
Evidence-only
OPSD [ 57 ]
51.1 ± 2.6
45.8
57.7
49.9
98.8
98.3 ± 0.6
97.2
80.3
Shared-weight
SFT on gold answers
56.8 ± 1.2
54.2
61.8
54.6
98.5
98.7 ± 0.7
97.3
82.8
SFT on teacher answers
57.2 ± 1.3
54.1
62.1
55.4
96.5
95.0 ± 2.3
95.2
76.9
Table 2: Results on synthetic skills. PriorSplit leads every ranked method in each short-side column, Long8 and Long32. Accuracy (%) of Qwen3-VL-4B over three training seeds. Short pools 0, the first 2 (Prefix 2) and 2 strided (Stride 2) of six frames, Interval all six, Conflict its out-of-order items, Long8 and Long32 hold 8 and 32 frames. Bold, underline: top two per column. Subscripts: seed standard deviations. Italic rows are not ranked: all-pair rows score untrained questions, the oracle reference uses true visibility and presented order.
Short-side gain over
Setting
Reader
OPSD
Strongest
Conflict
Miss
Mind2Web
Order
4B
13.3
10.4
− 1.7
3.6
Order
8B
13.5
11.9
− 1.1
1.9
HoloAssist
Order
4B
29.0
3.9
− 3.0
3.1
Table 3: Results on real frames. PriorSplit gains short-side points over OPSD and the strongest compared method in every row. Qwen3-VL-4B and 8B readers, points averaged over seeds.
Figure 7: Short contexts on real frames. PriorSplit is highest on every dataset. Short-side accuracy (%), Qwen3-VL-4B.
Variant
Short
Interval
Conflict
PriorSplit (switch)
73.5
98.6
96.5
Oracle switch
74.0
98.8
98.3
Reader only (OPSD)
51.1
98.8
98.3
Prior only
71.3
71.3
1.5
Reader plus prior
54.7
98.7
97.7
Reader ratio plus prior, checked
73.4
98.3
94.8
Table 4: Ablations. PriorSplit (switch) leads every ranked variant on the short side. Accuracy (%) with Qwen3-VL-4B. Reader ratio plus prior, checked: where the check says yes, log prior plus the reader’s score with minus without frames. Italic: not ranked.
Figure 8: Check misses bound the conflict loss. On every seed, setting and reader, PriorSplit loses at most the miss share (conflict items handed to the prior) against OPSD on conflict items. Seed means per setting and reader. Dashed: loss equal to the miss share.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Check
Kind
Within
Theorem 1
Conflict accuracy at the ρ=0 end of the band, all and informative cases
prediction
63/72 , 31/34
Theorem 2 (a)
Lemma S5 recomposes every accuracy and conflict loss, a check of the data
identity
16/16 , 9/9
Conflict loss of Lemma S5 with every quantity from development
prediction
5/5
Theorem 2 (b)
Appendix
Table S1 : Numerical checks. Route identities, conflict loss and prior predictions hold in every listed case. Selected checks on seen skills and held-out test splits: accuracy band, route identities, conflict loss predictions, episode and library priors. Within: cases meeting each criterion.
Figure S1 : Where the order is stored. Only 3.5% of PriorSplit’s conflict scores cross zero, against 16.2% for prior distillation. Log-odds z on frame-free ( ZF , light) and conflict items ( ZC , solid), Qwen3-VL-4B.
Episodes
Test questions
Question
Skills
Train
Dev
Test
All
Short
Conflict
Usual
Synthetic skills
Order
9
1600
300
1400
3200
1800
173
74.1
Mind2Web
Order
97
1600
300
1400
3200
1800
154
75.3
Next step
97
1600
500
700
1600
900
181
43.8
Appendix
Table S2 : Data per setting. The usual answer is gold on most order questions. Episodes: split sizes, sampled windows on synthetic skills and Mind2Web. Conflict: full-context departures in view. Usual: canonical or most counted gold (%). Blank: undefined.
Figure S2 : Construction of the synthetic skills. One stored test question needs memory under Prefix 2 and Stride 2 and is answered by the frames under Interval. Amber: canonical order, teal: presented frames, coral: conflict pair, dashed: unseen, outlined: named.
Setting
Question A (named unit)
Question B (next step)
Synthetic skills
Skill: name . Is an image labelled word among the images above?
none
Mind2Web, CaptainCook4D, HoloAssist (tag banner)
Skill: name . Is a step tagged tag among the steps above?
Skill: name . Is the step tagged tag followed by another step among the steps above?
Tag-free Mind2Web
Skill: name . Is a screenshot of the step that performs ‘ text ’ among the images above?
Skill: name . Is the step that performs ‘ text ’ followed by another step among the images above?
Appendix
Table S3 : Check wordings. The reader itself answers them, and no label trains the check. Asked once per named unit (A) or next-step question (B). Every prompt ends with “Answer yes or no.” Italic words are filled per question.
Mind2Web
Qwen2.5-VL-7B
Qwen3-VL-8B
Idefics3-8B
InternVL3.5-8B
Method
Short
Interval
Conflict
Short
Interval
Conflict
Short
Interval
Conflict
Short
Interval
Conflict
Short
Interval
Conflict
OPSD [ 57 ]
58.6
99.7
99.4
53.2
99.3
98.5
52.2
98.2
98.5
54.5
97.8
98.1
51.8
99.6
99.0
Subsequence OPSD
54.3
98.4
97.0
54.9
98.1
97.1
55.8
95.6
94.2
55.3
91.3
87.3
56.7
99.9
99.8
NCA [ 47 ]
55.1
94.3
90.9
54.5
92.3
87.1
56.0
85.6
75.9
56.6
80.3
70.1
55.1
95.4
91.1
Prior distillation
61.4
99.4
98.9
69.0
99.8
99.6
67.4
94.1
87.1
69.7
99.2
97.9
All-pair subsequence OPSD (5 × steps)
61.0
98.8
97.4
64.5
98.6
96.1
64.4
97.0
93.1
61.9
91.4
81.5
65.4
99.9
99.8
Appendix
Table S4 : Generality. PriorSplit leads on Short throughout. Seen Mind2Web websites (order, Qwen3-VL-4B) and synthetic skills, seed means (%). Bold, underline: best, second best per column and group. Shaded: PriorSplit. Italic: not ranked, all-pair rows on untrained questions only.
Method
Short
Interval
Conflict
OPSD
29.1
72.0
67.0
Subsequence OPSD
26.1
63.4
59.3
NCA
24.9
59.2
54.1
Prior distillation
26.9
66.6
60.4
PriorSplit
50.4
71.1
65.4
Appendix
Table S5 : Next step. PriorSplit leads on Short. Next-step questions on seen Mind2Web websites (which of four actions comes next), accuracy (%) over three training seeds.
Figure S3 : Short versus conflict accuracy. PriorSplit is the only ranked method above the frontier. Seen synthetic skills, seed means, arrows from OPSD. Off the axis: context distillation (42.6, 59.9) and counts in prompt with the base model (9.1, 70.0).
Variant
Short
0 frames
Prefix 2
Stride 2
Interval
Conflict
Long8
Long32
PriorSplit (switch)
73.5
71.3
78.5
70.6
98.6
96.5
97.4
83.0
Oracle switch
74.0
71.3
79.3
71.3
98.8
98.3
97.2
80.3
Reader only (OPSD)
51.1
45.8
57.7
49.9
98.8
98.3
97.2
80.3
Prior only
71.3
71.3
71.3
71.3
71.3
1.5
81.0
85.0
Reader plus prior
54.7
50.4
61.2
52.4
98.7
97.7
97.6
82.5
Reader ratio plus prior, checked
73.4
71.3
78.4
70.6
98.3
94.8
96.0
80.8
Appendix
Table S6 : Ablation. On Short the switch beats both parts and every unchecked combination. Seen synthetic skills, main model, three training seeds. Oracle switch: true visibility. Reader plus prior, checked, uses option scores. Reader ratio plus prior: every check set to yes.
Setting
Miss
False yes
Miss conflict
Short self
Short true
Interval
Conflict
Qwen3-VL-4B
5.3
2.1
2.3
22.3
22.8
− 0.2
− 1.7
Mind2Web
3.6
0.5
2.2
13.3
13.8
− 0.6
− 1.7
Qwen2.5-VL-7B
2.3
0.1
2.3
20.8
20.7
− 0.7
− 2.3
Qwen3-VL-8B
2.6
3.6
1.3
20.9
21.7
0.1
− 1.2
Idefics3-8B
25.0
0.1
16.2
19.4
19.5
− 4.3
− 16.0
InternVL3.5-8B
0.0
0.1
0.0
22.2
22.2
0.0
0.0
Appendix
Table S7 : Own check. PriorSplit gains on Short throughout. Seen skills and websites, seed means. PriorSplit minus OPSD (points), self: own check, true: true visibility (0.0 on Interval and Conflict, not shown). Miss conflict: conflict items handed to the prior (%).
200 steps
5 × steps
Counts in prompt
Cell
Questions
OPSD
Subsequence OPSD
Prior distillation
Subsequence OPSD
Prior distillation
PriorSplit
OPSD reader
Base model
Oracle reference
Gain over OPSD
m=0 , δ=0.3
200
51.2
51.8
53.3
50.0
50.5
78.0
78.5
77.0
78.5
26.8 [18.7, 34.7]
m=6 , δ=0.3
200
98.2
91.3
89.2
91.7
90.2
98.2
97.7
80.0
100.0
0.0 [ − 1.0, 1.0]
m=32 , δ=0.3
100
79.0
72.3
62.7
72.3
60.0
79.3
81.0
76.0
100.0
0.3 [ − 2.0, 2.7]
Appendix
Table S8 : Held-out skills. PriorSplit leads OPSD without frames and matches it with frames. Unranked controls, main model, seed mean accuracy (%). PriorSplit and the counted prior alone (78.0% in every cell) count those skills’ episodes. Gain: points [95% interval].
Method
Short
0 frames
Interval
Conflict
Held-out, 0 frames
OPSD
51.1
45.8
98.8
98.3
51.2
Prior only (counts)
71.3
71.3
71.3
1.5
78.0
Learned prior only
70.6
70.6
70.6
8.3
48.7
PriorSplit
73.5
71.3
98.6
96.5
78.0
PriorSplit, learned prior
72.8
70.6
98.6
96.5
48.7
Learned minus counted prior
− 0.7 [ − 1.9, 0.5]
− 0.7 [ − 1.9, 0.5]
− 0.7 [ − 1.9, 0.5]
6.7 [4.4, 9.2]
− 29.3 [ − 37.7, − 20.8]
Appendix
Table S9 : Learned prior. Within PriorSplit, counts match a learned prior on seen skills and beat it on held-out skills. Unranked controls, main model. Learned prior: adapter fitted to training counts, queried without frames. Seed means and differences [95% interval].
Method
Short
Interval
Conflict
Long8
Long32
Held-out, 0 frames
OPSD
51.1
98.8
98.3
97.2
80.3
51.2
Subsequence OPSD
56.0
96.2
95.6
94.6
76.4
51.8
Subsequence OPSD (5 × steps)
63.5
94.4
89.0
93.2
83.3
50.0
Prior distillation
69.1
92.7
83.8
91.3
75.4
53.3
Prior distillation (5 × steps)
72.6
93.8
85.4
89.8
75.9
50.5
PriorSplit
73.5
98.6
96.5
97.4
83.0
78.0
Appendix
Table S10 : Training budget. PriorSplit leads both 5 × rows on Interval and Conflict. Unranked controls, main model. Steps: 999 to 1007 versus 200. Prior distillation draws: 7992 to 8056 versus 1600, about 4.0 per training episode. Seed means and differences [95% interval].
Method
Short
0 frames
Prefix 2
Stride 2
Interval
Conflict
Held-out trajectories of seen websites (35 trajectories)
OPSD
61.9
59.0
69.3
57.4
94.0
88.6
Prior distillation
59.2
58.2
62.9
56.3
92.8
87.0
PriorSplit
73.4
71.2
78.4
70.5
94.0
88.6
Oracle switch
73.8
71.2
79.2
71.2
94.0
88.6
Held-out websites (25 trajectories)
Appendix
Table S11 : Held-out Mind2Web skills without tags. PriorSplit with the probe leads on the short side and matches OPSD with all frames. Seed mean accuracy (%), three training seeds, 550 questions per cell. Italic: oracle switch, true visibility.
Figure S4 : Real examples. In each row PriorSplit follows frames that contradict the usual order. Selected test questions on seen skills, one per dataset, picked to favor PriorSplit. Named steps outlined, +n counts omitted frames. Qwen3-VL-4B readers, seed 0 .
Figure S5 : The other situation of each dataset. PriorSplit is right in every row and the strongest compared method (Strongest) in none. Rows 1 and 2 show an unusual order, row 3 a missing frame. Selected examples, Qwen3-VL-4B, seed 0 .
Short accuracy
PriorSplit minus the strongest method
Setting
Strongest method
Strongest
PriorSplit
Short
Interval
Conflict
Qwen3-VL-4B
Prior distillation
69.1
73.5
4.4 [2.4, 6.4]
5.9 [4.6, 7.3]
12.7 [9.4, 16.2]
Qwen2.5-VL-7B
Prior distillation
69.0
74.0
5.0 [3.2, 6.9]
− 1.2 [ − 2.1, − 0.4]
− 3.5 [ − 6.2, − 1.2]
Qwen3-VL-8B
Prior distillation
67.4
73.1
5.7 [3.5, 7.9]
4.2 [2.9, 5.6]
10.2 [6.9, 13.7]
Idefics3-8B
PAD
58.4
73.9
15.6 [11.6, 19.4]
− 4.7 [ − 5.9, − 3.6]
− 17.0 [ − 19.8, − 14.1]
InternVL3.5-8B
Prior distillation
69.7
74.0
4.2 [2.5, 6.1]
0.4 [ − 0.1, 0.8]
1.2 [0.0, 2.5]
Appendix
Table S12 : Strongest method. PriorSplit leads on Short throughout the first two groups. Strongest ranked method by Short, seen skills and real test splits, points [95% interval]. Blank: unpaired. Italic: HoloAssist next step. (both): training on both question types.
Method
m=0
m=1
m=2
m=4
m=6
m=8
m=12
m=16
m=24
m=32
OPSD
44.8
50.7
53.8
99.8
98.4
96.4
92.1
87.8
84.7
80.0
Subsequence OPSD
53.7
52.9
56.3
98.4
96.4
94.9
92.0
87.1
82.6
76.1
Prior distillation
69.4
69.2
69.9
96.1
93.3
89.8
84.9
80.1
78.3
75.4
Subsequence OPSD (5 × steps)
61.1
61.2
62.2
95.7
94.1
93.1
91.0
88.2
81.9
79.2
Prior distillation (5 × steps)
72.2
72.2
73.3
96.8
93.9
89.1
82.8
78.0
76.6
73.1
PriorSplit
73.7
72.4
72.7
99.7
98.7
96.1
94.2
89.3
87.0
82.7
Appendix
Table S13 : Shown frames. PriorSplit leads OPSD up to two frames and every trained row from twelve. Seed mean accuracy (%) by shown frames m , seen skills, main model. Window: six frames for m≤2 , m frames for m≥4 . Differences in points.
Development
Test
Check
Miss
False yes
To prior
Within
Conflict
Tag-free Mind2Web order
Own check
11.5
29.9
0
81.7
Probe
0.2
0.9
3
89.5
Oracle switch
89.5
HoloAssist order, 4B
Appendix
Table S14 : Fitted probes. They meet every reported bound on all seeds and lift conflict accuracy. Qwen3-VL-4B readers, own checks exceeding development bounds. Rates: seed maxima. Within: seeds meeting reported bounds. Blank To prior: only miss and false yes checked.
Online
Departures
Reader
PriorSplit
Threshold
Oracle
PriorSplit
Threshold
Qwen3-VL-4B
56.5
72.1
73.7
97.6
83.2
Qwen3-VL-8B
66.7
73.1
73.7
98.6
97.3
Appendix
Table S15 : Threshold. PriorSplit leads on departures with both readers. HoloAssist next step, seed mean accuracy (%). Threshold: per reader and question, fitted on development checks (PriorSplit: 0 ). Online: next step not yet shown. Departures: next step not among the most counted.