Vanguard: Knowing When to Trust the Draft in Speculative Vision-Language Agents
Organizations: UIUC · CMU · NYU · Northeastern
Abstract
Vision-language agents that browse websites, operate phones and act in homes reason before every action, so every step waits for a large model, a latency and cost that stand in the way of deployment. Step-level speculation lets a small model draft each step and calls the large model only on rejected drafts, but its outcome rests on the judge, and a judge that reads only a step's text cannot tell whether the step fits the screen it acts on. We find that a few samples of a small model can contain a correct step about as often as the large model gets the step right, and not always on the same steps: checked against the screen, small drafts can be not only cheaper but better. We present Vanguard, a training-free judge that leaves the draft and target models unchanged: a general decision model, with the screenshot in view, gives each candidate step a probability of being right in one forward pass; the best candidate is kept if it clears a threshold, and otherwise the large model writes the step. On web, mobile and embodied agents, a 2B draft judged by Vanguard is at least as accurate as its 8B target and significantly more accurate on Mind2Web and EB-ALFRED, while the target writes only a minority of the steps; it completes far more closed-loop episodes, and with a 27B target from another family it is both faster and more accurate. Vanguard is the most accurate judge we compare on the agent tasks; it relies on the screenshot on web and mobile tasks, its calibrated threshold transfers to unseen websites, and across model families its gain over the target grows with the draft's pass@4 relative to the target's accuracy. Vanguard composes with token-level speculative decoding and matches trained process reward models on mathematics, turning speculation from a way to save the large model's work into one that can improve on it.
Figures & tables
| test_task | test_website | |||
| Method | Success | 8B steps | Success | 8B steps |
| Draft model alone (2B) | 42.4 | 0 | 43.8 | 0 |
| Target model alone (8B) | 64.4 | 100 | 63.3 | 100 |
| Random of four | 45.0 | 0 | 42.6 | 0 |
| Vote gate | 56.1 | 33 | 56.9 | 37 |
| Qwen-VL-PRM-7B | 63.0 | 36 | 61.0 | 39 |
| Draft target | Task | Target | Vanguard |
| Qwen2.5-VL 3B 7B | Mind2Web | 50.5 | 58.6 |
| EB-ALFRED | 67.9 | 76.3 | |
| Qwen3-VL-2B | Mind2Web | 64.1 | 66.6 |
| Gemma-3-27B | AndroidControl | 34.1 | 45.9 |
| EB-ALFRED | 76.6 | 78.3 | |
| Gemma-3 12B 27B | Mind2Web | 64.1 | 69.1 |
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
| Judge | No fallback | With fallback |
| Draft log-probability | 29.7 | 33.1 (39) |
| Vote gate | 30.1 | 47.6 (93) |
| Qwen3-VL-8B, same prompt | 39.6 | 46.9 (42) |
| Vanguard | 44.4 | 51.0 (19) |
| Draft alone 32.4 Random of four 32.4 Target alone 48.8 | ||
| EB-ALFRED | GoToLocal | PickupLoc | ||||
| Method | Succ. | Target | Succ. | Target | Succ. | Target |
| Draft alone (2B) | 60.6 | 0 | 37 | 0 | 16 | 0 |
| Target alone (8B) | 74.2 | 100 | 61 | 100 | 36 | 100 |
| 8B, same prompt | 74.7 | 66 | 63 | 90 | 53 | 83 |
| Vanguard | 77.4 | 28 | 86 | 25 | 65 | 25 |
| Judge | Mind2Web | AndroidControl |
| Draft model alone (2B) | 43.2 | 32.4 |
| Target model alone (8B) | 63.8 | 48.8 |
| Qwen3-VL-2B, self-check | 55.0 | 41.7 |
| Qwen3-VL-8B, 0–9 grade | 62.8 | 44.2 |
| Qwen3-VL-8B, same prompt | 65.3 | 44.0 |
| Qwen3-VL-32B, same prompt | 66.6 | 49.0 |
| Selection | Gate | Mind2Web | AndroidControl | |
| 1 | – | 43.2 | 32.4 | |
| 4 | random | 43.7 | 32.4 | |
| 4 | ✓ | 57.4 | 44.4 | |
| 1 | – | ✓ | 58.5 | 41.1 |
| 4 | ✓ | random | 58.9 | 45.1 |
| 4 | ✓ | ✓ | 66.6 | 51.0 |
| Mind2Web | AndroidControl | |||
| Image given to the judge | Success | AUROC | Success | AUROC |
| No screenshot | 63.1 | 0.764 | 45.1 | 0.529 |
| Another task’s screenshot | 60.8 | 0.673 | 35.0 | 0.544 |
| This step’s screenshot | 66.6 | 0.807 | 51.0 | 0.714 |
| Judge | Expected calibration error |
| Qwen3-VL-8B, same prompt | 0.282 |
| Qwen-VL-PRM-7B | 0.272 |
| Vanguard | 0.106 |
| Method | M2W task | M2W web |
| Target model (8B) | 65.7 / 1.00 | 58.0 / 1.00 |
| 8B + EAGLE-3 | 67.0 / 1.55 | 59.7 / 1.61 |
| Vanguard | 71.3 / 1.17 | 64.0 / 1.15 |
| + EAGLE-3 on both | 70.0 / 1.50 | 62.0 / 1.52 |
| projected, at 0.16 s | 70.0 / 1.86 | 62.0 / 1.84 |
| Method | Element | Task |
| Target model alone (8B) | 74.1 | 18.0 |
| Vanguard | 78.4 | 19.9 |
| Configuration | Step success (%) | Target steps (%) | Wins/losses vs. 8B ( ) | |
| Draft model alone (2B) | – | 42.4 | 0 | 48/228, 0.001 |
| Target model alone (8B) | – | 64.4 | 100 | – |
| Four candidates, random | none | 45.0 | 0.0 | 60/219, 0.001 |
| Four candidates, vote gate | none | 47.2 | 0.0 | 58/199, 0.001 |
| Four candidates, vote gate | 0.75 | 56.1 | 32.6 | 37/105, 0.001 |
| Four candidates, log-probability | none | 43.2 | 0.0 | 46/220, 0.001 |
| Configuration | Step success (%) | Target steps (%) | Wins/losses vs. 8B ( ) | |
| Draft model alone (2B) | – | 43.8 | 0 | 73/263, 0.001 |
| Target model alone (8B) | – | 63.3 | 100 | – |
| Four candidates, random | none | 42.6 | 0.0 | 88/290, 0.001 |
| Four candidates, vote gate | none | 46.3 | 0.0 | 85/251, 0.001 |
| Four candidates, vote gate | 0.75 | 56.9 | 36.6 | 56/118, 0.001 |
| Four candidates, log-probability | none | 43.1 | 0.0 | 83/280, 0.001 |
| Configuration | Step success (%) | Target steps (%) | Speedup over 8B | Wins/losses vs. 8B ( ) | |
| Draft model alone (2B) | – | 32.4 | 0 | 2.16 | 22/108, 0.01 |
| Target model alone (8B) | – | 48.8 | 100 | 1.00 | – |
| Four candidates, random | none | 32.4 | 0 | 1.59 | 38/124, 0.01 |
| Four candidates, vote gate | none | 30.1 | 0 | 1.59 | 29/127, 0.01 |
| Four candidates, vote gate | 0.75 | 44.4 | 82 | 0.69 | 5/28, 0.01 |
| Four candidates, vote gate | 1.0 | 47.6 | 93 | 0.64 | 3/9, 0.15 |
| Qwen3-VL (2B, 8B) | Qwen2.5-VL (3B, 7B) | |||
| Method | Success | Target | Success | Target |
| Draft model alone | 60.6 | 0 | 69.7 | 0 |
| Target model alone | 74.2 | 100 | 67.9 | 100 |
| Oracle (pass@4) | 76.3 | 0 | 83.2 | 0 |
| Random of four | 61.8 | 0 | 62.5 | 0 |
| Vote gate | 72.7 | 39 | 71.3 | 27 |
| GoToLocal | PickupLoc | |||
| Method | Success | Steps | Success | Steps |
| Draft model alone (2B) | 37 | 33.6 | 16 | 40.1 |
| Target model alone (8B) | 61 | 22.7 | 36 | 32.0 |
| Target model, same prompt | 63 | 19.4 | 53 | 25.6 |
| Vanguard | 86 | 13.1 | 65 | 20.7 |
| Method | Success | Steps | 8B steps | s / step | s / episode | TFLOP / step |
| Draft model alone (2B) | 12.1 | 16.5 | 0% | 1.57 / 1.20 | 25.8 | 5.6 |
| Target model alone (8B) | 28.0 | 13.9 | 100% | 2.99 / 2.76 | 41.4 | 21.6 |
| Vanguard | 29.3 | 14.9 | 29% | 2.79 / 1.83 | 41.7 | 13.7 |
| Mind2Web | EB-ALFRED | |||
| Method | Success | Target | Success | Target |
| Qwen2.5-VL [ 1 ] , 3B draft, 7B target | ||||
| Draft alone | 39.6 | 0 | 69.7 | 0 |
| Target alone | 50.5 | 100 | 67.9 | 100 |
| Oracle (pass@4) | 59.4 | 0 | 83.2 | 0 |
| Target, same prompt | 53.2 | 43 | 70.3 | 78 |
| Target | Alone | Oracle (2B pass@4) | Vanguard | W/L, |
| Gemma-3-27B | ||||
| Mind2Web | 64.1 | 62.8 | 66.6 (24) | 169/123, 0.008 |
| AndroidControl | 34.1 | 52.8 | 45.9 (6) | 125/63, |
| EB-ALFRED | 76.6 | 76.3 | 78.3 (32) | 112/80, 0.025 |
| Gemma-3-12B | ||||
| Mind2Web test_task | 59.6 | 62.3 | 64.0 (26) | 78/42, 0.001 |
| Family | Benchmark | Vanguard | PRM | Target, same q. |
| Qwen3-VL | Mind2Web | 66.2 (31) | 61.2 | 65.8 |
| AndroidControl | 50.7 (28) | 30.9 | 45.9 | |
| EB-ALFRED | 77.4 (28) | 69.6 | 70.8 | |
| Qwen3.5 | Mind2Web | 66.5 (19) | 57.8 | 62.8 |
| AndroidControl | 43.8 (30) | 26.9 | 41.3 | |
| EB-ALFRED | 73.2 (40) | 63.7 | 70.5 |
| Time per step (s) | Mind2Web | AndroidControl |
| Four draft candidates, | 1.26 | 1.57 |
| Decision model, | 0.56 | 0.42 |
| Target step, | 2.98 | 2.50 |
| 1.82 | 1.99 | |
| 2.26 | 2.02 |
| Setting (target 8B unless stated) | Predicted | Measured |
| Mind2Web test_task | 1.21 | 1.17 |
| Mind2Web test_website | 1.19 | 1.15 |
| AndroidControl | 1.06 | 1.06 |
| Mind2Web test_task, EAGLE-3 on the 8B | 1.32 | 1.28 |
| Mind2Web test_task, EAGLE-3 on both | 1.54 | 1.50 |
| Mind2Web test_website, EAGLE-3 on both | 1.55 | 1.52 |
| Vanguard, | |||||
| Test set | Target | Seed 0 | Seed 1 | Seed 2 | Success (Target) |
| Mind2Web test_task | 64.4 | 66.5 | 66.3 | 66.6 | 68.3 (15) |
| Mind2Web test_website | 63.3 | 66.8 | 67.2 | 67.1 | 68.6 (18) |
| AndroidControl | 48.8 | 51.0 | 49.1 | 49.3 | 50.7 (12) |
| Input of | Success | vs. Vanguard | AUROC |
| Vanguard (screen, history, action) | 66.6 | – | 0.807 |
| No screenshot | 63.1 | 45/108 | 0.764 |
| Screenshot of another task | 60.8 | 57/161 | 0.673 |
| No previous actions | 64.6 | 17/54 | 0.766 |
| With the draft’s reasoning | 62.5 | 45/119 | 0.756 |
| Wording of the question | |||
| Mind2Web | AndroidControl | EB-ALFRED | |
| Steps accepted | 76 | 81 | 72 |
| Accepted: draft / target right | 71.6 / 67.9 | 51.3 / 48.5 | 86.5 / 82.1 |
| Rejected: draft / target right | 12.7 / 50.9 | 15.0 / 50.0 | 30.2 / 53.7 |
| Fallbacks | 432 | 100 | 514 |
| target fixes a wrong draft | 189 (71) | 40 (14) | 169 (67) |
| target replaces a right draft | 55 (248) | 15 (45) | 155 (364) |
| Steps written by the target model | ||
| Judge | 2% | 12% |
| VisualPRM-8B | 63.2 (2.5) | 63.4 (13.4) |
| Qwen3-VL-8B, same prompt | 64.5 (2.1) | 66.7 (12.0) |
| Qwen-VL-PRM-7B | 67.6 (2.5) | 69.5 (15.1) |
| Vanguard | 67.9 (1.5) | 69.4 (11.9) |
| Draft alone 58.5 Random of four 60.4 | ||
| ProcessBench | F1, held-out subsets |
| Qwen3-8B, same prompt | 49.9 |
| Qwen3-32B, same prompt | 61.0 |
| Qwen2.5-Math-PRM-7B | 63.0 |
| Vanguard | 66.6 |
| MATH-500 test, best of four | Accuracy |
| Qwen3-8B as judge | 79.0 |
| Judge | GSM8K | MATH | Olymp. | Omni | Held-out | |
| Qwen3-8B, same (correct) | 0.08 | 55.8 | 56.2 | 46.9 | 46.6 | 49.9 |
| Qwen3-8B, same (compute) | 0.51 | 56.2 | 54.2 | 40.8 | 35.8 | 43.6 |
| Qwen3-8B, same (error) | 0.98 | 52.4 | 37.1 | 36.4 | 37.5 | 37.0 |
| Qwen3-32B, same (correct) | 0.51 | 74.4 | 67.2 | 59.0 | 56.9 | 61.0 |
| Qwen3-32B, same (compute) | 0.57 | 73.9 | 64.0 | 54.0 | 53.5 | 57.2 |
| Qwen3-32B, same (error) | 0.05 | 71.8 | 52.9 | 34.1 | 29.7 | 38.9 |
| Method | Acc. | Median | Mean |
| 8B alone | 78 | 1.0 | 1.0 |
| Vanguard ( ) | 80 | 2.0 | 1.2 |
| Task (main-text item) | Vanguard against | W/L | ||
| Mind2Web test_task ( Tab. 1 ) | Qwen-VL-PRM-7B | 820 | 70/42 | 0.010 |
| Vote gate | 820 | 117/32 | ||
| Random of four | 820 | 201/25 | ||
| Draft model alone | 820 | 224/27 | ||
| Mind2Web test_website ( Tab. 1 ) | Qwen-VL-PRM-7B | 975 | 104/48 | |
| Vote gate | 975 | 133/37 |
Vision-Language Agents
Vision-language agents use visual and language models to perceive environments, reason about goals, and take actions through interactive interfaces.
See all 48 papers