SpecLook: Speculating Where to Look for Fast and Reliable Agentic MLLMs
Organizations: University of Wisconsin–Madison
Abstract
Agentic multimodal large language models (MLLMs) that think with images interleave reasoning with zoom-in tool calls. However, this sequential loop is slow, and its answers carry no reliability score. Prior speculation of tokens, reasoning steps, or answers keeps the loop, and answer speculation also accepts confident errors. We find that the loop mostly adds one zoom-in step, with which a small model is ahead of the agent on most splits. Our key insight is to speculate the agent's next observation, which, unlike a drafted answer, can be verified, corrected, and used to test the answer. We propose SpecLook, a training-free speculate-and-verify framework with three levels. A small model drafts a glimpse and re-answers from it, and the question ends if the answer is unchanged and confident. Otherwise, the agent's large model verifies and corrects the glimpse without tools, and their answers are fused. Only questions that still lack evidence reach the agent, in one forward pass that SpecLook-Fast skips, so no question runs the loop. The trace scores every answer's reliability without labels. Extensive experiments on V*, HR-Bench, and POPE show that both are more accurate than every baseline on every split. SpecLook reaches 87.2% and 86.2% mean accuracy with 7.7x and 4.7x speedups over the DeepEyes-7B and Thyme-RL agents, and SpecLook-Fast reaches 9.9x and 6.3x.
Figures & tables
| V* | HR-Bench | POPE | All splits | ||||||||||||
| 4K | 8K | Rand. | Pop. | Adv. | |||||||||||
| Method | Acc. | Spd. | Acc. | Spd. | Acc. | Spd. | Acc. | Spd. | Acc. | Spd. | Acc. | Spd. | Acc. | s/q | Spd. |
| DeepEyes-7B as the expert | |||||||||||||||
| DeepEyes-7B agent | 89.0 | 1.0 | 74.6 | 1.0 | 70.8 | 1.0 | 92.1 | 1.0 | 85.5 | 1.0 | 81.8 | 1.0 | 82.3 | 9.92 | 1.0 |
| DeepEyes-7B w/o tools | 78.0 | 5.3 | 75.9 | 2.4 | 72.6 | 2.7 | 89.5 | 34.9 | 88.3 | 28.0 | 87.3 | 40.4 | 81.9 | 1.28 | 7.7 |
| Qwen3-VL-2B (draft only) | 80.6 | 6.7 | 71.5 | 3.5 | 67.8 | 2.4 | 92.1 | 37.2 | 89.1 | 30.2 | 87.6 | 36.3 | 81.5 | 1.20 | 8.3 |
| DeepEyes-7B | Thyme-RL | |||||||
| Variant | V* | HR | POPE | V* | HR | POPE | Err. | s/q |
| SpecLook | 93.2 | 79.3 | 90.9 | 90.6 | 77.7 | 90.5 | 234 | 1.78 |
| SpecLook-Fast | 92.1 | 78.5 | 90.7 | 90.6 | 76.3 | 90.4 | 240 | 1.48 |
| Speculate | ||||||||
| w/o glimpse | 81.7 | 71.5 | 89.6 | 82.2 | 70.2 | 89.2 | 700 | 0.22 |
| Route | ||||||||
| HR-Bench | POPE | All splits | ||||||
| Method | V* | 4K | 8K | Rand. | Pop. | Adv. | s/q | Speedup |
| DeepEyes-7B as the expert, batch size 1 | ||||||||
| ZoomEye | 86.9 | 73.0 | 72.5 | 87.3 | 86.5 | 85.8 | 10.03 | 0.6 |
| SpecEyes | 89.5 | 74.3 | 70.6 | 92.6 | 89.5 | 87.6 | 6.20 | 1.0 |
| SpecLook-Fast | 92.1 | 80.4 | 76.6 | 92.9 | 90.5 | 88.7 | 1.54 | 4.0 |
| SpecLook | 93.2 | 81.0 | 77.5 | 93.1 | 90.6 | 89.0 | 1.86 | 3.3 |
| Method | Batch | Acc. | s/q |
|---|---|---|---|
| DeepEyes-7B as the expert | |||
| SpecEyes | 1 | 89.5 | 6.67 |
| SpecEyes | 2 | 89.5 | 6.85 |
| SpecEyes | 4 | 89.5 | 5.25 |
| SpecEyes | 6 | 89.5 | 6.35 |
| SpecEyes | 8 | 89.5 | 6.70 |
| HR-Bench | POPE | ||||||
| Seconds per question | V* | 4K | 8K | Rand. | Pop. | Adv. | All |
| DeepEyes-7B as the expert | |||||||
| DeepEyes-7B agent | 11.64 | 15.84 | 20.54 | 9.06 | 7.31 | 8.86 | 9.92 |
| ZoomEye ‡ | 13.89 | 29.92 | 70.51 | 2.72 | 2.97 | 2.74 | 10.03 |
| SpecEyes | 6.35 | 11.07 | 19.16 | 3.25 | 4.10 | 4.23 | 5.57 |
| SpecLook-Fast | 2.14 | 2.37 | 3.42 | 0.64 | 0.56 | 0.72 | 1.00 |
| V* | HR-Bench | POPE | All questions | |||||
| Route | Share | Acc. | Share | Acc. | Share | Acc. | Share | Acc. |
| DeepEyes-7B as the expert | ||||||||
| Certified exit | 50.3 | 100.0 | 35.8 | 100.0 | 0.1 | 100.0 | 6.3 | 100.0 |
| Single-view exit | 0.0 | – | 1.7 | 77.8 | 43.3 | 94.0 | 36.4 | 93.9 |
| Escalated to the Overseer | 49.7 | 84.2 | 62.5 | 66.2 | 56.5 | 88.1 | 57.3 | 84.5 |
| Thyme-RL as the expert | ||||||||
| HR-Bench | POPE | All splits | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Scout | V* | 4K | 8K | Rand. | Pop. | Adv. | s/q | Speedup |
| DeepEyes-7B agent | – | 89.0 | 74.6 | 70.8 | 92.1 | 85.5 | 81.8 | 9.92 | 1.0 |
| SpecLook-Fast | Qwen3-VL-2B | 92.1 | 80.4 | 76.6 | 92.9 | 90.5 | 88.7 | 1.54 | 6.5 |
| SpecLook-Fast | Qwen3-VL-4B | 92.7 | 81.4 | 76.4 | 92.3 | 90.0 | 88.5 | 1.41 | 7.1 |
| Scout sees | Overseer sees | V* | HR-Bench | POPE |
|---|---|---|---|---|
| Accuracy (%) of the fused answer | ||||
| whole image | whole image | 80.9 | 76.5 | 86.0 |
| whole image | whole image + glimpse | 85.6 | 82.6 | 86.5 |
| glimpse | whole image + glimpse | 89.4 | 82.9 | 87.1 |
| glimpse | whole image | 90.4 | 84.1 | 87.4 |
| Error correlation between the two models | ||||
| V* | HR-Bench | POPE | ||||||
|---|---|---|---|---|---|---|---|---|
| Method | Attr. | Spatial | 4K | 8K | Rand. | Pop. | Adv. | s/q |
| SpecEyes | 91.3 | 86.8 | 74.4 | 70.5 | 92.7 | 89.5 | 87.7 | 5.57 |
| 7B zoom-refine (Overseer plans and reads its glimpse) | 92.2 | 85.5 | 76.3 | 67.8 | 90.1 | 88.4 | 87.3 | 1.26 |
| 2B zoom-refine (Scout plans and reads its glimpse) | 90.4 | 85.5 | 74.3 | 72.0 | 92.6 | 89.1 | 87.7 | 1.11 |
| SpecLook-Fast | 93.9 | 89.5 | 80.4 | 76.6 | 92.9 | 90.5 | 88.7 | 1.51 |
| DeepEyes-7B | Thyme-RL | ||||||||
| Method | V* | HR | POPE | Mean | V* | HR | POPE | Mean | Loop |
| Depth 0, confidence gate | 89.5 | 73.1 | 89.3 | 83.9 | 85.3 | 75.5 | 89.2 | 84.0 | 23.1 |
| Depth 1, confidence gate | 92.7 | 73.6 | 88.4 | 84.1 | 91.1 | 76.3 | 88.6 | 84.9 | 53.6 |
| Depth 1, agreement gate | 90.1 | 73.0 | 88.3 | 83.5 | 86.9 | 75.4 | 88.5 | 83.9 | 57.3 |
| SpecLook-Fast | 92.1 | 78.5 | 90.7 | 86.9 | 90.6 | 76.3 | 90.4 | 85.7 | 0.0 |
| SpecLook | 93.2 | 79.3 | 90.9 | 87.4 | 90.6 | 77.7 | 90.5 | 86.3 | 0.0 |
| HR-Bench | POPE | |||||||
| Method | V* | 4K | 8K | Rand. | Pop. | Adv. | Mean | Spd. |
| DeepEyes-7B as the expert | ||||||||
| SpecLook-Fast (one glimpse round) | 91.6 | 79.9 | 75.8 | 92.8 | 90.4 | 88.8 | 86.5 | 9.9 |
| + up to two more rounds | 91.6 | 78.6 | 75.3 | 92.8 | 90.4 | 88.8 | 86.2 | 8.3 |
| SpecLook | 92.7 | 81.0 | 76.8 | 93.1 | 90.6 | 89.0 | 87.2 | 7.7 |
| Thyme-RL as the expert | ||||||||
| Selective acc. at coverage | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Expert | Bench. | Score | AUROC | 95% CI | AURC | ECE | 50% | 70% | 90% |
| DeepEyes-7B | V* | Trace reliability (ours) | 93.4 | [88.9, 97.0] | 0.87 | 3.5 | 100.0 | 100.0 | 96.5 |
| Scout single-view conf. | 84.8 | [76.3, 91.3] | 1.70 | 15.8 | 99.0 | 97.8 | 94.2 | ||
| SpecEyes separability | 77.5 | [70.4, 83.8] | 3.24 | – | 97.9 | 93.3 | 91.3 | ||
| HR | Trace reliability (ours) | 87.1 | [83.1, 90.5] | 5.45 | 8.1 | 98.3 | 91.9 | 82.7 | |
| Scout single-view conf. | 77.1 | [71.9, 82.0] | 7.94 | 24.4 | 94.8 | 86.6 | 79.6 | ||
| Agent | HR-Bench | POPE | |||||||
| Method | loop | V* | 4K | 8K | Rand. | Pop. | Adv. | Mean | Spd. |
| DeepEyes-7B as the expert | |||||||||
| SpecLook-Fast | 0% | 91.6 | 79.9 | 75.8 | 92.8 | 90.4 | 88.8 | 86.5 | 9.9 |
| SpecLook | 0% | 92.7 | 81.0 | 76.8 | 93.1 | 90.6 | 89.0 | 87.2 | 7.7 |
| SpecLook-Fast + agent loop | 1% | 91.6 | 79.6 | 75.5 | 92.8 | 90.3 | 88.5 | 86.4 | 8.9 |
| SpecLook-Fast + agent loop | 5% | 92.7 | 78.4 | 75.3 | 93.0 | 90.0 | 88.2 | 86.2 | 5.8 |
| DeepEyes-7B | Thyme-RL | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Variant | V* | 4K | 8K | Mean | Spd. | V* | 4K | 8K | Mean | Spd. |
| SpecLook-Fast (Table 1) | 91.6 | 79.9 | 75.8 | 82.4 | 6.2 | 89.5 | 78.6 | 74.4 | 80.8 | 4.5 |
| SpecLook | 92.7 | 81.0 | 76.8 | 83.5 | 4.0 | 90.1 | 80.0 | 75.4 | 81.8 | 2.8 |
| w/o agent answer | 92.7 | 79.8 | 76.8 | 83.1 | 4.2 | 89.0 | 80.0 | 75.1 | 81.4 | 3.0 |
| w/o grounding | 91.6 | 81.0 | 76.4 | 83.0 | 4.5 | 89.0 | 79.9 | 74.4 | 81.1 | 3.0 |
| w/o grounding and agent answer | 90.6 | 80.3 | 76.3 | 82.4 | 4.8 | 88.0 | 79.8 | 74.5 | 80.7 | 3.3 |
| DeepEyes-7B | Thyme-RL | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Variant | Rand. | Pop. | Adv. | Mean | Spd. | Rand. | Pop. | Adv. | Mean | Spd. |
| SpecLook-Fast (Table 1) | 92.8 | 90.4 | 88.8 | 90.7 | 13.2 | 92.6 | 89.8 | 88.3 | 90.3 | 8.3 |
| SpecLook | 93.1 | 90.6 | 89.0 | 90.9 | 12.6 | 92.8 | 90.1 | 88.6 | 90.5 | 7.9 |
| w/o Scout union | 91.8 | 90.2 | 89.2 | 90.4 | 12.6 | 91.4 | 89.7 | 88.8 | 90.0 | 7.9 |
| w/o Overseer whole image | 93.3 | 90.2 | 88.1 | 90.5 | 12.6 | 93.1 | 89.9 | 87.9 | 90.3 | 7.9 |
| SpecLook-Fast + agent loop on searched | 93.5 | 89.4 | 86.8 | 89.9 | 5.7 | 93.0 | 89.6 | 87.1 | 89.9 | 4.6 |
| AURC (%) | DeepEyes-7B | Thyme-RL | ||||
|---|---|---|---|---|---|---|
| Method | V* | HR | POPE | V* | HR | POPE |
| SpecLook-Fast | 1.22 | 5.55 | 3.12 | 1.47 | 5.87 | 3.34 |
| SpecLook | 1.21 | 4.96 | 3.11 | 1.47 | 5.29 | 3.33 |
| DeepEyes-7B | Thyme-RL | |||||||
|---|---|---|---|---|---|---|---|---|
| V* | HR | POPE | All | V* | HR | POPE | All | |
| Searched questions | 40 | 478 | 908 | 1,426 | 44 | 500 | 921 | 1,465 |
| SpecLook-Fast | 70.0 | 51.9 | 64.0 | 60.1 | 65.9 | 50.6 | 60.6 | 57.3 |
| Agent loop | 75.0 | 50.6 | 56.4 | 55.0 | 65.9 | 57.0 | 57.3 | 57.5 |
| SpecLook | 75.0 | 55.4 | 66.4 | 63.0 | 68.2 | 54.4 | 63.1 | 60.3 |
| Fixed / broken | +3/ 1 | +36/ 19 | +59/ 37 | +98/ 57 | +3/ 2 | +27/ 8 | +62/ 39 | +92/ 49 |
| HR-Bench | POPE | |||||||
| Method | V* | 4K | 8K | Rand. | Pop. | Adv. | Mean | Spd. |
| DeepEyes-7B as the expert | ||||||||
| SpecLook-Fast | 91.6 | 79.9 | 75.8 | 92.8 | 90.4 | 88.8 | 86.5 | 9.9 |
| SpecLook | 92.7 | 81.0 | 76.8 | 93.1 | 90.6 | 89.0 | 87.2 | 7.7 |
| SpecLook + SenseNova-MARS-8B on searched | 92.7 | 84.0 | 80.1 | 93.1 | 90.6 | 89.0 | 88.2 | 4.7 |
| Thyme-RL as the expert | ||||||||
| Expert | Method | Compared with | Gain | Better | Worse | |
|---|---|---|---|---|---|---|
| DeepEyes-7B | SpecLook | DeepEyes-7B agent | +4.9 | 321 | 94 | |
| SpecLook | SpecEyes | +3.1 | 157 | 64 | ||
| SpecLook-Fast | DeepEyes-7B agent | +4.2 | 323 | 97 | ||
| SpecLook-Fast | SpecEyes | +2.5 | 150 | 74 | ||
| Thyme-RL | SpecLook | Thyme-RL agent | +3.7 | 312 | 115 | |
| SpecLook | SpecEyes | +1.8 | 170 | 99 |
Tool-Using Vision-Language Agents
Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.
See all 150 papers