cs.AINov 17, 2026

SpecLook: Speculating Where to Look for Fast and Reliable Agentic MLLMs

Authors: Xi Ding, Jiawei Zhang

Organizations: University of Wisconsin–Madison

Abstract

Agentic multimodal large language models (MLLMs) that think with images interleave reasoning with zoom-in tool calls. However, this sequential loop is slow, and its answers carry no reliability score. Prior speculation of tokens, reasoning steps, or answers keeps the loop, and answer speculation also accepts confident errors. We find that the loop mostly adds one zoom-in step, with which a small model is ahead of the agent on most splits. Our key insight is to speculate the agent's next observation, which, unlike a drafted answer, can be verified, corrected, and used to test the answer. We propose SpecLook, a training-free speculate-and-verify framework with three levels. A small model drafts a glimpse and re-answers from it, and the question ends if the answer is unchanged and confident. Otherwise, the agent's large model verifies and corrects the glimpse without tools, and their answers are fused. Only questions that still lack evidence reach the agent, in one forward pass that SpecLook-Fast skips, so no question runs the loop. The trace scores every answer's reliability without labels. Extensive experiments on V*, HR-Bench, and POPE show that both are more accurate than every baseline on every split. SpecLook reaches 87.2% and 86.2% mean accuracy with 7.7x and 4.7x speedups over the DeepEyes-7B and Thyme-RL agents, and SpecLook-Fast reaches 9.9x and 6.3x.

Figures & tables

Explore similar work

CardsList
  1. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

    Mar 24, 2026Haoyu Huang, Jinfa Huang, Zhongwei Wan +3Efficient Multimodal InferenceLLM Inference Acceleration

  2. SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search

    Jun 30, 2026Ming Dai, Zhihong Lu, Jinjie Gu +7Tool-Augmented Language Model AgentsAgentic Search

  3. MUSE: A Unified Agentic Harness for MLLMs

    Jun 2, 2026Jianglin Lu, Hailing Wang, Xu Ma +4Multimodal ReasoningMultimodal Agents