cs.CVNov 10, 2026

Looking One Step Ahead: An Anticipation-guided Agent for Object State Readout in Videos

Authors: Yuanhe Tian, Yan Song

Organizations: Zhongguancun Academy · University of Science and Technology of China

Abstract

Understanding the progress of an activity requires tracking how the objects involved change over time. Object state readout provides this information by describing a target object's states and transitions across video frames. Although multimodal large language models (MLLMs) generate detailed visual descriptions, these descriptions may leave the attributes needed to distinguish successive states insufficiently specified. In this paper, we propose look-ahead state readout (LASER), an anticipation-guided agent for object state readout that uses possible next-step changes to determine which attributes to inspect in the present. Given temporally ordered frames and a target object, LASER converts anticipated changes into targeted visual questions and inspects the current frame to establish attribute values. It then checks state readouts and forecasts against independent observations of subsequent frames, refining a state record used to generate the final description. Reusable textual observation skills guide this process and are refined by tracing discrepancies with reference captions to the responsible instructions. Candidate revisions are selected on separate videos based on improved caption quality without increased contradictions with reference facts. Model parameters remain frozen throughout adaptation and inference, and the revised skills remain fixed during evaluation. Experiment results and further analysis on OSCaR and PLM RCap show consistent improvements of LASER over strong baselines and existing approaches, which demonstrates the effectiveness of our approach.

Figures & tables

Explore similar work

CardsList
  1. Benchmarking Visual State Tracking in Multimodal Video Understanding

    Jun 2, 2026Sihyun Yu, Nanye Ma, Pinzhi Huang +8Multimodal Large Language ModelsVideo Reasoning

  2. Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes

    Aug 5, 2026Quynh Vo, Thong Nguyen, Vinh-Hien Do +2Cross-Modal RetrievalVideo Prediction

  3. Agentic Video Understanding: A Survey

    Sep 21, 2026Xinyu Deng, Siwen Luo, Daochang LiuTemporal Video UnderstandingMemory-Augmented Video Understanding