Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress

Yuelin Zhang; Sijie Cheng; Chen Li; Zongzhao Li; Yuxin Huang; Yang Liu; Wenbing Huang

cs.CVMar 18, 2026

Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress

Authors: Yuelin Zhang, Sijie Cheng, Chen Li, Zongzhao Li, Yuxin Huang, Yang Liu, Wenbing Huang

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE · Department of Computer Science and Technology, Tsinghua University ·

Abstract

Accurately estimating task progress is critical for embodied agents to plan and execute long-horizon, multi-step tasks. Despite promising advances, existing Vision-Language Models (VLMs) based methods primarily leverage their video understanding capabilities, while neglecting their complex reasoning potential. Furthermore, processing long video trajectories with VLMs is computationally prohibitive for real-world deployment. To address these challenges, we propose the Recurrent Reasoning Vision-Language Model ( $\text{R}^2$ VLM). Our model features a recurrent reasoning framework that processes local video snippets iteratively, maintaining a global context through an evolving Chain of Thought (CoT). This CoT explicitly records task decomposition, key steps, and their completion status, enabling the model to reason about complex temporal dependencies. This design avoids the high cost of processing long videos while preserving essential reasoning capabilities. We train $\text{R}^2$ VLM on large-scale, automatically generated datasets from ALFRED and Ego4D. Extensive experiments on progress estimation and downstream applications, including progress-enhanced policy learning, reward modeling for reinforcement learning, and proactive assistance, demonstrate that $\text{R}^2$ VLM achieves strong performance and generalization, achieving a new state-of-the-art in long-horizon task progress estimation. The models and benchmarks are publicly available at huggingface.

Figures & tables

Figure 1: $\text{R}^{2}$ VLM performs multi-turn recurrent reasoning. It takes a video snippet and the CoT from the previous iteration as input, and outputs the updated CoT and a progress estimation.

Figure 2: Framework of $\text{R}^{2}$ VLM. It performs recurrent reasoning over the newly observed video snippet, while incorporating the historical CoT as context. The CoT records the completed, pending, and incomplete steps from earlier video snippets and is dynamically updated throughout the recurrent reasoning process. The model estimates task progress based on the proportion of completed steps, and supports diverse applications including progress estimation, reward modeling, and proactive assistance.

Figure 3: An illustration of the CoT structure and its update process in recurrent reasoning.

Model	Size	Alfred				Ego4D
Model	Size	$p_{\text{mae}}\downarrow$	$\Delta p_{\text{mae}}\downarrow$	$bin\uparrow$	$acc\uparrow$	$p_{\text{mae}}\downarrow$	$\Delta p_{\text{mae}}\downarrow$	$bin\uparrow$	$acc\uparrow$
API-Based Models
GPT-5 [ 18 ]	-	18.35	15.48	0.505	0.901	25.04	15.89	0.259	0.749
Gemini-2.5-Pro [ 5 ]	-	16.27	17.37	0.481	0.830	28.22	16.71	0.217	0.713
Qwen3-VL-Plus [ 25 ]	-	27.30	15.29	0.379	0.777	28.20	10.14	0.251	0.652
Open-Source Models

Table 1: We evaluated our three model variants and all baselines on the Alfred and Ego4D benchmarks.

\text{R}^{2}

VLM outperforms every baseline on all four metrics across both datasets, reaching state-of-the-art performance.

Figure 4: Comparison of MAE and inference time between full and incremental video inputs at different video durations.

Figure 5: The comparison of the policy model’s performance on a three-step task before and after online reinforcement learning. The rewards in the reinforcement learning process are provided by $\text{R}^{2}$ VLM.

Figure 6: Demonstration of $\text{R}^{2}$ VLM as a proactive assistant. The monitoring task is “cutting a carrot on a chopping board”.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: We visualize the model’s progress estimation on four tasks, where key steps trigger sharp progress increases, and irrelevant steps yield zero progress increments.

Figure 8: A demo of $\text{R}^{2}$ VLM dynamically revising the CoT.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 627 papers

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding
Sep 23, 2026Xiaoda Yang, Can Wang, Yuxiang Liu +3Temporal Reasoning in VLMs Spatiotemporal Reasoning
Zhejiang University, Hangzhou, China · Qingdao University, Qingdao, China · Tianjin University, Tianjin, China +1
Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models
Mar 18, 2026Yuhao Dong, Zuyan Liu, Shulin Tian +2VLM Reasoning Multimodal Large Language Models
S-Lab, Nanyang Technological University, Tencent Hunyuan, and Tsinghua University
Recursive Belief Vision Language Action Models
Feb 24, 2026Vaidehi Bagaria, Bijo Sebastian, Nirav Kumar PatelVision-Language-Action Models Long-Horizon Robotic Manipulation
Department of Engineering Design, Indian Institute of Technology, Madras, Chennai, India

Sep 23, 2026J/K move · Enter open · S save

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

Xiaoda Yang, Can Wang, Yuxiang Liu, Pengfei Zhou+2

Zhejiang University, Hangzhou, China · Qingdao University, Qingdao, China · Tianjin University, Tianjin, China+1

Temporal Reasoning in VLMs Spatiotemporal Reasoning Visual Spatial Reasoning VLMs for Robotics VLM Robustness

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have advanced rapidly in static visual understanding, yet remain unreliable when judging how an egocentric task is progressing. Given a task instruction and two visual observations, a model should determine which state is closer to the goal by analyzing task-relevant object configurations and spatial relations, rather than relying on timestamps or presentation order. This distinction is critical in manipulation, where retries, corrective actions, and temporary regressions make progress inherently non-monotonic. We introduce EgoTSR, a unified framework for diagnosing and improving order-robust task-progress understanding. First, SpatialLogic-Bench evaluates each physical state pair in both original and order-swapped presentations across short- and long-horizon settings, exposing whether a model follows task-state evidence or chronological shortcuts. Second, our data construction pipeline converts successful, approximately monotonic manipulation and first-person trajectories into bidirectional supervision; LongTag further preserves intermediate subtask structure for long-horizon comparison, while failure-aware data extend learning to regressions and recoveries. Third, a progressive CoT-to-Tag curriculum first supervises evidence-grounded interpretation of task-relevant state changes and then consolidates the comparison rule through scalable label-only training. Experiments reveal substantial input-order bias in representative VLMs. EgoTSR achieves 92.4% long-horizon accuracy with a 0.1-point forward-inverse Gap. Failure-aware supervision further improves accuracy on non-monotonic trajectories by 11.8 points and Recovery Accuracy by 11.2 points, while maintaining broad visual and spatial capabilities. These results establish goal-conditioned state comparison as an explicit formulation of egocentric spatiotemporal reasoning for task-progress understanding.

Abstract

VLMs for Robotics

Explore similar work

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models

Recursive Belief Vision Language Action Models