Tracking Task Progress Through Errors and Recovery:A Benchmark and Recurrent Distributional Model

Chen Wang; Xinze Niu; Zihao Zhu; Keru Zhou; Yicheng Xu; Shihao Jin; Wenhao Wang; Bowen Jiang

cs.ROOct 29, 2026

Tracking Task Progress Through Errors and Recovery:A Benchmark and Recurrent Distributional Model

Authors: Chen Wang, Xinze Niu, Zihao Zhu, Keru Zhou, Yicheng Xu, Shihao Jin, Wenhao Wang, Bowen Jiang,

Organizations: Maniformer · Shanghai Jiao Tong University · Tsinghua University · Zhejiang University

Abstract

Operational strategies for long-horizon robotic tasks require reliable evaluation to determine whether execution is progressing, stalling, regressing, or recovering from errors. Yet real- world robot datasets that capture error recovery and provide frame-level error annotations remain scarce. To address this gap, we introduce and open-source ReSTEP, a dataset of 3,156 episodes across two manipulation tasks with semantic stage labels and frame-level er- ror annotations. ReSTEP supports training and benchmarking progress monitors through successful execution, mistakes, and recovery. Existing progress estimators can struggle to distinguish visually similar task stages and maintain temporally consistent predictions while responding to genuine setbacks. We therefore propose REMORA, an online progress esti- mator that combines recurrent visual memory, distributional prediction, and causal peak selection. Memory retains execution history to help resolve visual ambiguity, while distribu- tional prediction represents alternative progress hypotheses and causal peak selection tracks a supported hypothesis over time. Together, these components enable consistent progress tracking that remains responsive to errors and recovery. Experiments on both ReSTEP tasks demonstrate that REMORA outperforms the state-of-the-art SARM baseline in progress ac- curacy and temporal consistency.

Figures & tables

Figure 1: Stage-aware progress supervision in a representative bottle-transfer episode. Head-camera observations are partitioned into four task stages, whose start frames are shown below the trajectory. The white curve denotes the normalized progress target and the cyan curve denotes the corresponding value estimate. Progress is anchored at $V=0$ at the episode start and $V=1$ upon task completion.

Figure 2: Overview of REMORA. Recurrent memory enriches multi-camera visual features before the VLA backbone and categorical value head predict a progress distribution. During training, stage and error annotations provide soft progress targets for distributional supervision. During online inference, progress tracking selects a supported peak using the previous estimate. Colors indicate frozen and trainable modules in the frozen-backbone configuration.

Figure 3: Gated recurrent visual memory. Each camera maintains eight memory slots, with independent states and shared update parameters. Memory queries current visual tokens to form a candidate state, and a frame-gap-conditioned gate updates the memory. Current visual tokens then attend to the updated memory for residual fusion. The updated state is carried to the next frame with stopped gradients during training and reset at stream boundaries.

Source	Episodes	Frames	Mistake episodes
D1 (expert)	1,953	793,632	18
D2 (expert)	959	657,159	66
D2 (HIL)	244	241,599	179
Total	3,156	1,692,390	263

Table 1: ReSTEP composition by task and collection source. Mistake episodes contain an available mistake annotation; all episodes end in successful task completion.

Figure 4: Temporal input configurations. A2 - A4 use four explicit multi-view observations sampled at $(t-3,t-2,t-1,t)$ , $(t-30,t-10,t-3,t)$ and $(t-90,t-30,t-10,t)$ , respectively. REMORA uses only the current observation and retains temporal context through eight gated recurrent memory tokens.

Figure 5: Representative progress estimation on a D2 trajectory. The black curve denotes the stage- and error-aware target, while the shaded region indicates an annotated error interval. REMORA better captures the target decrease and subsequent recovery than SARM, achieving a lower episode-level MAE.

Figure 6: Representative progress estimation on a D1 trajectory. The black curve denotes the target progress. The REMORA accurately tracks the nonzero initial target and subsequent progression, whereas A2 - A4 exhibit transient underestimation at the instruction onset. REMORA achieves the lowest episode-level MAE.

Dataset	Method	$A\downarrow$	$\rho_{\mathrm{S}}\uparrow$	$S\downarrow$	$J\downarrow$
D1	REMORA	1.100	0.999	4.181	4.188
	A1	1.150	0.998	4.821	4.826
	A2	1.490	0.986	6.478	6.483
	A3	1.203	0.996	5.456	5.460
	A4	1.137	0.997	4.871	4.876
	A5	1.087	0.999	3.915	3.920

Table 2: Ablations and comparison with SARM on D1 and D2. Best values before rounding are bold.

Robot Task Progress Estimation

See all 20 papers

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context
Sep 29, 2026Jianshu Zhang, Keliang Wu, Chengxuan Qian +6Robot Task Progress Estimation Process Reward Models
Northwestern · UCSB · UIUC +1
See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation
Mar 10, 2026Tingjun Dai, Mingfei Han, Tingwen Du +6Vision-Language-Action Models Language-Conditioned Robot Manipulation
School of Information Science and Technology, University of Science and Technology of China, Anhui, China. · University of Technology Sydney, Ultimo, NSW, Australia. · Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. +3
Event2Reward: Learning Robotic Rewards from Progress Relations
Oct 29, 2026Junnan Nie, Zitian Fan, Jiachen Zhang +14Robotic RL Reward Design for RL
Peking University · JD Explore Academy

Sep 29, 2026J/K move · Enter open · S save

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context

Jianshu Zhang, Keliang Wu, Chengxuan Qian, Xiyuan Yang+5

Northwestern · UCSB · UIUC+1

Robot Task Progress Estimation Process Reward Models Robot Manipulation Benchmarks VLMs for Robotics

Open paper AskPDF arXiv

Embodied agents now take on ever longer tasks. For long tasks, knowing only whether a task finally succeeds or fails says little; the steps along the way matter. Progress Reward Models (PRMs) score how far a task has come at every step, and serve as dense rewards, verifiers and monitors. Yet in long tasks the current frame alone often cannot tell how far the task has come, because progress depends on what happened before. We call this problem context-dependent progress estimation. Existing benchmarks on progress estimation mostly focus on short tasks whose progress can be read from the current observation, and whether PRMs can estimate progress when context is needed remains underexplored. We therefore build ContextProgress-Bench, with 24 manipulation tasks for 120 episodes. The benchmark covers three settings: (i) State Recall, where information needed for progress appeared earlier but is not in the current frame; (ii) Sequence Tracking, where steps follow a fixed order, so progress requires knowing which steps are done and which comes next; and (iii) Recurrence Disambiguation, where look-alike frames sit at very different progress. We then run a paired diagnosis: each PRM keeps the same input format in both runs, and in one run its instruction integrates the right context. Even PRMs that read the entire history get lost in estimating progress, yet with the right context the same five models cut their progress error by 77-82%. Embodied PRMs are thus not incapable of progress estimation, but lost without the right context. We therefore propose ProgressCompass, an autonomous agentic loop that reorients an existing PRM and uses current general-purpose VLMs to supply the context the PRM needs. Wrapped in the loop, the same frozen PRM cuts its progress error by 63% and raises its rank agreement by 76%. With such a compass, PRMs estimate progress far better on longer, more complex tasks.

Abstract

Robot Task Progress Estimation

Explore similar work

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context

See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation

Event2Reward: Learning Robotic Rewards from Progress Relations