Not All Actions Contribute Equally: Learning Precise Progress Estimation from Imprecise Demonstrations

Huanqian Wang; Boshi Tang; Botong Xiao; Jie Zhang; Yang Yue; Danhua Tao; Shiji Song; Gao Huang

cs.ROOct 16, 2026

Not All Actions Contribute Equally: Learning Precise Progress Estimation from Imprecise Demonstrations

Authors: Huanqian Wang, Boshi Tang, Botong Xiao, Jie Zhang, Yang Yue, Danhua Tao, Shiji Song, Gao Huang

Organizations: BNRist, Tsinghua University

Abstract

Progress estimators play an essential role in providing dense reward signals for reinforcement learning in robotic manipulation, but accurately assessing task progress can be much challenging. Existing methods heavily rely on expert data and naively assume that all actions within a demonstration contribute to task success. This assumption often breaks down in real-world scenarios, especially in high-precision manipulation, where subtle geometric errors can lead to task failure. Therefore, these methods can erroneously interpret human mistakes as meaningful progress, further corrupting progress learning. We argue that a key limitation is the lack of a global prior regarding task evaluation criteria, which is crucial for distinguishing meaningful progress from human mistakes. To this end, we propose PRIME, a novel progress estimator capable of learning precise progress estimation from imprecise demonstrations by aggregating experience across trajectories. PRIME constructs a weighted graph over observed states, connecting states across trajectories based on spatial proximity and visual consistency. Edge weights encode spatial distances as transition costs, and shortest-path search approximates the geodesic distance to success. Changes in this distance naturally distinguish advancement, stagnation, and setbacks without assuming monotonic progress over time. Extensive experiments in both simulation and real world on precise manipulation tasks show that PRIME is consistently effective across diverse tasks, and robust to data with varying degrees of suboptimality. Notably, the advantage of PRIME gets more pronounced in higher-precision scenarios, yielding up to a $5\times$ improvement in policy success rate over the SFT baselines on charger insertion.

Figures & tables

Fig. 1 : Overview of suboptimal behaviors and their impact on policy training. (Top) Demonstrations for precision manipulation inevitably contain suboptimal behaviors, even after trajectory-level filtering. In peg-insertion, for example, removing successful yet low-quality trajectories improves the task success rate only from 50% to 66.7%, whereas our step-level filtering achieves a success rate of 100%. (Bottom) During data collection, human operators unavoidably introduce suboptimal behaviors such as drifting (unintended lateral deviation), overshooting (motion beyond the target), and idling (unnecessary pauses or inactivity). While prior works assume that all actions contribute equally to the final success, they fail to capture the real-time progress in the face of these behaviors.

Fig. 2 : Quantitative success rate comparison and qualitative illustration of progress estimation. (Left) Success rates of different methods across three real-world tasks: peg-in-hole, peg-in-hole (Hard) and charger insertion. (Right) Qualitative illustration of progress estimation on Peg-Hard task. PRIME reflects partial advancements during an undershoot, maintains a constant estimation during idling, and appropriately penalizes the progress when a drift occurs. (Bottom) Correspondence between suboptimal behaviors and progress estimation of PRIME. Our method accurately identifies different types of suboptimal behavior, and assigns progress estimation accordingly.

Fig. 3 : An overview of PRIME. (a): Graph construction. Each demonstration timestep is represented by a graph node which comprises multi-view observations and the corresponding robot configuration. (b): Edge establishment. Cross-trajectory edges are established through multi-view visual consistency and configuration proximity, with weights reflecting pose and gripper-state differences. (c): Shortest-path distance as progress for RL. The graph $\mathcal{G}$ defines the remaining distance to task completion as the weighted shortest-path distance from the current node to the goal. This distance serves as a measure of task progress.

Task	Demo	Success Rate
		Behavior Cloning	Linear Temporal Progress				Non-uniform Progress
		$\pi_{0.5}$ -SFT	Linear	RECAP	ViVa	RoboMeter	SARM	GSR-based	ProcVLM	Ours
Drawer	Noisy	86.7%	80.0%	90.0%	90.0%	93.3%	90.0%	46.7%	90.0%	96.7%
Drawer	Expert	100.0%	90.0%	100.0%	100.0%	100.0%	100.0%	86.7%	100.0%	100.0%
Peg	Noisy	33.3%	16.7%	40.0%	43.3%	50.0%	53.3%	26.7%	46.7%	73.3%
Peg	Expert	86.7%	76.7%	93.3%	86.7%	90.0%	73.3%	73.3%	86.7%	100.0%

TABLE I : Main results from the real experiments. We evaluate the performance of PRIME across four robotic manipulation tasks of varying complexity: Drawer Opening, Peg-in-Hole (Standard and Hard), and Charger Insertion. These evaluations are conducted under diverse data quality conditions, encompassing noisy data that is unqualified for pure imitation learning, as well as expert data that is qualified for training but inevitably contains human mistakes. Experimental results demonstrate that PRIME consistently outperforms standard baselines across various levels of data quality. Besides, the gains on expert data suggest that local progress information remains useful even when overall demonstration quality is high.

Method	Stage-level Ordering $\uparrow$	Local Precision $\uparrow$	Progress Continuity $(s^{-2})\downarrow$
Linear	73.3%	70.0%	8.14
SARM	67.5%	67.5%	3.47
RoboMeter	70.8%	61.7%	4.72
ViVa	79.2%	40.8%	1.74
ProcVLM	87.5%	67.5%	5.50
RECAP	98.3%	68.3%	2.69

TABLE II : Results on the diagnostic benchmark.

Fig. 4 : Evaluation on various robotic trajectories. We compare our method against baselines across three scenarios. (1) Successful Execution: For a clean trajectory with minor pauses, PRIME provides a smooth, monotonic progress estimate. (2) Suboptimal Execution: When a human collector makes a mistake and lifts the cylinder upward to adjust, PRIME captures this setback with a local decrease in progress (orange band). (3) Failure Detection: When the cylinder is misplaced, PRIME reflects a sharp drop in progress (red band) as the trajectory drifts toward failure. For visualization, we normalize our distance predictions as $1-\frac{d(s_{t})}{d(s_{0})}$ . More visualization results are provided in Appendix -C .

Reward Model Type	Success Rate
Reward Model Type	In-distribution	Out-of-distribution
Sparse reward	$54.0{\scriptstyle\pm 2.3}$	$27.1{\scriptstyle\pm 1.4}$
Time Progress	$60.8{\scriptstyle\pm 6.1}$	$31.1{\scriptstyle\pm 1.5}$
Ours	$\mathbf{67.4}{\scriptstyle\pm 5.6}$	$\mathbf{35.3}{\scriptstyle\pm 3.1}$

TABLE III : Main results from the simulation. We report the average success rates across 3 seeds (0, 1 and 2).

Method	Peg-Hard	Drawer
PRIME w/o geometric edge costs	33.3%	90.0%
PRIME w/o configuration-proximity constraint	26.7%	53.3%
Euclidean distance to goal (no graph)	56.7%	33.3%
PRIME	70.0%	96.7%

TABLE IV : Ablation of graph connectivity and distance computation in PRIME.

Fig. 5 : Effect of temperature

\tau

(left) and reward clipping bound

r_{\max}

(right).

Panel of Table 10 — TABLE V : Success rate under different percentile choices for neighbor distance thresholds $\epsilon_{xy}$ and $\epsilon_{z}$

Fig. 6 : Convergence and computational efficiency. (Left) The shortest-path deviation

\Delta d

stabilizes after 4 episodes. (Right) Computation time of visual encoding and graph construction with increasing numbers of graph nodes. All runtime measurements are obtained on an NVIDIA RTX3090 GPU.

Fig. 7 : An overview of our real-world experimental setup.

Hyper-parameters
Action Prediction Horizon		50
Action Horizon		50
Observation Horizon		1
Gradient Steps		80000
Batch Size		16
$r_{\max}$	0.1	$\epsilon_{vis}$	0.1

TABLE VI : Training Details of PRIME.

Fig. 8 : Expert demonstrations with pauses in peg-in-hole (Hard) task.

Fig. 9 : Suboptimal behaviors in charger insertion task.

Fig. 10 : Suboptimal behaviors in charger insertion task.

Fig. 11 : Demonstrations from an untrained human, exhibiting severe boundary violations.

Fig. 12 : The cylinder is placed in an incorrect location.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 239 papers

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context
Sep 29, 2026Jianshu Zhang, Keliang Wu, Chengxuan Qian +6Process Reward Models Robot Manipulation Benchmarks
Northwestern · UCSB · UIUC +1
RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
Jun 20, 2026Pengzhi Yang, Xinyu Wang, Pengyu Jing +7Long-Horizon Robotic Manipulation Robot Skill Learning
NUS Human-Centered Robotic Lab · ‡ Booking.com · University of Cambridge +1
ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation
May 9, 2026Youhe Feng, Hansen Shi, Haoyang Li +7Long-Horizon Robotic Manipulation Multimodal Reward Modeling
School of Information, Renmin University of China · Work done during the internship at Zhipu AI. · Zhipu AI +1

Sep 29, 2026J/K move · Enter open · S save

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context

Jianshu Zhang, Keliang Wu, Chengxuan Qian, Xiyuan Yang+5

Northwestern · UCSB · UIUC+1

Process Reward Models Robot Manipulation Benchmarks VLMs for Robotics

Open paper AskPDF arXiv

Embodied agents now take on ever longer tasks. For long tasks, knowing only whether a task finally succeeds or fails says little; the steps along the way matter. Progress Reward Models (PRMs) score how far a task has come at every step, and serve as dense rewards, verifiers and monitors. Yet in long tasks the current frame alone often cannot tell how far the task has come, because progress depends on what happened before. We call this problem context-dependent progress estimation. Existing benchmarks on progress estimation mostly focus on short tasks whose progress can be read from the current observation, and whether PRMs can estimate progress when context is needed remains underexplored. We therefore build ContextProgress-Bench, with 24 manipulation tasks for 120 episodes. The benchmark covers three settings: (i) State Recall, where information needed for progress appeared earlier but is not in the current frame; (ii) Sequence Tracking, where steps follow a fixed order, so progress requires knowing which steps are done and which comes next; and (iii) Recurrence Disambiguation, where look-alike frames sit at very different progress. We then run a paired diagnosis: each PRM keeps the same input format in both runs, and in one run its instruction integrates the right context. Even PRMs that read the entire history get lost in estimating progress, yet with the right context the same five models cut their progress error by 77-82%. Embodied PRMs are thus not incapable of progress estimation, but lost without the right context. We therefore propose ProgressCompass, an autonomous agentic loop that reorients an existing PRM and uses current general-purpose VLMs to supply the context the PRM needs. Wrapped in the loop, the same frozen PRM cuts its progress error by 63% and raises its rank agreement by 76%. With such a compass, PRMs estimate progress far better on longer, more complex tasks.

Abstract

RL for Robotics

Explore similar work

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context

RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation

ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation