Progress estimators play an essential role in providing dense reward signals for reinforcement learning in robotic manipulation, but accurately assessing task progress can be much challenging. Existing methods heavily rely on expert data and naively assume that all actions within a demonstration contribute to task success. This assumption often breaks down in real-world scenarios, especially in high-precision manipulation, where subtle geometric errors can lead to task failure. Therefore, these methods can erroneously interpret human mistakes as meaningful progress, further corrupting progress learning. We argue that a key limitation is the lack of a global prior regarding task evaluation criteria, which is crucial for distinguishing meaningful progress from human mistakes. To this end, we propose PRIME, a novel progress estimator capable of learning precise progress estimation from imprecise demonstrations by aggregating experience across trajectories. PRIME constructs a weighted graph over observed states, connecting states across trajectories based on spatial proximity and visual consistency. Edge weights encode spatial distances as transition costs, and shortest-path search approximates the geodesic distance to success. Changes in this distance naturally distinguish advancement, stagnation, and setbacks without assuming monotonic progress over time. Extensive experiments in both simulation and real world on precise manipulation tasks show that PRIME is consistently effective across diverse tasks, and robust to data with varying degrees of suboptimality. Notably, the advantage of PRIME gets more pronounced in higher-precision scenarios, yielding up to a 5× improvement in policy success rate over the SFT baselines on charger insertion.
Figures & tables
Fig. 1 : Overview of suboptimal behaviors and their impact on policy training. (Top) Demonstrations for precision manipulation inevitably contain suboptimal behaviors, even after trajectory-level filtering. In peg-insertion, for example, removing successful yet low-quality trajectories improves the task success rate only from 50% to 66.7%, whereas our step-level filtering achieves a success rate of 100%. (Bottom) During data collection, human operators unavoidably introduce suboptimal behaviors such as drifting (unintended lateral deviation), overshooting (motion beyond the target), and idling (unnecessary pauses or inactivity). While prior works assume that all actions contribute equally to the final success, they fail to capture the real-time progress in the face of these behaviors.
Fig. 2 : Quantitative success rate comparison and qualitative illustration of progress estimation. (Left) Success rates of different methods across three real-world tasks: peg-in-hole, peg-in-hole (Hard) and charger insertion. (Right) Qualitative illustration of progress estimation on Peg-Hard task. PRIME reflects partial advancements during an undershoot, maintains a constant estimation during idling, and appropriately penalizes the progress when a drift occurs. (Bottom) Correspondence between suboptimal behaviors and progress estimation of PRIME. Our method accurately identifies different types of suboptimal behavior, and assigns progress estimation accordingly.
Fig. 3 : An overview of PRIME. (a): Graph construction. Each demonstration timestep is represented by a graph node which comprises multi-view observations and the corresponding robot configuration. (b): Edge establishment. Cross-trajectory edges are established through multi-view visual consistency and configuration proximity, with weights reflecting pose and gripper-state differences. (c): Shortest-path distance as progress for RL. The graph G defines the remaining distance to task completion as the weighted shortest-path distance from the current node to the goal. This distance serves as a measure of task progress.
Task
Demo
Success Rate
Behavior Cloning
Linear Temporal Progress
Non-uniform Progress
π0.5 -SFT
Linear
RECAP
ViVa
RoboMeter
SARM
GSR-based
ProcVLM
Ours
Drawer
Noisy
86.7%
80.0%
90.0%
90.0%
93.3%
90.0%
46.7%
90.0%
96.7%
Expert
100.0%
90.0%
100.0%
100.0%
100.0%
100.0%
86.7%
100.0%
100.0%
Peg
Noisy
33.3%
16.7%
40.0%
43.3%
50.0%
53.3%
26.7%
46.7%
73.3%
Expert
86.7%
76.7%
93.3%
86.7%
90.0%
73.3%
73.3%
86.7%
100.0%
TABLE I : Main results from the real experiments. We evaluate the performance of PRIME across four robotic manipulation tasks of varying complexity: Drawer Opening, Peg-in-Hole (Standard and Hard), and Charger Insertion. These evaluations are conducted under diverse data quality conditions, encompassing noisy data that is unqualified for pure imitation learning, as well as expert data that is qualified for training but inevitably contains human mistakes. Experimental results demonstrate that PRIME consistently outperforms standard baselines across various levels of data quality. Besides, the gains on expert data suggest that local progress information remains useful even when overall demonstration quality is high.
Method
Stage-level Ordering ↑
Local Precision ↑
Progress Continuity (s−2)↓
Linear
73.3%
70.0%
8.14
SARM
67.5%
67.5%
3.47
RoboMeter
70.8%
61.7%
4.72
ViVa
79.2%
40.8%
1.74
ProcVLM
87.5%
67.5%
5.50
RECAP
98.3%
68.3%
2.69
TABLE II : Results on the diagnostic benchmark.
Fig. 4 : Evaluation on various robotic trajectories. We compare our method against baselines across three scenarios. (1) Successful Execution: For a clean trajectory with minor pauses, PRIME provides a smooth, monotonic progress estimate. (2) Suboptimal Execution: When a human collector makes a mistake and lifts the cylinder upward to adjust, PRIME captures this setback with a local decrease in progress (orange band). (3) Failure Detection: When the cylinder is misplaced, PRIME reflects a sharp drop in progress (red band) as the trajectory drifts toward failure. For visualization, we normalize our distance predictions as 1−d(s0)d(st) . More visualization results are provided in Appendix -C .
Reward Model Type
Success Rate
In-distribution
Out-of-distribution
Sparse reward
54.0±2.3
27.1±1.4
Time Progress
60.8±6.1
31.1±1.5
Ours
67.4±5.6
35.3±3.1
TABLE III : Main results from the simulation. We report the average success rates across 3 seeds (0, 1 and 2).
Method
Peg-Hard
Drawer
PRIME w/o geometric edge costs
33.3%
90.0%
PRIME w/o configuration-proximity constraint
26.7%
53.3%
Euclidean distance to goal (no graph)
56.7%
33.3%
PRIME
70.0%
96.7%
TABLE IV : Ablation of graph connectivity and distance computation in PRIME.
Fig. 5 : Effect of temperature τ (left) and reward clipping bound rmax (right).
TABLE V : Success rate under different percentile choices for neighbor distance thresholds ϵxy and ϵz
Fig. 6 : Convergence and computational efficiency. (Left) The shortest-path deviation Δd stabilizes after 4 episodes. (Right) Computation time of visual encoding and graph construction with increasing numbers of graph nodes. All runtime measurements are obtained on an NVIDIA RTX3090 GPU.
Fig. 7 : An overview of our real-world experimental setup.
Hyper-parameters
Action Prediction Horizon
50
Action Horizon
50
Observation Horizon
1
Gradient Steps
80000
Batch Size
16
rmax
0.1
ϵvis
0.1
TABLE VI : Training Details of PRIME.
Fig. 8 : Expert demonstrations with pauses in peg-in-hole (Hard) task.
Fig. 9 : Suboptimal behaviors in charger insertion task.
Fig. 10 : Suboptimal behaviors in charger insertion task.
Fig. 11 : Demonstrations from an untrained human, exhibiting severe boundary violations.
Fig. 12 : The cylinder is placed in an incorrect location.
Embodied agents now take on ever longer tasks. For long tasks, knowing only whether a task finally succeeds or fails says little; the steps along the way matter. Progress Reward Models (PRMs) score how far a task has come at every step, and serve as dense rewards, verifiers and monitors. Yet in long tasks the current frame alone often cannot tell how far the task has come, because progress depends on what happened before. We call this problem context-dependent progress estimation. Existing benchmarks on progress estimation mostly focus on short tasks whose progress can be read from the current observation, and whether PRMs can estimate progress when context is needed remains underexplored. We therefore build ContextProgress-Bench, with 24 manipulation tasks for 120 episodes. The benchmark covers three settings: (i) State Recall, where information needed for progress appeared earlier but is not in the current frame; (ii) Sequence Tracking, where steps follow a fixed order, so progress requires knowing which steps are done and which comes next; and (iii) Recurrence Disambiguation, where look-alike frames sit at very different progress. We then run a paired diagnosis: each PRM keeps the same input format in both runs, and in one run its instruction integrates the right context. Even PRMs that read the entire history get lost in estimating progress, yet with the right context the same five models cut their progress error by 77-82%. Embodied PRMs are thus not incapable of progress estimation, but lost without the right context. We therefore propose ProgressCompass, an autonomous agentic loop that reorients an existing PRM and uses current general-purpose VLMs to supply the context the PRM needs. Wrapped in the loop, the same frozen PRM cuts its progress error by 63% and raises its rank agreement by 76%. With such a compass, PRMs estimate progress far better on longer, more complex tasks.
Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.
Pengzhi Yang, Xinyu Wang, Pengyu Jing +7
NUS Human-Centered Robotic Lab · ‡ Booking.com · University of Cambridge +1
Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages, not merely whether the final outcome is successful. Existing reward models often rely on trajectory-level success labels or time-based interpolation, which can conflate elapsed time with true task progress and therefore fail to capture unfinished steps, stagnation, and failure states. We present ProcVLM, a progress-aware vision-language model that learns procedure-grounded progress as a dense reward signal for manipulation. Rather than deriving progress from terminal outcomes or temporal proxies, ProcVLM grounds progress estimation in procedural structure and intra-stage visual change, and further adopts a reasoning-before-estimation paradigm that infers the remaining atomic actions before estimating task progress. Specifically, we construct this supervision by synthesizing frame-level subtask-semantic annotations, assigning progress budgets according to subtask structure, and distributing each budget based on intra-subtask visual change. To train ProcVLM at scale, we build a standardized procedural supervision synthesis pipeline and construct ProcCorpus-60M from 30 embodied datasets with 60M annotated frames, from which we derive ProcVQA for procedure-aware pretraining, with progress estimation as the central task alongside action segmentation and future planning. Experiments on ProcVQA and reward-model benchmarks show that ProcVLM improves embodied procedural reasoning and yields more discriminative trajectory-internal progress estimates than representative baselines, supporting its use as a dense reward model for downstream reward-guided policy optimization. Project page: https://procvlm.github.io/
Youhe Feng, Hansen Shi, Haoyang Li +7
School of Information, Renmin University of China · Work done during the internship at Zhipu AI. · Zhipu AI +1