Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence · Tsinghua University · University of Sydney · Beihang University
Current robotic evaluation is still largely dominated by binary success rates, which collapse rich execution processes into a single outcome and obscure critical qualities such as progress, efficiency, and stability. To address this limitation, we propose PRM-as-a-Judge, a dense evaluation paradigm that leverages Process Reward Models (PRMs) to audit policy execution directly from trajectory videos by estimating task progress from observation sequences. Central to this paradigm is the OPD (Outcome-Process-Diagnosis) metric system, which explicitly formalizes execution quality via a task-aligned progress potential. We characterize dense robotic evaluation through two axiomatic properties: macro-consistency, which requires additive and path-consistent aggregation, and micro-resolution, which requires sensitivity to fine-grained physical evolution. Under this formulation, potential-based PRM judges provide a natural instantiation of dense evaluation, with macro-consistency following directly from the induced scalar potential. We empirically validate the micro-resolution property using RoboPulse, a diagnostic benchmark specifically designed for probing micro-scale progress discrimination, where several trajectory-trained PRM judges outperform discriminative similarity-based methods and general-purpose foundation-model judges. Finally, leveraging PRM-as-a-Judge and the OPD metric system, we conduct a structured audit of mainstream policy paradigms across long-horizon tasks, revealing behavioral signatures and failure modes that are invisible to outcome-only metrics.
Figures & tables
Figure 1: PRM-as-a-Judge and OPD. Binary success rate compresses an execution into a terminal outcome and obscures progress, efficiency, and stability. We use a Process Reward Model to induce a dense progress potential and derive OPD metrics at the outcome, process, and diagnosis levels. This yields fine-grained auditing that distinguishes near miss failures from early collapse and separates smooth successes from inefficient or unstable ones.
Level
Metric
Range
Pref.
Interpretation
Outcome
Milestone Coverage (MC)
{0,0.25,…,1}
↑
The furthest milestone reached during execution.
Max Progress (MP)
[0,1]
↑
The maximum progress score attained during the episode.
Process
Path-weighted Progress Length (PPL)
[0,1]
↑
Efficiency of the execution path relative to net progress and path variation.
Diagnosis
Cumulative Regret Area (CRA)
[0,1]
↓
Measures the severity and duration of regression from the best-so-far progress level.
Stagnation Ratio (STR)
[0,1]
↓
Fraction of time steps with negligible task-relevant progress change.
Table 1: Overview of the OPD Metric System. Range : Valid value domain, where MC takes discrete quartile values and others are continuous. Pref. : Preference direction ( ↑ : higher is better; ↓ : lower is better).
Figure 2: RoboPulse overview. Left: data composition across collection settings and embodiment–setting categories. Right: task semantic coverage via a token word cloud extracted from task names. RoboPulse is designed to probe micro-scale progress discrimination under diverse embodiments and visual domains.
Small
Medium
Large
AVG
Method
Real
Sim
UMI
Human
Avg
Real
Sim
UMI
Human
Avg
Real
Sim
UMI
Human
Avg
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)
0.56
0.52
0.58
0.58
0.56
0.56
0.64
0.52
0.50
0.56
0.60
0.69
0.63
0.51
0.61
0.57
CLIP ViT-L/14 (I2I)
0.52
0.56
0.55
0.56
0.54
0.61
0.65
0.50
0.59
0.59
0.64
0.72
0.68
0.54
0.65
0.59
CLIP ViT-B/32 (T2I)
0.48
0.49
0.56
0.49
0.51
0.51
0.48
0.35
0.34
0.42
0.38
0.42
0.61
0.54
0.49
0.47
CLIP ViT-L/14 (T2I)
0.50
0.46
0.48
0.52
0.49
0.49
0.42
0.35
0.52
0.45
0.46
0.42
0.42
0.51
0.45
0.46
Table 2: Pairwise progress-judgment accuracy on RoboPulse under different relative comparison scales. We report accuracy (0–1) over Small/Medium/Large hop ranges, stratified by four collection settings (Real, Sim, UMI, Human), together with per-scale and overall averages. All evaluators are tested zero-shot through their native input interfaces, so results reflect practical judging capability rather than a strictly input-matched comparison.
Blocks Ranking RGB
Handover Mic
Place Bread Basket
Outcome Level
P.
D.
Outcome Level
P.
D.
Outcome Level
P.
D.
MC
MP
PPL
CRA
STR
MC
MP
PPL
CRA
STR
MC
MP
PPL
CRA
STR
@25
@50
@75
@100
@25
@50
@75
@100
@25
@50
@75
@100
ACT
84
44
22
2
49.9
11.7
8.99
59.7
100
100
94
74
96.8
72.3
4.08
44.1
100
74
46
4
73.1
17.6
15.5
65.4
DP
94
40
18
0
51.7
4.07
16.3
43.8
100
94
88
44
93.8
66.0
5.49
57.2
100
94
74
16
87.6
21.3
16.9
48.0
RDT
100
62
30
0
61.2
6.19
16.3
39.0
100
100
100
100
100
84.2
1.45
39.8
100
100
78
8
90.4
16.6
22.7
37.1
Table 3: OPD auditing on RoboTwin 2.0. Five policy families on three representative long-horizon tasks (50 rollouts each). MC@25/50/75/100 denotes milestone coverage. P. denotes the process level (PPL), and D. denotes the diagnosis level (CRA and STR). Full results are in Appendix H .
Figure 3: Reachability and failure-stage decomposition by milestone coverage. For each task, we plot the fraction of rollout episodes that reach milestone thresholds (25/50/75/100%), revealing where execution progress tends to stall along the horizon.
Figure 4: Success-only execution quality on Handover Mic. We report success-conditioned OPD metrics and compare path efficiency, measured by PPL, against regression, measured by CRA, and stagnation, measured by STR, across policy families. Error bars denote standard deviation across successful episodes.
Figure 5: Failure-only OPD fingerprints. We normalize OPD metrics over failed episodes within each task to highlight policy-specific failure patterns.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Formal definition
Intended signal
Failure mode
PPE
PPE(τ)=∑t=0T−1∣Φt+1−Φt∣+ε1
Efficiency in progress space by penalizing back and forth motion.
Ill conditioned when progress variation is small, producing inflated scores for stalled behavior. It is completion agnostic and penalizes physically necessary corrections.
PTI
PTI(τ)=T+11t=0∑TΦt
Average maintained progress, intended to reward early and sustained advancement.
Overweights timing of progress and penalizes valid late completion patterns that arise in bottleneck tasks. It can prefer incomplete trajectories with moderate early progress over successful but late trajectories.
EAD
EAD(τ)=T1t=0∑T−1I[ΔΦt>ϵ]
Frequency of noticeable positive progress steps, intended to separate purposeful motion from dithering.
Strong dependence on control frequency and temporal discretization. It encourages bursty behavior and undervalues smooth continuous progress.
PJ
PJ(τ)=T−21t=1∑T−2∣ΔΦt+1−ΔΦt∣
Smoothness of progress evolution, intended to penalize jerkiness.
In contact rich manipulation, valid progress can occur through discrete contact events. The metric is sensitive to judge noise and can penalize correct executions.
CS
CSK(τ)=Var({ΦT−K,…,ΦT})
Terminal stability, intended to detect oscillatory behavior near termination.
Dependence on termination conventions and episode length. Its value can change substantially under different episode truncation rules.
RR
RR(τ)=t=1∑Tmax(0,Φt−1−Φt)
Accumulated local regressions, intended to quantify backward motion.
Ignores temporal persistence and severity relative to the best achieved progress. It cannot separate early exploration from catastrophic late stage drops.
Appendix
Table 4: Discarded progress based metrics. All candidates are defined on the same potential Φ for comparison. For each metric we list a formal definition, the intended signal, and the primary failure mode that motivated exclusion.
Robot Embodiment
Primary Data Sources
# Cases
Franka Emika Panda
DROID ( Khazatsky et al., 2024 ) , LIBERO ( Liu et al., 2023 ) , RoboCasa ( Nasiriany et al., 2024 )
Table 5: Distribution of RoboPulse cases by robot embodiment. RoboPulse spans diverse robot hardware, from industrial robots to humanoids.
Method
Noise = 0
Noise = 0.05
Noise = 0.10
Real.
Sim.
UMI
Human
AVG
Real.
Sim.
UMI
Human
AVG
Real.
Sim.
UMI
Human
AVG
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)
0.57
0.62
0.58
0.53
0.57
0.57
0.63
0.62
0.59
0.60
0.53
0.63
0.59
0.53
0.57
CLIP ViT-L/14 (I2I)
0.59
0.64
0.58
0.56
0.59
0.56
0.62
0.64
0.51
0.58
0.56
0.57
0.63
0.54
0.58
CLIP ViT-B/32 (T2I)
0.46
0.46
0.51
0.46
0.47
0.47
0.46
0.53
0.49
0.49
0.47
0.47
0.53
0.54
0.50
CLIP ViT-L/14 (T2I)
0.48
0.44
0.42
0.52
0.46
0.48
0.44
0.43
0.49
0.46
0.49
0.47
0.49
0.51
0.49
Appendix
Table 6: Comparison of discriminative similarity-based methods and progress reward model judges under different noise levels. We report the performance across Real-World, Simulation, UMI, and Human settings, along with the average (AVG) for Noise levels of 0, 0.05, and 0.10.
Real-World
Simulation
UMI
Human
Method
Agibot-World
Agilex
Droid
Galaxea R1Lite
Libero
RoboCasa
RoboTwin2.0
Pika
Egodex
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)
0.54
0.53
0.75
0.45
0.55
0.41
0.61
0.58
0.58
CLIP ViT-L/14 (I2I)
0.55
0.53
0.52
0.47
0.59
0.54
0.53
0.55
0.56
CLIP ViT-B/32 (T2I)
0.50
0.47
0.55
0.42
0.54
0.44
0.50
0.56
0.49
CLIP ViT-L/14 (T2I)
0.54
0.46
0.55
0.47
0.45
0.50
0.44
0.48
0.52
Appendix
Table 7: Performance comparison on the Small scale datasets.
Real-World
Simulation
UMI
Human
Method
Agibot-World
Agilex
Droid
Galaxea R1Lite
Libero
RoboCasa
RoboTwin2.0
Pika
Egodex
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)
0.57
0.49
0.57
0.62
0.69
0.59
0.66
0.52
0.50
CLIP ViT-L/14 (I2I)
0.65
0.55
0.54
0.73
0.81
0.50
0.66
0.50
0.59
CLIP ViT-B/32 (T2I)
0.51
0.47
0.57
0.48
0.44
0.60
0.40
0.35
0.34
CLIP ViT-L/14 (T2I)
0.49
0.53
0.46
0.48
0.33
0.45
0.47
0.35
0.52
Appendix
Table 8: Performance comparison on the Medium scale datasets.
Real-World
Simulation
UMI
Human
Method
Agibot-World
Agilex
Droid
Galaxea R1Lite
Libero
RoboCasa
RoboTwin2.0
Pika
Egodex
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)
0.65
0.52
0.61
0.61
0.76
0.64
0.65
0.63
0.51
CLIP ViT-L/14 (I2I)
0.69
0.58
0.56
0.73
0.80
0.63
0.73
0.68
0.54
CLIP ViT-B/32 (T2I)
0.45
0.32
0.32
0.44
0.33
0.59
0.35
0.61
0.54
CLIP ViT-L/14 (T2I)
0.58
0.40
0.40
0.45
0.24
0.58
0.46
0.42
0.51
Appendix
Table 9: Performance comparison on the Large scale datasets.
Method
Human
Real-World
Simulation
UMI
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)
0.54
0.57
0.61
0.58
CLIP ViT-L/14 (I2I)
0.56
0.58
0.64
0.58
CLIP ViT-B/32 (T2I)
0.48
0.46
0.46
0.52
CLIP ViT-L/14 (T2I)
0.52
0.49
0.44
0.43
General Foundation-Model Judges
Appendix
Table 10: Overall performance summary across different domains.
Model
Blocks Ranking RGB
Handover Block
Handover Mic
Outcome Level
Process
Diagnosis
Outcome Level
Process
Diagnosis
Outcome Level
Process
Diagnosis
MC
MP
PPL
CRA
STR
MC
MP
PPL
CRA
STR
MC
MP
PPL
CRA
STR
@25
@50
@75
@100
@25
@50
@75
@100
@25
@50
@75
@100
ACT
84
44
22
2
49.93
11.67
8.99
59.74
86
60
44
42
66.35
47.98
9.6
65.49
100
100
94
74
96.79
72.33
4.08
44.14
DP
94
40
18
0
51.72
4.07
16.26
43.77
92
52
50
44
66.88
62.18
1.05
69.64
100
94
88
44
93.8
65.97
5.49
57.18
RDT
100
62
30
0
61.23
6.19
16.3
39.03
94
82
62
38
78.86
53.13
9.88
65.09
100
100
100
100
100
84.23
1.45
39.82
Appendix
Table 11: OPD auditing on RoboTwin 2.0. Performance of different policy models on three tasks. MC@25/50/75/100 denotes milestone coverage; MP/PPL/CRA/STR are averaged episode-level metrics.
Model
Hanging Mug
Place Bread Basket
Outcome Level
Process
Diagnosis
Outcome Level
Process
Diagnosis
MC
MP
PPL
CRA
STR
MC
MP
PPL
CRA
STR
@25
@50
@75
@100
@25
@50
@75
@100
ACT
96
84
74
14
84.88
23.61
17.12
51.03
100
74
46
4
73.11
17.55
15.46
65.38
DP
100
100
98
14
86.4
30.82
18.72
57.03
100
94
74
16
87.55
21.33
16.88
47.96
RDT
98
96
92
20
88.5
48.4
11.35
65.84
100
100
78
8
90.4
16.57
22.68
37.1
Appendix
Table 12: OPD Auditing on RoboTwin 2.0: Task Set A. Results for Hanging Mug and Place Bread Basket (50 rollouts each).
Model
Place Bread Skillet
Place Can Basket
Outcome Level
Process
Diagnosis
Outcome Level
Process
Diagnosis
MC
MP
PPL
CRA
STR
MC
MP
PPL
CRA
STR
@25
@50
@75
@100
@25
@50
@75
@100
ACT
100
80
34
8
68.66
34.19
7.99
69.11
98
88
74
4
83.84
30.62
9.06
69.9
DP
100
58
30
4
62.51
31.45
8.36
78.07
88
60
34
12
60.91
39.87
5.83
81.04
RDT
100
80
50
6
76.42
27.87
12.62
63.74
100
100
94
16
95.94
52.41
4.6
69.02
Appendix
Table 13: OPD Auditing on RoboTwin 2.0: Task Set B. Results for Place Bread Skillet and Place Can Basket (50 rollouts each).
Figure 6: Reachability profiles over all RoboTwin 2.0 tasks. For each task, we plot the fraction of rollouts that reach milestone thresholds at 25/50/75/100%. The curves localize where progress saturates along the horizon and expose last-mile bottlenecks when MC@75 is high but MC@100 drops sharply.
Figure 7: Success-conditioned execution quality over all tasks. We compute PPL, CRA, and STR on successful episodes only and visualize cross-policy trade-offs. This separates execution quality from success frequency and highlights policies that succeed efficiently versus those that succeed with higher correction cost or hesitation.
Figure 8: Failure-only OPD fingerprints over all tasks. We aggregate MP, PPL, CRA, and STR over failed episodes and normalize scores within each task across policy families. For CRA and STR, we flip the sign before normalization so that higher normalized scores consistently indicate more desirable behavior.
Figure 9: Real-World Task: Stack the wooden blocks. The reward curve demonstrates accurate tracking of manipulation errors. When the robot picks the wrong block or misplaces it (red arrows), the score drops, reflecting negative progress. The agent eventually recovers, reaching a success state (Progress ≈1.0 ). The CRA of 4.59% quantifies these regression events.
Figure 10: Real-World Task: Clean the table. This task involves semantic understanding of “messiness.” The model correctly penalizes the agent for moving a correctly placed drink and jumbling tissues, causing deep valleys in the progress curve. The successful recovery is captured by the final rise, though the efficiency ( PPL 58.49% ) is impacted by these mistakes.
Figure 11: Human Task: Fold the trouser. Evaluating manipulation of deformable objects is challenging. Robo-Dopamine correctly identifies the regression when the trouser is accidentally unfolded (Step ∼18 ) and when the folded trouser is misplaced (Step ∼60 ). The curve aligns well with the visual state of the fabric.
Figure 12: Simulation Task: Blocks Ranking RGB. A long-horizon task ( ∼700 steps). The initial phase is dominated by empty grasps and incorrect interactions, leading to a long period of stagnation (high STR of 16.22% and flat curve). Once the correct sorting sequence begins (Step ∼500 ), the reward signal increases steadily.
Figure 13: Simulation Task: Place bread into the basket. The model exhibits sensitivity to spatial relations. Moving the bread away from the basket (Step ∼75 ) causes a dip in the score. The high PPL ( 75.06% ) indicates that despite minor deviations, the overall trajectory was relatively efficient compared to other tasks.
Figure 14: Simulation Task: Hanging the mug. Fine-grained manipulation tracking. The curve captures the “redundant removal” of the mug (Step ∼450 ) as a regression before the final successful placement. This granularity allows for the detection of suboptimal behavior even in successful trajectories.
Figure 15: RoboPulse qualitative examples for progress direction judging. Each panel shows the task, reference start/end frames, and a sampled BEFORE/AFTER pair from an episode. The arrow indicates the ground-truth progress direction. We report whether each judge predicts the correct direction. Robo-Dopamine is robust to appearance shifts and detects both improvement and regression, whereas CLIP can fail when visual similarity is misaligned with physical progress.
Figure 16: Prompt template for VLAC ( Zhai et al., 2025 ) on RoboPulse.
Figure 17: Prompt template for Gemini 3 Pro ( Google, 2025 ) , GPT-5.2 ( OpenAI, 2025 ) , Qwen3-VL ( Bai et al., 2025a ) pairwise progress judgment on RoboPulse. We provide the full system and user prompt, including task description, reference anchors, multiview BEFORE/AFTER observations, and the required output schema used for automated parsing.
Figure 18: Prompt template for RoboReward ( Lee et al., 2026 ) on RoboPulse.
Figure 19: Prompt template for GVL ( Ma et al., 2024 ) on RoboPulse.
Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.
Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts. This strong resource constraint in real-world testing necessitates both more informative performance measures and reliable and efficient evaluation procedures to properly assess model capabilities and benchmark progress in the field. This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice. Based on safe, anytime-valid inference (SAVI), our test procedure is sequential, allowing the evaluator to stop early when sufficient statistical evidence has accumulated to reach a decision at a pre-specified level of confidence. Unlike previous work developed for binary success, our unified approach addresses a wide range of informative metrics: from discrete partial credit task progress to continuous measures of episodic reward or trajectory smoothness, spanning both parametric and nonparametric comparison problems. Through extensive validation on simulated and real-world evaluation data, we demonstrate up to 70% reduction in evaluation burden compared to standard batch methods and up to 50% reduction compared to state-of-the-art sequential procedures designed for binary outcomes, with no loss of statistical rigor. Notably, our empirical results show that competing policies can be separated more quickly when using fine-grained task progress than binary success metrics.
David Snyder, Apurva Badithela, Nikolai Matni +4
University of Pennsylvania · Princeton University · Toyota Research Institute (TRI)
Although Vision--Action (VA) and Vision--Language--Action (VLA) policies have advanced robotic manipulation, their evaluation remains dominated by binary success rates, which obscure process-level differences among executions that complete the same task. We introduce Eval-Actions, a diagnostic evaluation methodology and real-robot benchmark for fine-grained execution-quality assessment of learned manipulation policies. Eval-Actions combines criteria-based Expert Grading (EG), Rank-Guided (RG) labels that align measurable motion indicators with expert rankings, and Chain-of-Thought-style (CoT) annotations that explain observable quality differences. The benchmark contains 13K+ teleoperated and policy-generated real-robot episodes covering 150+ tasks and approximately 52 hours of recordings with RGB-D videos, robot-state trajectories, task descriptions, and success/failure labels. Its densely annotated subset provides EG/RG/CoT supervision for training and evaluation. We further provide AutoEval, a reference multimodal evaluator that predicts quality scores, task outcomes, and diagnostic explanations from RGB temporal evidence and compact kinematic summaries. On the annotated Eval-Actions test split, AutoEval-S achieves Spearman rank correlations (SRCCs) of 0.81 and 0.84 under EG and RG, with success detection accuracies of 90.6% and 91.0%; AutoEval-P reaches 0.70 SRCC under CoT. Analyses of expert consistency, physical-metric baselines, modality ablations, structured generalization, and offline policy ranking show that Eval-Actions provides standardized, interpretable diagnostic signals complementary to success-rate evaluation.