PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing

Yuheng Ji; Yuyang Liu; Huajie Tan; Xuchuan Huang; Fanding Huang; Yijie Xu; Cheng Chi; Yuting Zhao

cs.ROMar 23, 2026

PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing

Authors: Yuheng Ji, Yuyang Liu, Huajie Tan, Xuchuan Huang, Fanding Huang, Yijie Xu, Cheng Chi, Yuting Zhao,

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence ·

Abstract

Current robotic evaluation is still largely dominated by binary success rates, which collapse rich execution processes into a single outcome and obscure critical qualities such as progress, efficiency, and stability. To address this limitation, we propose PRM-as-a-Judge, a dense evaluation paradigm that leverages Process Reward Models (PRMs) to audit policy execution directly from trajectory videos by estimating task progress from observation sequences. Central to this paradigm is the OPD (Outcome-Process-Diagnosis) metric system, which explicitly formalizes execution quality via a task-aligned progress potential. We characterize dense robotic evaluation through two axiomatic properties: macro-consistency, which requires additive and path-consistent aggregation, and micro-resolution, which requires sensitivity to fine-grained physical evolution. Under this formulation, potential-based PRM judges provide a natural instantiation of dense evaluation, with macro-consistency following directly from the induced scalar potential. We empirically validate the micro-resolution property using RoboPulse, a diagnostic benchmark specifically designed for probing micro-scale progress discrimination, where several trajectory-trained PRM judges outperform discriminative similarity-based methods and general-purpose foundation-model judges. Finally, leveraging PRM-as-a-Judge and the OPD metric system, we conduct a structured audit of mainstream policy paradigms across long-horizon tasks, revealing behavioral signatures and failure modes that are invisible to outcome-only metrics.

Figures & tables

Figure 1: PRM-as-a-Judge and OPD. Binary success rate compresses an execution into a terminal outcome and obscures progress, efficiency, and stability. We use a Process Reward Model to induce a dense progress potential and derive OPD metrics at the outcome, process, and diagnosis levels. This yields fine-grained auditing that distinguishes near miss failures from early collapse and separates smooth successes from inefficient or unstable ones.

Level	Metric	Range	Pref.	Interpretation
Outcome	Milestone Coverage (MC)	$\{0,0.25,\dots,1\}$	$\uparrow$	The furthest milestone reached during execution.
Outcome	Max Progress (MP)	$[0,1]$	$\uparrow$	The maximum progress score attained during the episode.
Process	Path-weighted Progress Length (PPL)	$[0,1]$	$\uparrow$	Efficiency of the execution path relative to net progress and path variation.
Diagnosis	Cumulative Regret Area (CRA)	$[0,1]$	$\downarrow$	Measures the severity and duration of regression from the best-so-far progress level.
Diagnosis	Stagnation Ratio (STR)	$[0,1]$	$\downarrow$	Fraction of time steps with negligible task-relevant progress change.

Table 1: Overview of the OPD Metric System. Range : Valid value domain, where MC takes discrete quartile values and others are continuous. Pref. : Preference direction (

\uparrow

: higher is better;

\downarrow

: lower is better).

Figure 2: RoboPulse overview. Left: data composition across collection settings and embodiment–setting categories. Right: task semantic coverage via a token word cloud extracted from task names. RoboPulse is designed to probe micro-scale progress discrimination under diverse embodiments and visual domains.

	Small					Medium					Large					AVG
Method	Real	Sim	UMI	Human	Avg	Real	Sim	UMI	Human	Avg	Real	Sim	UMI	Human	Avg
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)	0.56	0.52	0.58	0.58	0.56	0.56	0.64	0.52	0.50	0.56	0.60	0.69	0.63	0.51	0.61	0.57
CLIP ViT-L/14 (I2I)	0.52	0.56	0.55	0.56	0.54	0.61	0.65	0.50	0.59	0.59	0.64	0.72	0.68	0.54	0.65	0.59
CLIP ViT-B/32 (T2I)	0.48	0.49	0.56	0.49	0.51	0.51	0.48	0.35	0.34	0.42	0.38	0.42	0.61	0.54	0.49	0.47
CLIP ViT-L/14 (T2I)	0.50	0.46	0.48	0.52	0.49	0.49	0.42	0.35	0.52	0.45	0.46	0.42	0.42	0.51	0.45	0.46

Table 2: Pairwise progress-judgment accuracy on RoboPulse under different relative comparison scales. We report accuracy (0–1) over Small/Medium/Large hop ranges, stratified by four collection settings (Real, Sim, UMI, Human), together with per-scale and overall averages. All evaluators are tested zero-shot through their native input interfaces, so results reflect practical judging capability rather than a strictly input-matched comparison.

	Blocks Ranking RGB								Handover Mic								Place Bread Basket
	Outcome Level					P.	D.		Outcome Level					P.	D.		Outcome Level					P.	D.
	MC				MP	PPL	CRA	STR	MC				MP	PPL	CRA	STR	MC				MP	PPL	CRA	STR
	@25	@50	@75	@100	MP	PPL	CRA	STR	@25	@50	@75	@100	MP	PPL	CRA	STR	@25	@50	@75	@100	MP	PPL	CRA	STR
ACT	84	44	22	2	49.9	11.7	8.99	59.7	100	100	94	74	96.8	72.3	4.08	44.1	100	74	46	4	73.1	17.6	15.5	65.4
DP	94	40	18	0	51.7	4.07	16.3	43.8	100	94	88	44	93.8	66.0	5.49	57.2	100	94	74	16	87.6	21.3	16.9	48.0
RDT	100	62	30	0	61.2	6.19	16.3	39.0	100	100	100	100	100	84.2	1.45	39.8	100	100	78	8	90.4	16.6	22.7	37.1

Table 3: OPD auditing on RoboTwin 2.0. Five policy families on three representative long-horizon tasks (50 rollouts each). MC@25/50/75/100 denotes milestone coverage. P. denotes the process level (PPL), and D. denotes the diagnosis level (CRA and STR). Full results are in Appendix H .

Figure 3: Reachability and failure-stage decomposition by milestone coverage. For each task, we plot the fraction of rollout episodes that reach milestone thresholds (25/50/75/100%), revealing where execution progress tends to stall along the horizon.

Panel of Figure 4 — Figure 4: Success-only execution quality on Handover Mic. We report success-conditioned OPD metrics and compare path efficiency, measured by PPL, against regression, measured by CRA, and stagnation, measured by STR, across policy families. Error bars denote standard deviation across successful episodes.

Figure 5: Failure-only OPD fingerprints. We normalize OPD metrics over failed episodes within each task to highlight policy-specific failure patterns.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Formal definition	Intended signal	Failure mode
PPE	$\displaystyle\mathrm{PPE}(\tau)=\frac{1}{\sum_{t=0}^{T-1}\lvert\Phi_{t+1}-\Phi_{t}\rvert+\varepsilon}$	Efficiency in progress space by penalizing back and forth motion.	Ill conditioned when progress variation is small, producing inflated scores for stalled behavior. It is completion agnostic and penalizes physically necessary corrections.
PTI	$\displaystyle\mathrm{PTI}(\tau)=\frac{1}{T+1}\sum_{t=0}^{T}\Phi_{t}$	Average maintained progress, intended to reward early and sustained advancement.	Overweights timing of progress and penalizes valid late completion patterns that arise in bottleneck tasks. It can prefer incomplete trajectories with moderate early progress over successful but late trajectories.
EAD	$\displaystyle\mathrm{EAD}(\tau)=\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{I}\!\left[\Delta\Phi_{t}>\epsilon\right]$	Frequency of noticeable positive progress steps, intended to separate purposeful motion from dithering.	Strong dependence on control frequency and temporal discretization. It encourages bursty behavior and undervalues smooth continuous progress.
PJ	$\displaystyle\mathrm{PJ}(\tau)=\frac{1}{T-2}\sum_{t=1}^{T-2}\lvert\Delta\Phi_{t+1}-\Delta\Phi_{t}\rvert$	Smoothness of progress evolution, intended to penalize jerkiness.	In contact rich manipulation, valid progress can occur through discrete contact events. The metric is sensitive to judge noise and can penalize correct executions.
CS	$\displaystyle\mathrm{CS}_{K}(\tau)=\mathrm{Var}\!\left(\{\Phi_{T-K},\dots,\Phi_{T}\}\right)$	Terminal stability, intended to detect oscillatory behavior near termination.	Dependence on termination conventions and episode length. Its value can change substantially under different episode truncation rules.
RR	$\displaystyle\mathrm{RR}(\tau)=\sum_{t=1}^{T}\max\!\left(0,\Phi_{t-1}-\Phi_{t}\right)$	Accumulated local regressions, intended to quantify backward motion.	Ignores temporal persistence and severity relative to the best achieved progress. It cannot separate early exploration from catastrophic late stage drops.

Appendix

Table 4: Discarded progress based metrics. All candidates are defined on the same potential

\Phi

for comparison. For each metric we list a formal definition, the intended signal, and the primary failure mode that motivated exclusion.

Robot Embodiment	Primary Data Sources	# Cases
Franka Emika Panda	DROID ( Khazatsky et al., 2024 ) , LIBERO ( Liu et al., 2023 ) , RoboCasa ( Nasiriany et al., 2024 )	600
AGIBot-A2D	AGIBot-World ( Bu et al., 2025 )	200
Agilex Piper	RoboBrain-X ( FlagOpen, 2025 ) , RoboTwin ( Chen et al., 2025b )	400
Galaxea R1Lite	RoboBrain-X ( FlagOpen, 2025 )	200
Pika	RoboBrain-X ( FlagOpen, 2025 )	200
Human	Egodex ( Hoque et al., 2025 )	200

Appendix

Table 5: Distribution of RoboPulse cases by robot embodiment. RoboPulse spans diverse robot hardware, from industrial robots to humanoids.

Discriminative Similarity-Based Methods
Method	Noise = 0					Noise = 0.05					Noise = 0.10
Method	Real.	Sim.	UMI	Human	AVG	Real.	Sim.	UMI	Human	AVG	Real.	Sim.	UMI	Human	AVG
CLIP ViT-B/32 (I2I)	0.57	0.62	0.58	0.53	0.57	0.57	0.63	0.62	0.59	0.60	0.53	0.63	0.59	0.53	0.57
CLIP ViT-L/14 (I2I)	0.59	0.64	0.58	0.56	0.59	0.56	0.62	0.64	0.51	0.58	0.56	0.57	0.63	0.54	0.58
CLIP ViT-B/32 (T2I)	0.46	0.46	0.51	0.46	0.47	0.47	0.46	0.53	0.49	0.49	0.47	0.47	0.53	0.54	0.50
CLIP ViT-L/14 (T2I)	0.48	0.44	0.42	0.52	0.46	0.48	0.44	0.43	0.49	0.46	0.49	0.47	0.49	0.51	0.49

Appendix

Table 6: Comparison of discriminative similarity-based methods and progress reward model judges under different noise levels. We report the performance across Real-World, Simulation, UMI, and Human settings, along with the average (AVG) for Noise levels of 0, 0.05, and 0.10.

	Real-World				Simulation			UMI	Human
Method	Agibot-World	Agilex	Droid	Galaxea R1Lite	Libero	RoboCasa	RoboTwin2.0	Pika	Egodex
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)	0.54	0.53	0.75	0.45	0.55	0.41	0.61	0.58	0.58
CLIP ViT-L/14 (I2I)	0.55	0.53	0.52	0.47	0.59	0.54	0.53	0.55	0.56
CLIP ViT-B/32 (T2I)	0.50	0.47	0.55	0.42	0.54	0.44	0.50	0.56	0.49
CLIP ViT-L/14 (T2I)	0.54	0.46	0.55	0.47	0.45	0.50	0.44	0.48	0.52

Appendix

Table 7: Performance comparison on the Small scale datasets.

	Real-World				Simulation			UMI	Human
Method	Agibot-World	Agilex	Droid	Galaxea R1Lite	Libero	RoboCasa	RoboTwin2.0	Pika	Egodex
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)	0.57	0.49	0.57	0.62	0.69	0.59	0.66	0.52	0.50
CLIP ViT-L/14 (I2I)	0.65	0.55	0.54	0.73	0.81	0.50	0.66	0.50	0.59
CLIP ViT-B/32 (T2I)	0.51	0.47	0.57	0.48	0.44	0.60	0.40	0.35	0.34
CLIP ViT-L/14 (T2I)	0.49	0.53	0.46	0.48	0.33	0.45	0.47	0.35	0.52

Appendix

Table 8: Performance comparison on the Medium scale datasets.

	Real-World				Simulation			UMI	Human
Method	Agibot-World	Agilex	Droid	Galaxea R1Lite	Libero	RoboCasa	RoboTwin2.0	Pika	Egodex
Discriminative Similarity-Based Methods
CLIP ViT-B/32 (I2I)	0.65	0.52	0.61	0.61	0.76	0.64	0.65	0.63	0.51
CLIP ViT-L/14 (I2I)	0.69	0.58	0.56	0.73	0.80	0.63	0.73	0.68	0.54
CLIP ViT-B/32 (T2I)	0.45	0.32	0.32	0.44	0.33	0.59	0.35	0.61	0.54
CLIP ViT-L/14 (T2I)	0.58	0.40	0.40	0.45	0.24	0.58	0.46	0.42	0.51

Appendix

Table 9: Performance comparison on the Large scale datasets.

Discriminative Similarity-Based Methods
Method	Human	Real-World	Simulation	UMI
CLIP ViT-B/32 (I2I)	0.54	0.57	0.61	0.58
CLIP ViT-L/14 (I2I)	0.56	0.58	0.64	0.58
CLIP ViT-B/32 (T2I)	0.48	0.46	0.46	0.52
CLIP ViT-L/14 (T2I)	0.52	0.49	0.44	0.43
General Foundation-Model Judges

Appendix

Table 10: Overall performance summary across different domains.

Model	Blocks Ranking RGB								Handover Block								Handover Mic
	Outcome Level					Process	Diagnosis		Outcome Level					Process	Diagnosis		Outcome Level					Process	Diagnosis
	MC				MP	PPL	CRA	STR	MC				MP	PPL	CRA	STR	MC				MP	PPL	CRA	STR
	@25	@50	@75	@100	MP	PPL	CRA	STR	@25	@50	@75	@100	MP	PPL	CRA	STR	@25	@50	@75	@100	MP	PPL	CRA	STR
ACT	84	44	22	2	49.93	11.67	8.99	59.74	86	60	44	42	66.35	47.98	9.6	65.49	100	100	94	74	96.79	72.33	4.08	44.14
DP	94	40	18	0	51.72	4.07	16.26	43.77	92	52	50	44	66.88	62.18	1.05	69.64	100	94	88	44	93.8	65.97	5.49	57.18
RDT	100	62	30	0	61.23	6.19	16.3	39.03	94	82	62	38	78.86	53.13	9.88	65.09	100	100	100	100	100	84.23	1.45	39.82

Appendix

Table 11: OPD auditing on RoboTwin 2.0. Performance of different policy models on three tasks. MC@25/50/75/100 denotes milestone coverage; MP/PPL/CRA/STR are averaged episode-level metrics.

Model	Hanging Mug								Place Bread Basket
	Outcome Level					Process	Diagnosis		Outcome Level					Process	Diagnosis
	MC				MP	PPL	CRA	STR	MC				MP	PPL	CRA	STR
	@25	@50	@75	@100	MP	PPL	CRA	STR	@25	@50	@75	@100	MP	PPL	CRA	STR
ACT	96	84	74	14	84.88	23.61	17.12	51.03	100	74	46	4	73.11	17.55	15.46	65.38
DP	100	100	98	14	86.4	30.82	18.72	57.03	100	94	74	16	87.55	21.33	16.88	47.96
RDT	98	96	92	20	88.5	48.4	11.35	65.84	100	100	78	8	90.4	16.57	22.68	37.1

Appendix

Table 12: OPD Auditing on RoboTwin 2.0: Task Set A. Results for Hanging Mug and Place Bread Basket (50 rollouts each).

Model	Place Bread Skillet								Place Can Basket
	Outcome Level					Process	Diagnosis		Outcome Level					Process	Diagnosis
	MC				MP	PPL	CRA	STR	MC				MP	PPL	CRA	STR
	@25	@50	@75	@100	MP	PPL	CRA	STR	@25	@50	@75	@100	MP	PPL	CRA	STR
ACT	100	80	34	8	68.66	34.19	7.99	69.11	98	88	74	4	83.84	30.62	9.06	69.9
DP	100	58	30	4	62.51	31.45	8.36	78.07	88	60	34	12	60.91	39.87	5.83	81.04
RDT	100	80	50	6	76.42	27.87	12.62	63.74	100	100	94	16	95.94	52.41	4.6	69.02

Appendix

Table 13: OPD Auditing on RoboTwin 2.0: Task Set B. Results for Place Bread Skillet and Place Can Basket (50 rollouts each).

Figure 6: Reachability profiles over all RoboTwin 2.0 tasks. For each task, we plot the fraction of rollouts that reach milestone thresholds at 25/50/75/100%. The curves localize where progress saturates along the horizon and expose last-mile bottlenecks when MC@75 is high but MC@100 drops sharply.

Figure 7: Success-conditioned execution quality over all tasks. We compute PPL, CRA, and STR on successful episodes only and visualize cross-policy trade-offs. This separates execution quality from success frequency and highlights policies that succeed efficiently versus those that succeed with higher correction cost or hesitation.

Figure 8: Failure-only OPD fingerprints over all tasks. We aggregate MP, PPL, CRA, and STR over failed episodes and normalize scores within each task across policy families. For CRA and STR, we flip the sign before normalization so that higher normalized scores consistently indicate more desirable behavior.

Figure 9: Real-World Task: Stack the wooden blocks. The reward curve demonstrates accurate tracking of manipulation errors. When the robot picks the wrong block or misplaces it (red arrows), the score drops, reflecting negative progress. The agent eventually recovers, reaching a success state (Progress $\approx 1.0$ ). The CRA of $4.59\%$ quantifies these regression events.

Figure 10: Real-World Task: Clean the table. This task involves semantic understanding of “messiness.” The model correctly penalizes the agent for moving a correctly placed drink and jumbling tissues, causing deep valleys in the progress curve. The successful recovery is captured by the final rise, though the efficiency ( PPL $58.49\%$ ) is impacted by these mistakes.

Figure 11: Human Task: Fold the trouser. Evaluating manipulation of deformable objects is challenging. Robo-Dopamine correctly identifies the regression when the trouser is accidentally unfolded (Step $\sim 18$ ) and when the folded trouser is misplaced (Step $\sim 60$ ). The curve aligns well with the visual state of the fabric.

Figure 12: Simulation Task: Blocks Ranking RGB. A long-horizon task ( $\sim 700$ steps). The initial phase is dominated by empty grasps and incorrect interactions, leading to a long period of stagnation (high STR of $16.22\%$ and flat curve). Once the correct sorting sequence begins (Step $\sim 500$ ), the reward signal increases steadily.

Figure 13: Simulation Task: Place bread into the basket. The model exhibits sensitivity to spatial relations. Moving the bread away from the basket (Step $\sim 75$ ) causes a dip in the score. The high PPL ( $75.06\%$ ) indicates that despite minor deviations, the overall trajectory was relatively efficient compared to other tasks.

Figure 14: Simulation Task: Hanging the mug. Fine-grained manipulation tracking. The curve captures the “redundant removal” of the mug (Step $\sim 450$ ) as a regression before the final successful placement. This granularity allows for the detection of suboptimal behavior even in successful trajectories.

Figure 15: RoboPulse qualitative examples for progress direction judging. Each panel shows the task, reference start/end frames, and a sampled BEFORE/AFTER pair from an episode. The arrow indicates the ground-truth progress direction. We report whether each judge predicts the correct direction. Robo-Dopamine is robust to appearance shifts and detects both improvement and regression, whereas CLIP can fail when visual similarity is misaligned with physical progress.

Figure 16: Prompt template for VLAC ( Zhai et al., 2025 ) on RoboPulse.

Figure 17: Prompt template for Gemini 3 Pro ( Google, 2025 ) , GPT-5.2 ( OpenAI, 2025 ) , Qwen3-VL ( Bai et al., 2025a ) pairwise progress judgment on RoboPulse. We provide the full system and user prompt, including task description, reference anchors, multiview BEFORE/AFTER observations, and the required output schema used for automated parsing.

Figure 18: Prompt template for RoboReward ( Lee et al., 2026 ) on RoboPulse.

Figure 19: Prompt template for GVL ( Ma et al., 2024 ) on RoboPulse.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 125 papers

PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment
Aug 14, 2026Yuyang Liu, Yanqing Shen, Ruike Chen +22Robotic Policy Evaluation Robot Manipulation Benchmarks
Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison
Mar 13, 2026David Snyder, Apurva Badithela, Nikolai Matni +4Robotic Policy Evaluation Sequential Hypothesis Testing
University of Pennsylvania · Princeton University · Toyota Research Institute (TRI)
Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation
Jan 26, 2026Mengyuan Liu, Juyi Sheng, Peiming Li +4Robotic Policy Evaluation Robotic Manipulation

Aug 14, 2026J/K move · Enter open · S save

PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment

Yuyang Liu, Yanqing Shen, Ruike Chen, Jifan Zhao+21

Robotic Policy Evaluation Robot Manipulation Benchmarks Reward Model Evaluation

Open paper AskPDF arXiv

Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.

Abstract

Robotic Policy Evaluation

Explore similar work

PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment

Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation