PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment

Yuyang Liu; Yanqing Shen; Ruike Chen; Jifan Zhao; Yuxuan Tian; Yichi Zhang; Tianfeng Long; Zixuan Yin

cs.ROAug 14, 2026

PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment

Authors: Yuyang Liu, Yanqing Shen, Ruike Chen, Jifan Zhao, Yuxuan Tian, Yichi Zhang, Tianfeng Long, Zixuan Yin,

Abstract

Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.

Figures & tables

Figure 1: The evolution of robotic metrics. Evaluation progresses from coarse binary outcomes to holistic process understanding. PRM-as-a-Judge 1.0 introduces multi-dimensional process assessment using OPD metrics. Building upon this foundation, PRM-as-a-Judge 1.5 not only evaluates execution quality but also produces a Model Assessment Report for embodied models, including VLAs and WAMs.

Figure 2: Overall pipeline of PRM-as-a-Judge 1.5. After receiving the rollout video and related inputs, PRM-as-a-Judge 1.5 first uses the PRM to obtain a progress curve. Additionally, we update the metrics of version 1.0. Based on this progress curve and the designed metrics, users can obtain a comprehensive report. This report includes fine-grained evaluation and assessment results of the embodied models.

Metric	Pref.	Definition	Interpretation
Outcome–level Metrics
Milestone Coverage (MC@ $q$ )	$\uparrow$	The proportion of rollouts whose maximum progress reaches milestone $q$ .	Reveal where embodied models stop progressing.
Max Progress (MP)	$\uparrow$	The maximum progress value attained during the whole rollout.	The furthest progress reached.
Process–level Metrics
Path-weighted Progress Length (PPL)	$\uparrow$	Squared maximum progress divided by the total variation of the progress curve.	Measure how efficiently a rollout reaches its best progress.
Diagnosis- level Metrics

Table 1: Overview of the OPD Metric Suite. Pref. is Preference direction (

\uparrow

: higher is better, and vice versa).

{\dagger}

represents new metrics added in PRM-as-a-Judge 1.5.

Embodied Model	MC@25 $\uparrow$ 1	MC@50 $\uparrow$ 1	MC@75 $\uparrow$ 1	MP $\uparrow$ 2	SR $\uparrow$ 3	PPL $\uparrow$ 4	DRR $\uparrow$ 5	FNS $\uparrow$ 6	SQS $\uparrow$ 7	CRA $\downarrow$ 8	STR $\downarrow$ 9
$\pi_{0.5}$	85.29	60.59	38.82	59.90	17.06	28.93	100.00	45.39	82.08	5.22	21.81
InternVLA-A1	61.76	33.53	18.82	30.42	4.71	14.18	66.81	14.83	86.62	9.24	38.15
Xiaomi-Robotics-0	54.71	29.41	11.76	27.12	3.53	10.50	62.70	13.14	84.47	13.01	35.45
GalaxeaVLA (G0)	60.00	27.65	10.00	29.41	2.35	11.17	66.69	14.36	88.21	11.87	35.48
X-VLA	69.41	32.35	17.06	37.21	2.35	11.38	66.31	18.24	83.45	15.76	22.39
$\pi_{0}$	54.12	21.18	7.65	26.99	1.76	10.69	71.27	12.96	88.57	10.15	38.80

Table 2: Comparisons on RoboDojo-RealWorld.

\uparrow

represents a higher value, the better, the darker the color, and vice versa. Bold represents the best results, and underline represents the suboptimal results.

Embodied Model	MC@25 $\uparrow$ 1	MC@50 $\uparrow$ 1	MC@75 $\uparrow$ 1	MP $\uparrow$ 2	SR $\uparrow$ 3	PPL $\uparrow$ 4	DRR $\uparrow$ 5	FNS $\uparrow$ 6	SQS $\uparrow$ 7	CRA $\downarrow$ 8	STR $\downarrow$ 9
Hy-Embodied-0.5-VLA	67.01	46.53	28.82	47.68	11.46	18.81	83.06	19.99	77.77	9.75	35.80
$\pi_{0.5}$	73.61	45.83	27.43	46.60	8.68	16.88	87.46	21.03	78.76	10.18	30.36
Spatial Forcing (w/ $\pi_{0.5}$ )	70.83	44.44	29.17	44.88	8.68	16.98	93.47	20.15	74.19	11.45	29.59
X-WAM	73.96	41.67	23.26	41.19	7.29	16.03	80.51	19.38	77.86	10.89	30.88
GalaxeaVLA (G0)	51.05	31.82	17.13	26.14	6.29	10.15	74.13	11.55	77.07	13.62	38.52
X-VLA	62.59	41.61	23.08	39.84	6.29	13.82	65.52	17.64	81.74	11.17	28.41

Table 3: Comparisons on RoboDojo-Sim. Specific identification is the same as Table 2 .

Embodied Model	MC@25	MC@50	MC@75	MP	SR	PPL	DRR	FNS	SQS	CRA	STR	Avg.
$\pi_{0.5}$	2	2	3	2	2	3	2	1	6	3	3	1
Hy-Embodied-0.5-VLA	4	1	2	1	1	1	3	3	8	1	6	2
Spatial Forcing (w/ $\pi_{0.5}$ )	3	3	1	3	2	2	1	2	13	6	2	3
X-WAM	1	4	4	4	4	4	4	4	7	4	4	4
X-VLA	5	5	5	5	5	5	13	5	5	5	1	5
GigaWorld-Policy-0	6	6	6	6	11	6	14	6	9	2	11	6

Table 4: Rankings on RoboDojo-Sim. Specific values are shown in Table 3 .

Figure 3: Proportion of VLAs and WAMs ranked within the Top-3, Top-5, and Top-10 on RoboDojo-Sim. Percentages are normalized by the total number of evaluated models in each paradigm.

Figure 4: Model size versus overall ranking on RoboDojo-Sim. Each point represents one model. The dashed lines indicate the median model size and median average rank across metrics, dividing the models into four regions. The shaded regions highlight models whose size and performance lie on opposite sides of the two medians, and $n$ denotes the number of models in each region.

Figure 5: Performance comparison across different metrics in RoboDojo. The further out the indicator is, the better the performance. SQS is not compared in the real-world setting because most models have low SR for a reliable estimate; its values are set to a common radius.

Figure 6: Task-category process profiles on RoboDojo-Sim. Each point represents one model’s category-level mean, with tasks weighted equally, and each box summarizes the distribution. The panels report metrics for 24 Generalization , 8 Precision , 8 Long-Horizon , and 8 Open-Vocabulary task conditions.

Figure 7: Simulation-to-real changes in performance ranking for the shared model set. The three panels trace within-benchmark rank changes for SR, MP, and FNS across the nine shared models. Colors highlight

\pi_{0.5}

, InternVLA-A1, and StarVLA; all other models are shown in gray.

Figure 8: Simulation-to-real performance gaps in Max Progress (MP). Each cell reports the Real-minus-Sim MP difference in percentage points across the six shared tasks. Orange cells (negative values) indicate lower progress in Real, whereas blue cells (positive values) indicate higher progress in Real.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Task-semantic coverage of RoboPulse++.

Data source	Episodes	Tasks	Frames	Intervals
Real-World Data
RoboChallenge-Table30 ( Yakefu et al., 2025 )	79	25	3,448	882
RoboChallenge-Table30-V2 ( Yakefu et al., 2025 )	79	26	2,068	169
ViFailback ( Zeng et al., 2026 )	265	103	4,760	544
RoboFAC-Real ( Ye and others, 2025 )	16	6	261	33
Simulation Data

Appendix

Table 5: Data statistics of RoboPulse++ across different sources.

Judge Model	Rising			Falling			Overall
Judge Model	Precision	Recall	F1 Score	Precision	Recall	F1 Score	Macro-F1	Accuracy
Specialized PRMs
Robo-Dopamine (Forward)	0.92	0.92	0.92	0.79	0.50	0.61	0.77	0.84
Robo-Dopamine (Backward)	0.92	0.88	0.90	0.72	0.55	0.63	0.76	0.82
Robo-Dopamine (Incremental)	0.93	0.84	0.88	0.62	0.57	0.59	0.74	0.79
RoboMeter	0.89	0.86	0.87	0.47	0.54	0.50	0.69	0.80

Appendix

Table 6: Progress-direction performance of judge models on RoboPulse++ (700 episodes).

Failure type	# Intervals	Proportion
Interaction-relation failure	121	78.1%
Task-order failure	34	21.9%

Appendix

Table 7: Failure-type distribution of Falling intervals.

Evaluation subset	Robo-Dopamine (Forward)	RoboMeter
Rising	0.844	0.951
Falling	0.408	0.901
Overall	0.746	0.940

Appendix

Table 8: Accuracy of representative Pair Style and Sequence Style PRMs on context-dependent task patterns. Rising and Falling report class-conditioned accuracy, while Overall reports accuracy over all evaluated points.

Judge Model	Params. (B)	Peak Memory (GiB)	Runtime
Robo-Dopamine	4	11	29.3 min
Robo-Dopamine	8	19	31.7 min
VLAC	2	6	11.3 min
RoboMeter	4	12	46 s
TOPReward	8	17	12.3 min
PRIMO (w/ CoT)	7	17	19.6 h

Appendix

Table 9: Computational cost of representative progress judge models.

Setting	# Models	SR MAE (pp)	Mean Diff. (pp)	Spearman $\rho$
Simulation	16	1.57%	+1.29%	0.88
Real-World	9	1.32%	-0.35%	0.96

Appendix

Table 10: Model-level consistency between PRM-derived and benchmark-reported RoboDojo SR.

Figure 10: Drawdown recovery on the hang mugs task. The upper row shows the $\pi_{0.5}$ rollout, and the lower row shows the Xiaomi-Robotics-0 rollout. The annotated drawdown and recovery intervals indicate how much progress each policy regains after its maximum-drawdown trough.

Figure 11: Drawdown recovery on the sort nesting dolls by size task. The upper row shows the X-WAM rollout, and the lower row shows the Fast-WAM rollout. The annotated drawdown and recovery intervals distinguish sustained post-drawdown recovery from partial recovery.

Figure 12: Successful-trajectory quality on the stand up bottles task. The upper row shows the InternVLA-A1 rollout, and the lower row shows the $\pi_{0.5}$ rollout. The progress curves illustrate how SQS rewards efficient completion while penalizing regression and repeated corrective behavior.

Figure 13: Successful-trajectory quality on the put bottles into dustbin task. The upper row shows the X-WAM rollout, and the lower row shows the GalaxeaVLA rollout. The progress curves show how SQS distinguishes stable completion from a less efficient trajectory with intermediate regression.

Figure 14: Failure proximity on the plug in charger task. The upper row shows the $\pi_{0.5}$ rollout, and the lower row shows the $\pi_{0}$ rollout. Although both rollouts fail, FNS assigns a higher score to the trajectory that reaches the late-stage plug-in attempt than to the trajectory that repeatedly fails at grasping.

Figure 15: Failure proximity on the store laptop and headphones task. The upper row shows the AHA-WAM rollout, and the lower row shows the Fast-WAM rollout. FNS differentiates the near-complete trajectory from the rollout that remains stalled at the initial grasping stage.

Figure 16: Automatically Generated Evaluation Report. PRM-as-a-Judge 1.5 automatically generates a comprehensive report that summarizes model-level results and provides visual analyses of process metrics, success and failure profiles, failure progress, recovery, and individual trajectories.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 125 papers

PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing
Mar 23, 2026Yuheng Ji, Yuyang Liu, Huajie Tan +15Robotic Policy Evaluation Process Reward Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University +4
Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation
Jun 11, 2026Dayu Xia, Yue Shi, Yao Mu +7VLM Evaluation Robotic Manipulation
Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University +2
RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation
Oct 20, 2026Siyuan Ma, Yu Qi, Cheng Ye +8Vision-Language Model-as-a-Judge Video Generation Evaluation
The Chinese University of Hong Kong · Northeastern University · University of California, San Diego +3

Mar 23, 2026J/K move · Enter open · S save

PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing

Yuheng Ji, Yuyang Liu, Huajie Tan, Xuchuan Huang+14

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University+4

Robotic Policy Evaluation Process Reward Models Robot Manipulation Benchmarks Long-Horizon Robotic Manipulation

Open paper AskPDF arXiv

Current robotic evaluation is still largely dominated by binary success rates, which collapse rich execution processes into a single outcome and obscure critical qualities such as progress, efficiency, and stability. To address this limitation, we propose PRM-as-a-Judge, a dense evaluation paradigm that leverages Process Reward Models (PRMs) to audit policy execution directly from trajectory videos by estimating task progress from observation sequences. Central to this paradigm is the OPD (Outcome-Process-Diagnosis) metric system, which explicitly formalizes execution quality via a task-aligned progress potential. We characterize dense robotic evaluation through two axiomatic properties: macro-consistency, which requires additive and path-consistent aggregation, and micro-resolution, which requires sensitivity to fine-grained physical evolution. Under this formulation, potential-based PRM judges provide a natural instantiation of dense evaluation, with macro-consistency following directly from the induced scalar potential. We empirically validate the micro-resolution property using RoboPulse, a diagnostic benchmark specifically designed for probing micro-scale progress discrimination, where several trajectory-trained PRM judges outperform discriminative similarity-based methods and general-purpose foundation-model judges. Finally, leveraging PRM-as-a-Judge and the OPD metric system, we conduct a structured audit of mainstream policy paradigms across long-horizon tasks, revealing behavioral signatures and failure modes that are invisible to outcome-only metrics.

Abstract

Robotic Policy Evaluation

Explore similar work

PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing

Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation

RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation