Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

David Snyder; Apurva Badithela; Nikolai Matni; George Pappas; Anirudha Majumdar; Masha Itkina; Haruki Nishimura

cs.ROMar 13, 2026

Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

Authors: David Snyder, Apurva Badithela, Nikolai Matni, George Pappas, Anirudha Majumdar, Masha Itkina, Haruki Nishimura

Organizations: University of Pennsylvania · Princeton University · Toyota Research Institute (TRI)

Abstract

Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts. This strong resource constraint in real-world testing necessitates both more informative performance measures and reliable and efficient evaluation procedures to properly assess model capabilities and benchmark progress in the field. This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice. Based on safe, anytime-valid inference (SAVI), our test procedure is sequential, allowing the evaluator to stop early when sufficient statistical evidence has accumulated to reach a decision at a pre-specified level of confidence. Unlike previous work developed for binary success, our unified approach addresses a wide range of informative metrics: from discrete partial credit task progress to continuous measures of episodic reward or trajectory smoothness, spanning both parametric and nonparametric comparison problems. Through extensive validation on simulated and real-world evaluation data, we demonstrate up to 70% reduction in evaluation burden compared to standard batch methods and up to 50% reduction compared to state-of-the-art sequential procedures designed for binary outcomes, with no loss of statistical rigor. Notably, our empirical results show that competing policies can be separated more quickly when using fine-grained task progress than binary success metrics.

Figures & tables

Fig. 1 : The evaluation context of the N-SCORE procedure. (Left) We consider the general problem of policy comparison, which arises out of counterfactual design decisions in the policy synthesis process. (Middle) Evaluation on hardware or in high-fidelity simulation is the gold standard to assess the effect of such changes, but is costly to collect. (Right) N-SCORE is a sequential evaluation procedure that is statistically rigorous, sample efficient, and generalizes to rich, diverse measures of robot performance.

	Bernoulli Data		Nonparametric Data
	TTD	Power	TTD	Power
STEP	95.1	0.953	–	–
SAVI	117.6	0.962	–	–
N-SCORE 2	117.9	0.965	–	–
N-SCORE ∞	122.3	0.958	206.8	0.889
WSR	224.8	0.592	247.3	0.840

TABLE I : Average time-to-decision (TTD) and empirical power for each method on simulated data. For Bernoulli data (left), TTD is averaged over all 35 alternative hypotheses, while Power is averaged only the 9 hardest alternative hypotheses corresponding to a gap of

0.1

(all other 26 alternatives have empirical power 1). For each alternative, metrics are averaged over 250 independent redraws; each redraw has

N=1000

. We observe similar statistical power for all methods except WSR, which lags substantially due to high variance in the Bernoulli regime. For nonparametric data (right), TTD and power are averaged over 3000 independent redraws (there is only one alternative). Only N-SCORE ∞ and WSR are valid in the nonparametric regime; the remaining methods cannot be adapted to this case. We observe that N-SCORE ∞ outperforms WSR by approximately 15% on average in terms of time-to-decision, while improving the empirical power by approximately 5 percentage points.

Fig. 2 : Policy performance comparisons on crowd-sourced real-world evaluations on the DROID [ 39 ] setup from RoboArena [ 6 ] . The violin plots represent empirical distributions of observed results. Policies with different letters are statistically distinguishable by the method. Policies are compared at a global error bound of $\alpha=0.05$ with a Bonferroni correction.

Method $\rightarrow$	Progress Metrics		Binary Success/Failure Metrics
Task $\downarrow$	N-SCORE ∞	WSR	STEP	N-SCORE 2	$\theta$ -SAVI	WSR
DumpVegetables	38	36	154	–	–	–
PutContainer	33	36	169	–	–	–
PutFruit	10	10	40	46	45	52
SeparateFruit	18	20	77	98	98	103
TurnUpsideDown	–	–	–	–	–	–

TABLE II : Time-to-decision for all simulation (top) and hardware (bottom) policy comparisons for evaluation context in [ 9 ] . If a decision is not reached, the entry is left blank; for the purpose of computing evaluation savings, any blank entry is counted at

N

trials. All simulation tasks utilize

N=200

; all hardware tasks utilize

N=50

. The total number of trials is the column sum multiplied by two (because there are two policies being evaluated, and each must be run). Thus, a column in the top half could require up to 2000 simulated trajectories; one in the bottom half could require up to 500 hardware evaluations. Several important observations are: (1) sequential evaluation saves significant evaluation effort over batch methods (right); (2) more informative partial credit metrics can provide even greater savings (left). In fact, for these evaluations, the savings are up to 50% on hardware and 70% in simulation.

Comparison $\rightarrow$	PPO vs. DDPG		SAC vs. TD3
Task $\downarrow$ Method $\rightarrow$	WSR	N-SCORE ∞	WSR	N-SCORE ∞
Ant-v4	27	21	14	13
HalfCheetah-v4	8	8	18	15
Hopper-v4	13	12	30	20
InvertedPendulum-v4	677	267	–	–
Humanoid-v4	42	40	96	89

TABLE III : Time-to-decision for selected reinforcement learning policy comparisons on Mujoco benchmarks. If a decision is not reached, the entry is left blank; for the purpose of computing evaluation savings, any blank entry is counted at

N

trials. All simulated tasks utilize

N=1000

. We observe similar behavior between N-SCORE and WSR on easier instances (with lower times-to-decision); however, in harder instances significant improvements can be observed. In aggregate, the hard instances dominate sample complexity, resulting in substantial savings in evaluation burden.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Comparison $\rightarrow$	PPO vs. TD3		PPO vs. DDPG		PPO vs. SAC		SAC vs. DDPG		TD3 vs. DDPG		SAC vs. TD3
Task $\downarrow$ Method $\rightarrow$	WSR	N-SCORE	WSR	N-SCORE	WSR	N-SCORE	WSR	N-SCORE	WSR	N-SCORE	WSR	N-SCORE
Ant-v4	69	60	27	21	13	12	10	9	21	19	14	13
HalfCheetah-v4	10	9	8	8	8	7	40	38	26	23	18	15
Hopper-v4	142	79	13	12	46	41	17	14	11	10	30	20
InvertedPendulum-v4	39	39	677	267	39	39	46	46	46	46	–	–
Humanoid-v4	8	7	42	40	8	7	8	8	8	8	96	89

Appendix

TABLE A.I : Empirical time-to-decision for all reinforcement learning pairwise policy comparisons on Mujoco benchmark tasks using WSR and N-SCORE. If a decision is not reached, the entry is left blank; for the purpose of computing evaluation savings, any blank entry is counted at

N

trials. All simulated tasks utilize

N=1000

. Thus, as there are two algorithms being compared pairwise, the total number of batch trials for each column is

14

k. Due to not being optimized for policy comparison, WSR requires more trials to reach a decision, uniformly across tasks and learning algorithms. Though performance is similar in the low-variance regime (small TTD, or ‘easy problems’), there are multiple instances where significant variation in time-to-decision arises due to this suboptimal multiplier selection, most notably with PPO vs DDPG on InvertedPendulum-v4.

Fig. A.1 : Number of trials required to separate RL policies on InvertedPendulum-v4. This is one of the most stark instances of disparity between N-SCORE and WSR, the latter of which requires significantly more evaluation effort (by nearly a factor of three) to reach a decision.

Panel of Figure 8 — Fig. A.2 : Violin plots and the number of samples required for N-SCORE and WSR on multi-policy comparison of RL policies on Mujoco benchmarks. Policies with different letters are statistically distinguishable by the method. Policies are compared at a global error bound of $\alpha=0.05$ with a Bonferroni correction. In all cases, N-SCORE results in the same comparison conclusions as WSR with fewer samples, demonstrating its broadly improved efficiency. These results also serve as an alternate visualization of the time-to-decision results in Table A.I .

Task	DDPG	TD3	PPO	SAC
Ant-v4	418.8	2477.5	1849.0	4901.8
HalfCheetah-v4	9932.2	7782.5	1332.2	11759.0
Hopper-v4	1106.4	3234.6	2797.4	2488.7
InvertedPendulum-v4	927.2	998.6	848.2	1000.0
Pusher-v4	-38.5	-35.7	-47.6	-32.4
Walker2d-v4	1732.5	3513.4	1679.8	4544.1

Appendix

TABLE A.II : Empirical mean episodic return on evaluation instances for Mujoco benchmark tasks. Note, in tandem with Figure A.2 , that the distributions over rewards near optimality vary significantly in shape and scale. This emphasizes the generality of nonparametric procedures and their broad applicability to creative or nonstandard metrics of robot performance or behavior.

Fig. A.3 : Continuous task progress scores enable faster policy comparison on RoboArena policy evaluation data . Violin plots and time-to-decision for multi-policy comparison from RoboArena evaluations. Left: Policy comparison under continuous scores with N-SCORE. Right: policy comparison under Bernoulli scores with N-SCORE, WSR, and STEP. In the Bernoulli comparison setting, none of the methods are able to distinguish all the policies. Even though STEP is maximally efficient on Bernoulli comparisons, N-SCORE on continuous progress requires fewer total trials to distinguish policies. We emphasize that the partial credit and binary metrics arise from the same rollouts of each policy; the partial credit thus reflects precisely a more informative ‘representation’ of the rollout for evaluation purposes. The reduced time-to-decision and increased power (separating all of the policies successfully) highlights the fundamental advantage of fine-grained task progress scores over sparse binary success rates for efficient policy comparison. As can be observed in Figure 2 in the main text, N-SCORE also significantly reduces the time-to-decision with respect to WSR. The former requires approximately 1420 trials, while the latter needs an additional 450, while failing to distinguish $\pi_{0}$ from PG-Diff .

Panel of Figure 11 — Fig. A.4 : Anytime-valid p-values vs. number of trials on multi-policy comparisons on RoboArena. A test procedure terminates when its p-value is at or below the error threshold $\alpha$ . As can be seen, it is faster to distinguish policies under task progress metrics as opposed to binary metrics when the policy performance gap is large. PG-Binning-DROID has an empirical mean that is more than 30 $\%$ points lower than $\pi_{0}$ -DROID, $\pi_{0}$ -FAST-DROID, and PG-Diff-DROID.

Panel of Figure 12 — Fig. A.5 : Anytime p-values of test procedures on multi-policy comparisons on RoboArena data with small gaps in empirical performance of the policies. A test procedure terminates when its p-value is at or below the error threshold $\alpha$ . In this regime, binary metrics often fail to obtain a significant result at all, whereas methods that exploit more informative measures can stop significantly more quickly.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 125 papers

Active Real-World Factor-Based Evaluation for Generalist Robot Policies
Jul 16, 2026Andrew Liao, Hanchen Cui, Karthik Desingh +1Robotic Policy Evaluation Sequential Experimental Design
University of Minnesota Twin Cities
R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models
Sep 3, 2026Yidi Wang, Feixiang Ruan, Ruoqu Chen +4Robotic Policy Evaluation
Sharpa · Nanjing University · Tongji University +1
AnyRank: Reliable Anytime Policy Ranking via Sequential Testing
Oct 10, 2026Apurva Badithela, David Snyder, Tenny Yin +6Robotic Policy Evaluation Adaptive Rank Allocation
Princeton University · University of Pennsylvania · Toyota Research Institute

Jul 16, 2026J/K move · Enter open · S save

Active Real-World Factor-Based Evaluation for Generalist Robot Policies

Andrew Liao, Hanchen Cui, Karthik Desingh, Aryan Deshwal

University of Minnesota Twin Cities

Robotic Policy Evaluation Sequential Experimental Design Robotic Manipulation Robot Policy Generalization

Open paper AskPDF arXiv

Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, so current practice is to use narrow test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy's performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We conduct 2331 real-world evaluations across 3 tasks with 3 factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing.

Abstract

Robotic Policy Evaluation

Explore similar work

Active Real-World Factor-Based Evaluation for Generalist Robot Policies

R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models

AnyRank: Reliable Anytime Policy Ranking via Sequential Testing