AnyRank: Reliable Anytime Policy Ranking via Sequential Testing

Apurva Badithela; David Snyder; Tenny Yin; Jade Sceats; George Pappas; Nikolai Matni; Masha Itkina; Haruki Nishimura

cs.ROOct 10, 2026

AnyRank: Reliable Anytime Policy Ranking via Sequential Testing

Authors: Apurva Badithela, David Snyder, Tenny Yin, Jade Sceats, George Pappas, Nikolai Matni, Masha Itkina, Haruki Nishimura,

Organizations: Princeton University · University of Pennsylvania · Toyota Research Institute

Abstract

Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of hardware trials. However, finite-sample statistical uncertainty can lead to erroneous conclusions, especially when ranking multiple policies using a fixed batch of evaluation data. We propose AnyRank: a sequential method for multi-policy ranking that (i) adaptively allocates trials to hard-to-separate policies, (ii) provides (partial) rankings at any stopping time while bounding the probability of an incorrect ranking, and (iii) optionally exploits prior information to prioritize hardware evaluations more efficiently. This prior information could be informed from expert intuition, a small set of held-out hardware evaluations, or from policy rollouts in a simulator or an action-conditioned video world model. Notably, the prior can be imperfect; while a strong prior can provide efficiency gains on hardware, any inaccuracies in prior policy rankings do not invalidate the statistical assurances of AnyRank. We achieve the aforementioned properties through novel theoretical contributions that combine sequential testing with multiple-hypothesis graphical testing approaches. Experiments in simulation and real-world policy evaluation benchmarks show that AnyRank requires up to 50% fewer hardware evaluations compared to existing statistically rigorous approaches, while guarding against the incorrect rankings produced by heuristic methods used by state-of-the-art benchmarks.

Figures & tables

Figure 1: Reliable Policy Ranking with AnyRank . Prior information is leveraged to form a weighted multi-hypothesis graph, which in turn informs an active evaluation protocol to minimize expected sample complexity. The resulting ranking provably maintains a pre-specified error rate.

Figure 2: Graph propagation and risk reallocation as hypotheses are decided at FWER bound of

\alpha^{*}=0.3

. This is a batch example, with all evaluations collected before graphical testing commences.

	Methods	RoboArena4 Sample complexity: 1200	RoboArena7 Sample complexity: 5106	RoboArena7 Sample complexity: 2450
Ours	AnyRank	$\textbf{A}>\textbf{F}>\textbf{G}$ , $\{\textbf{A}\textbf{E}\}$ , $\{\textbf{E}\textbf{F}\}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}>\textbf{F}>\textbf{G}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\}}>\textbf{F}>\textbf{G},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{D}\textbf{E}\textbf{F}\}}$
	${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$	$\textbf{A}>\textbf{F}>\textbf{G}$ , $\{\textbf{A}\textbf{E}\}$ , $\{\textbf{E}\textbf{F}\}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}>\textbf{F}>\textbf{G}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\}}>\textbf{F}>\textbf{G},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{D}\textbf{E}\textbf{F}\}}$
	${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{\infty}$	$\textbf{A}>\textbf{E}>\textbf{G}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{F}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{E}\textbf{F}\}}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{F}\}}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{F}\}}$
Baselines	Bonferroni	$\textbf{A}>\textbf{F}>\textbf{G}$ , $\{\textbf{A}\textbf{E}\}$ , $\{\textbf{E}\textbf{F}\}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}>\textbf{F}>\textbf{G}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\}}>\textbf{F}>\textbf{G},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{D}\textbf{E}\textbf{F}\}}$
	Weighted Bonferroni	$\textbf{A}>\textbf{F}>\textbf{G}$ , $\{\textbf{A}\textbf{E}\}$ , $\{\textbf{E}\textbf{F}\}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}>\textbf{F}>\textbf{G}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\}}>\textbf{F}>\textbf{G},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}},{\color[rgb]{0.7227,0.5273,0.043}\{\textbf{D}\textbf{E}\textbf{F}\}}$
	BT (Classical)	${\color[rgb]{1,0,0}\textbf{E}>\textbf{A}}>\textbf{F}>\textbf{G}$	$\textbf{A}>{\color[rgb]{1,0.5,0}\textbf{E}>\textbf{D}>\textbf{B}>\textbf{C}}>\textbf{F}>\textbf{G}$	$\textbf{A}>{\color[rgb]{1,0.5,0}\textbf{D}>\textbf{E}>\textbf{C}>\textbf{B}}>\textbf{F}>\textbf{G}$

Table 1: Policy rankings for RoboArena [ 2 ] using world model evaluations as a prior (

r=0.05

). In descending order of empirical mean progress, the policies are: A :

\pi_{0}

-FAST (

\hat{\mu}=0.459

), B : PG-FAST (

\hat{\mu}=0.430

), C : PG-FAST-Spec (

\hat{\mu}=0.428

), D : PG-FSQ (

\hat{\mu}=0.419

), E : PG-Diff (

\hat{\mu}=0.403

), F :

\pi_{0}

(

\hat{\mu}=0.350

), and G : PG-Bin (

\hat{\mu}=0.051

). Policies grouped in gold are not statistically distinguishable by the method. Red implies incorrect rankings, and Orange indicates temporally inconsistent rankings.

Panel of Figure 3 — Figure 3: Consistency : AnyRank returns self-consistent rankings as the total sample complexity increases. Time to decision indicates the number of evaluations required for each policy.

Figure 4: RoboArena4 with heldout evaluations prior.

Panel of Figure 5 — Figure 5: Pareto Curves for RoboArena. Efficacy of ranking methods by the number of atomic hypotheses rejected as well as the (negated) total sample complexity. The upper right of the plot corresponds to the most effective method: maximizing hypotheses rejected with low sample complexity indicates greater progress towards achieving a full policy ranking in the fewest hardware evaluations.

Panel of Figure 6 — Figure 6: Pareto Curves for TRI-LBM. AnyRank is the most efficient, rejecting the largest number of hypotheses in the fewest number of samples, comparable to the no prior variant ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ .

Figure 7: R4. AnyRank improves with good priors. AnyRank ( $\gamma=10$ ) achieves full policy ranking with fewer evaluations compared to ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ .

Figure 8: AnyRank ranking 15 RoboArena policies using all available data. Bradley–Terry rankings for these policies are detailed in Table 3 .

$N_{\text{max}}\downarrow$	# of policies $M\rightarrow$	5	10	15	20
500	AnyRank	1102 (9/10)	2480 (44/45)	6705 (98/105)	9075 (182/190)
500	Bonferroni	1170 (9/10)	2855 (43/45)	6742 (93/105)	9220 (175/190)
5,000	AnyRank	1554 (10/10)	3259 (45/45)	38k (102/105)	57k (185/190)
5,000	Bonferroni	1683 (10/10)	5191 (45/45)	$>42$ k (102/105)	$>63$ k (185/190)

Table 2: Sample complexity and fraction of hypotheses rejected on synthetic data at significance

\alpha^{*}=0.1

and

N_{\max}\in\{500,5000\}

per policy.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Prior policy performance quality for RoboArena policies using 20 heldout evaluations and 40 random DROID scenes for world model evaluation.

Rank	Letter	Policy	BT score	Empirical mean	# comparisons
1	J	Dream-Zero	$1.228\pm 0.309$	0.606	58
2	I	$\pi_{0.5}$	$0.639\pm 0.140$	0.476	564
3	A	$\pi_{0}$ -Fast	$0.502\pm 0.133$	0.455	739
4	K	FB-195k	$0.428\pm 0.737$	0.479	8
5	H	FB-525k	$0.306\pm 0.373$	0.380	33
6	D	PG-FSQ	$0.283\pm 0.132$	0.418	762

Appendix

Table 3: Bradley-Terry-Davidson preference rankings on all 15 RoboArena policies.

Figure 10: Prior policy performance quality for tasks fold towel and pick up carrot tasks considered in DROID-task . In the carrot task, we have poor policy ranking information from the prior while for the fold towel task, we have a good policy ranking.

	Methods	RoboArena4	RoboArena7
Ours	AnyRank	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}$
	${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}$
	${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{\infty}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}$ , ${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{B}\textbf{C}\textbf{D}\textbf{F}\}}$
Baselines	Bonferroni	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$	$\textbf{A}>\textbf{E}>\textbf{F}>\textbf{G}$ , $\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\}$ , $\{\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}$
Baselines	Weighted Bonferroni	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{E}\}}>\textbf{F}>\textbf{G}$	${\color[rgb]{0.7227,0.5273,0.043}\{\textbf{A}\textbf{B}\textbf{C}\textbf{D}\textbf{E}\}}>\textbf{F}>\textbf{G}$

Appendix

Table 4: Ranking DROID policies from RoboArena [ 2 ] with all available evaluations using world model evaluations as a prior. Each capitalized alphabet represents a policy:

\pi_{0}

-FAST ( A ), PG-FAST

(\textbf{B})

, PG-FAST-Specialist

(\textbf{C})

, PG-FSQ

(\textbf{D})

, PG-Diff ( E ),

\pi_{0}

( F ), and PG-Bin ( G ). Policies in a group and highlighted in gold are not statistically distinguishable by the method.

Panel of Figure 11 — Figure 11: Pareto Curves for TRI-LBM under nominal evaluation environments.

Figure 12: Reliability against imperfect priors. AnyRank (at $\gamma=10$ ) requires slightly more trials (130) than ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ , which does not use any prior information.

Figure 13: Validating FWER Control with Synthetic Data. Setting HO-I : we demonstrate the improved efficiency of ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ over BT α on synthetic data. Here, there are five policies corresponding to 10 pairwise comparisons. Two policies share the same mean, resulting in an infeasible full ranking. We demonstrate here the time to a complete partial ranking (denoted by dashed lines) for each approach, as well as the time to full rankings (denoted by solid lines). Note that a full ranking in this context is equivalent to a Type-1 error, as one of the atomic comparisons [between the policies with equal means] should never decide. We observe that ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ is able to better utilize the risk budget $\alpha=0.05$ and active sampling in order to more quickly arrive at complete partial rankings.

Panel of Figure 14 — Figure 14: Demonstrating Bootstrap BT Inconsistency. We demonstrate by counterexample (constructed in Section F.2.1 ) the inconsistency of Bradley-Terry approaches in general, even in the presence of robust bootstrapping corrections.

Figure 15: Testing Statistical Power with Synthetic Data. Setting HE-F : we demonstrate the improved efficiency of ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ over BT α on synthetic data. Here, there are five policies corresponding to 10 pairwise comparisons. The policies have unequally spaced means, resulting in heterogeneity in time to feasible full ranking. We demonstrate here the time to $k$ -hypothesis partial rankings (dashed lines) for each approach, as well as the empirical time to full ranking (solid lines). We observe that ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ is able to better utilize the risk budget $\alpha=0.05$ and active sampling in order to more quickly arrive at partial and complete rankings across the spectrum of sample complexity, illustrated by the blue lines being above their red counterparts.

Figure 16: Empirical Performance is As Good as Preferences (Feasible Homogeneous).

Figure 17: Empirical Performance is As Good as Preferences (Feasible Inhomogeneous).

Figure 18: Empirical Performance is As Good as Preferences (Infeasible).

Figure 19: Testing Statistical Power with Synthetic Data. Setting HO-F : we demonstrate the improved efficiency of ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ over BT α on synthetic data. Here, there are five policies corresponding to 10 pairwise comparisons. The policies have equispaced means $\{0.1,0.3,0.5,0.7,0.9\}$ , resulting in an approximately homogeneous feasible full ranking. We demonstrate here the time to $k$ -hypothesis partial rankings (dashed lines) for each approach, as well as the empirical time to full ranking (solid lines). We observe that ${\color[rgb]{1,0.5,0}\textsc{AnyRank}}_{0}$ is able to better utilize the risk budget $\alpha=0.05$ and active sampling in order to more quickly arrive at partial and complete rankings across the spectrum of sample complexity, illustrated by the blue lines being above their red counterparts.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 125 papers

Mar 13, 2026

Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts. This strong resource constraint in real-world testing necessitates both more informative performance measures and reliable and efficient evaluation procedures to properly assess model capabilities and benchmark progress in the field. This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice. Based on safe, anytime-valid inference (SAVI), our test procedure is sequential, allowing the evaluator to stop early when sufficient statistical evidence has accumulated to reach a decision at a pre-specified level of confidence. Unlike previous work developed for binary success, our unified approach addresses a wide range of informative metrics: from discrete partial credit task progress to continuous measures of episodic reward or trajectory smoothness, spanning both parametric and nonparametric comparison problems. Through extensive validation on simulated and real-world evaluation data, we demonstrate up to 70% reduction in evaluation burden compared to standard batch methods and up to 50% reduction compared to state-of-the-art sequential procedures designed for binary outcomes, with no loss of statistical rigor. Notably, our empirical results show that competing policies can be separated more quickly when using fine-grained task progress than binary success metrics.

David Snyder, Apurva Badithela, Nikolai Matni +4

University of Pennsylvania · Princeton University · Toyota Research Institute (TRI)

Robotic Policy Evaluation Sequential Hypothesis Testing

Ask PDF

Jul 16, 2026

Active Real-World Factor-Based Evaluation for Generalist Robot Policies

Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, so current practice is to use narrow test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy's performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We conduct 2331 real-world evaluations across 3 tasks with 3 factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing.

Andrew Liao, Hanchen Cui, Karthik Desingh +1

University of Minnesota Twin Cities

Robotic Policy Evaluation Sequential Experimental Design Robotic Manipulation Robot Policy Generalization

Ask PDF

Mar 24, 2026

Efficient Benchmarking of AI Agents

Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, we find that absolute score prediction degrades under this shift, while rank-order prediction remains stable. Exploiting this asymmetry, we propose a simple optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates (30-70%). This mid-range difficulty filter, motivated by Item Response Theory, reduces the number of evaluation tasks by 44-70% while maintaining high rank fidelity under scaffold and temporal shifts. It provides more reliable rankings than random sampling, which exhibits high variance across seeds, and outperforms greedy task selection under distribution shift. These results suggest that reliable leaderboard ranking does not require full-benchmark evaluation.

Franck Ndzomga

AI Agent Evaluation AI Agent Benchmarks Automated Agent Evaluation Benchmark Construction AI Agent Reliability

Ask PDF

Abstract

Robotic Policy Evaluation

Explore similar work

Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

Active Real-World Factor-Based Evaluation for Generalist Robot Policies

Efficient Benchmarking of AI Agents