Reinforcement learning (RL) has enabled major advances in fields such as robotics and natural language processing. A key challenge in RL is measuring task complexity, which is essential for creating meaningful benchmarks and designing effective curricula. While there are numerous well-established metrics for assessing task complexity in tabular settings, relatively few exist in non-tabular domains. These include (i) Statistical analysis of the performance of random policies via Random Weight Guessing (RWG), and (ii) information-theoretic metrics Policy Information Capacity (PIC) and Policy-Optimal Information Capacity (POIC), which are reliant on RWG. In this paper, we evaluate these methods using progressively difficult robotic manipulation setups, with known relative complexity, with both dense and sparse reward formulations. Our empirical results reveal that measuring complexity is still nuanced. Specifically, under the same reward formulation, PIC suggests that a two-link robotic arm setup is easier than a single-link setup - which contradicts the robotic control and empirical RL perspective whereby the two-link setup is inherently more complex. Likewise, for the same setup, POIC estimates that tasks with sparse rewards are easier than those with dense rewards. Thus, we show that both PIC and POIC contradict typical understanding and empirical results from RL. These findings highlight the need to move beyond RWG-based metrics towards better metrics that can more reliably capture task complexity in non-tabular RL with our task framework as a starting point.
Figures & tables
Figure 1. Illustration of manipulators.
Figure 2. Learning curves of SAC algorithm across the six tasks. The left panel depicts agent performance in dense-reward settings, while the right panel is in sparse-reward settings. To accommodate wide and varying ranges of steps, results are plotted on a logarithmic scale to enhance interpretability. In the 2-link arm with sparse rewards, SAC results are presented with HER ( Andrychowicz et al., 2017 ) augmentation (SAC+HER) and without it. The results are obtained via evaluation of each task over 5 runs.
Figure 3. Performance distribution plots for the tasks: (a) 1-link (L=1.0m) , (b) 1-link (L=1.65m) , (c) 2-link arms with dense rewards, and (d) 2-link arm with sparse rewards. The left column shows a histogram of mean performances of the random policies ( Log-scale histogram of Mn ). The middle column depicts mean performance curves in black, i.e. mean performance Mn vs rank Rn . Moreover, all the cumulative rewards of the policies sa,n,e across the trials are represented by red dots (behind the black curve). The right column displays plots of standard deviation Vn vs mean performance Mn (often referred to as variance distribution ). The plots were made using 104 random policies.
Rewards
Arm [dim]
PIC (×10−3)
POIC (×10−3)
Dense
1-link [1.0]
4005±8.5
2.628±0.056
1-link [1.65]
4153±8.4
4.105±0.085
2-link [0.95,1.7]
4200±6.1
0.725±0.011
Sparse
1-link [1.0]
85.11±0.4
1.958±0.034
1-link [1.65]
71.21±0.2
1.197±0.031
2-link [0.95,1.7]
45.95±0.0
0.946±0.0079
Table 1. PIC and POIC values with N=104 samples (random policies). High PIC and POIC values correspond to easier tasks, while low values correspond to harder tasks.
Figure 4. 2D-scatter plots with Normalised scores (performance) computed using min-max scaling (Equation 11 ) over the returns of untrained random policies. The Normalised scores are plotted against PIC, POIC, variance of returns, along with entropies of optimality variable and cumulative reward (return) variable.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
learning rate
0.001
discount factor ( γ )
0.99
temperature coefficient ( α )
0.2
number of hidden layers (all networks)
2
number of hidden units per layer
256
number of samples per minibatch
64
Appendix
Table 2. SAC Hyperparameters.
Rewards
Tasks
Metric
Value
CI ( 95% )
Dense
1-link (L=1m)
PIC
4.005±0.0085
(3.979;4.012)
Dense
1-link (L=1m)
POIC
(2.628±0.056)⋅10−3
(2.520;2.738)⋅10−3
Dense
1-link (L=1.65m)
PIC
4.153±0.0084
(4.126;4.159)
Dense
1-link (L=1.65m)
POIC
(4.105±0.085)⋅10−3
(3.944;4.279)⋅10−3
Dense
2-link
PIC
4.205±0.0061
(4.195;4.219)
Dense
2-link
POIC
(0.722±0.011)⋅10−3
(0.701;0.746)⋅10−3
Appendix
Table 3. Complete PIC and POIC values based on bootstrapping.
Figure 5. Performance distribution plots for the tasks 1-link (L=1.0m) and 1-link (L=1.65m) arms with sparse rewards. The performance is normalised using min-max scaling to allow tasks to have the same range. Note that a in Sa,n,e represents the neural network architecture which is the same across all panels.
1-link 1.0
1-link 1.65
2-link 0.95,1.7
1-link 1.0
1-link 1.65
PIC
1-link 1.0
0.0 (1.0)
-
-
-
-
1 -link 1.65
-20.1 (2.3 ×10−7 )
0.0 (1.0)
-
-
-
2-link 0.95,1.7
-29.3 (2.9 ×10−8 )
-8.6 (2.6 ×10−5 )
0.0 (1.0)
-
-
1-link 1.0
948 (4.2 ×10−12 )
646 (2.8 ×10−11 )
618 (3.4 ×10−11 )
0.0 (1.0)
-
1-link 1.65
955 (5.7 ×10−12 )
649 (3.1 ×10−11 )
621 (3.7 ×10−11 )
25.4 (4.7 ×10−8 )
0.0 (1.0)
Appendix
Table 4. Statistical significance between PIC and POIC values using Welch’s t-test. Results are presented as: statistic (p-value) . D stands for dense-rewards and S stands for sparse-rewards.
Figure 6. Performance distribution plots for the 1-link ( L=1.65 m) arm with dense rewards across various policy network architectures and initialisation methods of weights. This shows how the random policies behave similarly across various settings. Note that a in Sa,n,e represents the neural network architecture which is the same across all panels.
Figure 7. 2-link arm with obstacle in the workspace.
Figure 8. Learning curves of SAC algorithm across tasks with dense rewards. To accommodate wide and varying ranges of steps, we draw the x-axis on a logarithmic scale to enhance interpretability. The results are obtained via evaluation of each task over 5 runs.
Figure 9. Performance distribution plots for the 2-link arm with dense rewards and an obstacle in the workspace. The left column shows a histogram of mean performances of the random policies ( Log-scale histogram of Mn ). The middle column depicts mean performance curves in black, i.e. mean performance Mn vs rank Rn . Moreover, all the cumulative rewards of the policies sa,n,e across the trials are represented by red dots (behind the black curve), where a represents the neural network architecture (same across all panels), n and e are as explained in the main text. The right column displays plots of standard deviation Vn vs mean performance Mn (often referred to as variance distribution ). The plots were made using 104 random policies.
Rewards
Arm [dim]
PIC (×10−3)
POIC (×10−3)
Dense
1-link [1.0]
4005±8.5
2.628±0.056
1-link [1.65]
4153±8.4
4.105±0.085
2-link [0.95,1.7]
4200±6.1
0.725±0.011
2-link [obstacle]
3671±14.3
21.31±0.33
Sparse
1-link [1.0]
85.11±0.4
1.958±0.034
1-link [1.65]
71.21±0.2
1.197±0.031
Appendix
Table 5. PIC and POIC values with N=104 samples (random policies). High PIC and POIC values correspond to easier tasks, while low values correspond to harder tasks.
1-link [1.0]
1-link [1.65]
2-link [0.95,1.7]
2-link [obstacle]
PIC
1-link [1.0]
0.0 (1.0)
-
-
-
1 -link [1.65]
-20.1 (2.3 ×10−7 )
0.0 (1.0)
-
-
2-link [0.95,1.7]
-29.3 (2.9 ×10−8 )
-8.6 (2.6 ×10−5 )
0.0 (1.0)
-
2-link [obstacle]
37.4 (5.4 ×10−9 )
48.5 (3.8 ×10−11 )
55.4 (1.3 ×10−11 )
0.0 (1.0)
POIC
Appendix
Table 6. Statistical significance between PIC and POIC values using Welch’s t-test. Results are presented as: statistic (p-value) . These are for only tasks with dense-rewards.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a high-fidelity simulation environment. We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies. Project website: https://research.nvidia.com/labs/srl/projects/robolab/.
Jenai Xuning Yang, Rishit Dagli, Alex Zook +5
NVIDIA · University of Toronto · Work done during internship at NVIDIA +1
Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts. This strong resource constraint in real-world testing necessitates both more informative performance measures and reliable and efficient evaluation procedures to properly assess model capabilities and benchmark progress in the field. This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice. Based on safe, anytime-valid inference (SAVI), our test procedure is sequential, allowing the evaluator to stop early when sufficient statistical evidence has accumulated to reach a decision at a pre-specified level of confidence. Unlike previous work developed for binary success, our unified approach addresses a wide range of informative metrics: from discrete partial credit task progress to continuous measures of episodic reward or trajectory smoothness, spanning both parametric and nonparametric comparison problems. Through extensive validation on simulated and real-world evaluation data, we demonstrate up to 70% reduction in evaluation burden compared to standard batch methods and up to 50% reduction compared to state-of-the-art sequential procedures designed for binary outcomes, with no loss of statistical rigor. Notably, our empirical results show that competing policies can be separated more quickly when using fine-grained task progress than binary success metrics.
David Snyder, Apurva Badithela, Nikolai Matni +4
University of Pennsylvania · Princeton University · Toyota Research Institute (TRI)
Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual instruction factors, e.g., reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, i.e., color ≥ object ≥ spatial ≥ verb ≥ size, with color dominant, and verb and size most under-grounded. We further show the diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines in simulation and on a real robot using half the demonstrations, thereby enabling more sample-efficient and generalizable policy learning.