Reinforcement learning (RL) has enabled major advances in fields such as robotics and natural language processing. A key challenge in RL is measuring task complexity, which is essential for creating meaningful benchmarks and designing effective curricula. While there are numerous well-established metrics for assessing task complexity in tabular settings, relatively few exist in non-tabular domains. These include (i) Statistical analysis of the performance of random policies via Random Weight Guessing (RWG), and (ii) information-theoretic metrics Policy Information Capacity (PIC) and Policy-Optimal Information Capacity (POIC), which are reliant on RWG. In this paper, we evaluate these methods using progressively difficult robotic manipulation setups, with known relative complexity, with both dense and sparse reward formulations. Our empirical results reveal that measuring complexity is still nuanced. Specifically, under the same reward formulation, PIC suggests that a two-link robotic arm setup is easier than a single-link setup - which contradicts the robotic control and empirical RL perspective whereby the two-link setup is inherently more complex. Likewise, for the same setup, POIC estimates that tasks with sparse rewards are easier than those with dense rewards. Thus, we show that both PIC and POIC contradict typical understanding and empirical results from RL. These findings highlight the need to move beyond RWG-based metrics towards better metrics that can more reliably capture task complexity in non-tabular RL with our task framework as a starting point.
Figures & tables
Figure 1. Illustration of manipulators.
Figure 2. Learning curves of SAC algorithm across the six tasks. The left panel depicts agent performance in dense-reward settings, while the right panel is in sparse-reward settings. To accommodate wide and varying ranges of steps, results are plotted on a logarithmic scale to enhance interpretability. In the 2-link arm with sparse rewards, SAC results are presented with HER ( Andrychowicz et al., 2017 ) augmentation (SAC+HER) and without it. The results are obtained via evaluation of each task over 5 runs.
Figure 3. Performance distribution plots for the tasks: (a) 1-link (L=1.0m) , (b) 1-link (L=1.65m) , (c) 2-link arms with dense rewards, and (d) 2-link arm with sparse rewards. The left column shows a histogram of mean performances of the random policies ( Log-scale histogram of Mn ). The middle column depicts mean performance curves in black, i.e. mean performance Mn vs rank Rn . Moreover, all the cumulative rewards of the policies sa,n,e across the trials are represented by red dots (behind the black curve). The right column displays plots of standard deviation Vn vs mean performance Mn (often referred to as variance distribution ). The plots were made using 104 random policies.
Rewards
Arm [dim]
PIC (×10−3)
POIC (×10−3)
Dense
1-link [1.0]
4005±8.5
2.628±0.056
1-link [1.65]
4153±8.4
4.105±0.085
2-link [0.95,1.7]
4200±6.1
0.725±0.011
Sparse
1-link [1.0]
85.11±0.4
1.958±0.034
1-link [1.65]
71.21±0.2
1.197±0.031
2-link [0.95,1.7]
45.95±0.0
0.946±0.0079
Table 1. PIC and POIC values with N=104 samples (random policies). High PIC and POIC values correspond to easier tasks, while low values correspond to harder tasks.
Figure 4. 2D-scatter plots with Normalised scores (performance) computed using min-max scaling (Equation 11 ) over the returns of untrained random policies. The Normalised scores are plotted against PIC, POIC, variance of returns, along with entropies of optimality variable and cumulative reward (return) variable.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
learning rate
0.001
discount factor ( γ )
0.99
temperature coefficient ( α )
0.2
number of hidden layers (all networks)
2
number of hidden units per layer
256
number of samples per minibatch
64
Appendix
Table 2. SAC Hyperparameters.
Rewards
Tasks
Metric
Value
CI ( 95% )
Dense
1-link (L=1m)
PIC
4.005±0.0085
(3.979;4.012)
Dense
1-link (L=1m)
POIC
(2.628±0.056)⋅10−3
(2.520;2.738)⋅10−3
Dense
1-link (L=1.65m)
PIC
4.153±0.0084
(4.126;4.159)
Dense
1-link (L=1.65m)
POIC
(4.105±0.085)⋅10−3
(3.944;4.279)⋅10−3
Dense
2-link
PIC
4.205±0.0061
(4.195;4.219)
Dense
2-link
POIC
(0.722±0.011)⋅10−3
(0.701;0.746)⋅10−3
Appendix
Table 3. Complete PIC and POIC values based on bootstrapping.
Figure 5. Performance distribution plots for the tasks 1-link (L=1.0m) and 1-link (L=1.65m) arms with sparse rewards. The performance is normalised using min-max scaling to allow tasks to have the same range. Note that a in Sa,n,e represents the neural network architecture which is the same across all panels.
1-link 1.0
1-link 1.65
2-link 0.95,1.7
1-link 1.0
1-link 1.65
PIC
1-link 1.0
0.0 (1.0)
-
-
-
-
1 -link 1.65
-20.1 (2.3 ×10−7 )
0.0 (1.0)
-
-
-
2-link 0.95,1.7
-29.3 (2.9 ×10−8 )
-8.6 (2.6 ×10−5 )
0.0 (1.0)
-
-
1-link 1.0
948 (4.2 ×10−12 )
646 (2.8 ×10−11 )
618 (3.4 ×10−11 )
0.0 (1.0)
-
1-link 1.65
955 (5.7 ×10−12 )
649 (3.1 ×10−11 )
621 (3.7 ×10−11 )
25.4 (4.7 ×10−8 )
0.0 (1.0)
Appendix
Table 4. Statistical significance between PIC and POIC values using Welch’s t-test. Results are presented as: statistic (p-value) . D stands for dense-rewards and S stands for sparse-rewards.
Figure 6. Performance distribution plots for the 1-link ( L=1.65 m) arm with dense rewards across various policy network architectures and initialisation methods of weights. This shows how the random policies behave similarly across various settings. Note that a in Sa,n,e represents the neural network architecture which is the same across all panels.
Figure 7. 2-link arm with obstacle in the workspace.
Figure 8. Learning curves of SAC algorithm across tasks with dense rewards. To accommodate wide and varying ranges of steps, we draw the x-axis on a logarithmic scale to enhance interpretability. The results are obtained via evaluation of each task over 5 runs.
Figure 9. Performance distribution plots for the 2-link arm with dense rewards and an obstacle in the workspace. The left column shows a histogram of mean performances of the random policies ( Log-scale histogram of Mn ). The middle column depicts mean performance curves in black, i.e. mean performance Mn vs rank Rn . Moreover, all the cumulative rewards of the policies sa,n,e across the trials are represented by red dots (behind the black curve), where a represents the neural network architecture (same across all panels), n and e are as explained in the main text. The right column displays plots of standard deviation Vn vs mean performance Mn (often referred to as variance distribution ). The plots were made using 104 random policies.
Rewards
Arm [dim]
PIC (×10−3)
POIC (×10−3)
Dense
1-link [1.0]
4005±8.5
2.628±0.056
1-link [1.65]
4153±8.4
4.105±0.085
2-link [0.95,1.7]
4200±6.1
0.725±0.011
2-link [obstacle]
3671±14.3
21.31±0.33
Sparse
1-link [1.0]
85.11±0.4
1.958±0.034
1-link [1.65]
71.21±0.2
1.197±0.031
Appendix
Table 5. PIC and POIC values with N=104 samples (random policies). High PIC and POIC values correspond to easier tasks, while low values correspond to harder tasks.
1-link [1.0]
1-link [1.65]
2-link [0.95,1.7]
2-link [obstacle]
PIC
1-link [1.0]
0.0 (1.0)
-
-
-
1 -link [1.65]
-20.1 (2.3 ×10−7 )
0.0 (1.0)
-
-
2-link [0.95,1.7]
-29.3 (2.9 ×10−8 )
-8.6 (2.6 ×10−5 )
0.0 (1.0)
-
2-link [obstacle]
37.4 (5.4 ×10−9 )
48.5 (3.8 ×10−11 )
55.4 (1.3 ×10−11 )
0.0 (1.0)
POIC
Appendix
Table 6. Statistical significance between PIC and POIC values using Welch’s t-test. Results are presented as: statistic (p-value) . These are for only tasks with dense-rewards.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.