When a single policy is trained in parallel across multiple environments of the same task, such as procedurally generated levels, randomized dynamics, or curricula, implementations commonly use one critic across all sampled environments. Yet different environments can assign different expected returns to the same input visible to the critic. A critic without environment information must then reconcile distinct value targets, systematically shifting the sampled advantages within individual environments. Using illustrative bandit models with multiple environments and a common optimal arm, we characterize how this value mismatch redistributes sampled policy updates, reinforcing unhelpful actions while attenuating or even reversing useful ones. The oracle processes using no baseline, the shared value, or the value specific to the sampled environment have the same mean logit update at a fixed policy and converge to the same optimal policy, yet their realized learning paths can differ sharply. The analysis motivates a minimal intervention: give only a logged environment index to the critic so that it can separate the value targets. Controlled CartPole and MuJoCo experiments expose the predicted shifted values, advantages, and performance gaps. In the more complex BipedalWalker and Procgen settings, the same intervention yields more stable learning and higher returns. Across all 16 Procgen games, the multihead conditional critic improves aggregate normalized return on 600 unseen levels per game by 40.8%. In conclusion, the theory identifies value mismatch as a direct mechanism through which critic sharing can degrade stochastic learning dynamics, not captured by scalar estimator variance alone, and the experiments show that conditioning on an index is broadly effective in parallel reinforcement learning.
Figures & tables
Figure 2: A sample trajectory of Algorithm 1 under the shared and the conditional value baseline ( qE=qH=21 , rE=(1,0.8,0.6) , rH=(0.4,0.2,0) , η=0.5 ): the conditional path increases at every update after an early transient, while the shared path keeps stepping backward.
Figure 3: A minimal tabular example with learned critics ( 2,000 seeds, η=1 , critic learning rate β=0.1 , common random numbers; logits and both critics initialized at zero, so the initial policy is uniform). (A) Two environments E and H, drawn 50% each, share exactly one state s† ; the same actions lead to endpoints worth (1,0.9) in E and (0.2,0) in H , so a1 is optimal in both. (B) The conditional critic means track each environment; the shared critic mean approaches their average ( →0.6 ). (C,D) Mean advantages in E and H : sharing promotes E ’s wrong action ( +0.3 ) and suppresses H ’s correct action ( −0.4 ), while conditioning gives the correct signs. (E) πt(a1∣s†) , mean ± 1 s.d.: the conditional baseline dominates the shared one throughout training.
Figure 4: CartPole with two gravities. (a–c) From the same initial state and the same push ( F=10 ), 0.3 s of dynamics tilts the pole to 37.7∘ under g=10 and 50.7∘ under g=50 : one observation, two futures. (d) Learned value of the initial state over training: the multihead critic means separate by gravity (blue: g=10 above, green: g=50 below), while the shared critic mean (red) settles between them — a signature consistent with the value mismatch mechanism. Curves in (d) show mean ± 1 s.d. over 20 seeds.
Figure 5: Controlled identification of the CartPole mechanism, mean ± 1 s.e. across 20 seeds. The top row uses two labeled levels with identical dynamics ( g=10 ); the bottom row uses g∈{10,50} . Columns report mean return, value loss for levels 0 and 1 , and the mean sampled raw GAE advantage for levels 0 and 1 . With identical dynamics, all variants behave similarly. Under heterogeneous gravity, the multihead critic initially pays a larger fitting cost on the hard level, but its hard level advantage recenters as the loss falls and its return subsequently approaches 200 . Routing every sample to the same head (level id ≡0 ) closely tracks the shared critic, while a learned scalar bias for each level provides only a partial correction.
evaluation return (600 unseen levels)
Game
shared
FiLM
multihead
+ PopArt
Bigfish
1.87 ± 0.17
2.88 ± 0.57
2.99 ± 0.61
3.56 ± 1.27
Bossfight
4.44 ± 2.56
7.44 ± 0.51
8.97 ± 0.29
8.11 ± 0.82
Caveflyer
1.08 ± 0.16
1.90 ± 0.23
2.65 ± 0.43
1.87 ± 0.31
Chaser
0.95 ± 0.26
0.94 ± 0.19
1.32 ± 0.35
0.75 ± 0.17
Climber
1.40 ± 0.19
1.74 ± 0.27
2.75 ± 0.22
2.13 ± 0.27
Table 1: Procgen, 16 games: final evaluation return on 600 unseen levels ( 25 M steps; mean ± 1 s.d. over 10 seeds per method). Normalized returns per run are computed by dividing the average test return per run for each environment by the corresponding average test return of the shared critic baseline over all runs. Bold = best in the row.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Environment
obs / act
levels
shared
FiLM
multihead
CartPole
4 / 2
2
4,675
4,933 ( +5.5% )
4,740 ( +1.4% )
BipedalWalker
24 / 4
100
11,849
15,625 ( +31.9% )
18,284 ( +54.3% )
Walker2d
17 / 6
10
11,085
11,765 ( +6.1% )
11,670 ( +5.3% )
Hopper
11 / 3
10
10,119
10,799 ( +6.7% )
10,704 ( +5.8% )
HalfCheetah
17 / 6
10
11,085
11,765 ( +6.1% )
11,670 ( +5.3% )
Procgen
642×3 / 15
200
626,256
638,160 ( +1.9% )
677,399 ( +8.2% )
Appendix
Table 2: Total trainable actor and critic parameter counts for the reference combinations of architecture and benchmark. Parentheses report increases relative to the shared model. FiLM retains the shared value head and adds a level embedding table and two affine generators; multihead replaces the shared value head by one head per level. The CartPole FiLM reference uses the minimal embedding dimension d=1 , and the Procgen counts use the large PLR CNN ( Jiang et al., 2021 ) . PopArt has the same trainable count as multihead. The table includes parameterizations not used in the main comparisons; the coverage paragraph below states which variants were evaluated.
Parameter
Value
Algorithm
PPO ( Schulman et al., 2017 )
Parallel environments
16
Rollout length
128
Total environment steps
2,048,000 (1,000 updates)
PPO epochs / minibatches
4 / 4
Clip parameter
0.2
Appendix
Table 3: CartPole mechanism study hyperparameters ( 20 seeds per variant and setting).
Parameter
Value
Algorithm
PPO
Levels
10
Environments per level
2 (20 parallel)
Rollout length
256 (5,120 transitions per update)
Total environment steps
10,000,000
PPO epochs / minibatches
10 / 8
Appendix
Table 4: MuJoCo suite hyperparameters ( 30 seeds per method and body).
Parameter
Value
Algorithm
PPO
Parallel environments
100 (one per terrain)
Rollout length
256
Total environment steps
60,000,000
PPO epochs / minibatches
10 / 8
Clip parameter
0.2
Appendix
Table 5: BipedalWalker hyperparameters ( 10 seeds per method).
Parameter
Value
Parallel environments
200 (one per pinned level)
Rollout length
256 steps per environment
Batch size per update
51,200
PPO epochs / minibatches
3 / 8
Clip parameter
0.2
Learning rate
5×10−4 (Adam, ϵ=10−5 )
Appendix
Table 6: Procgen PPO hyperparameters (per game and method).
Final evaluation return
Game
shared
FiLM
multihead
+ PopArt
Bigfish
5.89 ± 0.64
6.77 ± 0.58
7.36 ± 0.69
7.44 ± 1.86
Bossfight
5.11 ± 3.56
8.86 ± 0.27
10.13 ± 0.24
9.11 ± 0.59
Caveflyer
3.11 ± 0.17
3.83 ± 0.21
4.18 ± 0.21
3.03 ± 0.39
Chaser
1.27 ± 0.21
1.43 ± 0.17
1.96 ± 0.15
1.33 ± 0.16
Climber
3.87 ± 0.27
4.39 ± 0.15
5.29 ± 0.18
3.51 ± 0.34
Appendix
Table 7: Procgen, 16 games: final evaluation return on the 200 pinned training levels ( 25 M steps; mean ± 1 s.d. over 10 seeds per method). Normalized training returns per run are computed by dividing the average test return per run for each environment by the corresponding average test return of the shared critic baseline over all runs. Bold = best in the row.
Figure 14: Value loss diagnostics beyond CartPole. The clipped PPO value objective on BipedalWalker and the three MuJoCo tasks; each panel uses its own vertical scale. On BipedalWalker, FiLM reaches a slightly lower loss late in training and the early multihead excess closes to a comparable level. On HalfCheetah, Hopper, and Walker2d, the conditioned critics remain below the shared critic through most of training. Curves show mean ± 1 s.d. over 10 seeds on BipedalWalker and 30 seeds for each MuJoCo task.
Figure 15: Procgen value loss trajectories. The clipped PPO value objective over 25 M environment steps for all 16 games; each panel uses its own vertical scale. In late training, both the FiLM and multihead mean curves lie below the mean for the shared critic in 11 games. Curves show mean ± 1 s.d. over 10 seeds. The ordering varies by task, and conditioning does not increase the loss uniformly or create a systematic bottleneck in value fitting.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.