When a single policy is trained in parallel across multiple environments of the same task, such as procedurally generated levels, randomized dynamics, or curricula, implementations commonly use one critic across all sampled environments. Yet different environments can assign different expected returns to the same input visible to the critic. A critic without environment information must then reconcile distinct value targets, systematically shifting the sampled advantages within individual environments. Using illustrative bandit models with multiple environments and a common optimal arm, we characterize how this value mismatch redistributes sampled policy updates, reinforcing unhelpful actions while attenuating or even reversing useful ones. The oracle processes using no baseline, the shared value, or the value specific to the sampled environment have the same mean logit update at a fixed policy and converge to the same optimal policy, yet their realized learning paths can differ sharply. The analysis motivates a minimal intervention: give only a logged environment index to the critic so that it can separate the value targets. Controlled CartPole and MuJoCo experiments expose the predicted shifted values, advantages, and performance gaps. In the more complex BipedalWalker and Procgen settings, the same intervention yields more stable learning and higher returns. Across all 16 Procgen games, the multihead conditional critic improves aggregate normalized return on 600 unseen levels per game by 40.8%. In conclusion, the theory identifies value mismatch as a direct mechanism through which critic sharing can degrade stochastic learning dynamics, not captured by scalar estimator variance alone, and the experiments show that conditioning on an index is broadly effective in parallel reinforcement learning.
Figures & tables
Figure 2: A sample trajectory of Algorithm 1 under the shared and the conditional value baseline ( qE=qH=21 , rE=(1,0.8,0.6) , rH=(0.4,0.2,0) , η=0.5 ): the conditional path increases at every update after an early transient, while the shared path keeps stepping backward.
Figure 3: A minimal tabular example with learned critics ( 2,000 seeds, η=1 , critic learning rate β=0.1 , common random numbers; logits and both critics initialized at zero, so the initial policy is uniform). (A) Two environments E and H, drawn 50% each, share exactly one state s† ; the same actions lead to endpoints worth (1,0.9) in E and (0.2,0) in H , so a1 is optimal in both. (B) The conditional critic means track each environment; the shared critic mean approaches their average ( →0.6 ). (C,D) Mean advantages in E and H : sharing promotes E ’s wrong action ( +0.3 ) and suppresses H ’s correct action ( −0.4 ), while conditioning gives the correct signs. (E) πt(a1∣s†) , mean ± 1 s.d.: the conditional baseline dominates the shared one throughout training.
Figure 4: CartPole with two gravities. (a–c) From the same initial state and the same push ( F=10 ), 0.3 s of dynamics tilts the pole to 37.7∘ under g=10 and 50.7∘ under g=50 : one observation, two futures. (d) Learned value of the initial state over training: the multihead critic means separate by gravity (blue: g=10 above, green: g=50 below), while the shared critic mean (red) settles between them — a signature consistent with the value mismatch mechanism. Curves in (d) show mean ± 1 s.d. over 20 seeds.
Figure 5: Controlled identification of the CartPole mechanism, mean ± 1 s.e. across 20 seeds. The top row uses two labeled levels with identical dynamics ( g=10 ); the bottom row uses g∈{10,50} . Columns report mean return, value loss for levels 0 and 1 , and the mean sampled raw GAE advantage for levels 0 and 1 . With identical dynamics, all variants behave similarly. Under heterogeneous gravity, the multihead critic initially pays a larger fitting cost on the hard level, but its hard level advantage recenters as the loss falls and its return subsequently approaches 200 . Routing every sample to the same head (level id ≡0 ) closely tracks the shared critic, while a learned scalar bias for each level provides only a partial correction.
evaluation return (600 unseen levels)
Game
shared
FiLM
multihead
+ PopArt
Bigfish
1.87 ± 0.17
2.88 ± 0.57
2.99 ± 0.61
3.56 ± 1.27
Bossfight
4.44 ± 2.56
7.44 ± 0.51
8.97 ± 0.29
8.11 ± 0.82
Caveflyer
1.08 ± 0.16
1.90 ± 0.23
2.65 ± 0.43
1.87 ± 0.31
Chaser
0.95 ± 0.26
0.94 ± 0.19
1.32 ± 0.35
0.75 ± 0.17
Climber
1.40 ± 0.19
1.74 ± 0.27
2.75 ± 0.22
2.13 ± 0.27
Table 1: Procgen, 16 games: final evaluation return on 600 unseen levels ( 25 M steps; mean ± 1 s.d. over 10 seeds per method). Normalized returns per run are computed by dividing the average test return per run for each environment by the corresponding average test return of the shared critic baseline over all runs. Bold = best in the row.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Environment
obs / act
levels
shared
FiLM
multihead
CartPole
4 / 2
2
4,675
4,933 ( +5.5% )
4,740 ( +1.4% )
BipedalWalker
24 / 4
100
11,849
15,625 ( +31.9% )
18,284 ( +54.3% )
Walker2d
17 / 6
10
11,085
11,765 ( +6.1% )
11,670 ( +5.3% )
Hopper
11 / 3
10
10,119
10,799 ( +6.7% )
10,704 ( +5.8% )
HalfCheetah
17 / 6
10
11,085
11,765 ( +6.1% )
11,670 ( +5.3% )
Procgen
642×3 / 15
200
626,256
638,160 ( +1.9% )
677,399 ( +8.2% )
Appendix
Table 2: Total trainable actor and critic parameter counts for the reference combinations of architecture and benchmark. Parentheses report increases relative to the shared model. FiLM retains the shared value head and adds a level embedding table and two affine generators; multihead replaces the shared value head by one head per level. The CartPole FiLM reference uses the minimal embedding dimension d=1 , and the Procgen counts use the large PLR CNN ( Jiang et al., 2021 ) . PopArt has the same trainable count as multihead. The table includes parameterizations not used in the main comparisons; the coverage paragraph below states which variants were evaluated.
Parameter
Value
Algorithm
PPO ( Schulman et al., 2017 )
Parallel environments
16
Rollout length
128
Total environment steps
2,048,000 (1,000 updates)
PPO epochs / minibatches
4 / 4
Clip parameter
0.2
Appendix
Table 3: CartPole mechanism study hyperparameters ( 20 seeds per variant and setting).
Parameter
Value
Algorithm
PPO
Levels
10
Environments per level
2 (20 parallel)
Rollout length
256 (5,120 transitions per update)
Total environment steps
10,000,000
PPO epochs / minibatches
10 / 8
Appendix
Table 4: MuJoCo suite hyperparameters ( 30 seeds per method and body).
Parameter
Value
Algorithm
PPO
Parallel environments
100 (one per terrain)
Rollout length
256
Total environment steps
60,000,000
PPO epochs / minibatches
10 / 8
Clip parameter
0.2
Appendix
Table 5: BipedalWalker hyperparameters ( 10 seeds per method).
Parameter
Value
Parallel environments
200 (one per pinned level)
Rollout length
256 steps per environment
Batch size per update
51,200
PPO epochs / minibatches
3 / 8
Clip parameter
0.2
Learning rate
5×10−4 (Adam, ϵ=10−5 )
Appendix
Table 6: Procgen PPO hyperparameters (per game and method).
Final evaluation return
Game
shared
FiLM
multihead
+ PopArt
Bigfish
5.89 ± 0.64
6.77 ± 0.58
7.36 ± 0.69
7.44 ± 1.86
Bossfight
5.11 ± 3.56
8.86 ± 0.27
10.13 ± 0.24
9.11 ± 0.59
Caveflyer
3.11 ± 0.17
3.83 ± 0.21
4.18 ± 0.21
3.03 ± 0.39
Chaser
1.27 ± 0.21
1.43 ± 0.17
1.96 ± 0.15
1.33 ± 0.16
Climber
3.87 ± 0.27
4.39 ± 0.15
5.29 ± 0.18
3.51 ± 0.34
Appendix
Table 7: Procgen, 16 games: final evaluation return on the 200 pinned training levels ( 25 M steps; mean ± 1 s.d. over 10 seeds per method). Normalized training returns per run are computed by dividing the average test return per run for each environment by the corresponding average test return of the shared critic baseline over all runs. Bold = best in the row.
Figure 14: Value loss diagnostics beyond CartPole. The clipped PPO value objective on BipedalWalker and the three MuJoCo tasks; each panel uses its own vertical scale. On BipedalWalker, FiLM reaches a slightly lower loss late in training and the early multihead excess closes to a comparable level. On HalfCheetah, Hopper, and Walker2d, the conditioned critics remain below the shared critic through most of training. Curves show mean ± 1 s.d. over 10 seeds on BipedalWalker and 30 seeds for each MuJoCo task.
Figure 15: Procgen value loss trajectories. The clipped PPO value objective over 25 M environment steps for all 16 games; each panel uses its own vertical scale. In late training, both the FiLM and multihead mean curves lie below the mean for the shared critic in 11 games. Curves show mean ± 1 s.d. over 10 seeds. The ordering varies by task, and conditioning does not increase the loss uniformly or create a systematic bottleneck in value fitting.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Cooperative MARL is commonly evaluated through cooperation discovery from random initialization, leaving open whether continued optimization can destabilize learned cooperation. Actor-critic comparisons can also conflate critic presence with value gradients entering shared actor representations. We study cooperation maintenance, defined as the survival of a behaviorally verified cooperative policy under continued training. We formulate maintenance as a right-censored event-time problem and compare matched warm starts: X0 allows value loss gradients to update shared actor features, X1 retains the critic while blocking those gradients, and X5 removes the learned critic as a critic-free reference. This isolates direct value-gradient access while controlling initialization, critic computation, and evaluation. Positive reward scaling preserves strategic preferences and equilibria while perturbing learning dynamics. Gradient audits confirm the intended routing pathways, and frozen-policy torso perturbations probe whether route-induced updates align with local cooperation boundaries. In confirmatory MinEx and CleanUp-lite experiments, higher scales selectively increase maintenance sensitivity in X0; X1 remains near the censoring ceiling, and X5 has no confirmed events in the tested settings. In CleanUp-lite, route-by-scale displacement is associated with reduced local cooperation margins; MinEx shows a weaker, optimizer-dependent effect. These results identify a conditional, scale-sensitive maintenance risk associated with direct value-gradient routing rather than a universal failure of critics.
In reinforcement learning, critics typically estimate absolute state values V(s), estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function Δ(si,sj)=V(si)−V(sj). We introduce a pairwise Bellman operator and prove it is a γ-contraction with a unique fixed point equal to the true value differences, derive well-posed 1-step, n-step and λ-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.
Marc Höftmann, Jan Robine, Stefan Harmeling
Department of Computer Science, Technical University of Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence
We introduce Mutual Reinforcement Learning, a framework for concurrent RL post-training in which heterogeneous LLM policies exchange typed experience while keeping separate parameters, objectives, and tokenizers. The framework combines a Shared Experience Exchange (SEE), Multi-Worker Resource Allocation (MWRA), and a Tokenizer Heterogeneity Layer (THL) that retokenizes text and aligns token-level traces across incompatible vocabularies. This substrate makes the experience-sharing design question operational across model families. We instantiate three controlled probes on top of GRPO: data-level rollout sharing via Peer Rollout Pooling (PRP), value-level advantage sharing via Cross-Policy GRPO Advantage Sharing (XGRPO), and outcome-level success transfer via Success-Gated Transfer (SGT). A contextual-bandit analysis characterizes their structural positions on a stability-support trade-off: PRP pays density-ratio variance and THL residual costs, XGRPO preserves on-policy actor support while changing scalar baselines, and SGT supplies a rescue-set score direction toward verified peer successes. In the evaluated regime, outcome-level sharing occupies the favorable point of this trade-off.
Xiaoze Liu, Dhananjay Ram, Yuting Zhang +3
1Purdue University · *Work done during an internship at Amazon. · 2AWS Agentic AI