Policy Gradient for Parallel State Entropy maximization (PGPSE) expands state-space coverage by training independently parameterized policies in replicated copies of the same environment. However, its pooled team-entropy score measures only collective exploration and cannot identify policies that contribute non-redundant coverage. We introduce Marginal Coverage Credit for PGPSE (MCC-PGPSE), which combines leave-one-policy-out coverage with state-owner specialization to estimate policy-specific credit. MCC-PGPSE preserves PGPSE's pooled objective and redistributes non-negative auxiliary intrinsic rewards according to these credits without changing their total mass. This redistribution is designed to discourage redundant visitation and promote complementary coverage. We evaluated MCC-PGPSE in controlled environments, seven public discrete-state benchmarks, and representative Room and Maze settings from the original PGPSE protocol. Across all tested settings, MCC-PGPSE produced positive final window gains in normalized team state entropy and state support over the Entropy baseline. Controlled-task comparisons and the fixed-suite public aggregate were significant, whereas five-seed original-protocol comparisons were directionally consistent. Ablations and credit alignment controls indicate that most gains arise from leave-one-policy-out coverage rather than non-uniform weighting, mismatched credit, or neural novelty alone. These results support contribution-conditioned auxiliary reward allocation as an interpretable approach to improving complementary coverage among parallel policies in discrete state spaces.
Figures & tables
Figure 1: Overview of MCC-PGPSE. At each policy update, the shared team-entropy return is combined with auxiliary rewards redistributed by leave-one-policy-out coverage and state-owner specialization. The redistribution preserves total auxiliary reward mass.
Figure 2: Controlled environments: open-field, bottleneck-memory, branching-specialization, and stochastic-loops.
Environment
Type
Evaluation focus
Role
Open-field
Controlled
Basic open-region coverage and implementation stability
Sanity check
Bottleneck-memory
Controlled
Remote coverage through constrained passageways
Primary task
Branching-specialization
Controlled
Complementary exploration across alternative branches
Primary task
Stochastic-loops
Controlled
Robustness under stochastic transition loops
Robustness
PGPSE Room/Maze
Original protocol
Deterministic Room and stochastic Maze with 43 reachable states
Protocol replication
FrozenLake-v1
Public benchmark
Discrete exploration under slippery transitions
Cross-environment validation
Table 3
Figure 3: Bottleneck-memory results across eight seeds. (A,B) Smoothed mean learning curves with standard-error bands and final-window insets. (C,D) Paired seed-level final-window estimates with mean 95% confidence intervals.
Figure 4: Branching-specialization results across eight seeds. (A,B) Smoothed mean learning curves with standard-error bands and final-window insets. (C,D) Paired seed-level final-window estimates with mean 95% confidence intervals.
Env.
Variant
Objective
Objective 95% CI
Support
Support 95% CI
BM
Entropy
0.4634
[0.4604, 0.4669]
12.75
[12.60, 12.94]
BM
Count
0.4642
[0.4612, 0.4678]
12.79
[12.63, 12.98]
BM
ICM
0.4638
[0.4610, 0.4671]
12.77
[12.63, 12.95]
BM
RND
0.4633
[0.4605, 0.4668]
12.76
[12.61, 12.94]
BM
Online
0.4668
[0.4638, 0.4707]
12.90
[12.76, 13.11]
BM
Triad
0.4655
[0.4627, 0.4691]
12.84
[12.71, 13.03]
Table 6
Figure 5: Behavioral diagnostics for Entropy, Triad, and MCC-PGPSE in bottleneck-memory (A–C) and branching-specialization (D–F): pairwise overlap, unique-state fraction, and credit-weight standard deviation. Curves show smoothed means with standard-error bands.
Environment / variant
Objective
Support
Pairwise overlap
Unique-state fraction
BM / Triad
0.4656
12.86
0.445
0.108
BM / Specialization-only
0.4682
12.97
0.444
0.108
BM / LOO-only
0.4866
13.83
0.427
0.113
BM / Entropy-marginal
0.4715
13.10
0.441
0.108
BM / MCC-PGPSE
0.4911
14.04
0.423
0.114
BM / MCC-on-entropy
0.5621
18.16
0.352
0.136
Table 8
Figure 6: Credit-alignment controls across eight seeds. Paired MCC-PGPSE-minus-control differences are shown for Objective (A) and Support (B); diamonds and whiskers denote means and bootstrap 95% confidence intervals.
Env.
Variant
Objective
Support
BM
MCC-PGPSE
0.4911
14.04
BM
Static credit
0.4652
12.83
BM
Reversed credit
0.4615
12.66
BM
Triad
0.4656
12.86
BS
MCC-PGPSE
0.5963
20.82
BS
Static credit
0.5898
20.25
Table 10
Env.
Control
Δ Objective
Δ Support
Paired rank-biserial
Holm-adjusted p
BM
Triad
+0.0255
+1.185
1.00 / 1.00
0.0234 / 0.0234
BM
Static credit
+0.0258
+1.214
1.00 / 1.00
0.0234 / 0.0234
BM
Reversed credit
+0.0295
+1.384
1.00 / 1.00
0.0234 / 0.0234
BS
Triad
+0.0064
+0.560
1.00 / 1.00
0.0234 / 0.0234
BS
Static credit
+0.0065
+0.568
1.00 / 1.00
0.0234 / 0.0234
BS
Reversed credit
+0.0079
+0.705
1.00 / 1.00
0.0234 / 0.0234
Table 11
Figure 7: Original-protocol validation. Paired final-window Entropy and MCC-PGPSE results are shown for five seeds in stochastic Maze (A,B) and deterministic Room (C,D); Mean rows include bootstrap 95% confidence intervals.
Figure 8: Stochastic-loops robustness across eight matched seeds. (A,B) Mean learning curves with standard-error bands and final-window insets. (C,D) Paired final-window estimates with mean 95% confidence intervals.
Figure 9: Public-benchmark validation. Paired final-window gains of MCC-PGPSE over Entropy and Triad are shown for Objective (A) and Support (B) across seven tasks; whiskers denote unadjusted bootstrap 95% confidence intervals.
The Computing & Mathematical Sciences Department, California Institute of Technology, Pasadena, CA. · Department of Computer Science, ETH Zurich, Switzerland. · 3ETH AI Center, ETH Zurich, Switzerland.