What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor

Abhisek Keshari

cs.LGSep 26, 2026

What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor

Authors: Abhisek Keshari

Organizations: Independent researcher

Abstract

A generalization gap in reinforcement learning, return on training levels minus return on held-out levels, is usually reported without a reference point. We argue that it should be read against a measured random floor: the return of a uniform-random policy on the same levels under the same evaluation harness. On eight ProcGen environments with PPO at a compute-limited budget (8M steps, 16 parallel environments; three games extended to 25M), the floor changes what standard numbers mean. The test-time action rule decides which policy is measured: in miner, the sampled policy scores 5.1x the floor on held-out levels while its argmax scores below it in every run, and greedy evaluation places two environments significantly below the floor. Used as a convergence diagnostic, raw policy entropy flags six of eight environments, but 32-66% of that entropy lies on actions with identical effects; against the floor, five of eight sampled policies are clearly above it on held-out levels and heist's is not distinguishable from it. An audit of twelve ProcGen codebases finds that nine sample test-time actions with no explicit choice at the evaluation call site. We recommend that every reported gap state its action rule, seed its evaluation and specify its tests before analysis, and report the floor on both level sets.

Figures & tables

Codebase	Venue	Stoch.	In-loop
VSOP-3D ( Jesson & Jiang, 2024 )	arXiv’24	✓	✓
VSOP ( Jesson et al., 2024 )	ICML’24	✓	✓
ProcGen ( Cobbe et al., 2020 )	ICML’20	✓	✓
mixreg ( Wang et al., 2020 )	NeurIPS’20	✓	✓
RAD ( Laskin et al., 2020 )	NeurIPS’20	✓	✓
IBAC-SNI ( Igl et al., 2019 )	NeurIPS’19	✓	✓

Table 1: Code audit of twelve prior ProcGen codebases. Stoch.: test-time actions of the policy-gradient agent are sampled. In-loop: test returns come from a held-out rollout inside the training loop, reported as a running mean over episodes that span policy updates (in none of the six is the policy trained on that rollout). ∗ UCB-DrAC passes deterministic=False explicitly. † Impoola sets the rule by configuration (PPO samples, DQN greedy). ‡ EDE’s PPO baseline; EDE’s own agent is evaluated

\epsilon

-greedily. Commits and file/line evidence: Appendix G .

		Sampled (primary)					Greedy
Env	Floor	Return	$\Delta$ [95% CI]	Call	$z$	$k$ /6	Return	Call	$z$	$k$ /6
starpilot	$1.67\pm 0.11$	$18.35\pm 1.11$	$+16.68$ $[14.78,18.72]$	above	$+15.0$	6	$17.89\pm 1.40$	above	$+11.6$	6
fruitbot	$-2.66\pm 0.21$	$22.89\pm 0.39$	$+25.55$ $[24.72,26.34]$	above	$+58.0$	6	$25.79\pm 0.41$	above	$+61.7$	6
bigfish	$0.84\pm 0.06$	$2.82\pm 0.75$	$+1.98$ $[0.72,3.37]$	n.d.	$+2.6$	6	$3.19\pm 0.89$	n.d.	$+2.6$	6
coinrun	$3.28\pm 0.24$	$5.72\pm 0.31$	$+2.44$ $[1.73,3.17]$	above	$+6.3$	6	$3.57\pm 0.62$	n.d.	$+0.4$	3
miner	$1.17\pm 0.10$	$5.93\pm 0.30$	$+4.76$ $[4.19,5.33]$	above	$+14.9$	6	$0.60\pm 0.07$	below	$-4.6$	0

Table 2: Held-out return of the same 8M checkpoints under sampled (primary) and greedy action selection, against a uniform-random floor on identical level draws. Returns are mean

\pm

SEM over six runs (two encoders

\times

three seeds; each run averaged over three seeded draws of 128 episodes); the floor is mean

\pm

per-episode SEM over the same three draws.

\Delta

is sampled return minus floor with a 95% bootstrap CI. Call (primary, pre-specified): above/below if Holm-adjusted Welch

p<0.05

, equiv. if TOST passes, n.d. (not distinguishable) otherwise. Descriptive columns:

z

for a normal-approximation rule (

z>2

;

\Delta

over the combined SEM) and the number of runs above the floor,

k

/6.

Env	$H$	% max	Raw tier	Merged (state / game)	Share	Above-floor	Gap (s)	Gap (g)
starpilot	$1.794$	$66.2$	low	low / low	$37\%$	above	$-0.23$	$-1.26$
fruitbot	$2.078$	$76.7$	interm.	low / low	$66\%$	above	$+1.22$	$+0.88$
bigfish	$1.971$	$72.8$	low	low / low	$33\%$	n.d.	$+1.58$	$+1.43$
coinrun	$2.261$	$83.5$	interm.	interm. / low	$59\%$	above	$+0.02$	$+0.19$
miner	$2.087$	$77.1$	interm.	low / low	$66\%$	above	$+1.55$	$+0.61$
dodgeball	$2.427$	$89.6$	high	low / interm.	$32\%$	above	$+0.59$	$+0.46$

Table 3: Entropy screen and above-floor check at 8M, pooled over both encoders and three seeds.

H

: mean raw-action entropy on sampled-evaluation states, averaged over train and test levels (maximum

\ln 15=2.708

). Tiers: low (

<2.0

), intermediate (

[2.0,2.3]

), high (

>2.3

) residual entropy; merged tiers after summing the probabilities of functionally equivalent actions per state / per game, with cuts at the same fractions of the maximum. Share: fraction of raw entropy on equivalent actions. Above-floor: primary call of Table 2 . Gaps: pooled train

-

test return under sampled (s) and greedy (g) selection.

Figure 1: Raw-action policy entropy during training at 25M steps, from per-iteration training logs (sampled actions on training levels; mean over seeds per encoder, band

\pm 1

SEM;

n

in legend). Dashed lines: the screen’s 2.0 and 2.3 cuts; dotted:

\ln 15

Env	Encoder	Train	Test	Gap	Test	Gap	Tier
		Sampled (primary)			Greedy
starpilot	IMPALA	$31.04\pm 1.37$	$28.02\pm 1.55$	$3.02\pm 0.33$	$30.15\pm 1.97$	$2.55\pm 1.00$	low
starpilot	FPN	$28.92\pm 1.51$	$27.72\pm 1.73$	$1.20\pm 0.51$	$29.08\pm 1.97$	$2.03\pm 0.81$	low
fruitbot	IMPALA	$27.57\pm 0.47$	$21.39\pm 0.62$	$6.18\pm 0.75$	$23.79\pm 0.69$	$5.12\pm 0.82$	interm.
fruitbot	FPN	$26.19\pm 0.13$	$20.52\pm 0.36$	$5.66\pm 0.49$	$21.79\pm 0.34$	$5.80\pm 0.52$	interm.
heist	IMPALA	$7.14\pm 0.23$	$2.66\pm 0.26$	$4.47\pm 0.17$	$0.62\pm 0.15$	$3.31\pm 0.19$	high

Table 4: Final-checkpoint results at 25M steps, mean

\pm

SEM over seeds (starpilot

n{=}9

, heist

n{=}6

, fruitbot

n{=}3

per encoder); each run averaged over three seeded draws of 128 episodes per level set. Sampled is primary. Tier: raw-entropy screen on sampled-evaluation states.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Env	Budget	$n$	Sampled gap [95% CI]	Fisher $p$	Holm $p$	Stouffer $Z$	Greedy gap [95% CI]
starpilot	8M	6	$-0.23$ $[-0.79,0.48]$	$0.82$	$1.00$	$-0.45$	$-1.26$ $[-2.71,0.20]$
starpilot	25M	18	$+2.11$ $[1.39,2.77]$	$1.0\times 10^{-7}$	$1.0\times 10^{-7}$	$6.46$	$+2.29$ $[1.13,3.50]$
fruitbot	8M	6	$+1.22$ $[0.51,1.92]$	$4.5\times 10^{-4}$	$0.002$	$3.98$	$+0.88$ $[0.05,1.75]$
fruitbot	25M	6	$+5.92$ $[5.28,6.70]$	$<10^{-30}$	$<10^{-30}$	$19.53$	$+5.46$ $[4.60,6.19]$
heist	8M	6	$+1.54$ $[1.26,1.82]$	$2.3\times 10^{-25}$	$1.9\times 10^{-24}$	$10.95$	$+0.71$ $[0.44,0.98]$
heist	25M	12	$+3.94$ $[3.61,4.30]$	$<10^{-30}$	$<10^{-30}$	$39.74$	$+2.62$ $[2.14,3.10]$

Appendix

Table 5: Generalization gap (train

-

test) at both budgets for the three extended environments, sampled (primary) and greedy.

Env	$n$	Sampled gap [95% CI]	Fisher $p$	Holm $p$	$Z$	Greedy gap [95% CI]	Fisher $p$	Holm $p$
starpilot	6	$-0.23$ $[-0.79,0.48]$	$0.82$	$1.00$	$-0.45$	$-1.26$ $[-2.71,0.20]$	$0.001$	$0.003$
fruitbot	6	$+1.22$ $[0.51,1.92]$	$4.5\times 10^{-4}$	$0.002$	$3.98$	$+0.88$ $[0.05,1.75]$	$8.7\times 10^{-5}$	$2.6\times 10^{-4}$
bigfish	6	$+1.58$ $[0.92,2.42]$	$6.9\times 10^{-23}$	$4.2\times 10^{-22}$	$10.22$	$+1.43$ $[0.70,2.25]$	$1.3\times 10^{-19}$	$6.3\times 10^{-19}$
coinrun	6	$+0.02$ $[-0.22,0.23]$	$0.70$	$1.00$	$0.14$	$+0.19$ $[-0.03,0.40]$	$0.33$	$0.33$
miner	6	$+1.55$ $[1.05,1.97]$	$3.2\times 10^{-20}$	$1.6\times 10^{-19}$	$9.50$	$+0.61$ $[0.31,0.92]$	$2.0\times 10^{-24}$	$1.6\times 10^{-23}$
dodgeball	6	$+0.59$ $[0.37,0.78]$	$6.9\times 10^{-25}$	$4.8\times 10^{-24}$	$10.16$	$+0.46$ $[0.18,0.74]$	$1.3\times 10^{-22}$	$9.2\times 10^{-22}$

Appendix

Table 6: Generalization gap at 8M for all eight environments.

Env	Levels	Floor	Sampled	Greedy	Sampled call	Greedy call
starpilot (8M)	train	$1.40\pm 0.09$	$18.12\pm 0.92$	$16.64\pm 1.22$	above	above
	test	$1.67\pm 0.11$	$18.35\pm 1.11$	$17.89\pm 1.40$	above	above
fruitbot (8M)	train	$-2.43\pm 0.20$	$24.10\pm 0.44$	$26.68\pm 0.27$	above	above
	test	$-2.66\pm 0.21$	$22.89\pm 0.39$	$25.79\pm 0.41$	above	above
bigfish (8M)	train	$0.90\pm 0.07$	$4.40\pm 1.04$	$4.62\pm 1.19$	n.d.	n.d.
	test	$0.84\pm 0.06$	$2.82\pm 0.75$	$3.19\pm 0.89$	n.d.	n.d.

Appendix

Table 7: Uniform-random floor and pooled returns on both level sets and both budgets, same draws. Floor

\pm

per-episode SEM (384 episodes); returns

\pm

SEM over runs. Calls as in Table 2 .

	Raw						Merged (per state)
Env	$H$	$1.8$	$1.9$	$2.0$	$2.1$	$2.2$	$1.8$	$1.9$	$2.0$	$2.1$	$2.2$
starpilot	$1.794$	L	L	L	L	L	L	L	L	L	L
fruitbot	$2.078$	I	I	I	L	L	L	L	L	L	L
bigfish	$1.971$	I	I	L	L	L	L	L	L	L	L
coinrun	$2.261$	I	I	I	I	I	I	I	I	L	L
miner	$2.087$	I	I	I	L	L	L	L	L	L	L

Appendix

Table 8: Tier under a swept low/intermediate cut

T

(high cut fixed at 2.3; merged cuts at

T/\ln 15

and

2.3/\ln 15

of each state’s maximum). L/I/H = low/intermediate/high residual entropy. 8M, sampled-evaluation states.

Env	$k$ (global)	Raw $H$	Merged $H$ (state)	Merged $H$ (game)	Share	Greedy states	First 1024
starpilot	11	$1.794$	$1.124$	$1.217$	$37\%$	$1.851$	$1.785$
fruitbot	4	$2.077$	$0.697$	$0.791$	$66\%$	$2.123$	$2.006$
bigfish	9	$1.974$	$1.321$	$1.404$	$33\%$	$2.011$	$2.088$
coinrun	9	$2.261$	$0.928$	$1.523$	$59\%$	$2.263$	$2.030$
miner	5	$2.083$	$0.716$	$1.109$	$66\%$	$1.897$	$1.726$
dodgeball	10	$2.426$	$1.655$	$1.798$	$32\%$	$2.440$	$2.270$

Appendix

Table 9: Entropy (nats) on sampled-evaluation states at 8M (upper block) and 25M (lower block) before and after merging functionally equivalent actions; share is the fraction of raw entropy within classes, measured on the same states. Last two columns: raw entropy on greedy-evaluation states, and on the first 1024 sampled-evaluation states only.

Env	Floor	Greedy	Merged	Sampled	Merged call	Merged $-$ greedy	Disagree.
starpilot	$1.67$	$17.89\pm 1.40$	$14.70\pm 1.11$	$18.35\pm 1.11$	above	$-3.20$	$33\%$
fruitbot	$-2.66$	$25.79\pm 0.41$	$26.07\pm 0.67$	$22.89\pm 0.39$	above	$+0.28$	$21\%$
bigfish	$0.84$	$3.19\pm 0.89$	$2.67\pm 0.71$	$2.82\pm 0.75$	n.d.	$-0.52$	$43\%$
coinrun	$3.28$	$3.57\pm 0.62$	$2.71\pm 0.59$	$5.72\pm 0.31$	n.d.	$-0.86$	$61\%$
miner	$1.17$	$0.60\pm 0.07$	$0.47\pm 0.07$	$5.93\pm 0.30$	below	$-0.13$	$38\%$
dodgeball	$0.47$	$0.82\pm 0.05$	$0.13\pm 0.05$	$1.09\pm 0.13$	below	$-0.69$	$68\%$

Appendix

Table 10: Exploratory: held-out return under merged argmax at 8M, against plain argmax (greedy), sampling and the floor on the same draws (mean

\pm

SEM over six runs). Merged call: Holm within this analysis. Merged

-

greedy: paired difference over runs. Disagree.: share of steps on which merged argmax picks a different class from plain argmax.

			Sampled		Greedy	Uniform floor
Env	Budget	$R_{\min}$ , $R_{\max}$	Train	Test	Test	Train	Test
starpilot	8M	2.5, 64	0.25	0.26	0.25	$-$ 0.02	$-$ 0.01
fruitbot	8M	-1.5, 32.4	0.76	0.72	0.81	$-$ 0.03	$-$ 0.03
bigfish	8M	1, 40	0.09	0.05	0.06	$-$ 0.00	$-$ 0.00
coinrun	8M	5, 10	0.15	0.14	$-$ 0.29	$-$ 0.41	$-$ 0.34
miner	8M	1.5, 13	0.52	0.39	$-$ 0.08	$-$ 0.02	$-$ 0.03

Appendix

Table 11: Normalized scores (mean over runs; each run averaged over three seeded draws). Sampled is primary. The floor is the uniform-random policy on the same draws.

Codebase	Venue	Commit	S	L	Location (file:line)
UCB-DrAC	NeurIPS’21	1fbf2567cca1	✓	–	test.py:42 ( deterministic=False )
DAAC	ICML’21	2fe302029428	✓	–	test.py:31 ( act() , default False)
PLR	ICML’21	ccecf452ee33	✓	–	test.py:71 (default False, test.py:34 )
EDE	NeurIPS’23	89396f548aaa	✓	–	test.py:70 (PPO; default :30 ); train_rainbow.py:642 ( $\epsilon$ )
Impoola	RLC’25	1d71110451cc	✓	–	evaluation.py:30 ; ppo_training.py:47 (F), dqn_training.py:50 (T)
VSOP	ICML’24	7764611bdc98	✓	✓	ppo_procgen.py:368 ( probs.sample() ); :457 $\to$ :691

Appendix

Table 12: Commit hash and evidence location per codebase. S = stochastic test evaluation, L = in-loop test return. For L rows the location is where the held-out environment or worker is created, then where its running return is reported.

Hyperparameter	Value
Parallel environments	16 (8M); 256 (25M)
Rollout length	256
Total timesteps	8M; 25M
Learning rate	$5\times 10^{-4}$ (no annealing)
Discount factor $\gamma$	0.999
GAE $\lambda$	0.95

Appendix

Table 13: PPO hyperparameters, fixed across encoders and environments (CleanRL ProcGen hyperparameters, except the number of parallel environments, whose default is 64, and the absence of reward normalization and reward clipping).

		Sampled			Greedy
Env	Encoder	Train	Test	Gap	Train	Test	Gap
starpilot	IMPALA	$17.40\pm 0.30$	$17.23\pm 0.83$	$0.17\pm 0.64$	$16.57\pm 1.35$	$17.22\pm 0.82$	$-0.64\pm 1.52$
starpilot	FPN	$18.85\pm 1.90$	$19.47\pm 2.05$	$-0.62\pm 0.27$	$16.70\pm 2.37$	$18.57\pm 2.94$	$-1.87\pm 0.80$
fruitbot	IMPALA	$24.38\pm 0.08$	$22.76\pm 0.49$	$1.61\pm 0.54$	$26.81\pm 0.47$	$25.72\pm 0.67$	$1.09\pm 1.02$
fruitbot	FPN	$23.83\pm 0.95$	$23.01\pm 0.70$	$0.82\pm 0.58$	$26.55\pm 0.37$	$25.86\pm 0.62$	$0.68\pm 0.28$
bigfish	IMPALA	$2.10\pm 0.18$	$1.22\pm 0.15$	$0.88\pm 0.07$	$2.10\pm 0.15$	$1.25\pm 0.07$	$0.85\pm 0.07$

Appendix

Table 14: Final-checkpoint results at 8M per encoder, mean

\pm

SEM over three seeds, each run averaged over three seeded draws.

Budget	Rule	Train	Test	Gap
8M ( $n{=}6$ )	sampled	$4.42\pm 0.29$	$2.88\pm 0.21$	$1.54\pm 0.16$
8M ( $n{=}6$ )	greedy	$1.16\pm 0.14$	$0.45\pm 0.14$	$0.71\pm 0.16$
	uniform floor †	$3.54\pm 0.24$	$3.15\pm 0.24$	$+0.39$
25M ( $n{=}12$ )	sampled	$6.94\pm 0.14$	$3.00\pm 0.20$	$3.94\pm 0.19$
25M ( $n{=}12$ )	greedy	$3.55\pm 0.23$	$0.92\pm 0.13$	$2.62\pm 0.26$
	uniform floor †	$3.54\pm 0.24$	$3.15\pm 0.24$	$+0.39$

Appendix

Table 15: Heist at both budgets, the same checkpoints under both rules, pooled over encoders (mean

\pm

SEM over runs), with the uniform-random floor on the identical level draws. † For the floor rows, the Gap column is floor train

-

floor test.

Env	Enc.	As computed	Perm.	Analytic	A	C	B	D	PR
		$n{=}1024$			$2\times 2$ ( $n{=}512$ )
starpilot	IMPALA	0.020	0.001	0.001	0.008	0.013	0.003	0.002	1.4
starpilot	FPN	0.020	0.002	0.002	0.014	0.012	0.003	0.003	1.6
fruitbot	IMPALA	0.144	0.023	0.023	0.259	0.251	0.040	0.039	22.6
fruitbot	FPN	0.042	0.008	0.008	0.081	0.097	0.015	0.014	7.4

Appendix

Table 16: Controls on within-encoder train-versus-test CKA, 8M, mean over 3 seeds. “As computed”, permutation and analytic nulls at

n{=}1024

; the

2\times 2

block at

n{=}512

: A = timestep-aligned train vs. test, C = timestep-aligned train vs. train, B and D = the same with one side’s rows shuffled (mean of 200 permutations). PR: participation ratio.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error
Sep 13, 2026Hongliu CaoAI Agent Evaluation Automated Agent Evaluation
Amadeus SAS
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Oct 8, 2026Hexuan Deng, Zihao Yan, Xuebo Liu +8Reinforcement Learning RL for Language Model Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) · Beijing Zhongguancun Academy · XinzhuAI
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
Sep 8, 2026Jiacheng Xu, Feng Chen, Xiuneng Xu +1RL for Code Generation Policy Optimization
Nanyang Technological University, Singapore

Sep 13, 2026J/K move · Enter open · S save

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

Hongliu Cao

Amadeus SAS

AI Agent Evaluation Automated Agent Evaluation LLM Agent Reliability AI Agent Auditing

Open paper AskPDF arXiv

Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two

τ^2

-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.

Abstract

Reinforcement Learning

Explore similar work

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

GRPODropout: Less is More for Online Reinforcement Learning Rollouts

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation