Evaluating Model-Free Policy Optimization in Masked-Action Environments via an Exact Blackjack Oracle

Kevin Song

cs.LGMar 19, 2026

Evaluating Model-Free Policy Optimization in Masked-Action Environments via an Exact Blackjack Oracle

Authors: Kevin Song

Organizations: Department of Biomedical Engineering The University of Alabama at Birmingham

Abstract

Infinite-shoe casino blackjack provides a rigorous, exactly verifiable benchmark for discrete stochastic control under dynamically masked actions. Under a fixed Vegas-style ruleset (S17, 3:2 payout, dealer peek, double on any two, double after split, resplit to four), an exact dynamic programming (DP) oracle was derived over 4,600 canonical decision cells. This oracle yielded ground-truth action values, optimal policy labels, and a theoretical expected value (EV) of -0.00161 per hand. To evaluate sample-efficient policy recovery, three model-free optimizers were trained via simulated interaction: masked REINFORCE with a per-cell exponential moving average baseline, simultaneous perturbation stochastic approximation (SPSA), and the cross-entropy method (CEM). REINFORCE was the most sample-efficient, achieving a 46.37% action-match rate and an EV of -0.04688 after 10^6 hands, outperforming CEM (39.46%, 7.5x10^6 evaluations) and SPSA (38.63%, 4.8x10^6 evaluations). However, all methods exhibited substantial cell-conditional regret, indicating persistent policy-level errors despite smooth reward convergence. This gap shows that tabular environments with severe state-visitation sparsity and dynamic action masking remain challenging, while aggregate reward curves can obscure critical local failures. As a negative control, it was proven and empirically confirmed that under i.i.d. draws without counting, optimal bet sizing collapses to the table minimum. In addition, larger wagers strictly increased volatility and ruin without improving expectation. These results highlight the need for exact oracles and negative controls to avoid mistaking stochastic variability for genuine algorithmic performance.

Figures & tables

Parameter	Setting
Shoe	Infinite (i.i.d. draws)
Blackjack payout	3:2
Dealer rule	S17 (stands soft 17)
Dealer peek	Yes
Double	Any two cards
Double after split (DAS)	Yes

Table 1: Frozen benchmark ruleset.

AMR = action-match rate vs oracle. $\bar{\Delta}$ = mean cell-conditional regret. $\max\Delta$ = worst-cell regret.
Method	Budget	Wall-clock	Final EV	Oracle EV	EV gap	AMR (%)	$\bar{\Delta}$	$\max\Delta$
PG (REINFORCE)	$10^{6}$ hands	$148.9\,\mathrm{s}$	$-0.04688$	$-0.00161$	$0.04526$	$46.37$	$0.25685$	$2.96262$
SPSA	$4.8{\times}10^{6}$ evals	$572.5\,\mathrm{s}$	$-0.41750$	$-0.00161$	$0.41589$	$38.63$	$0.30709$	$2.93918$
CEM	$7.5{\times}10^{6}$ evals	$819.8\,\mathrm{s}$	$-0.33030$	$-0.00161$	$0.32869$	$39.46$	$0.30244$	$2.96262$

Table 2: Optimizer comparison (wall-clock times reported for a single CPU core).

Figure 1: Smoothed EV per hand as a function of hands played. Policy gradient (REINFORCE) rapidly improved, representing the only optimizer to cross the 95% and 99% gap-closing thresholds relative to the oracle EV ( $-0.00161$ ).

Figure 2: Empirical bet-size sweep evaluating expected return. The optimizer correctly identified the minimum legal bet as mathematically optimal.

Strategy	Mean EV/hand	Mean net profit	Mean ruin events
Min bet ( $b=1$ )	$-0.0042$	$-21.13$	$0.0000$
Mid bet ( $b=50.5$ )	$-0.2513$	$-921.62$	$0.0333$
Max bet ( $b=100$ )	$-0.5337$	$+1{,}693.33$	$0.4333$
Proportional ( $1\%$ )	$-0.0334$	$-167.18$	$0.0000$

Table 3: Bet-strategy simulation (30 trials

\times

5,000 hands,

10{,}000

-unit starting bankroll, oracle policy). Bankrolls were reset to baseline upon ruin.

Method	Parameter	Value
PG	Learning rate $\eta$	$3\times 10^{-3}$
	Batch size $\|B\|$	$64$
	Entropy coef. $\lambda_{H,0}$	$0.05$
	Entropy anneal	$0.99995$ per hand
	Baseline EMA $\alpha_{b}$	$0.02$
	Adam $(\beta_{1},\beta_{2})$	$(0.9,\,0.999)$

Table S1 : Hyperparameters used in all experiments.

Ruleset	EV	$\Delta$ vs. S17
S17 (benchmark)	$-0.00476$	N/A
H17	$-0.00692$	$-0.00216$
S17 + Surrender	$-0.00398$	$+0.00078$
S17, no DAS	$-0.00614$	$-0.00138$

Table S2 : EV sensitivity to rule variants (

10^{5}

hands each under respective oracle policies).

Figure S1 : SPSA regret heatmap for hard totals.

Figure S2 : SPSA regret heatmap for soft totals.

Figure S3 : SPSA regret heatmap for pair cells.

Figure S4 : CEM regret heatmap for hard totals.

Figure S5 : CEM regret heatmap for soft totals.

Figure S6 : CEM regret heatmap for pair cells.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 224 papers

Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms
Mar 10, 2026Renos Zabounidis, Roy Siegelmann, Mohamad Qadri +3Policy Gradient Methods
Carnegie Mellon University · Massachusetts Institute of Technology · Virginia Tech, Department of Mechanical Engineering
Confessions of a Masked MDP
Oct 20, 2026Sebastian Reboul, Lorenzo Croissant, Hélène HalconruyReinforcement Learning Constrained RL
SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris · Modal’X, Université Paris-Nanterre · Wiremind Cargo, AI & ML Group +1
Model-Based Reinforcement Learning with Double Oracle Efficiency in Policy Optimization and Offline Estimation
May 1, 2026Haichen Hu, Jian Qian, David Simchi-LeviRegret Minimization in RL Markov Decision Processes
Laboratory for Information and Decision Systems · Department of AI and Data Science

Mar 10, 2026J/K move · Enter open · S save

Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms

Renos Zabounidis, Roy Siegelmann, Mohamad Qadri, Woojun Kim+2

Carnegie Mellon University · Massachusetts Institute of Technology · Virginia Tech, Department of Mechanical Engineering

Policy Gradient Methods

Open paper AskPDF arXiv

In reinforcement learning environments with state-dependent action validity, action masking consistently outperforms penalty-based handling of invalid actions, yet existing theory only shows that masking preserves the policy gradient theorem. We identify a distinct failure mode of unmasked training: it systematically suppresses valid actions at states the agent has not yet visited. This occurs because gradients pushing down invalid actions at visited states propagate through shared network parameters to unvisited states where those actions are valid. We prove that for softmax policies with shared features, when an action is invalid at visited states but valid at an unvisited state

s^*

, the probability

π(a \mid s^*)

is bounded by exponential decay due to parameter sharing and the zero-sum identity of softmax logits. This bound reveals that entropy regularization trades off between protecting valid actions and sample efficiency, a tradeoff that masking eliminates. We validate empirically that deep networks exhibit the feature alignment condition required for suppression, and experiments on Craftax, Craftax-Classic, and MiniHack confirm the predicted exponential suppression and demonstrate that feasibility classification enables deployment without oracle masks.

Abstract

Policy Optimization

Explore similar work

Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms

Confessions of a Masked MDP

Model-Based Reinforcement Learning with Double Oracle Efficiency in Policy Optimization and Offline Estimation