SkyNet: Belief-Aware Planning for Partially-Observable Stochastic Games

Adam Haile

cs.AIMar 29, 2026

SkyNet: Belief-Aware Planning for Partially-Observable Stochastic Games

Authors: Adam Haile

Organizations: Dwight and Dian Diercks School of Advanced Computing Milwaukee School of Engineering 1025 N Broadway St, Milwaukee, WI 53202

Abstract

In 2019, Google DeepMind released MuZero, a model-based reinforcement learning method that achieves strong results in perfect-information games by combining learned dynamics models with Monte Carlo Tree Search (MCTS). However, comparatively little work has extended MuZero to partially observable, stochastic, multi-player environments, where agents must act under uncertainty about hidden state. Such settings arise not only in card games but in domains such as autonomous negotiation, financial trading, and multi-agent robotics. In the absence of explicit belief modeling, MuZero's latent encoding has no dedicated mechanism for representing uncertainty over unobserved variables. To address this, we introduce SkyNet (Belief-Aware MuZero), which adds ego-conditioned auxiliary heads for winner prediction and rank estimation to the standard MuZero architecture. These objectives encourage the latent state to retain information predictive of outcomes under partial observability, without requiring explicit belief-state tracking or changes to the search algorithm. We evaluate SkyNet on Skyjo, a partially observable, non-zero-sum, stochastic card game, using a decision-granularity environment, transformer-based encoding, and a curriculum of heuristic opponents with self-play. In 1000-game head-to-head evaluations at matched checkpoints, SkyNet achieves a 75.3% peak win rate against the baseline (+194 Elo, $p < 10^{-50}$ ). SkyNet also outperforms the baseline against heuristic opponents (0.720 vs.\ 0.466 win rate). Critically, the belief-aware model initially underperforms the baseline but decisively surpasses it once training throughput is sufficient, suggesting that belief-aware auxiliary supervision improves learned representations under partial observability, but only given adequate data flow.

Figures & tables

Figure 1: A player’s $3\times 4$ card grid during a game of Skyjo, with the deck shown at right. Cards range from $-2$ to $12$ , with lower total scores being more desirable. Image credit: 3rd Grade Thoughts ( https://www.3rdgradethoughts.com/2019/01/board-game-review-skyjo.html ).

Figure 2: Architecture comparison between baseline MuZero (left) and Belief-Aware MuZero / SkyNet (right). Both share the same representation, dynamics, and prediction networks. SkyNet adds an ego conditioning layer that injects player identity before prediction, and two auxiliary heads (winner and rank) that shape the latent representation via outcome-prediction objectives. Pink-shaded outputs indicate the additional belief heads.

Parameter	Baseline	Belief-Aware
Iterations	2000	2000
Self-play episodes/iter	32	32
Train steps/iter	96	96
Batch size	64	64
Unroll steps / TD steps	8 / 10	8 / 10
Discount ( $\gamma$ )	0.997	0.997

Table 1: Training hyperparameters.

Figure 3: Training loss curves for baseline MuZero and Belief-Aware MuZero. Raw values shown in light color; 20-iteration rolling averages in bold. Both models converge, with the belief-aware model’s total loss higher due to the additional auxiliary loss terms.

Figure 4: Convergence of the belief-aware model’s auxiliary prediction heads. The winner head loss (left) and rank head loss (right) both decrease substantially over training, confirming that the network learns to predict game outcomes with increasing accuracy.

Metric	Baseline	Belief-Aware
Mean eval win rate ( $\pm 1.96\sigma$ )	0.466 [0.32–0.61]	0.720 [0.58–0.87]
Std eval win rate	0.073	0.074
Max eval win rate	0.600	0.825
Min eval win rate	0.313	0.525
Mean truncation rate	0.002	0.000
Mean episode length	87.5	87.7

Table 2: Evaluation statistics over training (self-play vs. heuristic opponents). Bracketed ranges:

\mu\pm 1.96\sigma

across checkpoints.

Figure 5: Evaluation win rate against the heuristic bot curriculum over training. The belief-aware model consistently achieves higher win rates after an initial ramp-up period. The dashed line indicates the 50% random baseline.

Checkpoint	Belief Wins	Baseline Wins	Draws	Belief WR (95% CI)	$\Delta$ Elo
Iter 125	360	640	0	36.0% [33.1–39.0]	$-99$
Iter 250	422	578	0	42.2% [39.2–45.3]	$-55$
Iter 500	668	332	0	66.8% [63.8–69.6]	$+120$
Iter 750	742	252	6	74.2% [71.4–76.8]	$+184$
Iter 1000	753	240	7	75.3% [72.5–77.9]	$+194$

Table 3: Head-to-head results (1000 games per comparison). Belief WR 95% CI: Wilson score interval.

Figure 6: Head-to-head win rates between SkyNet (Belief-Aware) and baseline MuZero at matched training checkpoints (1000 games each). The crossover between iterations 250 and 500 shows SkyNet overtaking the baseline after sufficient training.

Checkpoint	Full Wins	Ablated Wins	Draws	Full WR (95% CI)	$\Delta$ Elo
Iter 500	690	305	5	69.0% [66.1–71.8]	$+139$
Iter 1000	811	181	8	81.1% [78.6–83.4]	$+253$

Table 4: Inference-time ablation: full SkyNet vs. ego-conditioning-off (1000 games, iter 500). WR 95% CI: Wilson score interval.

Figure 7: Training diagnostics for both models (20-iteration rolling averages). Gradient norms (left) stabilize in a healthy range for both models. Policy entropy (right) decreases gradually without premature collapse, indicating progressive strategy refinement.

Figure 8: Linear probe $R^{2}$ on frozen latent vectors for seven game-state features. Features left of the dashed line are directly observable or semi-observable; features to the right are hidden. Negative $R^{2}$ indicates worse-than-mean prediction. The ego-conditioned belief representation is the only variant that achieves positive $R^{2}$ on both hidden-card features.

Partially Observable Stochastic Games

See all 14 papers

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability
Oct 13, 2026Marcel Petran, Stefan EdelkampOpponent Modeling Multi-Agent Reinforcement Learning
Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic
Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation
Sep 11, 2026Kowei Shih, Lu Cheng, Zeyu Wang +2Opponent Modeling Partially Observable RL
Tsinghua University, Beijing, China · Stevens Institute of Technology, New Jersey, USA · University of California, Los Angeles, Los Angeles, USA +2
Mind-Studio: Executable World Models with Lookahead Evaluation for Partially Observable Games
Jun 14, 2026Yifei Dong, Mingen Zheng, Linquan Wu +2World Model Learning LLM-Based Program Synthesis
Hong Kong University of Science and Technology · City University of Hong Kong · University of Edinburgh +1

Oct 13, 2026J/K move · Enter open · S save

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

Marcel Petran, Stefan Edelkamp

Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

Opponent Modeling Multi-Agent Reinforcement Learning Partially Observable RL Partially Observable Stochastic Games Deep Q-Learning

Open paper AskPDF arXiv

Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent's intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density.

Abstract

Partially Observable Stochastic Games

Explore similar work

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation

Mind-Studio: Executable World Models with Lookahead Evaluation for Partially Observable Games