cs.AIMar 29, 2026

SkyNet: Belief-Aware Planning for Partially-Observable Stochastic Games

Authors: Adam Haile

Organizations: Dwight and Dian Diercks School of Advanced Computing Milwaukee School of Engineering 1025 N Broadway St, Milwaukee, WI 53202

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In 2019, Google DeepMind released MuZero, a model-based reinforcement learning method that achieves strong results in perfect-information games by combining learned dynamics models with Monte Carlo Tree Search (MCTS). However, comparatively little work has extended MuZero to partially observable, stochastic, multi-player environments, where agents must act under uncertainty about hidden state. Such settings arise not only in card games but in domains such as autonomous negotiation, financial trading, and multi-agent robotics. In the absence of explicit belief modeling, MuZero's latent encoding has no dedicated mechanism for representing uncertainty over unobserved variables. To address this, we introduce SkyNet (Belief-Aware MuZero), which adds ego-conditioned auxiliary heads for winner prediction and rank estimation to the standard MuZero architecture. These objectives encourage the latent state to retain information predictive of outcomes under partial observability, without requiring explicit belief-state tracking or changes to the search algorithm. We evaluate SkyNet on Skyjo, a partially observable, non-zero-sum, stochastic card game, using a decision-granularity environment, transformer-based encoding, and a curriculum of heuristic opponents with self-play. In 1000-game head-to-head evaluations at matched checkpoints, SkyNet achieves a 75.3% peak win rate against the baseline (+194 Elo, p<10−50p < 10^{-50}). SkyNet also outperforms the baseline against heuristic opponents (0.720 vs.\ 0.466 win rate). Critically, the belief-aware model initially underperforms the baseline but decisively surpasses it once training throughput is sufficient, suggesting that belief-aware auxiliary supervision improves learned representations under partial observability, but only given adequate data flow.

Figures & tables

Figure 1: A player’s 3×43\times 4 card grid during a game of Skyjo, with the deck shown at right. Cards range from −2-2 to 1212 , with lower total scores being more desirable. Image credit: 3rd Grade Thoughts ( https://www.3rdgradethoughts.com/2019/01/board-game-review-skyjo.html ).
Figure 2: Architecture comparison between baseline MuZero (left) and Belief-Aware MuZero / SkyNet (right). Both share the same representation, dynamics, and prediction networks. SkyNet adds an ego conditioning layer that injects player identity before prediction, and two auxiliary heads (winner and rank) that shape the latent representation via outcome-prediction objectives. Pink-shaded outputs indicate the additional belief heads.
ParameterBaselineBelief-Aware
Iterations20002000
Self-play episodes/iter3232
Train steps/iter9696
Batch size6464
Unroll steps / TD steps8 / 108 / 10
Discount ( γ\gamma )0.9970.997
Table 1: Training hyperparameters.
Figure 3: Training loss curves for baseline MuZero and Belief-Aware MuZero. Raw values shown in light color; 20-iteration rolling averages in bold. Both models converge, with the belief-aware model’s total loss higher due to the additional auxiliary loss terms.
Figure 4: Convergence of the belief-aware model’s auxiliary prediction heads. The winner head loss (left) and rank head loss (right) both decrease substantially over training, confirming that the network learns to predict game outcomes with increasing accuracy.
MetricBaselineBelief-Aware
Mean eval win rate ( ±1.96σ\pm 1.96\sigma )0.466 [0.32–0.61]0.720 [0.58–0.87]
Std eval win rate0.0730.074
Max eval win rate0.6000.825
Min eval win rate0.3130.525
Mean truncation rate0.0020.000
Mean episode length87.587.7
Table 2: Evaluation statistics over training (self-play vs. heuristic opponents). Bracketed ranges: μ±1.96σ\mu\pm 1.96\sigma across checkpoints.
Figure 5: Evaluation win rate against the heuristic bot curriculum over training. The belief-aware model consistently achieves higher win rates after an initial ramp-up period. The dashed line indicates the 50% random baseline.
CheckpointBelief WinsBaseline WinsDrawsBelief WR (95% CI)Δ\Delta Elo
Iter 125360640036.0% [33.1–39.0]−99-99
Iter 250422578042.2% [39.2–45.3]−55-55
Iter 500668332066.8% [63.8–69.6]+120+120
Iter 750742252674.2% [71.4–76.8]+184+184
Iter 1000753240775.3% [72.5–77.9]+194+194
Table 3: Head-to-head results (1000 games per comparison). Belief WR 95% CI: Wilson score interval.
Figure 6: Head-to-head win rates between SkyNet (Belief-Aware) and baseline MuZero at matched training checkpoints (1000 games each). The crossover between iterations 250 and 500 shows SkyNet overtaking the baseline after sufficient training.
CheckpointFull WinsAblated WinsDrawsFull WR (95% CI)Δ\Delta Elo
Iter 500690305569.0% [66.1–71.8]+139+139
Iter 1000811181881.1% [78.6–83.4]+253+253
Table 4: Inference-time ablation: full SkyNet vs. ego-conditioning-off (1000 games, iter 500). WR 95% CI: Wilson score interval.
Figure 7: Training diagnostics for both models (20-iteration rolling averages). Gradient norms (left) stabilize in a healthy range for both models. Policy entropy (right) decreases gradually without premature collapse, indicating progressive strategy refinement.
Figure 8: Linear probe R2R^{2} on frozen latent vectors for seven game-state features. Features left of the dashed line are directly observable or semi-observable; features to the right are hidden. Negative R2R^{2} indicates worse-than-mean prediction. The ego-conditioned belief representation is the only variant that achieves positive R2R^{2} on both hidden-card features.
More in this topic

Partially Observable Stochastic Games

See all 14 papers

Explore similar work

CardsList
Oct 13, 2026

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent's intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density.

Marcel Petran, Stefan Edelkamp

Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

Opponent ModelingMulti-Agent Reinforcement LearningPartially Observable RLPartially Observable Stochastic GamesDeep Q-Learning
AskPDF
Sep 11, 2026

Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation

Lux AI Season 3 requires agents to act under partial observability, randomized episode level dynamics, and a best of five match structure that rewards both tactical execution and fast adaptation. We present HORIZON, a hierarchical agent that combines symmetry aware spatial perception, dual memory belief tracking, relic centric graph attention, information gain driven exploration, and an opponent conditioned policy mixture. HORIZON separates short horizon control from cross match meta reasoning, while auxiliary belief and world model objectives stabilize learning. Trained with PPO in a large scale JAX simulator, the resulting agent explicitly infers hidden game parameters and opponent style. Experiments show consistent gains in match win rate, episode win rate, adaptation gain, and league rating over strong recurrent and feed forward baselines.

Kowei Shih, Lu Cheng, Zeyu Wang +2

Tsinghua University, Beijing, China · Stevens Institute of Technology, New Jersey, USA · University of California, Los Angeles, Los Angeles, USA +2

Opponent ModelingPartially Observable RLHierarchical RLMulti-Agent Path Finding
AskPDF
Jun 14, 2026

Mind-Studio: Executable World Models with Lookahead Evaluation for Partially Observable Games

World-model synthesis aims to turn interaction experience into an internal model of environment dynamics. Existing symbolic approaches often fit observed transitions or mixtures of local rules, but they do not produce a complete executable program that can run independently of the real environment. We present Mind-Studio, a framework that synthesizes executable pygame-style world models from state-action-next-state trajectories using large language models. Mind-Studio combines entropy-selected traces with a lightweight game skill file containing object, action, and static scene information extracted from screenshots. We evaluate synthesis quality with a K-step lookahead fidelity protocol that compares generated world-model rollouts against Real-ALE rollouts from the same state. On Montezuma's Revenge, Mind-Studio improves chosen-action next-state prediction from 0.3% for PoE-World to 48.7% while verifying 5 of 8 subgoals; across Alien, Assault, and Skiing, it achieves stronger branch-level fidelity than prior learned lookahead sources.

Yifei Dong, Mingen Zheng, Linquan Wu +2

Hong Kong University of Science and Technology · City University of Hong Kong · University of Edinburgh +1

World Model LearningLLM-Based Program SynthesisWorld Model Evaluation
AskPDF