Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent's intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density.
Figures & tables
Figure 1. The two foraging maps ( 11×11 ). map3 (left): five densely placed central goals; map4 (right): seven goals dispersed behind sectioned walls. Agents and foods spawn at fixed positions and compete for the resources.
Figure 2. Overview of the Transformer-based Spatial Opponent Model. The Transformer Encoder processes the sequence of past observations, attending to the current observation to output a 2D heatmap representing the inferred opponent subgoals. The Feature Embedder is detailed in Figure 3 . The diagram also shows expected tensor shapes where: B is the batch dimension, T is the trajectory dimension, F is the environment Feature dimension, H and W are the environment Height and Width dimensions, and dmodel is the embedding output dimension.
Figure 3. Feature embedding process inside the Spatial Opponent Model.
Algorithm 1 Hindsight Subgoal Labelling with Gaussian Smoothing
Figure 4. OM Dueling Q-Network. The observation and belief channels are concatenated with the inferred heatmap and processed by the CNN; Advantage and Value heads compute the final Q-values by Equation 6 .
Algorithm 2 DDQN with the Spatial OM (OM-DDQN) using HER
Figure 5. Example of Spatial OM predictions on map4 (vision radius 2). Left: the learner’s observation under fog; the learner (green dot) sees only its surroundings, while the opponent’s true position (dark red when obscured by fog) and the food (green squares) are shown for reference. Middle: the OM’s predicted subgoal distribution fed to the Q-network. Right: the opponent’s true subgoal distribution, that is not accessible during match. The OM concentrates mass on the opponent’s actual target even when the opponent is outside the learner’s vision radius.
Learning rate
Adam; 3⋅10−4 (Q), 1⋅10−4 (OM)
Discount γ / soft update τsoft
0.985 / 0.001
Batch size / train frequency
128 / every 4 steps
Replay capacity
450,000
Episodes
30,000 (60 epochs × 500 episodes), max 50 steps
Boltzmann τ
1.5→0.05 over 650k steps
Label smoothing σ2
2.0→0.5 over 150k steps
Table 1. Shared training hyperparameters and network sizes (both learning agents).
Figure 6. Per-epoch evaluation return against the training opponent, per seed: classic (blue) vs. om (orange), belief prior off. Top row map4 , bottom row map3 . classic degenerates at n=1 : on map4 one v=2 seed collapses outright, all three v=5 seeds finish below 1.7 of 3.5 , and on map3 all seeds in the v=2 cell finish below 2.0 of 2.5 (thin fog recovers). om never collapses on either map.
Figure 7. Zero-shot return as a share of the symmetric game value, by opponent and cell (dots: seeds; bar: mean; dashed: game value). Left: map4 (prior off; collapsed classic seed excluded). Right: map3 (prior on/off pooled, 6 seeds per cell). Both agents exceed 100% under thin fog (best-response exploitation of the suboptimal heuristic) and both saturate on the dense map wherever training succeeds.
Opponent
Fog
n=3
n=1
greedy (seen, forager)
v=2
+0.64±0.20
+0.71±0.08†
greedy (seen, forager)
v=5
+0.41±0.07
+1.50±0.26
simple (unseen, forager)
v=2
+0.31±0.31
+0.66±0.05†
simple (unseen, forager)
v=5
+0.52±0.10
+1.16±0.05
oracle-stalker (tracker)
v=2
+0.12±0.39
+0.52±0.13†
oracle-stalker (tracker)
v=5
+0.02±0.09
+1.13±0.27
Table 2. Paired zero-shot advantage Δ=\textscom−\textscclassic (mean ± std over seeds) by opponent, fog, and n -step horizon on map4 (prior off). n=3 : well-trained cells; n=1 : cells where classic training is unstable. Seen forager means that the learner has trained against the opponent; unseen forager means that the opponent is not in the learner’s training set. Dense-map ( map3 ) deltas are reported in the text and Table 3 .
Map / opponent
Cell
om
om blind
Δ
Δ blind
map4 , greedy
v=2 , n=3
3.33
2.23
+0.64
−0.47
map4 , greedy
v=2 , n=1
2.86
2.43
+0.71†
+0.28†
map4 , simple
v=2 , n=3
2.77
2.10
+0.31
−0.37
map4 , simple
v=2 , n=1
2.55
2.31
+0.66†
+0.42†
map4 , oracle-stalker
v=2 , n=3
2.09
1.71
+0.12
−0.26
map4 , oracle-stalker
v=2 , n=1
2.10
2.10
+0.52†
+0.52†
Table 3. Blindfold ablation: heatmap zeroed at deployment. Mean om return over seeds; Δ=\textscom−\textscclassic paired by training seed, zero-shot. map4 : prior off, n -split; map3 : n=1 , prior on/off pooled, 6 seeds; vs. the opponent shown.
Fog
n
Prior
classic
om
Δ
v=2
1
off
55.1
82.3
+0.68
v=2
1
on
65.8
89.4
+0.59
v=2
3
off
90.5
99.5
+0.23
v=2
3
on
98.4
99.8
+0.04
v=5
1
off
96.2
105.3
+0.23
v=5
1
on
104.0
106.9
+0.07
Table 4. Dense map3 (game value 2.5 by symmetry): epoch-60 eval return vs. the training opponent as % of game value, mean over 3 seeds per cell. Δ : paired om − classic in goals.