When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

Marcel Petran; Stefan Edelkamp

cs.GTOct 13, 2026

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability

Authors: Marcel Petran, Stefan Edelkamp

Organizations: Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

Abstract

Deep reinforcement learning agents in partially observable competitive games must act on beliefs about an unseen opponent's intent. We study a spatial opponent model: a transformer trained with hindsight-relabelled opponent goals that outputs a subgoal heatmap from the opponent's observed trajectory, fused early into a dueling double DQN. In symmetric constant-sum foraging games with partial observability, the module stabilises training. Plain TD(0) DQN suffers degenerate seeds in three of four one-step configurations, whereas the opponent-model agent never collapses and, wherever training succeeds, recovers the majority of the symmetric game value. Transfers zero-shot to a foraging opponent unseen by the learner, with a positive goal advantage over the baseline, and loses this edge exactly where an information argument predicts: against a full-information tracking adversary whose intent is the learner itself, and on resource-dense maps where both agents saturate. A blindfold ablation, zeroing the heatmap at deployment, decomposes the advantage into two mechanisms: training-time stabilisation, which dominates where the credit assignment is hard and survives blinding on the dense map, and deployment-time inference, whose reliance shrinks as fog thins on the sparse map. We conclude that opponent-goal inference is a stability-and-transfer mechanism whose value is governed by measurable boundary conditions: visibility (fog thickness), opponent intent, and resource density.

Figures & tables

Panel of Figure 1 — Figure 1. The two foraging maps ( $11{\times}11$ ). map3 (left): five densely placed central goals; map4 (right): seven goals dispersed behind sectioned walls. Agents and foods spawn at fixed positions and compete for the resources.

Figure 2. Overview of the Transformer-based Spatial Opponent Model. The Transformer Encoder processes the sequence of past observations, attending to the current observation to output a 2D heatmap representing the inferred opponent subgoals. The Feature Embedder is detailed in Figure 3 . The diagram also shows expected tensor shapes where:

B

is the batch dimension,

T

is the trajectory dimension,

F

is the environment Feature dimension,

H

and

W

are the environment Height and Width dimensions, and

d_{model}

is the embedding output dimension.

Figure 3. Feature embedding process inside the Spatial Opponent Model.

Panel of Figure 4 — Algorithm 1 Hindsight Subgoal Labelling with Gaussian Smoothing

Figure 4. OM Dueling Q-Network. The observation and belief channels are concatenated with the inferred heatmap and processed by the CNN; Advantage and Value heads compute the final Q-values by Equation 6 .

Panel of Figure 6 — Algorithm 2 DDQN with the Spatial OM (OM-DDQN) using HER

Figure 5. Example of Spatial OM predictions on map4 (vision radius 2). Left: the learner’s observation under fog; the learner (green dot) sees only its surroundings, while the opponent’s true position (dark red when obscured by fog) and the food (green squares) are shown for reference. Middle: the OM’s predicted subgoal distribution fed to the Q-network. Right: the opponent’s true subgoal distribution, that is not accessible during match. The OM concentrates mass on the opponent’s actual target even when the opponent is outside the learner’s vision radius.

Learning rate	Adam; $3{\cdot}10^{-4}$ (Q), $1{\cdot}10^{-4}$ (OM)
Discount $\gamma$ / soft update $\tau_{soft}$	0.985 / 0.001
Batch size / train frequency	128 / every 4 steps
Replay capacity	450,000
Episodes	30,000 (60 epochs $\times$ 500 episodes), max 50 steps
Boltzmann $\tau$	$1.5\to 0.05$ over 650k steps
Label smoothing $\sigma^{2}$	$2.0\to 0.5$ over 150k steps

Table 1. Shared training hyperparameters and network sizes (both learning agents).

Figure 6. Per-epoch evaluation return against the training opponent, per seed: classic (blue) vs. om (orange), belief prior off. Top row map4 , bottom row map3 . classic degenerates at

n{=}1

: on map4 one

v{=}2

seed collapses outright, all three

v{=}5

seeds finish below

1.7

3.5

, and on map3 all seeds in the

v{=}2

cell finish below

2.0

2.5

(thin fog recovers). om never collapses on either map.

Figure 7. Zero-shot return as a share of the symmetric game value, by opponent and cell (dots: seeds; bar: mean; dashed: game value). Left: map4 (prior off; collapsed classic seed excluded). Right: map3 (prior on/off pooled, 6 seeds per cell). Both agents exceed 100% under thin fog (best-response exploitation of the suboptimal heuristic) and both saturate on the dense map wherever training succeeds.

Opponent	Fog	$n{=}3$	$n{=}1$
greedy (seen, forager)	$v{=}2$	$+0.64\pm 0.20$	$+0.71\pm 0.08^{\dagger}$
greedy (seen, forager)	$v{=}5$	$+0.41\pm 0.07$	$+1.50\pm 0.26$
simple (unseen, forager)	$v{=}2$	$+0.31\pm 0.31$	$+0.66\pm 0.05^{\dagger}$
simple (unseen, forager)	$v{=}5$	$+0.52\pm 0.10$	$+1.16\pm 0.05$
oracle-stalker (tracker)	$v{=}2$	$+0.12\pm 0.39$	$+0.52\pm 0.13^{\dagger}$
oracle-stalker (tracker)	$v{=}5$	$+0.02\pm 0.09$	$+1.13\pm 0.27$

Table 2. Paired zero-shot advantage

\Delta=\textsc{om}-\textsc{classic}

(mean

\pm

std over seeds) by opponent, fog, and

n

-step horizon on map4 (prior off).

n{=}3

: well-trained cells;

n{=}1

: cells where classic training is unstable. Seen forager means that the learner has trained against the opponent; unseen forager means that the opponent is not in the learner’s training set. Dense-map ( map3 ) deltas are reported in the text and Table 3 .

Map / opponent	Cell	om	om blind	$\Delta$	$\Delta$ blind
map4 , greedy	$v{=}2$ , $n{=}3$	3.33	2.23	$+0.64$	$-0.47$
map4 , greedy	$v{=}2$ , $n{=}1$	2.86	2.43	$+0.71^{\dagger}$	$+0.28^{\dagger}$
map4 , simple	$v{=}2$ , $n{=}3$	2.77	2.10	$+0.31$	$-0.37$
map4 , simple	$v{=}2$ , $n{=}1$	2.55	2.31	$+0.66^{\dagger}$	$+0.42^{\dagger}$
map4 , oracle-stalker	$v{=}2$ , $n{=}3$	2.09	1.71	$+0.12$	$-0.26$
map4 , oracle-stalker	$v{=}2$ , $n{=}1$	2.10	2.10	$+0.52^{\dagger}$	$+0.52^{\dagger}$

Table 3. Blindfold ablation: heatmap zeroed at deployment. Mean om return over seeds;

\Delta=\textsc{om}{}-\textsc{classic}{}

paired by training seed, zero-shot. map4 : prior off,

n

-split; map3 :

n{=}1

, prior on/off pooled, 6 seeds; vs. the opponent shown.

Fog	$n$	Prior	classic	om	$\Delta$
$v{=}2$	1	off	55.1	82.3	$+0.68$
$v{=}2$	1	on	65.8	89.4	$+0.59$
$v{=}2$	3	off	90.5	99.5	$+0.23$
$v{=}2$	3	on	98.4	99.8	$+0.04$
$v{=}5$	1	off	96.2	105.3	$+0.23$
$v{=}5$	1	on	104.0	106.9	$+0.07$

Table 4. Dense map3 (game value 2.5 by symmetry): epoch-60 eval return vs. the training opponent as % of game value, mean over 3 seeds per cell.

\Delta

: paired om

-

classic in goals.

Opponent Modeling

Opponent modeling studies how agents infer others’ goals, preferences, policies, and hidden states to predict and adapt to their actions.

See all 17 papers

May 29, 2026

Generalized Intention Modeling in Multi-Agent Reinforcement Learning

Modeling an opponent's intent is critical for effective decision-making in non-cooperative, competitive, and general-sum multi-agent reinforcement learning. Existing opponent modeling methods encode intent using an embedding derived from episode information chosen a priori, such as the opponent's next action or a future environment state, and use this to guide the ego-agent's behavior. These approaches assume that the chosen information is universally representative of intent; however, we show empirically that this is not the case as intentions are often task- and environment-dependent. To address this, we introduce a task-adaptive opponent modeling framework that learns a performance-driven mixture of multiple intent representations. We further introduce a new intention representation that maximizes mutual information with the ego-agent's future returns, thereby capturing opponent information that is most directly relevant to performance. Our approach consistently matches or exceeds the performance of state-of-the-art baselines across diverse tasks and yields insights into when and why different opponent modeling strategies succeed.

Mateusz Odrowaz-Sypniewski, Jasmine Bayrooti, Ajay Shankar +1

Department of Computer Science and Technology, University of Cambridge, UK

Opponent Modeling Multi-Agent Reasoning Multi-Agent Reinforcement Learning

Ask PDF

May 27, 2026

Differentiable Belief-based Opponent Shaping

Human coordination often relies on the ability to influence the beliefs of others through strategic action. In multi-agent reinforcement learning, opponent shaping attempts to replicate this influence, though existing methods typically operate within an opponent's parameter, policy, or value space. Meanwhile, belief-manipulation techniques in hidden-role games often rely on hard-coded objectives, such as deception or belief saturation. We propose Differentiable Belief-based Opponent Shaping (D-BOS), a first-order method that treats each observer's belief as the shaped opponent state and differentiates through

k

-step softmax-Bayes belief dynamics. Rather than explicitly rewarding deceptive or cooperative behavior, our method treats the belief state as the target for shaping. This allows the optimal strategy to emerge naturally from the environment's reward structure. This belief-space formulation provides an opponent-shaping signal by differentiating through opponent belief updates, and naturally extends to multiple observers by aggregating gradients over their individual inferred belief trajectories. Empirically, D-BOS outperforms PPO and BBM in hidden-role games, with the largest gains in mixed-motive settings.

Aarav G Sane, Karthik Sivachandran, Rohan Paleja

Department of Computer Science Purdue University

Multi-Agent Coordination Imperfect-Information Games Multi-Agent Reinforcement Learning

Ask PDF

Sep 11, 2026

Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation

Lux AI Season 3 requires agents to act under partial observability, randomized episode level dynamics, and a best of five match structure that rewards both tactical execution and fast adaptation. We present HORIZON, a hierarchical agent that combines symmetry aware spatial perception, dual memory belief tracking, relic centric graph attention, information gain driven exploration, and an opponent conditioned policy mixture. HORIZON separates short horizon control from cross match meta reasoning, while auxiliary belief and world model objectives stabilize learning. Trained with PPO in a large scale JAX simulator, the resulting agent explicitly infers hidden game parameters and opponent style. Experiments show consistent gains in match win rate, episode win rate, adaptation gain, and league rating over strong recurrent and feed forward baselines.

Kowei Shih, Lu Cheng, Zeyu Wang +2

Tsinghua University, Beijing, China · Stevens Institute of Technology, New Jersey, USA · University of California, Los Angeles, Los Angeles, USA +2

Opponent Modeling Partially Observable RL Hierarchical RL Multi-Agent Path Finding

Ask PDF

Abstract

Opponent Modeling

Explore similar work

Generalized Intention Modeling in Multi-Agent Reinforcement Learning

Differentiable Belief-based Opponent Shaping

Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation