Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games

Nick Leenders; Roy Lindelauf; Joost van Oijen; Boris Cule

cs.AIOct 8, 2026

Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games

Authors: Nick Leenders, Roy Lindelauf, Joost van Oijen, Boris Cule

Organizations: NLR Royal Netherlands Aerospace Centre Amsterdam, The Netherlands · Data Science Center of Excellence Faculty of Military Sciences, Breda, NL · Departement Intelligent Systems Tilburg University, Tilburg, NL

Abstract

Deep reinforcement learning agents reach strong performance in real-time strategy games but can be brittle against opponents outside their training distribution. Separating strategic command selection from learned unit control allows different strategies to be selected for different opponents while reusing the same execution policy. This requires an executor that can follow different commands and measurable criteria for assessing whether it does so. We introduce a constrained command-conditioned Proximal Policy Optimization (PPO) policy, the executor, for MicroRTS, a real-time strategy environment. Discrete commands specify strategic objectives and behavioral requirements for economy, army composition, military posture, and worker policy over multiple environment steps; the executor determines the unit-level actions used to fulfill them. A Thompson-sampling bandit acts as the strategist, selecting command tuples from an estimate of the opponent's strategy built from in-game observations rather than opponent identity. In a controlled comparison with a flat PPO baseline trained with the same architecture, budget, curriculum and self-play league, the strategist-executor system wins significantly more often against three of the four strongest opponents on a training map, including the two strongest held-out ones (0.55 to 0.97 and 0.01 to 0.34), with no significant difference against the others.

Figures & tables

Figure 1: Overview of the framework. The tuple bandit (strategist) supplies commands to a shared executor during training and evaluation. Dashed arrows show training feedback: terminal game outcomes update the bandit, while measured command outcomes adapt the multipliers used in PPO.

Economy	Composition	Military posture	Worker policy
STABILIZE	LIGHT	OFFENSE	ECON_ONLY
GROW	HEAVY	DEFENSE	HARASS
HOLD	RANGED	SPLIT
DOUBLE_BARRACKS	LIGHT_HEAVY	FLANK
	LIGHT_RANGED
	HEAVY_RANGED

Table 1: Command vocabulary of the low-level RL executor. One command is active in each slot.

Command	Physical metric	Train rate	Reference	Intervention	Response [95% interval]
GROW	new workers	0.82	0.91	2.23	$+1.32$ [ $+1.00$ , $+1.68$ ]
HEAVY	Heavy share of new army	0.75	0.00	1.00	$+1.00$ [ $+1.00$ , $+1.00$ ]
RANGED	Ranged share of new army	0.80	0.00	1.00	$+1.00$ [ $+1.00$ , $+1.00$ ]
OFFENSE	army position	0.79	0.469	0.463	$-0.006$ [ $-0.017$ , $+0.006$ ]
HARASS	share workers in enemy half	0.79	0.109	0.088	$-0.021$ [ $-0.067$ , $+0.023$ ]

Table 2: Constraint satisfaction versus controllability. Train rate is the smoothed terminal success rate

\bar{\rho}_{c}

at the end of training (target

0.8

). Response is the paired difference between the intervention and the same-seed reference game, pooled over three opponents, with a 95% bootstrap interval over pairs. Metrics use the first 300 ticks after the command, or the whole window when a game ended earlier.

Rank	Opponent	Split	Flat	Bandit	Static	$p$
			Win rate $W/(W+L)$			Flat vs Bandit
1	RAISocketAI	holdout	0.010	0.344	0.470	$5.9\times 10^{-10}$ $*$
2	TMA	train	1.000	0.950	0.949	0.53
3	agent_sota	train	0.612	0.929	0.898	$6.7\times 10^{-7}$ $*$
4	obiBotKenobi	holdout	0.545	0.970	1.000	$2.0\times 10^{-12}$ $*$
5	mayari	train	1.000	0.980	1.000	1.00

Table 3: Win rates of Flat PPO, the strategist–executor (Bandit) and the executor holding one tuple per opponent, chosen on a separate screen (Static), on map 16x16/basesWorkers16x16A , 100 games per cell. Ranks are from Table 9 (1 is strongest; rank 10, the training bot rojo , is not evaluated). Bold marks the highest observed rate, including ties;

*

marks significant Flat–Bandit differences (

p<0.05

). Counts, test details and the head-to-head exception (

\dagger

) are in Appendix I .

	Most common	First	Games with a change
Opponent	economy sequence (games)	change	Composition	Posture	Worker
RAISocketAI	STABILIZE $\to$ GROW (86)	440	18 (H, L+H)	0	0
TMA	STABILIZE $\to$ GROW (97)	360	2 (H)	0	0
agent_sota	STABILIZE $\to$ GROW (94)	360	9 (H, L+H)	0	0
obiBotKenobi	STABILIZE $\to$ GROW $\to$ HOLD (50)	360	2 (H)	0	0
mayari	STABILIZE $\to$ DOUBLE_BARRACKS (46)	441	3 (R)	0	0

Table 4: Strategist commands in the Bandit games of Table 3 . Every game opens with ( STABILIZE , LIGHT , DEFENSE , ECON_ONLY ). First change : median tick of the first economy change, which occurs in at least 98 of 100 games per opponent. Games with a change : games in which the slot left its opening command; R = RANGED , H = HEAVY , L+H = LIGHT_HEAVY .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Command	Success	Failure	$D$
STABILIZE	Base, four workers, and a barracks; recovery exceptions below.	Unauthorized second barracks.	450
GROW	Living worker count increases by at least one from window start.	Unauthorized second barracks.	300
HOLD	No prohibited worker production for 140 ticks; a base or barracks survives.	New worker production without a worker deficit, or unauthorized second barracks.	250
DOUBLE_BARRACKS	A base and at least two barracks survive.	—	1200
LIGHT, HEAVY, RANGED	At least three new military units; requested type accounts for $\geq 75\%$ .	At least three new units and $>30\%$ non-target types.	700, 800, 700
LIGHT_HEAVY, LIGHT_RANGED, HEAVY_RANGED	Both requested types produced; each accounts for $\geq 25\%$ , together $\geq 80\%$ .	Both types produced, at least four new units, and $>35\%$ non-target types.	1300, 1200, 1300

Appendix

Table 5: Command-specific terminal criteria. A dash denotes no explicit failure condition; unmet objectives instead time out at deadline

D

. For composition, deadlines correspond to commands in the listed order.

Opponent	A	B	C	D	E
randomBiasedAI	24	18	8	2	2
workerRushAI	16	8	4	4	4
lightRushAI	–	10	8	4	4
rojo	–	4	4	–	–
mayari	–	–	16	14	8
coacAI	–	–	–	8	14

Appendix

Table 6: Scripted-bot environments per curriculum phase (out of 40).

Command	$\lambda_{c}$	$\bar{\rho}_{c}$	Command	$\lambda_{c}$	$\bar{\rho}_{c}$
STABILIZE	0.05	0.94	LIGHT_RANGED	0.05	0.92
GROW	0.20	0.82	HEAVY_RANGED	0.05	0.91
HOLD	0.45	0.79	OFFENSE	0.24	0.79
DOUBLE_BARRACKS	0.05	1.00	DEFENSE	0.05	0.96
LIGHT	0.83	0.80	SPLIT	0.05	1.00
HEAVY	0.47	0.75	FLANK	0.29	0.92

Appendix

Table 7: Multipliers

\lambda_{c}

and smoothed success rates

\bar{\rho}_{c}

stored with the final checkpoint (target

0.8

, bounds

[0.05,1.0]

). No constraint is flagged as stalled.

	Contract success			Wins/losses
Setting	coacAI	droplet	obiBotKenobi	coacAI	droplet	obiBotKenobi
reference	—	—	—	8/0	8/0	7/0
GROW	4/8	3/7	8/8	7/1	7/0	8/0
HEAVY	8/8	3/6	3/6	0/8	6/0	4/2
RANGED	8/8	4/7	7/8	8/0	7/0	1/7
OFFENSE	0/8	3/8	0/7	8/0	8/0	7/0

Appendix

Table 8: Intervention games: windows with at least one contract success for the varied slot, and game outcomes (wins/losses), over the games in which the intervention was installed (at most eight per cell). The reference row has no varied slot.

Figure 2: Behavior scores in the intervention games (134 games, three opponents). Rows give the command issued in a slot, columns the compared command whose behavior is scored on the same trajectories. A slot the executor follows is diagonal. Composition is; the rows of posture and worker policy are indistinguishable, so the issued command does not change which behavior is observed. Scores are the heuristic behavior scores of Appendix G.2 , not contract outcomes.

Opponent	Split	Pooled $s$	95% CI	A	D	I	as P1	Win
RAISocketAI	holdout	$3.10$	$[2.77,3.56]$	$4.96$	$3.06$	$2.66$	$2.73$	$0.91$
TMA	train	$2.59$	$[2.29,2.94]$	$2.75$	$2.83$	$2.99$	$3.19$	$0.87$
agent_sota (map A)	train	$2.30$	$[2.03,2.66]$	$2.72$	–	–	–	$0.83$
obiBotKenobi	holdout	$1.85$	$[1.67,2.08]$	$2.26$	$2.66$	$1.39$	$1.86$	$0.79$
mayari	train	$1.16$	$[0.94,1.41]$	$0.69$	$1.24$	$2.05$	$0.71$	$0.68$
coacAI	train	$0.69$	$[0.45,0.98]$	$1.31$	$-1.05$	$1.48$	$1.05$	$0.56$

Appendix

Table 9: Bradley-Terry strength of the opponents on the training maps. Pooled is the fit over all three maps with 95% bootstrap intervals; A, D and I are separate per-map fits; as P1 is the pooled strength of the bot in the player-1 seat, which is the seat it occupies in Table 3 (each bot enters that fit as two players, one per seat, with

h

fixed to

0

). Win is the raw fraction of games won with draws counted as half. Split is as in Table 3 : train marks opponents in the executor’s scripted curriculum or self-play league. agent_sota plays only on map A and only as player 0 (see text), so its pooled strength is its map-A strength.

Rank	Opponent	W/L/D	W/L/D	Selected tuple	W/L/D
		Flat	Bandit	Static
1	RAISocketAI	1/98/1	33/63/4	GROW+L	47/53/0
2	TMA	100/0/0	95/5/0	DOUBLE+L+R	94/5/1
3	agent_sota (map A)	60/38/2	92/7/1	STABILIZE+L	88/10/2
4	obiBotKenobi	54/45/1	97/3/0	GROW+L	100/0/0
5	mayari	100/0/0	98/2/0	STABILIZE+L	100/0/0

Appendix

Table 10: Wins/losses/draws for Table 3 , in the same opponent ranking order. Static tuples show economy and composition only: DOUBLE abbreviates DOUBLE_BARRACKS ; L , H and R denote Light, Heavy and Ranged. A pair denotes a mixed composition.

Figure 3: Units trained per game by Flat, Bandit and Static (means over the 100 games of Table 3 , on map A; starting units excluded). The label at each bar gives the win rate

W/(W+L)

and, for Static, its fixed tuple ( Table 10 ).

Rank	Opponent	L	H	R	L+H	L+R	H+R	HOLD
1	RAISocketAI	0.50	0.17	0.03	0.33	0.14	0.07	0.02
2	TMA	0.95	0.97	0.97	0.94	0.95	0.95	0.28
3	agent_sota	0.83	0.42	0.25	0.78	0.71	0.28	0.30
4	obiBotKenobi	0.97	0.58	0.11	0.82	0.77	0.36	0.38
5	mayari	0.98	0.99	1.00	1.00	1.00	1.00	0.95
6	coacAI	0.73	0.00	1.00	0.26	0.92	0.69	0.45

Appendix

Table 11: Mean score of the frozen executor under a fixed composition command, averaged over the economy commands STABILIZE , GROW and DOUBLE_BARRACKS (60 games per cell), on map A. L , H and R denote Light, Heavy and Ranged; a pair denotes a mixed composition. Ranks are as in Table 3 . The HOLD column averages HOLD over all six compositions (120 games per cell); it is low because HOLD halts worker production: the executor has at most four workers in

95\%

of these games, against a median peak of six under GROW .

Explore similar work

STRATA: Self-Learning Through Role-Aligned Tiered Agents for Real-Time Strategy Games
Sep 30, 2026Xinhe Tian, Xiaoyue Zhang, Ziyou Zhang +4Multi-Agent Coordination Game-Playing Agents
School of Instrumentation Science and Optoelectronic Engineering, Beijing University of Aeronautics and Astronautics · Tsinghua University · University of Chinese Academy of Sciences +2
RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models
Jun 17, 2026San Kim, Daechul Ahn, Reokyoung Kim +3Multi-Agent Coordination Game-Playing Agents
Seoul National University
Finding the Time to Think: Learning Planning Budgets in Real-Time RL
Jun 24, 2026Aneesh Muppidi, Firas Darwish, Dylan Cope +2Reinforcement Learning Policy Learning
British Open-ended Learning and Discovery Lab (BOLD), University of Oxford · Department of Statistics, University of Oxford · VGG, University of Oxford

Sep 30, 2026, cs.AIJ/K move · Enter open · S save

STRATA: Self-Learning Through Role-Aligned Tiered Agents for Real-Time Strategy Games

Xinhe Tian, Xiaoyue Zhang, Ziyou Zhang, Jiacheng Li+3

School of Instrumentation Science and Optoelectronic Engineering, Beijing University of Aeronautics and Astronautics · Tsinghua University · University of Chinese Academy of Sciences+2

Multi-Agent Coordination Game-Playing Agents Lifelong Learning Agents Memory-Augmented Agents Self-Evolving Agents

Open paper AskPDF arXiv

Real-time strategy (RTS) games require agents to coordinate economic development, production and construction, base defense, unit organization, and attack timing over long matches. Existing studies have applied large language models to command decision-making in RTS games, enabling agents to read textual game states and generate high-level plans. However, long inference latency can cause them to miss critical tactical events. The complexity and tactical diversity of full RTS matches also leave existing systems heavily dependent on manually written experience-based prompts, with limited ability to learn continuously from past games. We present STRATA, a role-aligned hierarchical system with cross-game self-learning for Red Alert. STRATA assigns in-game strategic, logistical, and tactical decisions to a Strategic Agent (SA), Logistics Agent (LA), and Tactical Agent (TA), respectively. The SA generates high-level directives based on the global game state and relevant experience cards, while the LA and TA handle logistics and tactical execution. After each match, a Review Agent (RA) derives candidate experience from game traces, validates and revises it using evidence from subsequent matches, and compresses strategic experience supported across multiple games into concise experience cards for SA retrieval. We evaluate STRATA through the formation of experience cards, full-match comparisons before and after learning, and experience learning against AI opponents with different play styles. Under a fixed scenario, using the learned experience cards increases the observed win rate from 30% to 100%. Sequential learning against AI opponents with different play styles also produces distinct long-term strategic experience.

Abstract

Explore similar work

STRATA: Self-Learning Through Role-Aligned Tiered Agents for Real-Time Strategy Games

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

Finding the Time to Think: Learning Planning Budgets in Real-Time RL