LLM-Guided Reinforcement Learning for Adaptive NPC Behavior in Multi-Agent Combat Games

Hrithika Deepu Nair; Kayvan Karim

cs.MAAug 27, 2026

LLM-Guided Reinforcement Learning for Adaptive NPC Behavior in Multi-Agent Combat Games

Authors: Hrithika Deepu Nair, Kayvan Karim

Organizations: Heriot-Watt University Dubai, U.A.E

Abstract

Scripted and rule-based non-player characters (NPCs) in combat video games often exhibit predictable behaviors that experienced players can exploit, while reinforcement learning (RL) agents typically retain a fixed policy after training and cannot readily adapt their strategy to different opponents. We investigate a runtime strategy-selection framework in which a large language model (LLM) guides a trained RL policy without modifying its underlying behavior. To demonstrate this, we train five NPC agents with a shared PPO policy in Unity and compare a baseline configuration, in which the policy acts independently, with an LLM-augmented configuration in which a locally hosted Mistral 7B model, accessed through Ollama, reads the live game state every five seconds and assigns one of four tactical tags. We evaluate both configurations against three scripted opponent types across 600 episodes and analyze outcomes using the Mann-Whitney U test. Against a Balanced opponent that changes tactics during an episode, the LLM-augmented agents more than doubled their win rate from 11% to 24% and produced significantly longer episodes. Against an Evasive opponent, the augmented agents achieved a higher win rate and faster kills, although their shorter episode duration did not satisfy the strict hypothesis definition. Against an Aggressive opponent, the LLM's near-constant preference for encirclement was counterproductive. Analysis of 2,430 strategy selections showed that Surround was selected in 83.8% of cases regardless of opponent type, indicating limited zero-shot strategic differentiation at this model scale. These results demonstrate both the potential and limitations of LLM-guided runtime strategy selection for adaptive multi-agent game AI.

Figures & tables

Figure 1. System overview comparing Combination A (RL-only baseline) and Combination B (RL plus Mistral 7B strategy selector via Ollama), both evaluated in the same Unity 3D arena.

Figure 2. Combination A: RL-only shared policy. Five NPCs share one PPO policy; the strategy tag is fixed at zero.

Figure 3. Combination B: RL with LLM strategy selector. The Python sidecar queries Mistral 7B every five seconds and writes a strategy tag conditioning all five NPC observations.

Figure 4. Top-down layout of the $30\times 30$ combat arena, showing obstacle placement and the NPC/bot spawn zones.

Entity	HP	Speed	Range	Dmg / CD
Bot	300	5 u/s	3.0 u	15 / 0.8s
NPC ( $\times 5$ )	60	4 u/s	2.5 u	15 / 0.5s

Table 1. Entity combat statistics

Component	Specification
CPU	AMD Ryzen 7 5000 Series
RAM	16 GB
GPU	NVIDIA GeForce RTX 4060 Laptop GPU
OS	Windows 11 Home 24H2
Storage	512 GB SSD + 1 TB HDD
Unity Editor	6.3.3 LTS (URP)

Table 2. Hardware and software specification

Block	Floats	Contents
Self	3	Normalised X position, normalised Z position, HP ratio
Bot	7	Bot X, bot Z, bot HP ratio, distance to bot (normalised), can-attack binary, nearest cover X, nearest cover Z
Allies	12	Four allies $\times$ (X, Z, HP ratio)
Strategy tag	1	Current tag / 3.0, bounded in $[0,1]$ ; always 0.0 in Combination A

Table 3. NPC observation vector (23 floats)

Idx	Name	Description
0	Move Forward	Move toward the bot’s current position.
1	Strafe Left	Move along the local left vector.
2	Strafe Right	Move along the local right vector.
3	Melee Attack	Execute melee attack via CombatHandler.
4	Retreat	Move directly away from the bot.
5	Ranged Attack	Fire orb projectile at the bot via OrbLauncher.

Table 4. Discrete action space (branch size = 6)

Tag	Name	Tactical Intent	Shaping Term
0	Surround	Encircle bot from multiple angles	$r=+0.005\times\|\sin(\theta)\|$
1	Aggressive	Rush and apply sustained pressure	$r=+0.01\times(1-d/d_{\text{max}})$
2	Flank	Approach from bot’s lateral arcs	$r=+0.0075$ if $60^{\circ}<\|\theta\|<120^{\circ}$
3	Retreat	Pull back and preserve HP	$r=\min(0.005\times d/d_{\text{max}},\ 0.005)$ ; $-0.005$ if $d<8$

Table 5. Strategy tags, tactical intent, and shaping formulations ( comboB_v02 )

Bot Type	Mean (s)	Median (s)	Std (s)	NPC Win %	Bot Win %	Timeout %
Aggressive	77.72	80.03	22.51	8.0%	90.0%	2.0%
Evasive	78.31	73.42	21.12	90.0%	1.0%	9.0%
Balanced	104.08	120.00	22.21	11.0%	29.0%	60.0%

Table 6. Combination A — descriptive statistics (

n=100

per bot type)

Bot Type	Mean (s)	Median (s)	Std (s)	NPC Win %	Bot Win %	Timeout %
Aggressive	66.54	61.98	26.51	8.0%	84.0%	8.0%
Evasive	70.48	67.15	16.21	93.0%	3.0%	4.0%
Balanced	105.82	120.01	22.71	24.0%	14.0%	62.0%

Table 7. Combination B — descriptive statistics (

n=100

per bot type)

Figure 5. Combination A time-to-defeat distributions by bot type (dashed: mean; solid: median).

Figure 6. Combination B time-to-defeat distributions by bot type (dashed: mean; solid: median).

Figure 7. Episode outcome distribution (NPC win, bot win, timeout) by combination and bot type.

Bot	U	p	r	Direction
Aggressive	6430.5	0.0005***	$-0.286$	B shorter
Evasive	6032.0	0.0117*	$-0.206$	B shorter
Balanced	3668.5	0.0011**	$+0.266$	B longer

Table 8. Mann-Whitney U test, A vs B (two-sided,

\alpha=0.05

n_{1}=n_{2}=100

)

Figure 8. Combination A vs B time-to-defeat, with Mann-Whitney $p$ -values annotated per bot type.

Tag	Name	Count	%
0	Surround	2036	83.8
1	Aggressive	103	4.2
2	Flank	107	4.4
3	Retreat	184	7.6

Table 9. LLM strategy tag selection frequency (2,430 total decisions)

Figure 9. LLM strategy tag selection frequency during Combination B evaluation, by bot type.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Component	Content
Role	You control a squad of NPC soldiers fighting an enemy bot. Choose the best squad tactic for the current situation.
State	Enemy bot HP, average NPC HP, NPCs alive, bot moving (bool), nearest NPC distance.
Action	Reply with exactly one digit: 0 = Surround, 1 = Aggressive, 2 = Flank, 3 = Retreat.
Generation	num_predict: 3 , temperature: 0.1 , top_p: 0.9 , timeout: 40s . First character parsed; invalid output retains previous tag.
Fallback	Empty, non-numeric, or out-of-range output retains the previous strategy tag and is logged.

Appendix

Table 10. Combination B LLM sidecar prompt details

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 143 papers

Choosing LLMs for Strategic Agent Interactions
Oct 13, 2026Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz +1Model Selection LLM Decision-Making
University of Exeter United Kingdom · IBM Research Switzerland
Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
Jun 18, 2026Jannik Hösch, Alessandro Sestini, Florian Fuchs +6Sequential MARL Multi-Agent Coordination
1Electronic Arts (EA) · 2KTH Royal Institute of Technology, Stockholm, Sweden
Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games
May 6, 2026Yidong He, Yutao Lai, Pengxu Yang +4Game-Playing Agents RL for Language Model Reasoning
School of Software Engineering, South China University of Technology · Department of Computer Science, University of Oxford

Oct 13, 2026J/K move · Enter open · S save

Choosing LLMs for Strategic Agent Interactions

Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz, Shiqiang Wang

University of Exeter United Kingdom · IBM Research Switzerland

Model Selection LLM Decision-Making AI Agent Benchmarks Game-Playing Agents

Open paper AskPDF arXiv

Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs

N

compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.

Abstract

LLM-Guided RL

Explore similar work

Choosing LLMs for Strategic Agent Interactions

Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution

Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games