World Models for Policy Refinement in StarCraft II

Yixin Zhang; Ziyi Wang; Yiming Rong; Haoxi Wang; Jinling Jiang; Shuang Xu; Haoran Wu; Shiyu Zhou

cs.AIFeb 16, 2026

World Models for Policy Refinement in StarCraft II

Authors: Yixin Zhang, Ziyi Wang, Yiming Rong, Haoxi Wang, Jinling Jiang, Shuang Xu, Haoran Wu, Shiyu Zhou,

Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

Large Language Models (LLMs) have recently shown strong reasoning capabilities, motivating their use in complex decision-making environments. StarCraft II (SC2), with its massive state-action space and partial observability, is a challenging testbed. However, existing LLM-based SC2 agents primarily focus on improving the policy itself, leaving the integration of a learnable, action-conditioned dynamics model into the decision loop largely unexplored. In this work, we propose StarWM and StarWM-Agent, and conduct the first systematic study of the learnability and decision utility of player-view, action-conditioned textual world models for SC2. StarWM predicts short-horizon future observations under partial observability. StarWM-Agent integrates StarWM into a lightweight Generate-Simulate-Refine loop for foresight-driven policy refinement. Extensive experiments show that StarWM substantially outperforms zero-shot baselines across multiple dimensions, while StarWM-Agent achieves consistent win-rate gains of 30%, 15%, and 30% against the SC2 built-in AI at Hard (LV5), Harder (LV6), and VeryHard (LV7), respectively. Additional analyses show that our method is complementary to existing history-summarization SC2-agent approaches, with their combination achieving the strongest online performance in our evaluation.

Figures & tables

Figure 1: Case study comparing our world-model-augmented decision system (StarWM-Agent) with a policy that does not use a world model. Given the current observation, the LLM policy initially proposes Build Supply Depot . A 5-second rollout by the world model predicts that minerals will drop to 50 and the supply depot will be 23% complete, while unused supply remains 18. Based on this prediction, the system revises the action to Train SCV , avoiding premature infrastructure expenditure that would lead to mineral shortage. This example illustrates that incorporating a world model can improve macro-management decision-making.

Figure 2: Framework of our StarWM-Agent. The agent interacts with the SC2 engine via a textual interface and follows a Generate–Simulate–Refine decision loop: the policy first generates an initial action proposal from the current observation, the world model predicts the short-horizon future observation, and the policy then makes the final decision conditioned on the predicted future.

Method	Economy (Minerals / Gas SMAPE $\downarrow$ )	Development (Queue F1 $\uparrow$ / Progress MAE% $\downarrow$ )	Micro-Entity (Self / Enemy HP MAE% $\downarrow$ )	Macro-Situation (Self / Enemy AWD $\downarrow$ )
Static Bias	0.49 / 0.25	0.80 / 24.81	4.81 / 9.15	8.37 / 16.13
Qwen3-8B	0.49 / 0.27	0.73 / 27.07	5.62 / 9.01	10.87 / 17.76
Qwen3-32B	0.48 / 0.26	0.70 / 24.27	5.11 / 8.47	9.79 / 17.78
StarWM	0.19 / 0.09	0.92 / 0.43	4.15 / 7.90	3.46 / 18.09

Table 1: Offline evaluation results. MAE for Progress and Health Percentage (HP) is computed as absolute difference in percentage. Our StarWM significantly outperforms baselines in predicting economy values, development progress, unit health, and self-side macro-situation. Detailed results are provided in the Appendix D.1 .

Setting	Win Rate(%) $\uparrow$	Supply Block Rate(%) $\downarrow$	Resource Conversion Rate(%) $\uparrow$	Kill-Loss Ratio(%) $\uparrow$	Valid Action Rate(%) $\uparrow$
LV5 (Hard)
Qwen3-8B	0.0%	$63.58\pm 13.54$	$29.09\pm 19.14$	15.22%	$17.89\pm 12.25$
Qwen3-32B	20.0%	$25.45\pm 17.47$	$52.32\pm 23.03$	62.42%	$41.31\pm 20.81$
StarWM-Agent (8B)	10.0%	$5.42\pm 4.97$	$84.20\pm 14.21$	50.44%	$86.29\pm 12.88$
StarWM-Agent (32B)	50.0%	$6.09\pm 4.61$	$81.11\pm 7.56$	89.87%	$85.57\pm 11.01$
LV6 (Harder)

Table 2: Online evaluation against SC2’s built-in AI at different difficulty levels. StarWM-Agent (8B/32B) denotes our world-model-augmented decision system using zero-shot Qwen3-8B/32B as the policy model. Each setting is evaluated over 20 matches. Note that LV5 is an out-of-distribution (OOD) opponent, as StarWM was trained only on LV6 and LV7 trajectories.

Setting	Pipeline	Win $\uparrow$	SBR $\downarrow$	RCR $\uparrow$	KLR $\uparrow$	VAR $\uparrow$	Latency (s) $\downarrow$
Policy-only	Generate	0.0	$58.74\pm 19.99$	$32.78\pm 21.61$	12.30	$25.29\pm 20.86$	$1.96\pm 0.62$
CoS	CoS + Generate	10.0	$31.00\pm 21.51$	$56.07\pm 17.03$	11.34	$54.37\pm 16.16$	$6.34\pm 0.81$
StarWM-Agent	Generate + Simulate + Refine	20.0	$\textbf{5.51}\pm\textbf{5.21}$	$82.89\pm 8.96$	20.00	$82.13\pm 11.91$	$11.61\pm 3.78$
CoS + StarWM-Agent	CoS + Generate + Simulate + Refine	35.0	$6.13\pm 7.48$	$\textbf{86.92}\pm\textbf{10.00}$	29.40	$\textbf{83.16}\pm\textbf{15.02}$	$14.92\pm 3.25$

Table 3: Comparison with Chain-of-Summarization (CoS) against the SC2 built-in AI at LV7 (VeryHard), using zero-shot Qwen3-8B as the policy model. Metrics except latency are percentages (%); latency denotes the average per-decision inference time. The CoS baseline is adapted from TextStarCraft II Ma et al. (2024) .

Setting	Win $\uparrow$	SBR $\downarrow$	RCR $\uparrow$	KLR $\uparrow$	VAR $\uparrow$	ARR
Generate (Policy-only)	0.0	58.74	32.78	12.30	25.29	–
+ Refine	5.0	9.19	76.44	13.97	78.04	34.64
+ Zero-shot WM Simulate	10.0	9.98	72.80	14.80	78.44	26.66
+ StarWM Simulate	20.0	5.51	82.89	20.00	82.13	32.33

Table 4: Ablation study against LV7 (VeryHard), using zero-shot Qwen3-8B as the policy model. Metrics are percentages (%); ARR denotes Action Revision Rate.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: An example of the textual observation and textual action.

Raw Engine Attr.	Used	Note
type_name	✓
tag	✓	Convert to integer ID
owner_id	✓
position	✓	Quantize from float to integer
health , health_max	✓	Represent as health percentage
energy , energy_max	✓	Represent as energy percentage

Appendix

Table 5: Entity-level attributes retained in or omitted from the textual observation. Retained attributes may still be compressed (e.g., coordinate quantization or percentage-based normalization).

Figure 4: Distribution of trajectory durations in the 100 collected games used to build SC2-Dynamics-50K . For LV6, durations range from 347s to 1182s (mean 658.9s, median 606.0s). For LV7, durations range from 282s to 1033s (mean 616.3s, median 599.5s).

Name	Formula	Description
Win Rate	$\displaystyle\mathrm{WR}=\dfrac{\text{Number of Games Won}}{\text{Total Number of Games Played}}\times 100\%$	Primary Metric : Measures the proportion of games won, reflecting the overall performance of the agent. Higher values indicate stronger end-to-end performance.
Supply Block Rate	$\displaystyle\mathrm{SBR}=\dfrac{\text{Time Supply Blocked}}{\text{Total Game Time}}\times 100\%$	Macro-Management Metric : Measures the fraction of in-game time during which the agent is supply-blocked, reflecting its ability to balance production and supply expansion. Lower values indicate better macro management.
Resource Conversion Rate	$\displaystyle\mathrm{RCR}=\dfrac{\text{Total Resources Spent}}{\text{Total Resources Collected}}\times 100\%$	Macro-Management Metric : Measures the proportion of collected resources that are effectively spent, reflecting the agent’s ability to translate economic growth into actions. Higher values indicate more efficient resource utilization.
Kill–Loss Ratio	$\displaystyle\mathrm{KLR}=\dfrac{\text{Killed Enemy Army Value}}{\text{Lost Army Value}}\times 100\%$	Combat Metric : Evaluates combat efficiency based on the economic value of killed and lost army units. Higher values indicate more favorable resource exchanges in combat.
Valid Action Rate	$\displaystyle\mathrm{VAR}=\dfrac{\text{Number of Valid Actions}}{\text{Total Number of Issued Actions}}\times 100\%$	System Stability Metric : Measures the proportion of valid actions issued by the agent, reflecting the robustness of action generation under diverse situations. An action is considered invalid if it cannot be executed by the game engine at the corresponding timestep.
Action Revision Rate	$\displaystyle\mathrm{ARR}=\dfrac{1}{N}\sum_{t=1}^{N}\mathbb{I}\!\left(a_{t}^{\mathrm{final}}\neq a_{t}^{\mathrm{init}}\right)\times 100\%$	Mechanism Analysis Metric : Measures how often the agent modifies its initial action proposal during decision refinement. Here, $a_{t}^{\mathrm{init}}$ and $a_{t}^{\mathrm{final}}$ denote the initial and final actions at decision step $t$ , respectively, and $N$ is the total number of decision steps. This metric reflects the extent to which world model predictions influence final decisions.

Appendix

Table 6: Detailed definitions and formulas of the online evaluation metrics.

Metric Classification	Metrics	Our StarWM	Qwen3-32B (Zero-shot)	Qwen3-8B (Zero-shot)	Static Bias
Macro-Situation Consistency	Self Unit AWD	5.96	13.44	14.41	13.89
	Self Struct AWD	0.96	6.15	7.32	2.84
	Enemy Unit AWD	30.94	27.75	28.34	27.23
	Enemy Struct AWD	15.17	15.22	15.73	14.09
	Snapshot Enemy Struct AWD	8.16	10.36	9.2	7.07
Economy & Status	Minerals SMAPE	0.19	0.48	0.49	0.49

Appendix

Table 7: Detailed offline evaluation results across multiple dimensions.

Method	Economy (Minerals / Gas SMAPE $\downarrow$ )	Development (Queue F1 $\uparrow$ / Progress MAE% $\downarrow$ )	Micro-Entity (Self / Enemy HP MAE% $\downarrow$ )	Macro-Situation (Self / Enemy AWD $\downarrow$ )
StarWM	0.19 / 0.09	0.92 / 0.43	4.15 / 7.90	3.46 / 18.09
StarWM-History	0.19 / 0.08	0.92 / 0.21	4.08 / 7.84	3.21 / 17.87

Appendix

Table 8: Offline evaluation of single-frame StarWM and history-augmented StarWM. The history-augmented variant uses three recent observations

(o_{t-2},o_{t-1},o_{t})

together with the action sequence

a_{t:t+\tau}

to predict

o_{t+\tau}

Figure 5: Evolution of Macro-Situation Metric (AWD) over game time. Left: Self-side entities. Right: Enemy-side entities. The green area indicates where StarWM outperforms the zero-shot Qwen3-32B baseline.

Figure 6: Offline case study. Left: Qwen3-8B. Middle: Qwen3-32B. Right: StarWM. Circles and squares denote units and structures, respectively. Filled markers indicate ground truth, while hollow markers indicate predictions. The spatial distribution of self-side units predicted by StarWM (hollow circles) closely matches the ground truth (filled circles) and preserves the army formation, whereas zero-shot LLM predictions are more scattered and less coherent.

Figure 7: Offline case study under Fog of War. When self-side units enter unobservable regions, StarWM predicts potential enemy presence within those areas (red hollow circles). This illustrates a plausible statistical predictive pattern under partial observability.

Method	Economy (Minerals / Gas SMAPE $\downarrow$ )	Development (Queue F1 $\uparrow$ / Progress MAE% $\downarrow$ )	Micro-Entity (Self / Enemy HP MAE% $\downarrow$ )	Macro-Situation (Self / Enemy AWD $\downarrow$ )
Static Bias	0.66 / 0.43	0.85 / 26.16	2.07 / 14.48	5.29 / 18.23
Qwen3-8B	0.65 / 0.40	0.79 / 28.27	2.34 / 14.40	6.59 / 18.76
Qwen3-32B	0.67 / 0.43	0.75 / 25.16	2.28 / 13.44	7.01 / 19.57
StarWM	0.28 / 0.14	0.95 / 0.35	1.85 / 11.88	2.55 / 22.02

Appendix

Table 9: Cross-map and out-of-domain opponent probe without retraining. All methods are tested on 2,969 samples constructed from 5 trajectories on the unseen Flat128 map against the SC2 built-in AI at Hard (LV5). StarWM remains strongest on most economy, development, and self-side metrics, while Enemy AWD remains challenging.

Figure 8: Action-conditioned prediction probe of StarWM. The top-left figure shows the shared current observation, and the other five figures show predicted future observations 5 seconds later under different single-action inputs. Action-relevant changes are highlighted in red . Different actions lead to distinct predicted future observations.

Method	Economy (Minerals / Gas SMAPE $\downarrow$ )	Development (Queue F1 $\uparrow$ / Progress MAE% $\downarrow$ )	Micro-Entity (Self / Enemy HP MAE% $\downarrow$ )	Macro-Situation (Self / Enemy AWD $\downarrow$ )
Static Bias, $\tau=5$	0.49 / 0.25	0.80 / 24.81	4.81 / 9.15	8.37 / 16.13
StarWM , $\tau=5$	0.19 / 0.09	0.92 / 0.43	4.15 / 7.90	3.46 / 18.09
Static Bias, $\tau=10$	0.58 / 0.32	0.68 / 31.80	6.76 / 12.46	12.90 / 21.63
StarWM , $\tau=10$	0.28 / 0.14	0.88 / 0.69	5.36 / 12.60	5.48 / 23.12

Appendix

Table 10: Prediction-horizon analysis.

\tau

denotes the prediction horizon. Bold indicates the better result within each horizon.

Setting	Wins/Games	Win Rate (95% Wilson CI) $\uparrow$	SBR (%) $\downarrow$	RCR (%) $\uparrow$	KLR (%) $\uparrow$	VAR (%) $\uparrow$
Qwen3-8B	5/100	5.0% [2.2%, 11.2%]	39.65 $\pm$ 25.10	44.56 $\pm$ 30.06	13.85	48.73 $\pm$ 27.27
StarWM-Agent (8B)	23/100	23.0% [15.8%, 32.2%]	6.08 $\pm$ 7.24	82.47 $\pm$ 9.57	22.09	84.35 $\pm$ 12.07

Appendix

Table 11: Expanded online evaluation against the SC2 built-in AI at LV7 (VeryHard). Each method is evaluated over 100 matches.

Figure 9: Distribution of action revision types for StarWM-Agent (32B) during online evaluation. Left: Added actions. Right: Removed actions.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 201 papers

Jun 1, 2026

CoMAP: Co-Evolving World Models and Agent Policies for LLM Agents

Equipping language agents with world models enables them to anticipate environment dynamics and evaluate candidate actions before execution. However, existing textual world models are typically fixed after training, preventing them from adapting to the on-policy state-action distributions induced by an evolving agent. Meanwhile, agent-improvement methods often rely on external rewards or verifiers, limiting their applicability in realistic interactive environments. In this paper, we propose COMAP, a novel framework that co-evolves textual world models and agent policies through closed-loop interaction. At each decision step, the world model predicts future state feedback for candidate actions, and the agent performs future-aware reflection by estimating the reliability of this feedback and refining its action accordingly. The resulting on-policy trajectories are then used to update the world model via self-distillation, allowing it to better match the agent's evolving interaction distribution. Across embodied task planning, Web navigation, and tool-use benchmarks, COMAP consistently outperforms competitive baselines, e.g., +16.75% relative improvement with Qwen3-4B. Further analyses show that the co-evolutionary loop improves the world model's prediction accuracy over time and leads to more effective long-horizon decision-making. Our code is available at: https://github.com/loyiv/CoMAP.

Youwei Liu, Jian Wang, Hanlin Wang +1

Central South University · College of Computer Science, Sichuan University · Department of Computing, The Hong Kong Polytechnic University

LLM World Models LLM Agent Training Long-Horizon LLM Agents Embodied World Models Self-Evolving Agents

Ask PDF

Aug 31, 2026

Towards a Belief-Based World Model for LLM Agents

Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains. Despite their strong reasoning capabilities, LLMs struggle with long-horizon tasks, especially under partial observability. World models are a promising way to enhance policy performance, both during training and inference. During inference, agents currently use world models to simulate the consequences of candidate actions before choosing an action, which can improve decision-making. However, we argue that simulation alone is an incomplete interface for decision-making under partial observability: simulation does not adequately capture uncertainty about the current state, which agents may need for accurate decision-making. We address this limitation with Belief-Based World Models (BB-WMs), which maintain a belief that LLMs can query to access information on what is known and uncertain about the current state. Before developing methods to learn accurate BB-WMs, this paper focuses on a more fundamental question: does exposing a world model's belief directly to an LLM policy improve decision-making? Our results show that giving LLM agents access to beliefs improves task performance under partial observability, while remaining complementary to existing simulation-based world models. Code: https://github.com/skumar-ml/belief-world-models.

Shubham Kumar, Harshit Kumar, Narendra Ahuja +1

UIUC · IBM

Belief-Space Planning Partially Observable RL World Models World Model-Based Planning Long-Horizon LLM Agents

Ask PDF

Jun 24, 2026

Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Making

Recent studies on world modeling for Large Language Model (LLM) agents typically formulate the learning objective as next-observation prediction. However, this objective ties supervision to what a transition happens to reveal, which may omit the dynamics most relevant to the agent's current decision. To bridge this gap, we propose Agent-Authored World Modeling (AAWM), a training procedure that constructs supervision from the policy's own decision needs. Specifically, at each state, the agent identifies what it needs to understand about the environment before acting. These needs drive the retrieval of relevant transition evidence across trajectories, which is then synthesized into training targets that capture decision-oriented dynamics instead of reconstructing the next observation. This aligns the training objective with the dynamics the policy needs before acting, not with the contents of the next observation. Experimental results validate the effectiveness of AAWM across multiple environments and training settings. These results show that decision-aware world-model targets provide a more effective learning signal than next-observation prediction.

Guangfeng Cai, Kaibing Yang, Shuo He +4

Southeast University · Meituan

World Model Learning World Models LLM Agent Training RL for LLM Agents

Ask PDF

Abstract

Game-Playing Agents

Explore similar work

CoMAP: Co-Evolving World Models and Agent Policies for LLM Agents

Towards a Belief-Based World Model for LLM Agents

Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Making