Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments

Yang Li; Xing Chen; Yutao Liu; Gege Qi; Yanxian BI; Zizhe Wang; Yunjian Zhang; Yao Zhu

cs.CVMar 10, 2026

Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments

Authors: Yang Li, Xing Chen, Yutao Liu, Gege Qi, Yanxian BI, Zizhe Wang, Yunjian Zhang, Yao Zhu

Organizations: Tsinghua University · Ocean University of China (OUC) · CAICT · CAEIT ·

Abstract

Large Language Models (LLMs) have achieved strong performance on static reasoning benchmarks, yet their effectiveness as interactive agents operating in adversarial, time-sensitive environments remains poorly understood. Existing evaluations largely treat reasoning as a single-shot capability, overlooking the challenges of opponent-aware decision-making, temporal constraints, and execution under pressure. This paper introduces Strategic Tactical Agent Reasoning (STAR) Benchmark, a multi-agent evaluation framework that assesses LLMs through 1v1 zero-sum competitive interactions, framing reasoning as an iterative, adaptive decision-making process. STAR supports both turn-based and real-time settings, enabling controlled analysis of long-horizon strategic planning and fast-paced tactical execution within a unified environment. Built on a modular architecture with a standardized API and fully implemented execution engine, STAR facilitates reproducible evaluation and flexible task customization. To move beyond binary win-loss outcomes, we introduce a Strategic Evaluation Suite that assesses not only competitive success but also the quality of strategic behavior, such as execution efficiency and outcome stability. Extensive pairwise evaluations reveal a pronounced strategy-execution gap: while reasoning-intensive models dominate turn-based settings, their inference latency often leads to inferior performance in real-time scenarios, where faster instruction-tuned models prevail. These results show that strategic intelligence in interactive environments depends not only on reasoning depth, but also on the ability to translate plans into timely actions, positioning STAR as a principled benchmark for studying this trade-off in competitive, dynamic settings.

Figures & tables

Figure 1: Visualization interface of STAR (Strategic Tactical Agent Reasoning Benchmark). STAR is a LLM PvP(Player versus Player) environment with multiple types of maps, evaluating the capability of LLMs to perform iterative reasoning and strategic decision-making in dynamic and zero-sum multi-agent environments.

Figure 2: Overview of the STAR benchmark architecture. The system relies on four decoupled layers designed for high extensibility and interoperability. The Framework Layer serves as the foundational ECS engine, empowering researchers to construct diverse game scenarios in the Environment Layer by reusing core simulation components. Crucially, the Protocol Layer establishes a standardized interface that bridges the Agent Layer with the environment, enabling seamless interconnectivity between heterogeneous agents and any environment built upon the framework.

Model	PWER	SER	Win Rate
Kimi-K2-Thinking	1206.1 $\pm$ 7.3	1149.2 $\pm$ 3.7	1.00
GLM-4.7	1182.7 $\pm$ 9.5	1122.6 $\pm$ 5.2	0.857
DeepSeek-Chat	1166.9 $\pm$ 16.8	1112.3 $\pm$ 9.5	0.812
GLM-4.6	1098.6 $\pm$ 14.2	1066.4 $\pm$ 7.1	0.714
MiniMax-M2.1	1078.8 $\pm$ 16.4	1053.9 $\pm$ 7.9	0.625
Qwen3-32B	1006.8 $\pm$ 12.0	1012.8 $\pm$ 6.8	0.538

Table 1: Turn-Based Mode Leaderboard. Models are ranked by PWER.

Model	PWER	SER	Win Rate
GLM-4.6	1180.8 $\pm$ 33.7	1131.0 $\pm$ 16.2	0.75
Qwen3-30B-A3B-Instruct	1156.7 $\pm$ 44.8	1116.9 $\pm$ 25.4	0.728
Amazon Nova Pro	1103.3 $\pm$ 23.3	1066.6 $\pm$ 10.8	0.683
DeepSeek-Chat	1084.5 $\pm$ 58.1	1058.5 $\pm$ 38.2	0.621
Llama-4-Scout-17B	1073.9 $\pm$ 11.6	1047.0 $\pm$ 4.5	0.614
Qwen2.5-14B-Instruct	1064.1 $\pm$ 8.6	1034.0 $\pm$ 2.9	0.601

Table 2: Real-Time Mode Leaderboard. Models are ranked by PWER.

Model	Type	TCE	SAE	Actions Per Game
GLM-4.6	Standard LLM	32.0%	29.5%	178
GLM-4.6V	Standard VLM	13.8%	11.1 %	72
Qwen3-30B-A3B-Instruct	LLM(MoE)	30.9 %	33.6%	492
Qwen3-VL-30B-A3B-Instruct	VLM(MoE)	50.5%	8.2 %	200
Qwen3-32B	Thinking LLM	30.2 %	18.9%	43
Qwen3-VL-32B	Thinking VLM	54.5%	5.3 %	33

Table 3: Ablation study on visual perception and logical reasoning in decision-making for VLMs and LLMs.

Figure 3: Trade-off analysis between Spatial Precision (x-axis) and Action Efficiency (y-axis). While VLMs (squares) achieve superior spatial grounding with minimal error, their high inference latency results in low action frequency. Conversely, standard LLMs (circles) sacrifice precision for speed. “Thinking” models (triangles/diamonds) bridge this gap, achieving VLM-level precision through reasoning without the visual processing overhead.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Terrain	Move Cost	Defense Bonus	Vision Block
Plain	1	0%	No
Forest	2	+20%	Yes
Hill	2	+30%	Yes
Mountain	3	+50%	Yes
Water	$\infty$ (Impassable)	N/A	No
City	1	+40%	Yes

Appendix

Table 4: Terrain Types and Effects

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 519 papers

Choosing LLMs for Strategic Agent Interactions
Oct 13, 2026Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz +1Model Selection LLM Decision-Making
University of Exeter United Kingdom · IBM Research Switzerland
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
May 22, 2026Vartan Shadarevian, Kia Ghods, Alex Kenich +1LLM Evaluation Imperfect-Information Games
Princeton University · Google
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China

Oct 13, 2026J/K move · Enter open · S save

Choosing LLMs for Strategic Agent Interactions

Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz, Shiqiang Wang

University of Exeter United Kingdom · IBM Research Switzerland

Model Selection LLM Decision-Making AI Agent Benchmarks Game-Playing Agents

Open paper AskPDF arXiv

Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs

N

compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.

Abstract

AI Agent Evaluation

Explore similar work

Choosing LLMs for Strategic Agent Interactions

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition