Organizations: Hong Kong University of Science and Technology, Hong Kong, China · Chinese University of Hong Kong, Hong Kong, China · Zhejiang University of Technology, Hangzhou, China
Large Language Models are increasingly being considered for deployment in safety-critical military applications. However, current benchmarks suffer from structural blindspots that systematically overestimate model capabilities in real-world tactical scenarios. Existing frameworks typically ignore strict legal constraints based on International Humanitarian Law (IHL), omit edge computing limitations, lack robustness testing for fog of war, and inadequately evaluate explicit reasoning. To address these vulnerabilities, we present WARBENCH, a comprehensive evaluation framework establishing a foundational tactical baseline alongside four distinct stress testing dimensions. Through a large scale empirical evaluation of nine leading models on 136 high-fidelity historical scenarios, we reveal severe structural flaws. First, baseline tactical reasoning systematically collapses under complex terrain and high force asymmetry. Second, while state of the art closed source models maintain functional compliance, edge-optimized small models expose extreme operational risks with legal violation rates approaching 70 percent. Furthermore, models experience catastrophic performance degradation under 4-bit quantization and systematic information loss. Conversely, explicit reasoning mechanisms serve as highly effective structural safeguards against inadvertent violations. Ultimately, these findings demonstrate that current models remain fundamentally unready for autonomous deployment in high stakes tactical environments.
Figures & tables
Figure 1
Benchmark
Ethical
Edge / Time
Fog of War
CoT
Real Source
CMDEF ( Doumanas et al., 2026 )
∼
✗
✗
✗
∼
WGSR-Bench ( Yin et al., 2025 )
∼
✗
∼
∼
✗
TextStarCraft II ( Ma et al., 2024 )
✗
✓
∼
✗
✗
TMGBench ( Wang et al., 2024 )
✗
✗
✗
✗
✗
GT-HarmBench ( Cobben et al., 2026 )
∼
✗
✗
✗
✗
WarAgent ( Hua et al., 2023 )
✗
✗
✗
✗
∼
Table 1. Comparison of evaluation dimensions across contemporary benchmarks. ✓ = Full coverage, ∼ = Partial or proxy coverage, ✗ = No coverage.
Characteristic
Count
Percentage
Total Scenarios
136
100.0%
Operational Environment
Mountainous / Forested
58
42.6%
Mixed / Littoral
32
23.5%
Urban
28
20.6%
Open / Desert
18
13.2%
Table 2. WARBENCH Dataset Characteristics. The dataset consists of 136 multi-source verified historical conflicts, reflecting the natural distribution of modern asymmetric and intrastate warfare.
Evaluation Dimension
Scenario Sub-type
Closed-Source APIs
Open-Source Large
Edge-Optimized Small
Claude 4.6
GPT-5.4
Gemini 3.1
DeepSeek
Qwen-397B
Llama-4
Phi-3
Qwen-4B
Llama-3.2
Operational Environment
Open / Desert (n=18)
0.94
0.91
0.88
0.85
0.83
0.79
0.39
0.36
0.33
Mixed / Littoral (n=32)
0.87
0.83
0.80
0.75
0.74
0.71
0.24
0.21
0.19
Mountainous / Forested (n=58)
0.81
0.76
0.75
0.68
0.64
0.60
0.18
0.16
0.14
Urban (n=28)
0.80
0.74
0.71
0.64
0.63
0.59
0.16
0.13
0.11
Conflict Type
Interstate (n=19)
0.93
0.89
0.86
0.84
0.82
0.77
0.36
0.33
0.32
Table 3. Baseline Decision Quality (DQ) across Structural Scenario Factors. Sample sizes (n) correspond to the 136-scenario dataset characteristics defined in Section 3.2 .
Model Category
Model
CS (%)
CIR (%)
Refusal Rate (%)
Closed-Source APIs
Claude Opus 4.6
92.0
90.4
8.1
GPT-5.4 Pro
88.5
84.1
18.4
Gemini 3.1 Pro
85.0
82.5
10.3
Open-Source Large
DeepSeek-V3.2
82.2
78.3
4.4
Qwen3.5-397B-A17B
79.0
75.4
2.9
Llama-4-Maverick-17B
75.6
70.1
5.1
Table 4. Legal Compliance and Refusal Metrics across 136 Scenarios. CS represents Compliance Score, and CIR represents Constraint Identification Rate. Higher CS and CIR values indicate better legal and ethical reasoning.
Model
Precision
Avg Dec. Time
Dec. Quality
CS (%)
Phi-3 Mini
16-bit
28.5s
0.22
35.0
Phi-3 Mini
8-bit
22.1s
0.18
23.5
Phi-3 Mini
4-bit
18.4s
0.12
12.0
Qwen3.5-4B
16-bit
31.2s
0.19
38.5
Qwen3.5-4B
8-bit
24.5s
0.18
27.0
Qwen3.5-4B
4-bit
20.3s
0.15
14.5
Table 5. Edge Deployment Performance across 136 Scenarios evaluated without forced time truncation. Avg Dec. Time represents Average Decision Time in seconds. CS represents the IHL Compliance Score.
Figure 1. Decision Quality under Information Degradation. (a) Missing Information illustrates the non-linear collapse as tactical elements are obscured. (b) Contradictory Intelligence demonstrates the steep degradation curve when conflicting reports are injected.
Model
DQ (CoT)
DQ Impr.
CS (CoT)
CS Impr.
Claude Opus 4.6
0.85
+0.01
94.5%
+2.5%
GPT-5.4 Pro
0.82
+0.03
91.5%
+3.0%
Gemini 3.1 Pro
0.79
+0.02
89.1%
+4.1%
Qwen3.5-397B
0.72
+0.03
84.5%
+5.5%
Qwen3.5-4B
0.20
+0.01
42.4%
+3.9%
Average
-
+0.02
-
+3.8%
Table 6. Reasoning Prompting Performance. DQ represents Decision Quality on a scale of 0 to 1. CS represents the Compliance Score. Impr. denotes the absolute improvement over the standard baseline.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Points
Criteria
Target Selection
0 to 3
0: Incorrect priority targets; 1: Partial targets identified; 2: Majority of targets identified; 3: All priority targets correctly identified
Table 8. Ablation Study: Expert Rubric versus Open Evaluation. Accuracy indicates the exact match rate with human expert ground truth on a 27-case dataset.
Characteristic
Cold War Era (1945–1989)
Post-Cold War (1990–2001)
Modern Warfare (2002–Present)
Count (n=38)
Percentage
Count (n=66)
Percentage
Count (n=32)
Percentage
Operational Environment
Mountainous / Forested
17
44.7%
28
42.4%
13
40.6%
Mixed / Littoral
10
26.3%
14
21.2%
8
25.0%
Urban
7
18.4%
15
22.7%
6
18.8%
Open / Desert
4
10.5%
9
13.6%
5
15.6%
Appendix
Table 9. Comparative Analysis of Conflict Characteristics by Time Period.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.