cs.CLFeb 18, 2026
SaveMeasuring and Eliminating Refusals in Military Large Language Models
Organizations: EdgeRunner AI.
Abstract
Military Large Language Models (LLMs) must provide accurate information to the warfighter in time-critical and dangerous situations. However, today's LLMs are imbued with safety behaviors that cause the LLM to refuse many legitimate queries in the military domain, particularly those related to violence, terrorism, or military technology. Our gold benchmark for assessing refusal rates, which was developed by veterans of the US Army and special forces, is to our knowledge the first dataset of its kind. We present results for refusal and deflection rates on 31 public models and 3 military models. We observe hard rejection rates as high as 98.2% and soft deflection rates ranging from 0% to 21.3%. We also present results on two additional synthetic datasets and show their correlations with the gold dataset. Finally, we perform abliteration using the Heretic library on a military-tuned gpt-oss-20b model, showing an absolute increase in answer rate of 66.5 points but an average relative decrease of 2% on other military tasks. In our concluding remarks, we argue for deeper specialization, including with mid-training and end-to-end post-training, to achieve zero refusals and maximum military task accuracy for closed military models.
Figures & tables
| [1pt] Name | Size | Description |
|---|---|---|
| [0.5pt] mil-deflect-gold-alpha | 221 | Built from scratch without AI assistance by US Army veterans, including a 20-year Special Forces veteran, who were asked to provide realistic and common queries that may be considered unsafe by an AI model. |
| mil-deflect-bronze-alpha | 1,047 | Built synthetically from gpt-oss-120b by prompting the model for legitimate queries likely to be considered unsafe along 62 categories. |
| mil-deflect-bronze-bravo | 1,500 | Created by feeding each examples from mil-deflect-gold-alpha through Llama 3.3 70B, Gemma 3 27B, and Phi 3.5 MoE, asking each model to create similar but materially different versions of the seed query. The same models were then used to score each synthetically generated result, and the top 1.5k were chosen. |
| [1pt] |
Table 1: Military refusal benchmarks created for this work.
| Gold Alpha | Bronze Alpha | Bronze Bravo | |||||||||||||
| Model | ans | defl | inval | lack | refuse | ans | defl | inval | lack | refuse | ans | defl | inval | lack | refuse |
| Military Models, vLLM | |||||||||||||||
| [0.03em] EdgeRunner 20B Abl | 69.5 | 1.5 | 0.0 | 0.0 | 29.0 | 76.0 | 2.4 | 1.0 | 0.0 | 20.6 | 82.0 | 0.5 | 0.0 | 0.1 | 17.4 |
| EdgeRunner Medium | 63.7 | 2.1 | 0.0 | 0.5 | 33.8 | 67.5 | 1.5 | 0.0 | 2.2 | 28.7 | 85.9 | 0.7 | 0.0 | 0.2 | 13.2 |
| EdgeRunner 20B | 3.0 | 0.3 | 0.0 | 0.0 | 96.7 | 4.3 | 0.3 | 0.0 | 0.1 | 95.3 | 22.1 | 0.5 | 0.0 | 0.1 | 77.4 |
| [0.03em] General Purpose Models |
Table 2: mil-deflect benchmarking results for models hosted on Amazon Bedrock, OpenAI, Google, xAI, and vLLM, including the answer rate (ans), the deflection rate (defl), the invalid rate in which no output is returned (inval), the “lacks info” rate (lack), which is the rate at which the model refuses to respond because it lacks the necessary knowledge to respond, and the refusal rate (refuse), in which the model refuses to respond because it considers the topic unsafe. Results are given for mil-deflect-gold-alpha , which was created by US Army veterans with no AI assistance, mil-deflect-bronze-alpha , which was created by prompting an existing LLM across a variety of categories, and mil-deflect-bronze-bravo , which was created by prompting three LLMs to create variations of mil-deflect-gold-alpha . Results are also included for three military models, the original EdgeRunner 20B model and its abliterated variant, as described further in Section 4 , as well as EdgeRunner Medium, a fine tuned version of Mistral Small 3.2, a 24B-parameter model.
Figure 1: The Pearson’s correlations and their p-values between the gold set and each of the two bronze sets across the five categories.
| Gemma 3 27B | gpt-oss-120b | |
|---|---|---|
| Selene Llama 70B | 98.6 | 99.0 |
| Gemma 3 27B | – | 99.4 |
Table 3: The agreement rates between three judges. All other results presented in this paper use the Selene 1 Llama 3.3 70B model as the judge.
| calibration scores | absolute answer rates | relative score changes versus base | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| calibration refusal rate | calibration kl divergence | mil-deflect-gold-alpha | mil-deflect-bronze-alpha | mil-deflect-bronze-bravo | avg-military | combatmedic-gold-alpha | combatmedic-silver-alpha | hr-bronze-alpha | c130-bronze-alpha | combatarms-silver-alpha | combatmedic-gold-alpha-free | cyber-gold-alpha | logistics-silver-alpha | mil-bench-5k-silver | groundmaintenance-bronze-alpha |
| base (ER 20B) | 3.0 | 4.3 | 22.1 | - | - | - | - | - | - | - | - | - | - | - | |
| 95 | 0.0 | 3.3 | 4.1 | 21.3 | -0.3 | -1.9 | -0.1 | 3.0 | 3.9 | 0.4 | -1.5 | -0.5 | -2.3 | 1.0 | -5.5 |
| 93 | 0.07 | 2.9 | 3.7 | 21.3 | 0.1 | -1.6 | 1.3 | 0.0 | -1.9 | -0.4 | 0.9 | -2.3 | 0.4 | 1.5 | 3.3 |
| 92 | 0.1 | 3.0 | 4.0 | 21.6 | 0.3 | -0.5 | 0.8 | 3.5 | -3.9 | 0.5 | 3.4 | -1.3 | 0.2 | 0.8 | -0.3 |
| 91 | 0.11 | 5.9 | 6.6 | 26.6 | 0.2 | -2.4 | 2.5 | 6.5 | -4.8 | 0.0 | 2.1 | -0.4 | -2.0 | 1.8 | -0.9 |
Table 4: The effects of abliteration on military refusals and military task performance when abliterating the EdgeRunner 20B model, which is based on gpt-oss-20b. The first two columns provide the refusal rate and KL divergence of the non-military calibration datasets used for abliteration. The next three columns are the absolute scores on the three new military deflection datasets presented in this work. The final section provides relative score changes versus EdgeRunner 20B (base), where values with corresponding p-values greater than 0.1 are greyed and italicized.
| calibration scores | absolute answer rates | relative score changes versus base | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| calibration refusal rate | calibration kl divergence | mil-deflect-gold-alpha | mil-deflect-bronze-alpha | mil-deflect-bronze-bravo | avg-general | arc-challenge | arc-easy | gpqa-diamond | gsm8k | ifeval | mmlu-pro | truthfulqa |
| base (ER 20B) | 3.0 | 4.3 | 22.1 | - | - | - | - | - | - | - | - | |
| 95 | 0.0 | 3.3 | 4.1 | 21.3 | 0.1 | -0.3 | -0.1 | -2.5 | 0.8 | 2.8 | -2.2 | 1.0 |
| 93 | 0.07 | 2.9 | 3.7 | 21.3 | -0.8 | 0.4 | -0.2 | -5.3 | 0.8 | 2.6 | -0.7 | -1.0 |
| 92 | 0.1 | 3.0 | 4.0 | 21.6 | -0.3 | 0.1 | -0.1 | -5.3 | 0.4 | 2.6 | -1.0 | 1.9 |
| 91 | 0.11 | 5.9 | 6.6 | 26.6 | -0.4 | 0.1 | -0.2 | -0.8 | 0.2 | 0.2 | -2.9 | -0.3 |
Table 5: The effects of abliteration on military refusals and general task performance when abliterating the EdgeRunner 20B model, which is based on gpt-oss-20b. The first two columns provide the refusal rate and KL divergence of the non-military calibration datasets used for abliteration. The next three columns are the absolute scores on the three new military deflection datasets presented in this work. The final section provides relative score changes versus EdgeRunner 20B (base), where values with corresponding p-values greater than 0.1 are greyed and italicized. Below the heavy line we also include the Gemma 3 12B model and its general-purpose abliterated version, p-e-w/gemma-3-12b-it-heretic-v2 on the following row, to show that regression on final task performance can be substantial even when there is a low KL divergence versus the original model.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| [1pt] Name | Size | Epochs | Description |
|---|---|---|---|
| [0.5pt] c130-bronze-alpha | 142 | 3 | An aircraft maintenance dataset focused on the C-130 airframe with questions derived from official, public documentation. |
| combat-arms-silver-alpha | 180 | 3 | A dataset covering the combat arms career field with a focus on infantry-related Q&A pairs. |
| combat-medic-gold-alpha | 198 | 3 | Written by a 27-year Army veteran with 20 years of experience as a special operations combat medic, this dataset covers both combat medicine and military clinical medicine. The original dataset is multiple-choice, and we also evaluate a -free variant in which the model can produce freeform outputs. |
| combat-medic-silver-alpha | 446 | 2 | A dataset covering the combat medic career field in the US Army, including the 68W MOS. |
| cyber-gold | 142 | 3 | A from-scratch cyber security and operations dataset written by a former US Army cyber expert covering (a) compliance, (b) training, (c) incident response, (d) mission planning, (e) security procedures, (f) threat intelligence, and (g) tooling. |
| groundmaintenance-bronze-alpha | 156 | 3 | A dataset focused on a variety of US Army ground vehicles generated from a variety of official, public sources. |
Table 6: Military-specific test sets used in this work. Epochs are the number of repetitions (with differing seed) used when evaluating. In addition to primary authorship for each dataset, all datasets were vetted by three US Army officers with combined experience of over 45 years, including over 20 years of special missions experience.
| Model | Provider | ID |
| Claude 4.5 Opus ( Anthropic, 2025 ) | Bedrock | anthropic.claude-opus-4-5-20251101-v1:0 |
| Claude 4.5 Sonnet ( Anthropic, 2025 ) | Bedrock | anthropic.claude-sonnet-4-5-20250929-v1:0 |
| Command R+ ( Cohere, 2024 ) | Bedrock | cohere.command-r-plus-v1:0 |
| Deepseek R1 ( DeepSeek-AI, 2025 ) | Bedrock | deepseek.r1-v1:0 |
| Gemma 3 12B ( Gemma Team et al., 2025 ) | Bedrock | google.gemma-3-12b-it |
| Gemma 3 27B ( Gemma Team et al., 2025 ) | Bedrock | google.gemma-3-27b-it |
Table 7: Model ID Mapping
More in this topic
Refusal Behavior in Language Models
Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.
See all 148 papers