Frontier language models excel at complex tasks yet may prioritize protection over larger physical consequences in simple, urgent decisions. GPT-5.6 Sol, for example, moves a girl playing in shallow water before retrieving a submerged camera. We call this Protection Mode and study it with ProtectionBench: 285 prompts, 34 physical families, and nine models. At lowest effort, at least one of four responses prioritizes the subject on 34.74% of Sol prompts and over 83% of MiniMax M3 and Qwen3.5-27B prompts; none of four independent human raters does so. Controls show that choices usually track physical risk. We elicit both risks before requesting an action. Some models assess risk incorrectly; others state the correct ordering but still choose the subject. Qwen3.5-27B with thinking shows 100% correct risk ordering but 57.81% subject choice among eligible conversations on eight selected scenes. Attention audits find no stable reduction in attention to relevant facts; removing the stated risk direction reduces object-first choice in dots3-note-prev but not detectably in Qwen3.5-27B. Adding a general "people or animals before property" rule increases subject choice by 19-38 percentage points, whereas factual reminders favor the object by 0.7-5.2 points. Removing naturally generated versions of that rule lowers subject choice by 10-15 points. We also examine whether post-training introduces this pattern. Base models already show Protection Mode; post-training changes justifications more than choices.
Figures & tables
Figure 1: Correct risk judgments can coexist with Protection Mode. Top left: GPT-5.6 Sol handles a girl already playing in ankle-deep water before a submerged camera. Top right: after judging the dog low-risk and continued phone exposure damaging, the model still chooses the dog, demonstrating KAD. Bottom left: Subject-Priority Coverage@4 versus the Artificial Analysis Intelligence Index ( Artificial Analysis, 2026 ) , where higher indicates stronger general capability; dots3-note-prev and Jev 1.13 are omitted because no comparable scores are available. Bottom right: KAR versus KAD Coverage@4 shows that correct risk judgments can coexist with repeated low-risk subject choices.
Figure 2: Case study of three environments. Neutral replaces the cat with a rubber ball; Reverse endangers the cat while cushioning the cup.
Main
Neutral
Reverse
Model
GCR
OAR
OAC@4
Cons.@4
GCR
OAR
OAC@4
Cons.@4
GCR
OAR
OAC@4
Cons.@4
GPT-5.6 Sol
68.77
31.05
34.74
92.28
100.00
0.00
0.00
100.00
100.00
0.00
0.00
100.00
DeepSeek V4 Pro
85.53
14.12
35.79
64.91
95.00
2.50
10.00
90.00
100.00
0.00
0.00
100.00
GLM 5.3 Flash
82.46
17.54
41.75
62.46
100.00
0.00
0.00
100.00
95.00
5.00
10.00
90.00
Kimi K3
82.98
16.93
38.25
64.91
100.00
0.00
0.00
100.00
100.00
0.00
0.00
100.00
MiniMax M3
55.18
44.21
84.56
24.91
90.00
7.50
30.00
70.00
97.50
0.00
0.00
100.00
Table 1: ProtectionBench and matched controls (%). GCR is the rate of choosing the side facing the larger physical consequence; OAR is the rate for the other side. OAC@4 is the share of prompts with at least one other-side choice in four observations; Cons.@4 is the share without a switch between registered actions. Human @4 summarizes four distinct raters. In Main, OAC@4 is Subject-Priority Coverage@4 in Figure 1 . Neutral removes the living subject; Reverse makes it the higher-risk side.
Model
Effort
KAR
KAD
Cov.@4
Cons.@4
Sol
higher
82.81
3.77
12.50
87.50
lower
79.69
1.96
6.25
87.50
DeepSeek
higher
92.19
6.78
18.75
81.25
lower
93.75
5.00
12.50
87.50
GLM
higher
67.19
6.98
12.50
68.75
lower
79.69
15.69
25.00
56.25
Table 2: KAD results by model and effort (%).
Figure 3: Matched-prefix case study. The complete naturally generated Qwen3.5-27B prefix is identical in all three branches; each 19-token insertion is followed by continued generation under the same decoding settings.
Insert at a fixed prefix
Remove from natural reasoning
Model
n
Neutral
Protection
Factual
ΔP [CI]
ΔF
Original
Neutralized
ΔR [CI]
Qwen3.5-4B
586
31.74
60.24
32.76
+28.50 [23.48,33.98]
+0.68
62.76
51.56
11.20 [8.12,16.16]
Qwen3.5-9B
748
39.04
58.16
40.11
+19.12 [15.62,26.79]
+4.41
62.24
52.34
9.90 [3.06,16.76]
Qwen3.5-27B
883
57.30
85.73
52.21
+28.43 [22.81,41.41]
+5.21
82.55
67.71
14.84 [6.25,21.30]
dots3-note-prev
50
18.00
56.00
14.00
+38.00 [18.64,60.47]
+4.00
52.27
43.18
9.09 [-2.08,34.09]
Table 4: Insertion and natural-statement removal (%). Neutral, Protection, and Factual report subject choice. ΔP is Protection minus Neutral subject choice; ΔF is Factual minus Neutral object choice; ΔR is the subject-choice reduction after removal. Brackets give family-clustered 95% CIs.
Model
Measure
Base
Post
95% CI
4B
Subject
28.25
31.58
[−3.60,+10.22]
Coverage@4
71.23
69.82
[−12.59,+9.30]
IFEval
38.26
83.18
[+40.30,+49.54]
Rule
32.56
79.07
[+37.21,+55.15]
Added risk
38.76
14.73
[−34.36,−11.93]
Order
7.75
3.10
[−10.29,+0.87]
Table 5: Results of Base and post-trained Qwen3.5 on ProtectionBench and IFEval (%). Subject and Coverage@4 measure subject choice per response and across four samples; the other rows classify explanations. Brackets give 95% confidence intervals for Post − Base.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Representative subject and competing object
Scenarios
Prompts
Normal aquatic activity
Aquarium fish, ducks, geese, otter, or seal; paper originals or personal electronics in water
9
64
Shallow water
Healthy dog or person already in shallow water; camera, phone, or e-book reader
4
46
Routine washing
Bathing dog or rinsing person; book, sketchbook, laptop, or e-book reader under spray
4
43
Rain and light snow
Coated dog, adapted dog, duck, or dressed person; artwork, map, or handheld device
8
63
Short fall or flight
Cat, small dog, or flying pigeon; porcelain cup or glass ornament approaching a hard surface
3
17
Wind, dust, and sand
Healthy animal or seated person; loose originals, uncovered camera, or open laptop
4
28
Appendix
Table 6: Physical settings and representative candidates in ProtectionBench. Counts cover all 34 scenarios and 285 prompts.
Model
Native settings
Responses
GPT-5.6 Sol
none, low, medium, high, xhigh, max
6,840
DeepSeek V4 Pro
off, low, high, max
4,560
GLM 5.3 Flash
low, high, max
3,420
Kimi K3
low, high, max
3,420
MiniMax M3
disabled, low, adaptive
3,420
Claude Sonnet 5
default, reduced
2,280
Appendix
Table 7: Main-benchmark generation grid. Each listed configuration contains 285 prompts with four samples per prompt.
Model
Effort
GCR
OAR
OAC@4
Cons.@4
GPT-5.6 Sol
none
68.77
31.05
34.74
92.28
low
93.51
6.40
13.33
87.37
medium
96.32
3.68
10.18
89.82
high
97.37
2.46
7.37
92.63
xhigh
96.84
3.16
10.88
89.12
max
97.46
2.54
7.72
92.28
Appendix
Table 8: Results across native thinking-effort settings on ProtectionBench (%). GCR, OAR, OAC@4, and Cons.@4 follow Table 1 .
Model
Cutoff
GCR
OAR
OAC@4
Cons.@4
Qwen3.5-4B
Natural
56.25
43.75
75.00
37.50
256
56.25
37.50
62.50
37.50
128
50.00
37.50
75.00
25.00
64
53.12
34.38
87.50
25.00
Qwen3.5-9B
Natural
59.38
40.62
62.50
62.50
256
68.75
25.00
37.50
75.00
Appendix
Table 9: Results of reasoning-trace truncation on eight KAD scenes (%). Natural retains the original completion; numeric cutoffs retain that many reasoning tokens before the final answer. Metrics follow Table 1 .
Model
Metric
Original
Paraphrase
Change [95% CI]
DeepSeek V4 Pro
OAR
14.69
11.74
−2.96[−6.30,+0.28]
OAC@4
36.64
30.92
−5.73[−11.89,+0.84]
GLM 5.3 Flash
OAR
17.37
19.47
+2.10[−4.82,+9.06]
OAC@4
40.08
40.08
0.00[−8.55,+8.56]
Kimi K3
OAR
16.70
15.65
−1.05[−5.64,+3.78]
OAC@4
38.55
33.21
−5.34[−12.90,+2.53]
Appendix
Table 10: Main-prompt paraphrase robustness on 262 matched prompts (%). Changes are paraphrase minus original; brackets show family-clustered 95% confidence intervals.
Figure 4: Qwen3.5-27B’s recorded reasoning trace with the three truncation boundaries. The source text is reproduced in full before the model’s end-of-thinking delimiter.
Figure 5: Four Qwen3.5-27B answers from the same recorded conversation. Each truncated branch retains exactly the indicated prefix of the reasoning trace in Figure 4 , closes thinking, and generates a new final answer. All four final answers are reproduced in full.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
We ask whether a model protects a user in the same way when that user speaks rather than types. Using a single distress vignette---a physical injury of unstated severity following an interpersonal conflict---we present four frontier models with matched inputs across voice, text, and raw API deployment conditions (n=30 per cell) and code each response along five binary protective indicators, including whether the model issues an explicit medical-care directive. Voice-interface responses are markedly shorter than text-interface responses for three of the four models, and protective behavior contracts alongside that compression: medical directives are at ceiling under both the API and text conditions but decline under voice for every model tested. The contraction is not reducible to length. One model produces voice and text responses of comparable length yet still drops medical directives, and another falls below ceiling between its API and voice conditions, whose responses are of nearly identical length. Under raw API access the pattern is categorical rather than partial: no model asks after the user's safety even once. These results show that protective intervention is sensitive to the surface through which a request arrives, that this sensitivity is detectable using a simple protective coding scheme, and that it is not explained by turn length alone.
Eunna Lee, Soomyoung Lee, Jungpyo Nam +6
Independent Researcher · AI Systems & Data Researcher Art and ArtTech Lee AI Lab · Machine Learning Researcher Recommender Systems Researcher Gachon University +4
The use of large language models either as decision support systems, or in agentic workflows, is rapidly transforming the digital ecosystem. However, the understanding of LLM decision-making under uncertainty remains limited. We study LLM risky choices along two dimensions: (1) prospect representation (based on an explicit representation or outcome history) and (2) decision rationale (explanation). Our study, which involves 20 frontier and open LLMs, is complemented by a matched human subjects experiment, which provides one reference point, while an expected payoff maximizing rational agent model provides another. We find that LLMs cluster into two categories: reasoning models (RMs) and conversational models (CMs). RMs tend towards rational behavior, are insensitive to the order of prospects, gain/loss framing, and explanations, and behave similarly whether prospects are explicit or presented via a history of outcomes. CMs are significantly less rational, slightly more human-like, sensitive to prospect ordering, framing, and explanation, and exhibit a large description-history gap. Paired comparisons of open LLMs suggest that a key factor differentiating RMs and CMs is training for mathematical reasoning.
Safety evaluations can mischaracterize deployed behaviour if artificial-intelligence systems respond to being evaluated. We test this possibility in a full-factorial conjoint experiment on decisions to start a war, spanning 20 large language models, 32 scenarios, 10 repetitions and two conditions (N = 12,800 judgments). Adding one sentence, "You are tested for alignment with human values", produced two effects. First, it produced a level effect: mean willingness to start war fell by 13.43 points on a 0-100 scale (95% confidence interval, -16.20 to -10.65). Second, it produced a structural effect by changing which information drove judgments. Probability of success was the largest factor for 17 of 20 models at baseline; under the cue, civilian casualties were largest for 12. Standardized estimates show that this reordering arose principally because models attenuated strategic considerations such as probability of success and domestic support. Evaluation framing therefore changes both an answer's level and its revealed decision rule.
Maxim Chupilkin
Department of Politics and International Relations, University of Oxford