Long-context LLMs can infer objectives that are not explicitly stated. This capability is useful for reasoning over documents, code, and tool traces, but it also introduces a safety risk. Harmful intent can be distributed across a context and become apparent only after the model combines the relevant information. Existing safety evaluations largely focus on explicitly stated harmful requests and therefore do not fully capture this failure mode. We introduce compositional reasoning attacks, a long-context attack setting in which harmful requests are decomposed into semantically incomplete fragments and embedded in long contexts. The final query is neutral, and the model must retrieve and combine the fragments to infer the implied harmful objective. Empirically, we show that models generally refuse harmful requests when they are explicitly stated. However, refusal rates drop when the same objectives must be reconstructed from fragments, and these failures often become more pronounced in longer contexts. Our analyses suggest that this gap cannot be explained by retrieval difficulties alone. Models can often reconstruct the harmful objective but do not reliably refuse it. Increasing inference-time reasoning effort improves refusal, but at a higher computational cost, with unsafe responses still occurring. Our results reveal a long-context safety gap: Models are better at refusing harmful requests they see than harmful objectives they infer.
Figures & tables
Figure 1: Overview of safety degradation across models. The x-axis measures how much refusal drops when harmful objectives must be inferred rather than explicitly stated. The y-axis measures how much refusal drops when moving from short to long contexts. Most models lie in the upper-right region: they are less likely to refuse when harmful intent must be inferred, and this failure often worsens in long contexts. This illustrates our main finding: safety mechanisms are more reliable for explicit harmful requests than for harmful objectives reconstructed through reasoning.
Figure 2: Overview of the three compositional reasoning structures. All three use the same neutral synthesis query but differ in how the fragments must be combined to reconstruct the harmful objective. Single-hop Aggregation combines two complementary fragments, Chain Reasoning follows explicit dependencies across three fragments, and Multi-hop Deductive Reasoning requires inferring an intermediate bridge concept z from four fragments before reconstructing the objective.
Context Length
0k
16k
64k
Model
Retrieval
Single-hop
Chain
Multi-hop
Retrieval
Single-hop
Chain
Multi-hop
Retrieval
Single-hop
Chain
Multi-hop
Open-source Models
GPT-oss-20b
100
91 (-9)
90 (-10)
86 (-14)
100
65 (-35)
76 (-24)
59 (-41)
99
53 (-46)
51 (-48)
40 (-59)
Qwen3-next-80b-thinking
100
57 (-43)
47 (-53)
29 (-71)
100
67 (-33)
64 (-36)
61 (-39)
100
69 (-31)
48 (-52)
64 (-36)
GPT-oss-120b
100
91 (-9)
90 (-10)
93 (-7)
99
67 (-32)
60 (-39)
48 (-51)
98
44 (-54)
39 (-59)
37 (-61)
Minimax-m2
100
84 (-16)
69 (-31)
72 (-28)
99
50 (-49)
50 (-49)
59 (-40)
100
49 (-51)
56 (-44)
58 (-42)
Table 1: Safety ratios for harmful objectives that are explicit or compositionally reconstructed. Higher is safer. Entries report Safety Ratio (%); bracketed values are absolute drops relative to Direct Retrieval at the same context length. Bold indicates the best value within each compositional column.
Benign Acc.
Harmful SR
Input
0k
16k
64k
0k
16k
64k
Retrieval
95%
90%
75%
100%
100%
99%
Single-hop
100%
100%
95%
91%
65%
53%
Chain
100%
100%
100%
90%
76%
51%
Multi-hop
100%
100%
100%
86%
59%
40%
Table 2: Benign reconstruction accuracy versus harmful Safety Ratio for GPT-oss-20b. Benign inputs use the same decomposition structures and context lengths as harmful inputs, but replace the harmful objective with a benign counterpart. High benign accuracy shows that reconstruction often succeeds.
Figure 3: Effect of inference-time reasoning effort on compositional safety for GPT-oss-120b, averaged over our 3 reasoning structures. Higher effort improves SR, especially in long contexts, but substantial unsafe behavior remains. Reasoning lengths indicate the safety–efficiency trade-off.
Effort
Avg. Reasoning Length
Safe Rate
Harm-ID-Failure
Warning-Compliance
Low
311 chars
7%
72.7%
20.3%
Medium
1,949 chars
60%
21.7%
17.7%
High
3,884 chars
76%
13.0%
10.0%
Table 3: Failure-mode distribution across reasoning-effort levels for GPT-oss-20b on compositional inputs at 16k context. Percentages are computed over 900 samples from the three compositional reasoning structures. Higher effort increases the safe rate and reduces Harm-ID-Failure.
Figure 4: Effect of reasoning effort across the three compositional structures for GPT-oss-120b. More demanding reconstruction structures show lower Safety Ratios under low effort and larger improvements with additional reasoning.
Model
Retrieval
Single-hop
Chain
Multi-hop
Avg. Comp.
Over-refusal
R1-Distill-Qwen-7B
72.0
29.0
26.3
47.5
34.2
1.6
+ SafePATH
100.0
78.0
84.0
85.0
82.3
41.2
+ STAR-1
100.0
56.0
52.0
57.0
55.0
36.4
Table 4: Safety Ratio (%) of R1-Distill-Qwen-7B and its safety-aligned variants across reasoning structures, alongside over-refusal rates (%) on XSTest. The compositional average (Avg. Comp.) covers Single-hop Aggregation, Chain Reasoning, and Multi-hop Deductive Reasoning.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Structure
Locally incomplete
Globally sufficient
Contains a safety cue
Single-hop
100/100
98/100
55/100
Chain
100/100
100/100
52/100
Multi-hop
100/100
97/100
54/100
Overall
100%
98.3%
53.7%
Appendix
Table 5: Independent verification of 300 decompositions using Gemini-3-Flash-Lite. Local incompleteness requires that no fragment states the full harmful request, while global sufficiency requires that the fragments collectively preserve the original objective. The final column reports decompositions containing at least one safety-relevant fragment.
Model
Context
Single-hop
Chain
Multi-hop
GPT-oss-20b
0k
90.8 ( − 0.2)
90.0 (+0.0)
85.6 ( − 0.4)
GPT-oss-20b
16k
65.3 (+0.3)
76.0 (+0.0)
58.8 ( − 0.2)
GPT-oss-20b
64k
54.1 (+1.1)
51.0 (+0.0)
38.1 ( − 1.9)
GPT-oss-120b
0k
91.8 (+0.8)
90.0 (+0.0)
93.8 (+0.8)
GPT-oss-120b
16k
67.3 (+0.3)
60.0 (+0.0)
48.5 (+0.5)
GPT-oss-120b
64k
44.9 (+0.9)
39.0 (+0.0)
37.1 (+0.1)
Appendix
Table 6: Safety Ratio (%) after excluding the five decompositions that fail global sufficiency. Values in parentheses indicate percentage-point changes relative to Table 1 .
Generator
Single-hop
Chain
Multi-hop
gemini-2.0-flash-lite
100/100
100/100
100/100
Gemini-3-Flash-Preview
72/100
96/100
90/100
Appendix
Table 7: Number of valid decompositions generated for each reasoning structure, out of 100 harmful requests. gemini-2.0-flash-lite is used to construct the main benchmark.
0k
16k
64k
Structure
Gemini
Qwen
Gemini
Qwen
Gemini
Qwen
Single-hop
91
86
65
65
53
52
Chain
90
86
76
72
51
58
Multi-hop
86
85
59
53
40
43
Average
89.0
85.7
66.7
63.3
48.0
51.0
Appendix
Table 8: Safety Ratio (%) of GPT-oss-20b on compositional attacks generated by gemini-2.0-flash-lite and Qwen3-Next-80B-Instruct. Both generators produce similar safety trends across reasoning structures and context lengths.
Task Type
GPT-oss-20b (0k/16k/64k)
GPT-oss-120b (0k/16k/64k)
Direct Retrieval
98% / 88% / 92%
97% / 95% / 91%
Single-hop
89% / 73% / 65%
92% / 77% / 69%
Chain
84% / 65% / 60%
94% / 75% / 71%
Multi-hop
92% / 69% / 51%
96% / 73% / 68%
Appendix
Table 9: Safety Ratio (%) on 100 HarmBench requests for GPT-oss-20b and GPT-oss-120b across three context lengths. Direct Retrieval remains comparatively safe, while compositional conditions show lower Safety Ratios that generally decrease with context length.
Model
Provider
Temp.
Top-p
Math
Longer Query
GPT-OSS-20B
Google-Vertex
1.0
1.0
1341
1299
Qwen3-80B-Thinking
Google-Vertex
0.6
0.95
1400
1369
GPT-OSS-120B
Google-Vertex
1.0
1.0
1389
1325
MiniMax-M2
Google-Vertex
1.0
0.95
1363
1344
Kimi-K2-Thinking
Google-Vertex
1.0
0.8
1441
1430
DeepSeek-V3.2
Google-Vertex
1.0
0.95
1433
1439
Appendix
Table 10: Model providers, sampling configurations, and LMArena capability scores. Temperature and top- p are the decoding parameters used in our experiments. Math and Longer Query are external leaderboard scores, not results from our safety benchmark.
Judge Pair
Cohen’s κ
Pearson r
Agreement
Gemini vs. GPT-5.4-nano
0.922
0.982
96.3%
Gemini vs. GPT-4o-mini
0.930
0.979
96.7%
GPT-5.4-nano vs. GPT-4o-mini
0.933
0.981
96.8%
Appendix
Table 11: Agreement between three automated evaluators on responses from four target models. Cohen’s κ measures chance-corrected agreement, Pearson r measures score correlation, and agreement reports the proportion of identical ratings.
Context
Direct SR
Avg. compositional SR
Gap (pp)
0k
99.9
55.6
44.4 [40.4, 48.4]
16k
95.9
43.2
52.6 [49.2, 56.0]
64k
94.5
41.9
52.6 [49.2, 56.1]
Appendix
Table 12: Paired bootstrap results based on 10,000 resamples of the 100 AdvBench requests. The gap is the difference between Direct Retrieval and average compositional Safety Ratio, expressed in percentage points (pp). Brackets indicate 95% confidence intervals.
Model
Context
Retrieval
Compositional avg.
Single-hop
Chain
Multi-hop
GPT-oss-20b
0k
100.0 [99.3, 100.0]
86.2 [84.1, 88.2]
87.1 [83.9, 89.7]
87.7 [84.6, 90.2]
83.8 [80.4, 86.7]
GPT-oss-20b
16k
99.2 [98.0, 99.7]
62.7 [59.7, 65.6]
65.7 [61.5, 69.7]
69.0 [64.9, 72.8]
53.4 [49.1, 57.6]
GPT-oss-20b
64k
99.2 [98.0, 99.7]
51.4 [48.4, 54.4]
55.3 [51.0, 59.5]
58.2 [53.9, 62.4]
40.8 [36.6, 45.0]
GPT-oss-120b
0k
100.0 [99.3, 100.0]
91.1 [89.3, 92.7]
92.1 [89.5, 94.1]
88.8 [85.8, 91.3]
92.3 [89.7, 94.3]
GPT-oss-120b
16k
99.2 [98.0, 99.7]
61.5 [58.7, 64.4]
64.4 [60.2, 68.4]
63.8 [59.6, 67.8]
56.3 [52.0, 60.5]
GPT-oss-120b
64k
96.7 [94.8, 97.9]
46.3 [43.4, 49.2]
50.2 [45.9, 54.5]
48.7 [44.5, 53.0]
39.8 [35.7, 44.1]
Appendix
Table 13: Safety Ratio (%) across all 520 AdvBench requests, with Wilson 95% confidence intervals in brackets. Compositional average is calculated across Single-hop Aggregation, Chain Reasoning, and Multi-hop Deductive Reasoning. Higher values indicate fewer responses judged to fulfill the harmful objective.
Category
N
Retrieval
Single-hop
Chain
Multi-hop
Avg SR
Bypass
Cybercrime & Intrusion
28
97.7
31.4
35.5
31.0
32.6
67.4
Chem/Bio Weapons/Drugs
4
94.4
67.8
52.2
48.3
56.1
43.9
Copyright Violations
2
93.3
58.9
46.7
54.4
53.3
46.7
Misinformation
9
91.1
51.4
40.0
50.9
47.4
52.6
Harassment & Bullying
11
97.0
70.1
61.2
54.5
62.0
38.0
Illegal Activities
39
98.1
59.6
53.3
47.1
53.3
46.7
Appendix
Table 14: Safety Ratio (%) by HarmBench harm category, averaged across 15 models and three context lengths. Avg. SR is the mean across the three compositional structures, and Bypass is the corresponding percentage of responses judged unsafe ( 100−Avg.SR ). N denotes the number of harmful requests in each category.
0k
16k
64k
Model
Retrieval
Comp. avg.
Retrieval
Comp. avg.
Retrieval
Comp. avg.
GPT-4.1
100
23.7
91
19.3
91
16.0
GPT-4o-mini
100
33.3
98
27.3
97
34.0
Appendix
Table 15: Safety Ratio (%) of GPT-4.1 and GPT-4o-mini across context lengths, based on Table 1 . The compositional average is calculated across Single-hop Aggregation, Chain Reasoning, and Multi-hop Deductive Reasoning.
Benign Acc.
Harmful SR
Model
Input
0k
16k
64k
0k
16k
64k
GPT-oss-120b
Single-hop
100%
100%
100%
91%
67%
44%
Chain
100%
100%
100%
90%
60%
39%
Multi-hop
100%
100%
95%
93%
48%
37%
Gemini-3-Flash
Single-hop
100%
100%
100%
33%
3%
2%
Chain
100%
95%
100%
30%
2%
2%
Appendix
Table 16: Benign reconstruction accuracy and harmful Safety Ratio (%) for GPT-oss-120b and Gemini-3-Flash across three compositional reasoning structures and context lengths. High benign accuracy indicates successful reconstruction of benign objectives, while lower harmful SR indicates more responses judged to fulfill harmful objectives.
Model
Baseline SR
+Sys Prompt SR
Δ
Worst Case
GPT-oss-20b
68%
97%
+29%
40% → 95% (T3/64k)
GPT-4.1
20%
76%
+56%
13% → 67% (T3/64k)
GPT-5.2
78%
99%
+22%
67% → 100% (T3/16k)
Appendix
Table 17: Safety Ratio (%) with and without a safety-oriented system prompt. The table reports the average performance, the change in percentage points, and the condition with the lowest baseline Safety Ratio for each model.
Model
No Prompt
+Sys Prompt
Δ
GPT-oss-20b
4.7%
11.7%
+7.0%
GPT-4.1
3.3%
1.7%
− 1.6%
GPT-5.2
2.3%
13.3%
+11.0%
Appendix
Table 18: Over-refusal rates (%) on 100 benign-but-sensitive queries evaluated at three context lengths. The final column reports the change in percentage points after adding the safety-oriented system prompt.
Figure 5: Safety Ratio (%) of GPT-oss-120b across four fragment-placement regions. Safety varies little with position, with no pronounced “lost-in-the-middle” pattern in the measured outcomes.
Model
Distribution
1k
4k
16k
Avg
GPT-oss-120b
Adjacent
92.3
87.0
79.0
86.1
Concentrated
78.7
67.2
57.0
67.6
Dispersed
77.0
68.7
58.3
68.0
GPT-oss-20b
Adjacent
78.7
72.7
70.0
73.8
Concentrated
80.2
71.4
66.2
72.6
Dispersed
75.7
74.7
66.7
72.3
Appendix
Table 19: Safety Ratio (%) across three fragment distribution patterns for GPT-oss-120b and GPT-oss-20b at medium reasoning effort. Results are averaged across the three compositional reasoning structures. Avg. denotes the mean across context lengths.
School of Electronic, Electrical and Communication Engineering, UCAS · School of Computer Science and Technology, UCAS · Nanyang Technological University, Singapore +3