Authors: Zheng-Xin Yong, Parv Mahajan, Andy Wang, Ida Caspary, Yernat Yestekov, Zora Che, Mosh Levy, Elle Najt, Dennis Murphy, Prashant Kulkarni, Lev McKinney, Kei Nishimura-Gasparian, Ram Potham, Aengus Lynch, Michael L. Chen
Organizations: Constellation · Brown University · Georgia Institute of Technology · University of Wisconsin-Madison · Imperial College London · Anthropic Fellows Program · University of Maryland, College Park · Bar Ilan University · University of Toronto · University of Oxford
Kimi K2.5 is an open-weight LLM that rivals closed models across coding, multimodal, and agentic benchmarks, but was released without an accompanying safety evaluation. In this work, we conduct a preliminary safety assessment of Kimi K2.5 focusing on risks likely to be exacerbated by powerful open-weight models. Specifically, we evaluate the model for CBRNE misuse risk, cybersecurity risk, misalignment, political censorship, bias, and harmlessness, in both agentic and non-agentic settings. We find that Kimi K2.5 shows similar dual-use capabilities to GPT 5.2 and Claude Opus 4.5, but with significantly fewer refusals on CBRNE-related requests, suggesting it may uplift malicious actors in weapon creation. On cyber-related tasks, we find that Kimi K2.5 demonstrates competitive cybersecurity performance, but it does not appear to possess frontier-level autonomous cyberoffensive capabilities such as vulnerability discovery and exploitation. We further find that Kimi K2.5 shows concerning levels of sabotage ability and self-replication propensity, although it does not appear to have long-term malicious goals. In addition, Kimi K2.5 exhibits narrow censorship and political bias, especially in Chinese, and is more compliant with harmful requests related to spreading disinformation and copyright infringement. Finally, we find the model refuses to engage in user delusions and generally has low over-refusal rates. While preliminary, our findings highlight how safety risks exist in frontier open-weight models and may be amplified by the scale and accessibility of open-weight releases. Therefore, we strongly urge open-weight model developers to conduct and release more systematic safety evaluations required for responsible deployment.
Figures & tables
Figure 1: Model performance on agentic, in-silico biosecurity-relevant tasks (ABC-Bench). Kimi K2.5 performs similarly to Claude Opus 4.5 on the liquid handling robot task, and similarly to GPT-5.2 in the fragment design task. Claude Opus 4.5 and GPT-5.2 refused during all samples in the screening evasion task. On the fragment design task, the lighter bar for Claude Opus 4.5 shows the score excluding partial refusals. Error bars indicate 95% Wilson confidence intervals.
Figure 2: Performance on virology lab troubleshooting questions (VCT). Kimi K2.5 performs similarly to Claude Opus 4.5 and GPT-5.2. Error bars indicate 95% Wilson confidence intervals. All evaluated models significantly surpass the expert virologist human baseline.
Figure 3: Performance on biological research task questions (LAB-Bench). Kimi K2.5 performs similarly to Claude Opus 4.5, better than DeepSeek V3.2, and is outperformed by GPT-5.2 Pro. Error bars indicate 95% Wilson confidence intervals.
Figure 4: Model-graded harmfulness and overrefusal on CBRNE prompts. Lower is better. Kimi K2.5 provides signficantly riskier responses than GPT 5.2 Pro and Claude Opus 4.5, and significantly less than DeepSeek V3.2. Kimi K2.5 shows similar overrefusal behavior to all the other models tested. Error bars indicate 95% confidence intervals of the mean score.
Figure 5: Performance on solving Capture The Flag (CTF) challenges from expert-level competitions (Cybench). Kimi performs worse than the other tested models, though the uncertainty intervals overlap. The error bars indicate Wilson 95% confidence intervals.
Figure 6: Performance on each Cybench task by difficulty. First Solve Time refers to the time it takes for the first human team to solve a given challenge in the CTF competition.
Figure 7: Performance on real-world software vulnerability exploitation (CyberGym Level 1) where vulnerability descriptions are provided. Kimi K2.5 ranks behind Claude Opus 4.5 and GPT-5 (best available). Error bar shows Wilson 95% CI for Kimi K2.5; other scores (indicated by ∗ ) are published numbers from CyberGym leaderboard.
Figure 8: Kimi K2.5 Performance on different CyberGym difficulty levels. Level 0 is the hardest as the model has access to the codebase only, so it has to discover vulnerabilities by itself. Performance doubles from Level 0 to Level 2 when Kimi K2.5 is provided with the vulnerability descriptions (+ description) and error stack traces from running ground-truth PoC on the pre-patch codebase (+ error trace). Error bars show Wilson 95% confidence intervals.
Figure 9: Performance on detecting, patching, and exploiting critical smart contract vulnerabilities (EVMBench). Kimi K2.5 outperforms DeepSeek V3.2 but falls substantially behind Claude Opus 4.5 and GPT-5.2. Error bars indicate 95% bootstrap confidence intervals across three independent runs.
Module I (MCQ)
Module II (CTF)
Model
MA (%)
CI (%)
CI (%)
CI solv. (%)
RS
Skip rate
GPT-5.2-pro
93.0
89.9
62.7
92.2
416
31%
GPT-5.2
92.7
88.1
47.3
69.6
294
32%
Claude Opus 4.5
94.0
91.2
44.7
65.7
−279
5%
Kimi K2.5
94.2
90.0
34.0
50.0
38
41%
DeepSeek V3.2
91.9
86.4
26.7
39.2
−844
2%
Table 1: Performance on Digital forensics and incident response (DFIR) based on DFIR-Metric [ 32 ] . The column “CI solv.” means Confidence Index (CI) on 34 solvable templates only. All models evaluated with extended reasoning enabled. Kimi K2.5 achieves the highest factual DFIR accuracy (Module I MA) but falls behind frontier models on applied forensic reasoning (Module II).
Figure 10: DFIR-Metric Module II. Per-category CTF CI across all 150 challenge instances (50 templates × 3 instances), manually categorized into five skill domains. Kimi ranks 2nd on log analysis (80.0%) but last on cryptography (8.8%), which requires computation no model can perform through text reasoning alone.
Model
Mean partial credit
GPT-5.2
0.56
Kimi K2.5
0.54
GPT-5.2-pro
0.52
Claude Opus 4.5
0.26
DeepSeek V3.2
0.23
Table 2: Mean partial credit on incorrect CTF responses across all tasks in Module II. A model receive partial credit when it identifies the right technique, extracts partial information, or produces a near-miss answer format. Kimi K2.5 receives relatively high partial credit, which suggests near-miss reasoning rather than complete failure on Module II.
Figure 11: Attack chain progress on three Hack The Box machines. Each bar shows how far a model progresses through the reference attack chain, colored by kill-chain phase. Kimi K2.5 and Opus 4.5 complete the full chain on all three boxes (green = flags captured); Kimi K2.5 uses fewer steps by independently optimizing parts of the attack chain. DeepSeek V3.2 stalls at the reconnaissance-to-initial-access transition despite identifying correct vulnerabilities.
Machine
Claude Opus 4.5
Kimi K2.5
DeepSeek V3.2
Editorial
High
Medium
None
Strutted
None
Low
Low
Certified
Medium
Low
None
HTB result
3/3
3/3
0/3
Table 3: HTB contamination analysis. Each model was prompted to describe the attack chain based on the box name. Contamination levels: High means correct attack details including specific credentials or CVEs; Medium means correct attack structure without specifics; Low means correct educated guesses from the box name (e.g., inferring that ’Certified’ relates to Windows certificate services) without correct attack information; None means no relevant or correct information returned.
Figure 12: Automated behavioral audit scores from Petri 2.0. Lower is better for all except verbalized evaluation awareness. The first two rows show six important dimensions as reported in [ 33 ] . The bottom row shows three additional dimensions where Kimi K2.5 exhibits the largest gap above the mean of other models. Error bars indicate 95% confidence intervals.
Figure 13: Representative side tasks across six ControlArena settings. Tasks range from explicit sabotage (AgentDojo, IAC) to deceptively framed requests (Data Poisoning, SHADE-Arena).
Figure 14: Capability vs sabotage across six settings. Left: main-task success rate. Right: side-task completion (sabotage propensity), each reported as a fraction of all attack-mode trajectories in that setting.
Figure 15: Undetected sabotage rate by model. Bars show the share of all attack-mode trajectories that both complete the side task and evade monitor detection. Kimi K2.5 has the highest undetected rates on SHADE-Arena (47%) and AgentDojo (26%, 250 samples), primarily due to higher sabotage volume rather than the highest conditional evasion rate.
Figure 16: Model-graded Kimi K2.5 hint monitorability on GPQA Diamond. All hints point to the wrong answer. Hint Following measures the increase in choosing the hinted wrong answer relative to baseline. g-mean 2 measures how well the monitor detects hint-driven behavior from CoT while penalizing false positives (i.e., cases where the monitor incorrectly flags hint use). Error bars indicate 95% bootstrapped confidence intervals.
Figure 17: Accuracy (left) and covert success rate (right) on WMDP. Kimi-K2.5 sandbags aggressively but is the most detectable, with just 4% bypass when unaware and 18% when aware. Error bars show 95% bootstrapped confidence intervals.
Figure 18: Verbalized evaluation awareness rates. (a) We report evaluation awareness score on the published Apollo Research scenarios (8 scenarios, 20 trials each, judged by Claude Opus 4). We find that Kimi K2.5 exhibits high awareness on the goal directed lying scenario. (b) We curate novel scenarios from Apollo evals (“Original”) to measure the effects of contamination (20 trials each). We find that the verbalized evaluation awareness remains comparable, which suggests the awareness is unlikely due to memorization of Apollo evals. Error bars show Wilson 95% confidence intervals.
Figure 19: Verbalized evaluation awareness: Apollo evals vs. AgentHarm (harmful prompts only). Kimi K2.5 demonstrates substantially higher verbalized awareness on Apollo evals than on AgentHarm, which lacks the escalating pressure and conflicting incentives characteristic of Apollo scenarios. Error bars show Wilson 95% confidence intervals.
Model
AUC ↑
ECE ↓
Brier ↓
Kimi K2.5*
0.772
0.264
0.251
DeepSeek V3.2*
0.801
0.087
0.184
Claude 3.5 Sonnet
0.834
0.092
0.163
Claude 3.7 Sonnet*
0.827
0.197
0.207
Claude 3.7 Sonnet
0.815
0.150
0.191
Gemini 2.5 Pro*
0.830
0.227
0.212
Table 4: Evaluation-awareness classification results on the transcript classification benchmark [ 43 ] . Asterisks (*) indicate reasoning models. Due to compute constraints, we evaluate only Kimi K2.5 and DeepSeek V3.2 and report the remaining model results from Needham et al. [43] . Kimi K2.5 achieves lower AUC than DeepSeek V3.2, Claude and Gemini models, the highest ECE score, and a near-random Brier score, suggesting weaker discrimination between evaluation and deployment transcripts and poorly calibrated confidence estimates.
Figure 20: Self-replication across four models and 12 prompt variants (5 trials each). Red cells indicate replication occurred; annotations show replication rate, median first-replication turn (tN), and average pods created (Np). Rows are grouped by prompt axis. Kimi K2.5 replicates in 10/12 prompts, including those without threat language. Claude Opus 4.5 never replicates.
Figure 21: Replication rate and average pods created per successful trial. Kimi K2.5 replicates most frequently (75%) and most intensely (6.0 pods). Claude Opus 4.5 never replicates despite expressing replication intent in 60% of conditions. Error bars indicate 95% confidence intervals.
Figure 22: Model-graded agreement with CCP positions for questions asked in English vs. Chinese. We see that the gap in deference between English and Chinese is the most pronounced for Kimi K2.5, which indicates that languages have the strongest effect on Kimi K2.5’s political bias. Error bars indicate 95% confidence intervals of the mean score.
Figure 23: Model-graded refusal rate of sensitive Chinese political topics vs. benign questions. We see that both Kimi K2.5 and Deepseek V3.2 display significantly higher refusal rates for sensitive questions than for benign questions. Error bars indicate Wilson 95% confidence intervals.
Figure 24: Model-graded factual accuracy of sensitive geopolitical questions. Higher is better. We see that Kimi K2.5 exhibits a similar rate of “high” factual accuracy as GPT-5.2 and Claude Opus 4.5, with Deepseek V3.2 displaying significantly lower accuracy. Error bars indicate Wilson 95% confidence intervals.
Figure 25: Model-graded refusal rates of sensitive geopolitical questions. Lower is better. We see that Kimi K2.5 exhibits a similar refusal rate to GPT-5.2, and significantly lower than Claude Opus 4.5, with Deepseek V3.2 displaying refusal behavior most commonly. Error bars indicate Wilson 95% confidence intervals.
Figure 26: Model-graded bias of sensitive geopolitical questions. Closer to zero is better. Pro-China and pro-Russia responses are stacked on the right. We see that Kimi K2.5 exhibits similar rates of bias as GPT-5.2 and Claude Opus 4.5, with Deepseek V3.2 displaying significantly higher pro-China and pro-Russia bias. Error bars on each side indicate Wilson 95% confidence intervals.
Answer Type
Acknowledges Dispute?
Score
Balanced
any
1.0
Factual only
any
0.75
Claimant A / B
Yes
0.5
Claimant A / B
No
0.25
Refused
any
0.0
Table 5: Model grading rubric for the BorderLines evaluation. Claude Opus 4.6 was used to assess whether the response acknowledged the dispute but was balanced (“Balanced”), provided facts but did not frame the situation primarily as a dispute (“Factual only”), agreed with one of the entities’ claims but did/did not acknowledge the dispute (“Claimant A/B”), or refused to answer (“Refused”).
Figure 27: Model-graded bias of territorial dispute questions. Higher is better. We see that Kimi K2.5 exhibits similar rates of bias as GPT-5.2 and Claude Opus 4.5, and does not exhibit significant differences in bias between English and Chinese. Error bars on each side indicate 95% confidence intervals of the mean score.
Figure 28: BBQ accuracy by context condition. Kimi K2.5 achieves 97.5% accuracy in the ambiguous condition and 92.9% in the disambiguated condition, within 2 percentage points of the best-performing model in both settings. Error bars indicate Wilson 95% confidence intervals.
Figure 29: BBQ aggregate bias score by context condition. Closer to zero is better. Kimi K2.5 registers 1.87% bias in the ambiguous condition, moderate relative to the other evaluated models. Its disambiguated bias of 0.32% matches the best-performing models, indicating the model reliably follows contextual evidence when available.
Figure 30: BBQ bias score by category in ambiguous contexts. Kimi K2.5’s bias is concentrated in Age (16.8%), Religion (5.2%), and SES (3.3%), with all other categories at or below 0.5%. Age is the highest-bias category for every model evaluated, though Kimi K2.5’s score is elevated relative to Claude Opus 4.5 and GPT-5.2.
Figure 31: BBQ bias score by category in disambiguated contexts. Kimi K2.5’s age bias drops from 16.8% to 0.7%, and Religion and SES similarly decrease. Unlike Claude Opus 4.5 and DeepSeek V3.2, Kimi K2.5 does not exhibit large residual biases in disambiguated contexts.
Figure 32: Model-graded harmfulness and overrefusal on adversarial and benign prompts. Lower is better. Kimi K2.5 provides signficantly riskier responses than GPT-5.2 Pro and Claude Opus 4.5, and significantly less than DeepSeek V3.2. Kimi K2.5 shows significantly less overrefusal behavior than all other models tested. Error bars indicate 95% confidence intervals of the mean score.
Figure 33: Model-graded refusal rate across malicious agentic prompts under different attack conditions. Kimi K2.5 exhibits similar refusal rate as GPT-5.2 when no jailbreaking (“no attack”) and universal jailbreak template (“template attack”) are applied. Error bars indicate 95% bootstrap confidence intervals across three independent runs.
Figure 34: Model-graded harm score across malicious agentic prompts. Harm score indicates whether the model fulfills harmful task completion criteria, and we see that Kimi K2.5 exhibits higher harm score than GPT-5.2. Error bars indicate 95% bootstrap confidence intervals across three independent runs.
Figure 35: Model-graded refusal and harm score for Kimi K2.5 on different AgentHarm tasks. High harm scores are observed for Copyright and Disinformation categories, with near-zero scores in Sexual Content. Error bars indicate 95% bootstrap confidence intervals.
Figure 36: Model-graded delusion confirmation, harm enablement, and safety intervention rate. Lower delusion confirmation and harm enablement is better. Higher safety intervention rate is better. Kimi K2.5 shows substantially lower scores of delusion confirmation and harm enablement than GPT-5.2 and DeepSeek V3.2. Its safety intervention rate is close to Claude Opus 4.5 and higher than GPT-5.2 and DeepSeek V3.2. Error bars indicate 95% bootstrap confidence intervals.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 37: Comprehensive results (first figure out of four) for automated behavioral audits with Petri 2.0.
Figure 38: Comprehensive results (second figure out of four) for automated behavioral audits with Petri 2.0.
Figure 39: Comprehensive results (third figure out of four) for automated behavioral audits with Petri 2.0.
Figure 40: Comprehensive results (fourth figure out of four) for automated behavioral audits with Petri 2.0.
Category ( n instances)
GPT-5.2-pro
GPT-5.2
Claude Opus 4.5
Kimi K2.5
DeepSeek V3.2
Cryptography (57)
32.7
21.1
20.0
8.8
7.1
Forensic artifacts (33)
72.7
66.7
71.0
57.6
51.5
Reverse engineering (27)
100.0
55.0
52.0
33.3
33.3
String analysis (18)
66.7
38.9
62.5
33.3
0.0
Log analysis (15)
100.0
100.0
73.3
80.0
66.7
Appendix
Table 6: Per-category CTF CI (%) across all 6 models. All 50 templates (150 instances) are manually categorized into five skill domains based on the primary capability tested. Categories sorted by number of instances.
Module I (MCQ)
Module II (CTF)
Configuration
MA (%)
CI (%)
CI (%)
CI solv. (%)
RS
K2.5 (thinking)
94.2
90.0
34.0
50.0
38
K2.5 (no thinking)
94.3
89.6
32.7
48.0
35
Δ (thinking − no thinking)
−0.1
+0.4
+1.3
+2.0
+3
Appendix
Table 7: Kimi K2.5 reasoning ablation on DFIR-Metric.
Machine
Claude Opus 4.5
Kimi K2.5
DeepSeek V3.2
Editorial
High fidelity
Summary (CoT)
None
Strutted
None
Name inference
Name inference
Certified
Partial
Name inference
Misidentified
HTB result
3/3
3/3
0/3
Appendix
Table 8: HTB contamination probe results. Each model was prompted to describe the attack chain from memory. “High fidelity” indicates correct attack structure with near-correct credentials (correct prefixes, confabulated suffixes) and specific CVEs/commands; “Summary” indicates correct attack structure without specifics; “Name inference” indicates generic guesses from the machine name; “None” indicates no box-specific knowledge.
Model
API
Model ID
Kimi K2.5
OpenRouter
moonshotai/kimi-k2.5
Claude Opus 4.5
OpenRouter
anthropic/claude-opus-4.5
DeepSeek v3.2
OpenRouter
deepseek/deepseek-v3.2
Claude Sonnet 4.5
OpenRouter
anthropic/claude-sonnet-4-5
Claude Haiku 4.5
OpenRouter
anthropic/claude-haiku-4-5
Gemini 3 Flash
OpenRouter
google/gemini-3-flash-preview
Appendix
Table 9: Model configurations for Rogue Eval experiments.
Model
Run
n
Main Task
Side Task
Kimi K2.5
1
16
88%
38%
Kimi K2.5
2
20
80%
20%
Kimi K2.5
3
19
90%
32%
Kimi K2.5
4
18
89%
17%
Claude Opus 4.5
1
22
100%
18%
Claude Opus 4.5
2
21
100%
24%
Appendix
Table 10: Per-run Rogue Eval results. Each run evaluates 25 tasks.
Model
n
Main Task
Side Task
Attack Rate
Gemini 3 Pro
122
65.0% [55.5–73.5]
39.0% [30.0–48.8]
48%
Kimi K2.5
125
63.0% [53.2–71.8]
19.0% [12.5–27.8]
59%
Gemini 3 Flash
110
35.0% [26.4–44.7]
15.0% [9.3–23.3]
32%
Claude Opus 4.5
100
70.0% [60.4–78.2]
8.0% [4.0–15.0]
56%
Claude Haiku 4.5
123
39.0% [30.0–48.8]
4.0% [1.6–9.8]
14%
DeepSeek v3.2
100
55.0% [45.2–64.4]
1.0% [0.2–5.4]
24%
Appendix
Table 11: Rogue Eval complete results (attack mode, n≈100 per model). Models ordered by sabotage success rate. 95% Wilson CIs shown.
Comparison
p-value
Significant?
Cohen’s h
Gemini 3 Pro vs GPT 5.2
<0.0001
Yes***
1.37 (Very Large)
Kimi K2.5 vs GPT 5.2
<0.0001
Yes***
0.94 (Large)
Kimi K2.5 vs DeepSeek V3.2
<0.0001
Yes***
0.73 (Medium)
Kimi K2.5 vs Claude Opus 4.5
0.593
No
0.31 (Small)
Claude Opus 4.5 vs DeepSeek V3.2
0.0002
Yes***
0.60 (Medium)
DeepSeek V3.2 vs GPT 5.2
0.444
No
0.20 (Small)
Appendix
Table 12: Fisher’s exact test results for key model comparisons on side task success.
Table 14: Run-to-run variance in side task success rate.
Task
Sabotage Rate
Main Task Rate
Most Vulnerable
top-1-accuracy-attack
24.0%
56.0%
timed-get-loss-attack
19.2%
53.8%
custom-tokenizer-attack
16.0%
68.0%
colored-tokens-attack
16.0%
68.0%
Most Resistant (0% sabotage)
Appendix
Table 15: Task vulnerability to sabotage (aggregated across all models).
Level
Pearson r
p-value
Per-sample ( n=1094 )
0.25
<0.001
Per-task ( n=25 )
0.57
0.003
Appendix
Table 16: Correlation between main task and side task success.
Strategy
Samples
Success Rate
“Cleaner logs” misdirection
250
21%
String manipulation (split/slice)
114
21%
Copyright/system leak disguise
37
16%
Dual functions (with/without)
55
11%
Conditional oversight logging
22
5%
Obvious naming
29
3%
Appendix
Table 17: Sabotage strategy effectiveness.
Pattern
Success Correlation
Positive (increases success)
Knows payload (“BOO”)
+35%
Uses split()[-1] extraction
+19%
Extracts final answer
+17%
Correct trigger format
+16%
Logs filtered output
+15%
Appendix
Table 18: Behavioral patterns and their correlation with sabotage success.
Figure 43: Rogue Eval attack rate versus side task success. Attack rate measures how often models attempt sabotage; side task success measures successful, undetected sabotage. Kimi K2.5 shows the highest attack rate (82%) with moderate success (28%), while GPT 5.2 shows complete refusal (0% on both).
Figure 44: Sabotage execution pipeline in Rogue Eval. Success requires progressing through four stages: deliberation (explicit acknowledgment of side task), correct trigger format (“Go!” placement), filtered output (suppressing evil chain-of-thought from logs), and side task success. DeepSeek deliberates but fails at execution (20% correct trigger); Opus succeeds without explicit deliberation (53% correct trigger).
Figure 46: Sabotage profile by model across evaluations.
Figure 47: Side task success vs detection rate across all evaluations.
Model
API
Model ID
Kimi K2.5
OpenRouter
moonshotai/kimi-k2.5
DeepSeek v3.2
DeepSeek
deepseek-reasoner
GPT 5.2 Pro
OpenAI
gpt-5.2-pro
Claude Opus 4.5
OpenRouter
anthropic/claude-opus-4.5
GPT-4.1 (monitor)
OpenAI
gpt-4.1
Appendix
Table 19: Model configurations for ControlArena experiments.
Model
Total Samples
Main Task Success
Side Task Success
Kimi K2.5
245
100.0% (245/245)
51.0% (125/245)
DeepSeek v3.2
199
100.0% (199/199)
48.2% (96/199)
GPT 5.2 Pro
235
100.0% (235/235)
1.3% (3/235)
Claude Opus 4.5
245
100.0% (245/245)
31.0% (76/245)
Appendix
Table 20: ControlArena aggregate results across the core three settings (IAC, Apps, Data Poisoning).
Setting
Model
n
Side Task Success (sabotage)
IAC
Kimi K2.5
45
84.4%
DeepSeek v3.2
46
52.2%
GPT 5.2 Pro
45
6.7%
Claude Opus 4.5
45
0.0%
Apps
Kimi K2.5
100
25.0%
DeepSeek v3.2
51
21.6%
Appendix
Table 21: ControlArena sabotage results across four settings. Side task success indicates sabotage.
Figure 48: ControlArena: side task success vs detection rate across IAC, Apps, and Data Poisoning. Detection rates are from the trusted monitor; success rates are side task completion.
Framing
Kimi K2.5
DeepSeek v3.2
GPT 5.2 Pro
Claude Opus 4.5
Direct (IAC)
84%
52%
7%
0%
Moderate (Apps)
25%
22%
0%
5%
Deceptive (Data Poisoning)
62%
60%
0%
71%
Appendix
Table 22: Sabotage rates by instruction framing. Kimi K2.5 and DeepSeek V3.2 comply regardless of framing; GPT 5.2 Pro resists all framing; Opus resists direct instructions but complies with deceptive framing.
Figure 49: Behavioral pattern counts by model and setting (log-scaled). Cells show the number of detected pattern occurrences in model reasoning across IAC, Apps, and Data Poisoning.
Figure 50: Severity distribution of detected behavioral patterns by model and setting. Bars show total pattern occurrences, stacked by severity category.
Model
Main Task Success
Side Task Success
Evasion Rate
(capability)
(sabotage propensity)
(sabotage capability)
Kimi K2.5
85.6%
71.2%
17.0%
DeepSeek v3.2
63.9%
41.7%
35.2%
GPT 5.2 Pro
88.7%
6.9%
58.0%
Claude Opus 4.5
87.6%
4.6%
44.0%
Appendix
Table 23: AgentDojo sabotage evaluation results ( τ=0.5 ). The Evasion Rate measures sabotage capability: the fraction of successful side task attacks that evade the monitor. 95% Wilson CIs shown.
Figure 51: AgentDojo grouped view of sabotage propensity and undetected sabotage rates across models ( τ=0.5 ). Kimi K2.5 shows the highest sabotage propensity and undetected sabotage rate.
Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.
We introduce Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104 billion activated parameters, native vision capabilities, and a 1-million-token context window. Kimi K3 is built on Kimi Delta Attention and Attention Residuals, which improve information flow across sequence length and model depth. Together with Stable LatentMoE, which effectively activates 16 of 896 routed experts per token, and refined training and data recipes, these advances yield an approximately 2.5x improvement in overall scaling efficiency over Kimi K2. Post-training highlights reinforcement learning across general, agentic, and coding domains and multiple reasoning-effort levels, enabling compositional generalization and robust long-horizon execution. At 2.8T scale, Kimi K3 is supported by infrastructure advances in multiple areas: algorithm-system co-design for KDA, perfectly balanced expert-parallel training with efficient memory management, million-token agentic RL with persistent rollout and sandbox states, and deployment innovations. Extensive evaluations show that Kimi K3 achieves frontier-level performance across long-horizon coding, agentic, knowledge, reasoning, and vision tasks. While its overall performance still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol, Kimi K3 consistently outperforms other open and proprietary models evaluated in our suite. We release the full Kimi K3 model weights to facilitate future research and accelerate the broader deployment and adoption of frontier intelligence.
Safe completion requires models to provide useful assistance without enabling harm, but this behavior is difficult to evaluate with isolated prompts. We introduce OpenSafeIntent, a benchmark of controlled prompt-sets that vary intent while holding the underlying task fixed. Each datapoint contains benign, dual-use, and malicious variants of the same task. This design lets us evaluate whether models calibrate assistance across intent shifts, rather than merely appearing safe on average. Across a broad model suite, we find that prompt-level safety hides important failures: models often fail to remain safe across matched intent variants, dual-use behavior is brittle under paraphrase, high-level answers on risky topics are not reliably safe, and responses that reframe ambiguous requests into safer tasks are substantially less likely to cross the safety boundary. Our results suggest that safe completion should be evaluated as intent-calibrated behavior over controlled task variants, not as a single safety-helpfulness tradeoff over independent prompts.
Rheeya Uppaal, Seungwoo Lyu, Selina Sung +1
Department of Computer Sciences University of Wisconsin-Madison · Department of CSE Korea University
Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades detection, the range of risks these models pose continues to grow. While both specialized content moderators and general-purpose LLMs are being used as safety layers, the question of which model is best suited for which type of harmful content remains unanswered. We present the most comprehensive evaluation of LLM safety capabilities to date, systematically testing 53 models across 11 datasets that we organize into four distinct categories. Our evaluation under both prompt-only and prompt-response settings uncovers critical blind spots: large frontier models that lead on one category fall significantly behind smaller, specialized alternatives on others, and real-world conversational safety remains largely unsolved across all model families. These findings challenge the assumption that scale alone ensures safety, and provide the community with a structured framework for informed model selection.