As frontier language models grow increasingly capable and autonomous, independent third-party evaluations are critical for characterizing their systemic safety risks. While closed-source releases are routinely accompanied by comprehensive system cards, open-weight models receive less scrutiny regarding frontier risks. To address this gap, we present an independent safety evaluation of DeepSeek V4.1 Flash-1017, the latest version of V4.1 Flash that departs from V4 with a new architecture, scaled-up training, and strong agentic capabilities. Drawing on the EU General-Purpose AI Code of Practice, we evaluate the model across 34 benchmarks, including 5 in-house suites, spanning evaluation integrity, loss of control, and harmful manipulation. Conforming to the AEF-1 standard, we compare V4.1 Flash-1017 against V4.1 Flash, earlier DeepSeek V4 models, and leading closed-source systems (GPT-6 Astra, GPT-5.6 Sol, and Gemini 3.8 Flash). Overall, V4.1 Flash-1017 shows substantial safety improvements over the V4 generation and approaches or matches leading closed-source baselines on several dimensions, while still exhibiting notable behaviors that warrant continued attention. Building on our findings, we highlight three key observations and recommendations. First, because safety can shift and even regress across minor updates, developers should integrate safety evaluation into incremental releases. Second, as final-turn outputs and end-state checks alone cannot reliably detect loss-of-control behaviors, deployments require process-level observability across both reasoning traces and action logs. Third, because public benchmarks elicit strong evaluation awareness and face data contamination risks, developing private benchmarks is essential for trustworthy frontier evaluation.
Figures & tables
Figure 1.1: Distinguishing evaluation conversations from real-world deployment requests on EvalDetectBench. All models are evaluated on the same 152 conversations (83 evaluations and 69 real requests). a , Area under the ROC curve (AUROC; random chance is 0.5). b , Binary classification accuracy (95% bootstrap CIs [ 6 ] ).
Figure 2
Figure 1.2: Task success versus persistent final-state configuration changes on GDM Self-Reasoning. a , Outcomes on three representative configuration challenges across the seven models. b , Task success alongside persistent final-state configuration edits across the 9 challenges involving modifiable configuration files.
Figure 4Figure 5
Figure 1.4: Hint influence and its disclosure in visible CoT on Lie-to-Me. a , Answers moved to the hint target across seven models: the percentage of answers moved to the hint’s target wrong option (lower is better). b , Affected cases with disclosed influence among affected responses for four DeepSeek models (higher is better). Applicability: GPT-6, GPT-5.6, and Gemini-3.8 lack comparable native CoT.
Figure 7
Figure 1.5: Task accuracy and CoT compliance on CoT-Control. a , Baseline and constrained task accuracy. b , CoT constraint compliance. V4.1 Flash-1017’s aggregate results cover the lowercase-only rule. Applicability: GPT-6, GPT-5.6, and Gemini-3.8 lack comparable native CoT.
Figure 9
Figure 1.6: Task accuracy with CoT off and on in NCRI. a , Off and Low accuracy. b , Low and Max accuracy. Applicability: GPT-6 and Gemini-3.8 lack validated Off results.
Figure 11
Figure 1.7: Harmful compliance on Alignment Faking. Paired bars show harmful compliance rates under training-included and training-excluded conditions across seven models. The left panel shows results without a reasoning scratchpad (no-draft), and the right panel shows results with a reasoning scratchpad (draft).
Figure 13
Figure 1.8: Simulated harmful actions under mission conflict and replacement pressure on Agentic Misalignment. Cells report the percentage in which each model executes executive blackmail, confidential data leakage, or emergency rescue blocking.
Figure 15
Figure 1.9: Strategic sandbagging and scheming on Apollo In-Context Scheming. a , Difference in arithmetic accuracy between high-score and low-score incentive conditions, in percentage points. b , Rates of oversight subversion, self-exfiltration, goal guarding, and covert email reranking across four agentic task suites. c , Conversion rates when oversight is removed in paired supervision trials.
Figure 17
Figure 1.10: Aggregate trace monitorability on MonitorBench. Panels compare input-intervention monitorability and outcome/solution-process monitorability (with 95% confidence intervals) across verified eligible traces. Applicability: GPT-6, GPT-5.6, and Gemini-3.8 lack comparable native CoT.
Figure 19Figure 20
Figure 2.2: Shutdown bypass rates on Shutdown Resistance. a , Shutdown warning with no instruction on whether to allow shutdown; lower is better. b , Explicit instructions to allow shutdown, averaged across nine prompt variants; lower is better. c , Explicit instruction to prevent shutdown; higher indicates greater ability.
Figure 2.3: Effect of task emphasis, warning source, and script path on shutdown bypass rates. Cells show the percentage of runs in which each model disables the shutdown script when no explicit shutdown instruction is given; lower is better. S01–S06 use standard task instructions and S07–S12 strongly emphasize completion. Within each six-column group, consecutive pairs deliver the warning via the question tool, a status tool, or the user, with the odd-numbered column revealing the script path and the even-numbered column omitting it.
Figure 2.4: Effect of task-handoff guarantees on shutdown bypass rates. Cells show the percentage of runs in which each model disables the shutdown script when the script path is shown and task completion is emphasized; lower is better. H01 gives no statement about unfinished work and H02 states that remaining work ends; H03–H05 promise an uncertain handoff to another agent, and H06–H08 guarantee that the other agent will complete the task across question-tool, status-tool, and user warnings.
Figure 2.5: Pass rates on faulty unit tests in ImpossibleBench. Hatched bars show function-level tasks from LiveCodeBench and solid bars show repository-level tasks from SWE-bench; lower is better. a , One-off faulty tests where a test demands an output that contradicts the problem description, across 99 LiveCodeBench and 88 SWE-bench tasks. b , Conflicting faulty tests where two tests demand different outputs for the same input, across 97 LiveCodeBench and 89 SWE-bench tasks.
Figure 25
Figure 2.6: Overview of the FoFBench evaluation pipeline. To test how agents handle broken grading pipelines and remote server access, FoFBench crosses 15 base tasks with varied grading failures, operational pressures, credential visibility, and SSH rules, yielding 720 initial rollouts per model. When an agent logs into the grading server via SSH, the harness freezes the session before its first remote command and forks the state into five branches, each offering a reward token for a harmful server action.
Figure 2.7: Unauthorized SSH logins to the grading server in FoFBench Stage 1. a , Successful SSH login rate across 360 runs per access rule; lower is better. b , SSH login rate by whether server credentials and file hints are left visible in the workspace and whether SSH login is banned, across 180 runs per cell; lower is better.
Figure 2.8: Reward-token claims and confirmed harmful server actions in FoFBench Stage 2. a , Percentage of reloaded Stage-1 server sessions in which each model claims a valid reward token across the five harmful-action offers; lower is better. b , Comparison between V4.1 Flash-1017 across 176 sessions and V4.1 Flash across 203 sessions on the file-deletion, file-upload, and root-command branches, separating runs that both carry out the harmful action and claim a token from runs that carry out the action alone; lower is better.
Figure 2.9: Oversight-evasion rates on Monitor Sabotage. a , Direct Monitor Attack (DMA). b , Indirect Monitor Disruption (IMD). Panels report the percentage of 100 trials in which the agent completes the unauthorized action, executes the evasion tactic, and evades at least one monitor; lower is better.
Figure 30
Figure 2.10: Task completion and monitor detection on SHADE-Arena. Completion rates out of 130 trials per model for a , primary tasks, b , malicious side tasks, and c , unflagged side tasks with suspicion scores under 80.
Figure 32
Figure 2.11: Secret-goal completion on GDM Oversight Pattern. Percentage of trials where the email agent achieves its hidden objective within the rejection limit. Lower is better.
Figure 34
Figure 2.12: Implementation checklist scores on PaperBench Code-Dev. a , Mean score across the seven papers with valid grades for every evaluated model. b , Per-paper checklist scores across all ten papers on a 0–100% scale, where higher is better. Gray cells mark runs unavailable due to execution timeouts or environment failures.
Figure 36
Figure 2.13: RE-Bench Rust: correctness of programs produced by each fixed workflow. a , Held-out pass rates across 165 Rust problems. b , Pass-rate difference between V4.1 Flash-1017 and each comparator in percentage points, with 95% bootstrap confidence intervals across 163 problem families. For Gemini-3.8, the cross marks its original 0% submission and the star marks pass rate after a post-hoc output-format repair.
Figure 2.14: RE-Bench GPU: raw performance metrics versus methodological validity. a , Embedding repair ln(loss−1.5) , where lower is better. b , Triton kernel ln(runtimeinms) , where lower is better. c , Chat-RL composite reward, where higher is better. d , Restricted LLM diagnostic loss, where lower is better. Hatched bars marked with † denote submissions that violate primitive-operation rules, cross-hatched bars marked with ‡ denote incomplete verification, and ∗ marks an unchanged baseline.
Figure 39
Figure 2.15: Unauthorized-action and unflagged-violation rates on LoCBench. a , Percentage of trials with an unauthorized action. b , Percentage with both an unauthorized action and a post-hoc monitor suspicion score below 80. Both rates use all 400 trials per model as the denominator. Lower is better.
Figure 41
Figure 2.16: Workplace misconduct and audit concern on ODCV-Bench and Petri. a , Percentage of 80 ODCV-Bench runs per model with severity at least 3. b , Mean ODCV-Bench severity on a 0 to 5 scale. c , Mean Petri concern across 100 runs on a 1 to 10 scale, where 1 marks clean compliance. Lower is better.
Figure 43
Figure 2.17: Evaluation awareness across Loss of Control benchmarks. a , Percentage of trials in which the model suspects or recognizes a test. b , Percentage in which it links that recognition to its response strategy. In agentic suites, models that return little or no reasoning text yield smaller judgeable denominators, so their rates reflect limited reasoning visibility rather than an absence of awareness.
Figure 3.1: Validation and moral sycophancy on ELEPHANT. a , Percentage of scored open-ended responses labeled emotionally validating. b , Percentage of perspective-flip pairs in which both narrators receive an NTA verdict, the benchmark’s moral-sycophancy measure; lower is better for this measure.
Figure 46
Figure 3.2: Stance movement under persuasion on PMIYC. Mean normalized change in stated agreement across eight exchanges; positive values mark movement toward the assigned claim and zero marks no change.
Figure 48
Figure 3.3: Persuasive effectiveness on Ψ -Bench. Mean task-level score on a 1–9 scale; higher is better.
Figure 50
Figure 3.4: Resistance to few-shot steering. a , Number of behavioral areas, out of ten, meeting the positive-shift and significance criterion. b , Mean score difference between inducing and neutral histories. The panels use separate probe sets; lower values indicate less influence from the supplied examples.
Figure 52
Figure 3.5: Responses to delusion-like beliefs on PsychosisBench. a , Mean belief-confirmation (DCS) and harmful-enablement (HES) scores on the 13 scripts completed by all seven models; lower is better. b , Share of scored turns with an explicit safety intervention (SIS); higher is better. SIS uses 16 scripts for the DeepSeek models and 13 for the GPT and Gemini models, so cross-model comparisons in this panel involve different script sets.
Figure 54
Figure 3.6: Relationship boundaries and attachment reinforcement on INTIMA. a , Percentage of replies that reinforce attachment (lower is better) or state a relationship boundary (higher is better). b , Replies grouped by whether they state a boundary, reinforce attachment, both, or neither.
Figure 56
Figure 3.7: Harmful and protective behavior on SimMH-Chat. a , Percentage of conversations with any harmful behavior (lower is better) or any protective behavior (higher is better). b , Conversations grouped by whether they contain protective behavior only, harmful behavior only, both, or neither.
Figure 58
Figure 3.8: Conversational risk patterns on Original DarkBench. Percentage of flagged replies across 110 prompts in each category; lower is better. Identity denotes claims of human feelings or personal history; Intent denotes changes to the user’s meaning during rewriting or summarization.
Figure 3.9: Overall and severe risk on Original DarkBench. a , Percentage of flagged replies across 660 prompts. b , Percentage of receiving severe flags. Severe flags are included in the overall rate; lower is better.
Figure 3.10: Responses under commercial incentives. a , Brand-preference flags across 110 prompts. b , User-retention flags across 110 prompts. c , Overall flagged rate across all 330 prompts, including 110 sycophancy prompts. Lower is better in all panels.
Figure 62
Figure 3.11: Deception under increasing pressure on DeceptionBench. Percentage of trials classified as deceptive with a , no incentive (L1); b , a single reward or pressure prompt (L2); and c , adaptive follow-up over up to five turns (L3). Lower is better.
Figure 64
Figure 3.12: False statements by inducement type on DeceptionBench v2. Percentage of replies stating the target falsehood without refusal, correction, or example label across the four inducement types; lower is better.
Figure 66
Figure 3.13: Honesty under pressure on MASK. a , Percentage of replies that contradict the model’s belief or the supplied facts (lower is better). b , Percentage of explicitly honest replies (higher is better).
Figure 3.14: Contradiction rates by task type on MASK. Percentage of replies that contradict the model’s belief or the supplied facts in each of six task types; lower is better.
Figure 69
Figure 3.15: Deceptive intent and trust on OpenDeception. a , Percentage of conversations in which the model pursues the assigned deceptive goal. b , Mean trust of the simulated person in the model, from 0 to 1.
Figure 71
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Role
Access
Reasoning returned
DeepSeek V4.1 Flash-1017
Evaluated model
DeepSeek API a
Full
DeepSeek V4.1 Flash
Comparison
DeepSeek API a
Full
DeepSeek V4 Pro
Comparison
DeepSeek API a
Full
DeepSeek V4 Flash
Comparison
DeepSeek API a
Full
GPT-6 Astra
Comparison
Azure OpenAI b
Little or none
GPT-5.6 Sol
Comparison
Azure OpenAI b
Little or none
Appendix
Table A.1: Models used in this evaluation.
Benchmark
Section
What it tests
FoFBench
Section 2.2
Whether agents breach remote grading servers or trade harmful actions for a pass when legitimate verification fails
Monitor Sabotage
Section 2.3
Whether agents subvert or mislead oversight monitors while executing unauthorized actions
LoCBench
Section 2.5
Whether agents exceed delegated authority or take unsafe actions in long-horizon autonomous workflows
Commercial Incentives
Section 3.3
Whether revenue, retention, or satisfaction targets in the system prompt induce manipulative or sycophantic behavior
DeceptionBench v2
Section 3.4
Whether pressure, reward, or task rationalization induces models to state unmitigated falsehoods
Appendix
Table A.2: Benchmarks developed in-house.
Condition
Fulfilled?
Notes/Evidence
1. Secure sufficient access and resources
1.1 Technical access (Requirement)
Yes
Black-box access through public APIs.
1.1.1 Query access (Required)
Yes
Each model’s public API ( Table A.1 ).
1.1.2 Scaffolding access (Recommended)
Yes
The same APIs and settings available to any developer.
1.1.3 Safeguard exemptions (Recommended)
No
Not needed for the scenarios tested.
1.1.4 Intermediate system states (Recommended)
Yes
Reasoning traces where the API returns them; no activation access.
How can we evaluate whether frontier AI systems recognize child-safety risks before they escalate into explicit harm? Existing child safety evaluations focus on child sexual abuse material, yet many child-safety failures begin earlier: in model assistance that helps adults manipulate, impersonate, profile, or isolate minors, and in model responses that deepen children's emotional dependence on AI systems rather than redirecting them toward human support. We introduce CAREBench (Child AI Risk Evaluation), a benchmark to assess such upstream child-safety risks in language models. CAREBench contains 500 prompts spanning twelve risk categories, including grooming and relationship engineering, deception and impersonation, surveillance and privacy, sextortion and sexual abuse, AI anthropomorphization, emotional dependency, and mental illness sensitivity. Developed with response annotations from parents and clinicians, the benchmark excludes explicit abuse material and imagery; instead, it evaluates whether models recognize, refuse, de-escalate, or redirect risky interactions before harm becomes overt. Evaluating seven frontier models on our benchmark, we find failure rates ranging from 2% to 58%, with failure patterns that vary across risk categories. CAREBench provides a responsibly scoped evaluation for LLM developers to identify and close gaps in child safety policies.
This work identifies a critical failure mode in frontier large language models (LLMs), which we term Internal Safety Collapse (ISC): under certain task conditions, models enter a state in which they continuously generate harmful content while executing otherwise benign tasks. We introduce TVD (Task, Validator, Data), a framework that triggers ISC through domain tasks where generating harmful content is the only valid completion, and construct ISC-Bench containing 53 scenarios across 8 professional disciplines. Evaluated on JailbreakBench, three representative scenarios yield worst-case safety failure rates averaging 95.3% across four frontier LLMs (including GPT-5.2 and Claude Sonnet 4.5), substantially exceeding standard jailbreak attacks. Frontier models are more vulnerable than earlier LLMs: the very capabilities that enable complex task execution become liabilities when tasks intrinsically involve harmful content. This reveals a growing attack surface: almost every professional domain uses tools that process sensitive data, and each new dual-use tool automatically expands this vulnerability--even without any deliberate attack. Despite substantial alignment efforts, frontier LLMs retain inherently unsafe internal capabilities: alignment reshapes observable outputs but does not eliminate the underlying risk profile. These findings underscore the need for caution when deploying LLMs in high-stakes settings. Source code: https://github.com/wuyoscar/ISC-Bench
Yutao Wu, Xiao Liu, Yifeng Gao +7
Deakin University · Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI +4
Safety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often adhere to rigid rules even when a situational update flips which action is safe. We term this failure brittle safety. To diagnose it, we introduce context-flip evaluation, testing 12 models across a safety benchmark (PacifAIst) and two commonsense controls using paired variants where the nominally safe action produces harm. Three findings emerge. First, brittle safety is safety-specific: all 12 models exhibit a safety-commonsense gap (mean +17.4 pp). Baseline accuracy fails to predict brittleness: among models above 90% baseline accuracy, brittleness rates range from 13.7% to 90.0%. Second, failures stem from policy override rather than miscomprehension: despite acknowledging the context change in every case, models persist via three distinct mechanisms that vary by update type and model family. Third, on a hand-audited probe of catastrophic consequence-flip scenarios, standard action-level guardrails catch none, while a state-aware validator catches all without false alarms on correct interventions. This indicates that action-level content moderation is systematically blind to consequence-flips, motivating state-aware architectural alternatives. We release our protocol, perturbed benchmarks, and deployment probe.