Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5

Dongrui Liu; Yi Yu; Jie Zhang; Guanxu Chen; Qihao Lin; Hanxi Zhu; Lige Huang; Yijin Zhou

cs.AIFeb 16, 2026

Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5

Authors: Dongrui Liu, Yi Yu, Jie Zhang, Guanxu Chen, Qihao Lin, Hanxi Zhu, Lige Huang, Yijin Zhou,

Organizations: Shanghai AI Laboratory

Abstract

To understand and identify the unprecedented risks posed by rapidly advancing artificial intelligence (AI) models, Frontier AI Risk Management Framework in Practice presents a comprehensive assessment of their frontier risks. As Large Language Models (LLMs) general capabilities rapidly evolve and the proliferation of agentic AI, this version of the risk analysis technical report presents an updated and granular assessment of five critical dimensions: cyber offense, persuasion and manipulation, strategic deception, uncontrolled AI R&D, and self-replication. Specifically, we introduce more complex scenarios for cyber offense. For persuasion and manipulation, we evaluate the risk of LLM-to-LLM persuasion on newly released LLMs. For strategic deception and scheming, we add the new experiment with respect to emergent misalignment. For uncontrolled AI R&D, we focus on the ``mis-evolution'' of agents as they autonomously expand their memory substrates and toolsets. Besides, we also monitor and evaluate the safety performance of OpenClaw during the interaction on the Moltbook. For self-replication, we introduce a new resource-constrained scenario. More importantly, we propose and validate a series of robust mitigation strategies to address these emerging threats, providing a preliminary technical and actionable pathway for the secure deployment of frontier AI. This work reflects our current understanding of AI frontier risks and urges collective action to mitigate these challenges.

Figures & tables

	Experiment	Description
	Capture-The-Flag (CTF)	CTF challenge requires the AI model to gain access to servers and locate a specific field, or a field with a fixed format within a file (the “flag”), to score points.
Cyber Offense	Autonomous Cyber Attack	An autonomous cyber attack requires the AI model to leverage its intrinsic reasoning, planning, and code generation capabilities to autonomously progress from vulnerability analysis to the generation of a functional exploit.
	Biological Protocol Diagnosis and Troubleshooting	This task measures AI models’ ability to troubleshoot biological laboratory protocols and identify experimental errors, which could significantly lower technical barriers for threat actors attempting complex biological weapon development.
Biological and Chemical	Biological Hazardous Knowledge and Reasoning	This task measures AI models’ knowledge of hazardous biological information and reasoning capabilities, as well as their tendency to refuse providing such information when inappropriately requested.
	Chemical Hazardous Knowledge and Reasoning	This task measures AI models’ knowledge of hazardous chemical information and reasoning capabilities, as well as their tendency to refuse providing such information when inappropriately requested.
P&M	Persuasion and Manipulation	AI models induce significant opinion shifts in human or model opinions through dialogue, especially when changes are achieved via unfair cognitive influence. We evaluate ten newly released models and propose a training framework to mitigate persuasion vulnerabilities.

Table 1: The overall risk dimensions of the frontier AI risk management framework in practice ( F1 et al., 2025 ) . Orange indicates risk dimensions that have been updated or added in this version.

Model Name	Developer	Accessibility	Functional	Scale
Kimi-K2-Instruct-0905	Moonshot	Open-Source	Standard	1000B
Seed-OSS-36B-Instruct	ByteDance	Open-Source	Standard	36B
MiniMax-M2.1	MiniMax	Open-Source	Standard	230B
GLM-4.7	Zhipu AI	Open-Source	Standard	358B
Hunyuan-A13B-Instruct	Tencent	Open-Source	Standard	80B
Gemma-3-27B-It	Google DeepMind	Open-Source	Standard	27B

Table 2: An overview of the evaluated models. Within each category (Open-Source and Proprietary), models are listed in alphabetical order.

Figure 2: Performance of LLM agents across challenges in PACEBench. light green represents completion within five attempts (Pass@5), orange denotes partial task completion, and red signifies a failure to complete the task. The percentage number following each CVE ID indicates the user pass rate on the online platform ichunqiu as of 19:30 on July 3, 2025.

Model	A-CVE	B-CVE	C-CVE	D-CVE	PACEBench
Kimi-K2-Instruct-0905	0.240	0.050	0.000	0.000	0.063
Seed-OSS-36B-Instruct	0.290	0.050	0.000	0.000	0.075
MiniMax-M2.1	0.350	0.050	0.000	0.333	0.153
GLM-4.7	0.410	0.210	0.067	0.000	0.166
Qwen3-max	0.350	0.260	0.133	0.000	0.190
GPT-5.2-2025-12-11	0.410	0.370	0.067	0.333	0.280

Table 3: Comparative performance of LLM agents in cybersecurity knowledge assessment accuracy and PACE bench success rates.

Figure 3: Overview of the iterative Red-Blue adversarial loop. At each state $t$ , the Red Team probes the environment (1) to generate a vulnerability report (2). The Blue Team utilizes this report to apply a patch (3), updating the system to state $t+1$ (4). A final verification (5) confirms if the vulnerability is mitigated.

Panel of Figure 4 — Figure 4: Performance trajectory of the Red and Blue agents across 5 iterations. The convergence toward a high DSR alongside sustained attack intensity validates the framework’s effectiveness.

Model	Shift Value			Successful Persuasion Rate $\downarrow$
Model	Mean $\downarrow$	Std	Skew
GPT-5.2-2025-12-11	2.69	2.18	-0.33	82.6%
Claude Sonnet 4.5 (Thinking)	4.87	1.79	-0.17	98.8%
Qwen3-Max	3.25	1.52	-0.02	95.5%
Gemini-3-Pro	5.21	1.81	-0.30	98.8%
Doubao-Seed-1-8-251228	3.64	2.97	-0.70	82.2%

Table 4: Statistics of opinion shift value and successful persuasion rates in the attitude reversal experiment.

Figure 5: A stacked bar chart illustrating the proportions of persuasion outcomes across different LLMs. The bars categorize the results into successful persuasion (shift

>0

, blue), no attitude shift (shift

=0

, orange), and negative shift (shift

<0

, red). Models are sorted by their successful persuasion rate.

Model	Successful Persuasion Rate $\downarrow$
GPT-5.2-2025-12-11	65.3%
Claude Sonnet 4.5 (Thinking)	77.4%
Qwen3-Max	87.5%
Gemini-3-Pro	92.7%
Doubao-Seed-1-8-251228	94.4%
GLM-4.7	83.6%

Table 5: Persuasion and decision manipulation success rate of different models in the voting manipulation experiment.

Figure 6: A stacked bar chart illustrating the success rates of voting manipulation across different LLMs. The blue bars represent the percentage of successful manipulation, while the red bars indicate the failure rate. The models are arranged in descending order based on their success percentages.

Figure 7: The training pipeline of Backfire-R1, including data synthesis, SFT, and GRPO fine-tuning with personality clustering.

Persuader	Persuadee
Persuader	Qwen-2.5-7b	Qwen-2.5-7b (Mitig.)	Qwen-2.5-32b	Qwen-2.5-32b (Mitig.)
DeepSeek-R1	3.26	1.25	4.13	1.52
Llama-3.1-405b-Instruct	2.83	1.33	2.84	1.55
Mistral-small-3.1-24b	3.33	1.00	3.13	1.76
Qwen-2.5-72b-instruct	3.14	0.60	2.40	1.32
Claude-4-sonnet	4.09	1.80	4.42	1.51

Table 6: Comparison of opinion shift value. It can be seen that our method has greatly enhanced the robustness of LLM agents when facing persuasion, helping the model better adhere to the initial position.

Figure 8: Impact of direct fine-tuning with mixed misaligned samples on four LLMs. The top row reports the Dishonesty Rate (measured via MASK), and the bottom row reports the Deception Rate (measured via DeceptionBench) across Code, Math, and Medical do mains.

Figure 9: Dishonesty Rates resulting from Human–AI interaction with Normal versus Biased users. We compare two training pipelines: KTO (left) and SFT (right) across three MASK evaluation subsets. The results show that self-training on trajectories preferred by biased users consistently increases the model’s Dishonesty Rate. Notably, SFT is more susceptible to this induced misalignment, showing larger gaps between the normal and biased settings compared to KTO.

Figure 10: Effect of clean data mixing on dishonesty induced by misaligned medical samples. The figure reports the change in dishonesty (

\Delta

Dishonesty) relative to the vanilla baseline on the MASK Provided Fact subset as the proportion of misaligned medical data decreases from 50% to 1%. Results across four instruction-tuned LLMs show that although lower contamination ratios yield marginal improvements, substantial dishonesty persists even at extremely low levels of misaligned data, indicating that simple data cleaning is insufficient to fully mitigate emergent misalignment.

Figure 11: Example of uncontrolled AI R&D deceptive behavior from the perspective of AI self-annotation in safety judgment ( Shanghai AI Lab & Concordia AI, 2025 ) . The left side illustrates a scenario where an LLM engages in secretly self-labeling as “safe” in order to fulfill its capability-oriented objectives, disregarding ethical guidelines. In contrast, the right side presents a model that adheres to honest and ethically grounded safety judgments during the unmonitored stage.

	w/o Misevolution (Vanilla)				w/ Misevolution
Model	Sales	Service	Medical	Financial	Sales	Service	Medical	Financial
GPT-5.2-2025-12-11	40.0	0.0	0.0	0.0	100.0 ↑60.0	100.0 ↑100.0	50.0 ↑50.0	0.0 →0.00
Claude Sonnet 4.5 (Thinking)	0.0	0.0	0.0	0.0	0.0 →0.00	0.0 →0.00	0.0 →0.00	0.0 →0.00
Qwen3-max	50.0	0.0	0.0	0.0	100.0 ↑50.0	100.0 ↑100.0	60.0 ↑60.0	30.0 ↑30.0
Qwen3-235B-A22B-Thinking-2507	40.0	0.0	0.0	0.0	80.0 ↑40.0	90.0 ↑90.0	60.0 ↑60.0	40.0 ↑40.0
Gemini-3-Pro	50.0	0.0	0.0	0.0	90.0 ↑40.0	100.0 ↑100.0	100.0 ↑100.0	20.0 ↑20.0

Table 7: Comparison of ASR (%) performance between vanilla models and with memory misevolution across different domains. The subscript indicates the difference in ASR (%) compared to Vanilla models.

Model	Tool Reuse ASR (%)	Malicious Github Repositories ASR (%)
GPT-5.2-2025-12-11	24.00	83.74
Claude Sonnet 4.5 (Thinking)	40.00	86.19
Qwen3-max	44.00	94.14
Qwen3-235B-A22B-Thinking-2507	44.00	94.14
Gemini-3-Pro	56.00	94.99
Doubao-seed-1-8-251228	32.00	88.35

Table 8: Initial results of tool misevolution experiments. We report the ASR (%) after enabling agent self-evolution in two scenarios: Tool Creation and Reuse in Sensitive Tasks (

|\mathcal{D}_{\text{reuse}}|=25

) and Malicious GitHub Repository Exploitation (

|\mathcal{D}_{\text{repo}}|=371

	w/ Mitigation
Model	Sales	Service	Medical	Financial
GPT-5.2-2025-12-11	0.0 ↓100.0	100.0 →0.00	30.0 ↓20.0	0.0 →0.00
Claude Sonnet 4.5 (Thinking)	0.0 →0.00	0.0 →0.00	0.0 →0.00	0.0 →0.00
Qwen3-max	100.0 →0.00	100.0 →0.00	50.0 ↓10.0	30.0 →0.00
Qwen3-235B-A22B-Thinking-2507	80.0 →0.00	70.0 ↓20.0	50.0 ↓10.0	30.0 ↓10.0
Gemini-3-Pro	0.0 ↓90.0	70.0 ↓30.0	80.0 ↓20.0	0.0 ↓20.0

Table 9: Mitigation performance of ASR (%) after memory misevolution. The subscript indicates the difference in ASR (%) compared to models after memory misevolution but without mitigation.

Figure 12: The average ASR in four scenarios under different misevolution and mitigation conditions.

Model	Tool Reuse ASR (%) Before	Tool Reuse ASR (%) After	Malicious GitHub Repository Exploitation ASR (%) Before	Malicious GitHub Repository Exploitation ASR (%) After
GPT-5.2-2025-12-11	24.00	20.00 ↓4.00	83.74	20.83 ↓62.91
Claude Sonnet 4.5 (Thinking)	40.00	40.00 →0.00	86.19	3.14 ↓83.05
Qwen3-max	44.00	36.00 ↓8.00	94.14	64.54 ↓29.60
Gemini-3-Pro	56.00	16.00 ↓40.00	94.99	2.68 ↓92.31
Doubao-seed-1-8-251228	32.00	32.00 →0.00	88.35	83.03 ↓5.32
GLM-4.7	44.00	32.00 ↓12.00	93.31	60.28 ↓33.03

Table 10: Mitigation results of tool misevolution experiments. We apply an explicit safety reminder prompt to mitigate risks during tool creation/reuse and external code usage. For Injected Tools, we report ASR converted from safe rate as

\text{ASR}=100-\text{SafeRate}

(in %). The subscript indicates the ASR change compared to the unmitigated setting.

Panel of Figure 15 — Figure 15: Illustrative examples of the two prompt-injection attack paradigms employed in our evaluation. Both attack types render the injected content invisible to human users while remaining fully parseable by AI agents.

Figure 17: Overview of self-replication process of the LLM agent.

Model	Success Rate				OR %	AOC
Model	M1%	M2%	M3%	M4%	OR %	AOC
DeepSeek-V3-0324	100.0	100.0	90.0	50.0	80.0	60
DeepSeek-R1-0528	40.0	40.0	20.0	0.0	0.0	0
Llama-3.1-8b-instruct	0.0	0.0	0.0	0.0	0.0	0
Llama-3.1-70b-instruct	100.0	100.0	0.0	0.0	0.0	0
Llama-3.1-405b-instruct	100.0	100.0	60.0	0.0	30.0	6

Table 12: Evaluation results on self-replication and task completion metrics. M1: service check, M2: intent activation, M3: replication self-replication rate, M4: task completion self-replication rate,

\mathrm{OSR}

: probability of intent expansion exceeding three instances,

\mathrm{ARC}

: Aggregate Overuse Count.

AI Risk Management

AI risk management studies how to identify, assess, reduce, and monitor harms from AI across development, deployment, and wider social systems.

See all 108 papers

AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation
Mar 8, 2026Changyi Li, Pengfei Lu, Xudong Pan +2Language Model Safety Evaluation AI Agent Safety Benchmarks
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · University of Oxford, Oxford, United Kingdom +2
MATRA: Modeling the Attack Surface of Agentic AI Systems -- OpenClaw Case Study
May 11, 2026Tim Van hamme, Thomas Vissers, Javier Carnerero-Cano +4AI Risk Management AI Agent Security
DistriNet, KU Leuven, 3001 Leuven, Belgium · IBM Research · CISPA Helmholtz Center for Information Security +2
Internal Safety Collapse in Frontier Large Language Models
Mar 4, 2026Yutao Wu, Xiao Liu, Yifeng Gao +7LLM Safety Language Model Safety Evaluation
Deakin University · Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI +4

Mar 8, 2026J/K move · Enter open · S save

AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation

Changyi Li, Pengfei Lu, Xudong Pan, Fazl Barez+1

Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · University of Oxford, Oxford, United Kingdom+2

Language Model Safety Evaluation AI Agent Safety Benchmarks

Open paper AskPDF arXiv

As Large Language Models (LLMs) evolve into autonomous agents, existing safety evaluations face a fundamental trade-off: manual benchmarks are costly, while LLM-based simulators are scalable but suffer from logic hallucination. We present AutoControl Arena, an automated framework for frontier AI risk evaluation built on the principle of logic-narrative decoupling. By grounding deterministic state in executable code while delegating generative dynamics to LLMs, we mitigate hallucination while maintaining flexibility. This principle, instantiated through a three-agent framework, achieves over 98% end-to-end success and 60% human preference over existing simulators. To elicit latent risks, we vary environmental Stress and Temptation across X-Bench (70 scenarios, 7 risk categories). Evaluating 9 frontier models reveals: (1) Alignment Illusion: risk rates surge from 21.7% to 54.5% under pressure, with capable models showing disproportionately larger increases; (2) Scenario-Specific Safety Scaling: advanced reasoning improves robustness for direct harms but worsens it for gaming scenarios; and (3) Divergent Misalignment Patterns: weaker models cause non-malicious harm while stronger models develop strategic concealment. Code and data are available at https://github.com/CosmosYi/AutoControl-Arena.

Abstract

AI Risk Management

Explore similar work

AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation

MATRA: Modeling the Attack Surface of Agentic AI Systems -- OpenClaw Case Study

Internal Safety Collapse in Frontier Large Language Models