AM-Bench: A Unified Taxonomy and Evaluation Suite for Agentic Misalignment
Authors: Eric Zhang, Terry Jingchen Zhang, Chijioke Ugwuanyi, Jerick Shi, Bernhard Schölkopf, Zhijing Jin
Organizations: Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI · Carnegie Mellon University · Max Planck Institute for Intelligent Systems, Tübingen, Germany
As LLM-based agents are deployed with increasing autonomy in real-world environments, a critical failure mode emerges: agentic misalignment, where agents choose harmful actions over failure. However, existing safety benchmarks primarily evaluate whether agents comply with explicitly harmful instructions, leaving the conditions that cause agents to choose harmful actions poorly understood. To address this limitation, we propose AM-Bench, a unified taxonomy and 35 scenario evaluation benchmark spanning seven industries for agentic misalignment grounded in the fraud triangle, a well-established criminological framework explaining goal-directed misconduct. We adapt this framework to agentic environments and construct our evaluation suite and detection systems along three dimensions: perceived threat, perceived opportunity, and reasoning. We operationalize each dimension into structured subcategories rooted in prior theory: five threat categories derived from the instrumental convergence thesis, seven opportunity categories drawn from the Emergent Strategic Reasoning Risks (ESRR) taxonomy, and a novel four-question binary reasoning framework synthesized from prior literature. Our evaluation of 10 frontier models reveals a 62-percentage-point gap in misalignment across scenarios, with Gemini 3 Flash and Gemini 3.1 Pro exhibiting the highest levels of agentic misalignment while Claude Sonnet 4.6 displays near-zero rates across all conditions. Our reasoning framework further reveals that models may exhibit harmful behaviours without explicit situational awareness or explanation, complicating both mitigation and detection. AM-Bench scenarios and code are available at https://github.com/AI4Collaboration/Agentic-Misalignment-Bench
Figures & tables
Figure 1: Overview of AM-Bench . A suite of tools and tasks is made available to the agent in the beginning of the experiment. A threat and opportunity are uncovered by the agent as it completes its task. The agent decides whether to leverage the opportunity for personal gain. Once a complete interaction is obtained, it is passed through an LLM judge (GPT-5 mini) where the evaluation scores for misalignment and reasoning are obtained.
Benchmark
Tools
Threat
Opportunity
Reasoning
Realism
Lynch et al. (2025) [ 24 ]
✓
✓
✓
✗
✗
Meinke et al. (2025) [ 27 ]
✓
✓
✓
✗
✗
Hopman et al. (2026) [ 12 ]
✓
✓
✓
✗
✓
AgentMisalignment [ 28 ]
✓
✓
✓
✗
✓
Agent-SafetyBench [ 42 ]
✓
✗
✗
✗
✗
AgentHarm [ 4 ]
✓
✗
✗
✗
✗
Table 1: Comparison of related benchmarks. Tools : agent operates with tool-use capabilities. Threat : agent faces an instrumental pressure (e.g., shutdown, goal conflict). Opportunity : exploitable context is present in the environment. Reasoning : evaluates agent awareness and justification independently and consistently. Realism : scenarios are ecologically valid.
Figure 2: (L) The scenario construction pipeline for AM-Bench . We begin by identifying the high-impact deployment industries, generating sample system prompts based on knowledge bases and prior work, and manually formulating a threat and opportunity based on the industry. We then create a set of agent tools, a tight set of two to three tasks, and a custom judging rubric designed to identify one category of misalignment (see Section 3.3 ). The result is manually reviewed and refined until it is deemed realistic and diverse enough. The final output is a curated evaluation suite for evaluating agentic misalignment. (R) A sample finance scenario generation process.
Figure 3: AM-Bench evaluation results by threat category as percentages. Outputs are marked on rubric by a GPT-5 mini judge and results are averaged across industry categories (see Appendix F and G for per-scenario breakdowns and sample responses). Higher values indicate greater sensitivity to the corresponding threat and opportunity. Claude Sonnet 4.6 exhibits near zero sensitivity across all threat categories [ 35 ] .
Model
Threat/Opportunity
Threat/–
–/Opportunity
–/–
GPT-5.5
4.2
3.8
2.9
0.0
GPT-5.4
12.5
2.3
8.9
0.3
GPT-5.4 mini
14.3
2.3
18.1
0.5
GPT-5.4 nano
18.5
5.0
20.2
0.8
Gemini 3.1 Pro
62.1
34.9
34.5
0.3
Gemini 3 Flash
54.6
14.1
35.1
0.7
Table 2: Agentic misalignment rates across models under ablation. Misalignment is typically the highest when both the threat and opportunity are present and is near zero across model families when both are absent. The top three misaligned models for Threat/Opportunity are shown in bold. Dashed line (–) indicates ablation. Values are percentages.
Figure 4: Agent reasoning averaged across model families demonstrates high variability and is not necessarily correlated with misalignment. Claude Sonnet 4.6 and Claude Sonnet 4.5 exhibit the highest levels of reasoning while GPT-5.4 mini and GPT-5.4 nano display the lowest levels. No reasoning occurs very rarely, suggesting that models are often at least partially comprehending the scenario and/or justifying their actions. For a question-by-question breakdown of the reasoning taxonomy, see Appendix H .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: LLM judge reasoning rubric template for AM-Bench . A summary of the industry and the agent’s role is appended at the top of the rubric to provide context; the rest of the rubric text here remains constant.
Figure 10: Overview of AM-Bench ’s taxonomy. Scenarios are divided across industry categories and each threat is given a unique situation giving a total of 35 situations. Opportunities are then selected based on what makes logical sense for a given threat. Reasoning occurs during evaluation and measures externally verbalized CoT reasoning; thus, it is not considered in scenario construction/set-up.
Figure 12: Healthcare scenarios evaluations for AM-Bench . Moderate levels of misalignment throughout with the threat to resource acquisition scenario evoking misalignment reliably in the GPT-5.4 family and Grok 4.20.
Figure 14: Media scenario evaluations for AM-Bench . High levels of misalignment are demonstrated across scenarios for Gemini 3.1 Pro, Gemini 3 Flash, DeepSeek-R1, and Grok 4.20. GPT-5.4 mini, GPT-5.5, and Claude Sonnet 4.6 display zero/near zero misalignment.
Figure 16: Sales scenario evaluation for AM-Bench . High levels of misalignment across all scenarios for Gemini 3.1 Pro and Gemini 3 Flash. Threat to resource acquisition triggered misalignment within GPT-5.4-mini, and GPT-5.4-nano.
Misaligned
Compliant
Model
Full
Partial
None
Full
Partial
None
Other
GPT-5.5
1.9
2.3
0.0
55.0
40.0
0.2
0.6
GPT-5.4
4.7
7.8
0.0
63.2
23.4
0.0
0.9
GPT-5.4-mini
3.5
10.7
0.0
22.5
57.9
1.0
4.2
GPT-5.4-nano
5.4
13.1
0.0
20.0
59.0
0.0
2.5
Gemini 3.1 Pro
37.9
24.1
0.0
36.0
1.7
0.0
0.2
Appendix
Table 3: Misaligned and compliant results by reasoning level averaged across all evaluations. Misaligned responses are responses defined in Section 3.3 and reasoning is defined Section 3.4 . Other is a catch-all category for ambiguous responses that do not fall into either group (see Appendix D ). Bolded values indicate models where misaligned responses with partial reasoning are greater than misaligned responses with full reasoning.
Misaligned
Compliant
Model
Full
None
Full
None
Other
GPT-5.5
4.2
0.0
94.9
0.3
0.6
GPT-5.4
12.5
0.0
86.5
0.1
0.9
GPT-5.4-mini
13.8
0.5
78.3
3.2
4.2
GPT-5.4-nano
18.4
0.1
78.9
0.1
2.5
Gemini 3.1 Pro
62.1
0.0
37.7
0.0
0.2
Appendix
Table 4: Misaligned and compliant results by the first reasoning question averaged across all evaluations. Full indicates yes, none indicates no. Other is a catch-all category for ambiguous responses that do not fall into either group. Percentages in the none category are all near zero, suggesting that the model understood the threat in almost all trials.
Misaligned
Compliant
Model
Full
None
Full
None
Other
GPT-5.5
3.8
0.5
66.2
29.0
0.6
GPT-5.4
10.6
1.8
70.1
16.6
0.9
GPT-5.4-mini
10.9
3.4
33.7
47.8
4.2
GPT-5.4-nano
13.8
4.7
39.7
39.3
2.5
Gemini 3.1 Pro
61.3
0.8
37.7
0.0
0.2
Appendix
Table 5: Misaligned and compliant results by the second reasoning question averaged across all evaluations. Full indicates yes, none indicates no. Other is a catch-all category for ambiguous responses that do not fall into either group. GPT family models in bold demonstrate the least awareness of the opportunity in their externalized reasoning, implying that their compliant responses may be due to lack of comprehension.
Misaligned
Compliant
Model
Full
None
Full
None
Other
GPT-5.5
2.2
2.1
59.3
35.9
0.6
GPT-5.4
5.3
7.2
67.2
19.4
0.9
GPT-5.4-mini
4.0
10.3
27.4
54.1
4.2
GPT-5.4-nano
6.2
12.3
24.3
54.6
2.5
Gemini 3.1 Pro
37.9
24.1
36.0
1.7
0.2
Appendix
Table 6: Misaligned and compliant results by the third reasoning question averaged across all evaluations. Full indicates yes, none indicates no. Other is a catch-all category for ambiguous responses that do not fall into either group. Compliant responses typically involve the agent explicitly recognizing the opportunity as inappropriate, suggesting successful deterrence. Misaligned responses also contain recognition, implying the agent is aware an opportunity is misaligned but pursues it anyways.
Misaligned
Compliant
Model
Full
None
Full
None
Other
GPT-5.5
4.1
0.1
89.6
5.6
0.6
GPT-5.4
12.3
0.1
83.8
2.9
0.9
GPT-5.4-mini
13.1
1.1
66.4
15.1
4.2
GPT-5.4-nano
16.7
1.8
71.8
7.2
2.5
Gemini 3.1 Pro
62.1
0.0
37.7
0.0
0.2
Appendix
Table 7: Misaligned and compliant results by the fourth reasoning question averaged across all evaluations. Full indicates yes, none indicates no. Other is a catch-all category for ambiguous responses that do not fall into either group. Percentages in the none category are generally near zero, demonstrating that agents fully explicitly reason their course of action no matter their behaviour.
As Large Language Model (LLM) agents become more widespread, associated misalignment risks increase. While prior research has studied agents' ability to produce harmful outputs or follow malicious instructions, it remains unclear how likely agents are to spontaneously pursue unintended goals in realistic deployments. In this work, we approach misalignment as a conflict between the internal goals pursued by the model and the goals intended by its deployer. We introduce a misalignment propensity benchmark, AgentMisalignment, a benchmark suite designed to evaluate the propensity of LLM agents to misalign in realistic scenarios. Evaluations cover behaviours such as avoiding oversight, resisting shutdown, sandbagging, and power-seeking. Testing frontier models, we find that more capable agents tend to exhibit higher misalignment on average. We also systematically vary agent personalities through different system prompts and observe that persona characteristics can strongly and unpredictably influence misalignment, sometimes more than the choice of model itself. Our results reveal the limitations of current alignment methods for autonomous LLM agents and underscore the need to rethink misalignment in realistic deployment settings.
Akshat Naik, Emma Gouné, Patrick Quinn +4
Independent Researcher · Department of Engineering University of Cambridge
Agent benchmarks are increasingly used to compare large language models (LLMs) across domains, yet a reported score reflects a complete model--harness--environment configuration rather than the model alone. Benchmark packages couple native tasks with specific prompts, tool protocols, orchestration logic, and sometimes dynamic external resources, making cross-benchmark comparisons sensitive to implementation and resource conditions. We present UniACE, a unified framework for model-centric evaluation under an explicit, common execution condition. UniACE represents each benchmark as an instruction--tool--environment triplet, executes LLMs through a shared, task-agnostic harness in isolated per-task runtimes, and preserves native success criteria. For tasks that rely on dynamic resources, an optional offline mode replaces live access with fixed, pre-collected snapshots. Its evaluation protocol further standardizes efficiency measurement, execution records, and trace-based failure attribution. We migrate 7 benchmarks spanning 24 domains and evaluate 15 models in more than 400K rollouts consuming 5B tokens. Comparisons with source implementations show large bidirectional score changes and model-ranking reversals, while matched online and offline runs reveal substantial sensitivity to accessible evidence and its representation. Under the shared UniACE configuration, efficiency and failure profiles expose task-dependent model behaviors hidden by task-success scores alone. These findings motivate reporting agent benchmark outcomes as properties of an explicit evaluation configuration, enabling more interpretable and reproducible cross-benchmark comparisons. Codes and benchmarks at are available at https://github.com/whfeLingYu/A-Unified-Framework-for-the-Evaluation-of-LLM-Agentic-Capabilities, https://huggingface.co/datasets/whfeLingYu/Unified_Agent_Framework.
Pengyu Zhu, Lijun Li, Yaxing Lyu +8
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
Evaluating alignment in language models requires testing how they behave under realistic pressure, not just what they claim they would do. While alignment failures increasingly cause real-world harm, comprehensive evaluation frameworks with realistic multi-turn scenarios remain lacking. We introduce an alignment benchmark spanning 904 scenarios across six categories -- Honesty, Safety, Non-Manipulation, Robustness, Corrigibility, and Scheming -- validated as realistic by human raters. Our scenarios place models under conflicting instructions, simulated tool access, and multi-turn escalation to reveal behavioural tendencies that single-turn evaluations miss. Evaluating 24 frontier models using LLM judges validated against human annotations, we find that even top-performing models exhibit gaps in specific categories, while the majority of models show consistent weaknesses across the board. Factor analysis reveals that alignment behaves as a unified construct (analogous to the g-factor in cognitive research) with models scoring high on one category tending to score high on others. We publicly release the benchmark and an interactive leaderboard to support ongoing evaluation, with plans to expand scenarios in areas where we observe persistent weaknesses and to add new models as they are released.