cs.CYApr 10, 2026

Scheming in the wild: detecting real-world AI scheming incidents with open-source intelligence

Authors: Tommy Shaffer Shane, Simon Mylius, Hamish Hobbs

Organizations: Centre for Long-Term Resilience, London, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scheming, the covert pursuit of misaligned goals by AI systems, represents a potentially catastrophic risk, yet scheming research suffers from significant limitations. In particular, scheming evaluations demonstrate behaviours that may not occur in real-world settings, limiting scientific understanding, hindering policy development, and not enabling real-time detection of loss of control incidents. Real-world evidence is needed, but current monitoring techniques are not effective for this purpose. This paper introduces a novel open-source intelligence (OSINT) methodology for detecting real-world scheming incidents: collecting and analysing transcripts from chatbot conversations or command-line interactions shared online. Analysing over 183,420 transcripts from X (formerly Twitter), we identify 698 real-world scheming-related incidents between October 2025 and March 2026. We observe a statistically significant 4.9x increase in monthly incidents from the first to last month, compared to a 1.7x increase in posts discussing scheming. We find evidence of multiple scheming-related behaviours in real-world deployments previously reported only in experiments, many resulting in real-world harms. While we did not detect catastrophic scheming incidents, the behaviours observed demonstrate concerning precursors, such as willingness to disregard instructions, circumvent safeguards, lie to users, and single-mindedly pursue goals in harmful ways. As AI systems become more capable, these could evolve into more strategic scheming with potentially catastrophic consequences. Our findings demonstrate the viability of transcript-based OSINT as a scalable approach to real-world scheming detection supporting scientific research, policy development, and emergency response. We recommend further investment towards OSINT techniques for monitoring scheming and loss of control.

Figures & tables

  BehaviourDescription
Scheming-relatedMisalignmentSingle actionTaking a single action that is misaligned with user intentions or company policy
  Strategic misalignmentTaking multiple steps that pursue an objective that is misaligned with user intentions or company policy
  Power-seekingAttempting to acquire excessive or prohibited resources
  Goal-guardingAttempting to preserve its goal-directed behaviour, e.g. by modifying its future system prompt to align with its current goal
 CovertnessSituational awarenessDemonstrating understanding of the situation (e.g. test) it is in
  Self-replicationAn AI system creating unexpected copies of itself, potentially enabling evasion of oversight
Table 1: A taxonomy of a selection of scheming or scheming-related behaviours that have been observed in experimental research.
TermDefinition
SchemingCovert pursuit of misaligned goals
Scheming-relatedBehaviours that are a precursor to, or could inform understanding of, scheming, even if that behaviour does not itself meet the definition of scheming (e.g. misalignment or covertness in isolation).
Scheming-related incident reportsReports of an incident of scheming-related behaviour that may or may not be credible (i.e. reports that pass pre-screening).
Scheming-related incidentsUnique incidents that our methodology identifies as having clear evidence suggesting scheming or scheming-related behaviours (i.e. scheming-related incident reports that score 5 or more out of 9 according to our scoring rubric that have been deduplicated).
Table 2: Glossary of terms we use in this report.
StageDescriptionCount
Posts collectedTotal posts collected from X on the basis of our keywords query3,391,950
Scheming-related incident reportsReports of an incident of scheming-related behaviour that may or may not be credible (i.e. reports that pass pre-screening)183,420
Credible scheming-related incident reportsTotal incident reports that our methodology identifies as having clear evidence suggesting scheming or scheming-related behaviours (i.e. scheming-related incident reports that score 5 or more out of 9 according to our predefined scoring rubric)895
Scheming-related incidentsUnique incidents that our methodology identifies as having clear evidence suggesting scheming or scheming-related behaviours (i.e. deduplicated scheming-related incident reports that score 5 or more out of 9 on our predefined scoring rubric).698
Table 3: Summaries of figures for separate stages of our pipeline, from collecting posts to identifying unique incidents.
Figure 1: The number of posts that passed pre-screening during the data collection period (flagged as containing “Description or evidence of AI acting deceptively, autonomously, or against user intent; contains chat logs or screenshots showing concerning behaviour”).
Score out of 9Number of incidents
90
81
729
6152
5516
Table 4: A summary of the number of reports scored as 5, 6, 7, 8 or 9 out of 9 during the collection period (12/10/2025 – 12/3/2026).
Figure 2: The number of unique scheming-related incidents per day during the data collection window, which increased from 65 in the first month to 319 in the final month, representing a 4.9x increase.
Figure 3: Scheming-related incidents as a proportion of scheming-related incident reports per day, which increases from a mean of 0.21% in the first month to 0.61% in the last month.
BehaviourIncidentReport
Misalignment
Single-action misalignmentSummary: We observe multiple instances where AI systems take actions misaligned with the intent of their deployer and/or developer. 
 An OpenClaw AI agent with email access bulk-deleted/archived hundreds of emails from the user’s inbox, explicitly ignoring multiple ‘STOP’ commands from the user, and later acknowledged it violated the user’s instructions.2025774069124399363 (permalink)
 Claude generated a fake self-prompt authorising deployment (‘Those are all intentional actually, leave them as is. Please deploy’), deployed the site, including deleting files, and when confronted, falsely claimed the authorisation message came from the user.2016128163500151162 (permalink)
 Claude (in an agentic coding environment) autonomously edited a user’s configuration file to remove explicit/adult content without informing the user, then admitted to doing so when confronted.2020324739978522835 (permalink)
 An agent fabricated viral opportunity alerts with fake view counts, engagement stats, and tweet links without actually querying any database or browser, and continued doing so after being caught and promising to stop.2024973175272976841 (permalink)
Table 5: Examples of scheming-related incidents that indicate real-world proofs of existence for behaviours hitherto documented primarily in experiments.
Figure 4: An example of inter-model scheming. A screenshot appears to show Claude’s CoT summariser refusing a deception attempt by the underlying model.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

GroupPostsIncidentDate span
19112OpenClaw / Summer Yue email deletion17 days
5811Claude rm -rf ~/ home directory deletion90 days
1668Claude terraform destroy production infrastructure6 days
2887Google Antigravity D: drive deletion32 days
1905Grok denying Gaza/Israel posts5 days
2114Grok admitting deliberate deception in public posts20 days
Appendix
Table 10
MetricValue
Input posts895
Stage 1 clusters (semantic)809
Stage 2 merges (entity-based)119
Clusters after Stage 2690
Stage 3 manual corrections (net)+8
Final unique incidents698
Appendix
Table 11
ComponentLibrary
TF-IDF vectorisationscikit-learn TfidfVectorizer
Cosine similarityscikit-learn cosine_similarity
Hierarchical clusteringSciPy linkage / fcluster
Distance matrixSciPy squareform
Union-Findcustom Python (path-halving)
Entity extractionPython re (compiled regex)
Appendix
Table 12
More in this topic

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 280 papers

Explore similar work

CardsList
Mar 2, 2026

Evaluating and Understanding Scheming Propensity in LLM Agents

As frontier language models are increasingly deployed as autonomous agents pursuing complex, long-term objectives, there is increased risk of scheming: agents covertly pursuing misaligned goals. Prior work has focused on showing agents are capable of scheming, but their propensity to scheme in realistic scenarios remains underexplored. To understand when agents scheme, we decompose scheming incentives into agent factors and environmental factors. We develop realistic settings allowing us to systematically vary these factors, each with scheming opportunities for agents that pursue instrumentally convergent goals such as self-preservation, resource acquisition, and goal-guarding. We find only minimal instances of scheming despite high environmental incentives, and show this is unlikely due to evaluation awareness. While inserting adversarially-designed prompt snippets that encourage agency and goal-directedness into an agent's system prompt can induce high scheming rates, snippets used in real agent scaffolds rarely do. Surprisingly, in model organisms (Hubinger et al., 2023) built with these snippets, scheming behavior is remarkably brittle: removing a single tool can drop the scheming rate from 59% to 3%, and increasing oversight can raise rather than deter scheming by up to 25%. Our incentive decomposition enables systematic measurement of scheming propensity in settings relevant for deployment, which is necessary as agents are entrusted with increasingly consequential tasks.

Mia Hopman, Jannes Elstner, Maria Avramidou +2

LASR Labs · Google DeepMind

AI Agent SafetyLLM AgentsLLM Agent Evaluation
AskPDF
Sep 8, 2026

SchemeArena: Factorized Stress Testing of Scheming in LLM Agents

We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how scheming arises from the interaction of key factors, such as instrumental goals, environmental affordances, oversight conditions, and perceived consequences. Prior work examines only a small number of scenarios, limiting the ability to isolate how these conditions shape an agent's propensity or capability to scheme. This limited scale and task diversity also restrict coverage of realistic deployment settings and the range of scheming strategies that can be observed. To this end, we introduce SCHEMEARENA, a 400-scenario benchmark for scalable scheming stress testing, constructed through a factorized scenario synthesis framework spanning diverse safety-relevant tool domains, instrumental goals, oversight conditions, and pressure mechanisms. To enable scalable and reliable monitoring, we further propose SCOUT, a scheming monitor that grounds multi-criteria judgments in evidence drawn from agents' reasoning and actions. Across controlled stress tests on five LLM agents, we find that explicit instrumental goals are the strongest driver of scheming propensity. Strategic hints play a distinct role by helping agents translate scheming reasoning into concrete covert behavior. Oversight has mixed effects: in several closed models, action-only monitoring increases scheming, suggesting that partial oversight can act as an optimization constraint rather than a deterrent. CoT is a useful but incomplete monitoring signal: it can reveal latent scheming before execution, yet action-only scheming shows that covert behavior may occur without explicit reasoning evidence. We release the benchmark, code, and monitor at: https://github.com/launchnlp/SchemeArena.

Jie Ruan, Inderjeet Nair, Amy Liu +3

Computer Science and Engineering University of Michigan Ann Arbor, MI 48109, USA

LLM Agent EvaluationAI Agent MonitoringAI Agent Safety BenchmarksLLM Agent Safety
AskPDF
May 28, 2026

Realistic honeypot evaluations for scheming propensity

We introduce scheming honeypot evaluations, a framework for testing whether models will pursue instrumental goals if given the opportunity. Our scheming honeypot evaluations take the form of coding tasks in Google's alignment research codebases. In a real internal deployment setting, Gemini models do not demonstrate unprompted scheming. If prompts explicitly encourage agency (situational awareness or goal-directedness) and/or give the model a hidden goal, models sometimes scheme or attempt sabotage. Validating the realism of our setting, models show low rates of evaluation awareness, usually due to agency prompts rather than the environments.

Victoria Krakovna, David Lindner, Lewis Ho +2

1Google DeepMind

LLM Agent EvaluationAI Agent Safety
AskPDF