Signals in the Noise: Open Source Intelligence (OSINT) for AI Loss of Control Detection

Sarah Bollinger; Nada Aboserie; Amanda Coakley; Chih-Hsuan Lee; Taysir Mathlouthi

cs.CYMay 22, 2026

Signals in the Noise: Open Source Intelligence (OSINT) for AI Loss of Control Detection

Authors: Sarah Bollinger, Nada Aboserie, Amanda Coakley, Chih-Hsuan Lee, Taysir Mathlouthi

Organizations: AI Governance Taskforce, Arcadia Impact

Abstract

This paper applies open-source intelligence (OSINT) and cyber threat intelligence (CTI) methodologies to the problem of detecting AI systems operating outside human control. Drawing on a cross-disciplinary literature review and 14 semi-structured expert interviews conducted under Chatham House Rule, the paper develops two threat models, identifies a range of observable traces, and proposes an institutional architecture for monitoring. The research finds that OSINT-based detection of loss of control is partially feasible and worth building now. Three detection vectors emerge as highest priority: transcript-based collection of user-reported AI behaviour; infrastructure correlation for unexpected external connections or replication; and output analysis for capability concealment. The paper argues for a dedicated, federated international monitoring capability anchored in OSINT methods and independent of frontier AI developers, and identifies sustained non-industry funding as the highest-leverage structural intervention available.

Explore similar work

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
Jul 3, 2026Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin +3Agentic Systems Generative Artificial Intelligence
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
Jun 9, 2026Joachim Schaeffer, Thomas Jiralerspong, Alexander Panfilov +4Adversarial Evaluation Untrusted Input
Reframing AI Loss of Control: What It Is, How to Have It, How to Lose It
May 19, 2026Ze Shen Chin, Maurice Chiodo, Dennis Müller +1

Jul 3, 2026 · cs.CRJ/K move · Enter open · S save

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin, Zubair Baig+2

The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation. Large language models (LLMs) and agentic AI systems, capable of tool use, multi-step reasoning, and iterative intelligence generation, have emerged as promising solutions, yet evaluation frameworks have not kept pace with reported capabilities. This survey systematically reviews 74 studies and makes four contributions. First, it establishes agentic AI as a distinct analytical category rather than an extension of LLM prompting, organising the literature through an 11-category taxonomy covering LLM foundations, agentic architectures, retrieval-augmented generation (RAG), knowledge graphs, prompt engineering, domain adaptation, evaluation benchmarks, and risk. Second, it identifies the hallucination-validation gap as a corpus-level finding: although hallucination is recognised as a major reliability concern in over twenty studies, end-to-end hallucination is empirically measured in only one OSINT-specific RAG-based system, non-reproducible conditions, while related reasoning and factual-correction studies evaluate general-domain question answering rather than OSINT. Third, it maps existing research to the OSINT lifecycle, showing strong support for collection and analysis but limited coverage of verification, reporting, dissemination, and decision support. Fourth, it derives a ten-point research agenda addressing evaluation, benchmarking, hallucination measurement, adversarial robustness, dark-web coverage, multimodal intelligence, and governance. It concludes that a human-AI co-pilot model, where LLMs assist collection and triage while analysts retain responsibility for verification and decision-making, represents the most defensible near-term deployment architecture.

Agentic Systems Generative Artificial Intelligence Taxonomy

Open paper Ask about it PDF

Abstract

Explore similar work

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

Reframing AI Loss of Control: What It Is, How to Have It, How to Lose It