Realistic Threat Model

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 63

All topics

Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements
Oct 6, 2026Jeremy CanaleRealistic Threat Model
Distillation Defenses Easily Break After Reinforcement Learning
Sep 28, 2026Shidan Javaheri, Alexander Panfilov, Oliver Britton +2Probe-Logit Distillation LLM Defense Mechanisms
University of Oxford · ELLIS Institute Tübingen & MPI for Intelligent Systems
On Identifying Adversarial Intent Injection in AI-Native 6G Networks
Sep 14, 2026Nilesh Chakraborty, Petar Djukic, Burak KantarciIntrusion Detection Realistic Threat Model
University of Ottawa, Ottawa, ON, Canada · Nokia Bell Labs, 600 March Road, Kanata, ON K2K 2E6, Canada
Protocol effects on feature-based hardware-Trojan detection across Trust-Hub families
Sep 7, 2026Hang Xiao, Chuhong Xu, Kainan Zhou +2Malware Gating
Fortinet, Inc. Sunnyvale, USA · Sony Corporate of America San Jose, USA · Google LLC Mountain View, USA
Mind the Gap: Robustness Risks in PII Detection Systems
Sep 3, 2026Adeel Zafar, Slawomir NowaczykRealistic Threat Model Named-Entity Recognition
Center for Applied Intelligent Systems Research, Halmstad University, Sweden
Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario
Aug 8, 2026Satoshi MatsuokaRealistic Threat Model Large Language Model Agents
RIKEN Center for Computational Science, Kobe, Japan
Fairis: Fairness-Aware Aggregation with Provable Influence Containment against Fairness Poisoning Attacks in Collaborative Machine Learning
Aug 6, 2026Devharsh Trivedi, Nesrine Kaaniche, Nikos Triandopoulos +2Algorithmic Fairness Federated Learning
Department of Computer Science, Bowie State University, MD, USA · SAMOVAR, T´el´ecom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Department of Computer Science, Brown University, RI, USA +1
Breadcrumbing Search Agents
Aug 5, 2026Xuebin Li, Hanqing Zhao, Siyuan Liang +4Search Agents Instruction Injection Attacks
School of Cyber Science and Technology, University of Science and Technology of China · College of Computing and Data Science, Nanyang Technological University
Behavioral Grammar: Detecting Adaptive Malware via Tiny Language Model Priors and Second-Order Temporal Analysis
Aug 1, 2026Zihan LuoMalware Evasion
Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
Jul 30, 2026Fazhong Liu, Zhuoyan Chen, Haozhen Tan +3World Models Realistic Threat Model
Revisiting the Adversarial Robustness of Graph-Based Traffic Forecasting
Jul 30, 2026Qingzhao ZhangAccurate Traffic Forecasting Realistic Threat Model
The University of Arizoan
ThreatForest: Multi-Agent Attack Tree Generation with Pluggable TTP Framework Mapping
Jul 29, 2026Cristian Leo, Anton Dykyi, Danny Cortegaca +2Realistic Threat Model Threat
Amazon Web Services, Inc., 410 Terry Avenue North, Seattle, WA, 98109, United States · Amazon Web Services EMEA SARL, UK Branch, 1 Principal Place, Worship Street, London, EC2A 2FA, United Kingdom
Agentic Permissions Policy Algebra for Taint Confinement in LLM Agents
Jul 27, 2026Arseny Kravchenko, Vadim Liventsev, Innokentii Konstantinov +2Realistic Threat Model Information-Flow Control
Archestra AI
The Chronos Vulnerability: A Taxonomy of Temporal Persistence and Memory-Based Deception in Agentic AI
Jul 20, 2026Om Narayan, Ramkinker Singh, Praveen BaskarRealistic Threat Model LLM Defense Mechanisms
Department of Computer Science New York University New York, NY, USA · Carnegie Mellon University Pittsburgh, PA, USA · Independent Researcher Chicago, IL, USA
An Early Warning of Emerging Biosecurity Risks in Frontier LLMs
Jul 20, 2026Zhida He, Xia Hu, Baichen Le +20Large Language Model Safety Life Sciences Research
Shanghai Artificial Intelligence Laboratory
Signal-based Model Access Risk Analysis for AI System Operations Security
Jul 17, 2026Maria Mahbub, Steven Young, Amir Sadovnik +4Artificial Intelligence Risk Realistic Threat Model
Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA · Oak Ridge Institute for Science and Education, Oak Ridge, TN 37830, USA · DHS Science and Technology Directorate, Washington, DC 20528, USA
Rethinking Penetration Testing for AI-Enabled Systems: From Resource Compromise to Behavioral Objective Violation
Jul 15, 2026Mohammad Allahbakhsh, Mohammad Hassan Bahari, Moslem Attar-RaoufPenetration Testing Adversarial Evaluation
Computer Engineering Department, Ferdowsi University of Mashhad, Mashhad, Iran. · Sensifai BV, Brussels, Belgium.
A Threshold Exceedance Framework for CBRN Uplift Evaluation in Frontier Language Models
Jul 13, 2026Rahul Gupta, Abhinav Mohanty, Payal Motwani +8Realistic Threat Model Rise
Amazon Nova Responsible AI · Work done while at Amazon. · Nemesys Insights
The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators
Jul 12, 2026Dominik SchwarzRealistic Threat Model Model Auditing
Independent Researcher
Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability
Jul 11, 2026Lingwei Wei, Dou Hu, Wei Zhou +2Misinformation Realistic Threat Model
Institute of Information Engineering, Chinese Academy of Sciences, China · University of Illinois Chicago, USA · State Key Laboratory of Media Convergence and Communication, Communication University of China, China +1
Unicode TAG-Block Concealment of Tool-Metadata Payloads in the Model Context Protocol: An Approval-View Fidelity Gap Across Three Independent Server Implementations
Jul 7, 2026Mohammadreza RashidiModel Context Protocol Metadata
Department of Computer Science AI and Media Analysis Lab Berlin, Germany
From Forgeries to Foundation Models: A Systematic Survey of Identity Document Attack and Detection
Jul 1, 2026Gourab Das, Pavan Kumar C, Raghavendra RamachandraForgeries Realistic Threat Model
Indian Institute of Information Technology Dharwad (IIIT Dharwad), India · SAFE Center, Norwegian University of Science and Technology (NTNU), Gjøvik, Norway
On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models
Jun 25, 2026Dewank Pant, Shruti Lohani, Avijit KumarPrompt-Injection Detectors Untrusted Content
Independent Researcher
Hybrid privacy-aware semantic search: SVD-truncated document geometry and CKKS-encrypted query reranking under a restricted threat model
Jun 24, 2026Sergey KurilenkoVector Database Realistic Threat Model
Moscow Institute of Physics and Technology, Dolgoprudny, Moscow Region, Russia.
Falcon: Functional Assembly and Language for Compositional Reasoning in X-ray
Jun 24, 2026Yonathan Michael, Mohamad Alansari, Natnael Takele +2X-Ray Recent Vision-Language Models
Computer Science Department Khalifa University · Center of Cyber-physical Systems (C2PS) Khalifa University
Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems
Jun 24, 2026Balamurugan Palanisamy, G S S Chalapathi, Vikas Hassija +1Agentic Retrieval-Augmented Generation Systems Large Language Model Safety
Department of Electrical and Electronics Engineering Birla Institute of Technology and Science, Pilani, Pilani Campus Vidya Vihar, Pilani, Rajasthan 333031, India · Department of Computer Engineering, KIIT University Bhubaneshwar, Odisha 751024, India · Quantum Cloud Computing and Distributed Systems (qCLOUDS) Laboratory Department of Computing and Information Systems The University of Melbourne, Melbourne, Australia
What Does It Mean to Break a Distillation Defense?
Jun 23, 2026Lena Libon, Pura Peetathawatchai, Michael Aerni +2Realistic Threat Model LLM Defense Mechanisms
ETH Zurich
TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction
Jun 17, 2026Moon Ye-Bin, Nam Hyeon-Woo, Baek Seong-Eun +2Privacy Leakage
Dept. of Electrical Engineering, POSTECH · Grad. School of Artificial Intelligence, POSTECH · Grad. School of Metaverse, KAIST +1
Continual Backdoor Training in IoT/CPS
Jun 12, 2026Oxana Salish, Kuniyilh SClean Label Backdoor Attack Internet Of Thing
Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers
Jun 10, 2026Jun Wen LeongDistribution Shifts Threat Detection
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
Jun 9, 2026Joachim Schaeffer, Alexander Panfilov, Thomas Jiralerspong +4Adversarial Evaluation Realistic Threat Model
MATS · ELLIS Institute Tübingen, MPI for Intelligent Systems & Tübingen AI Center · Mila – Quebec AI Institute +4
Assessing Automated Prompt Injection Attacks in Agentic Environments
Jun 9, 2026David Hofer, Edoardo Debenedetti, Florian TramèrIndirect Prompt Injection Large Language Model Agents
ETH Zurich Zurich, Switzerland
Brain-Prompt Injection: A Route-Safety Audit for BCI-LLM Agents
Jun 8, 2026Jianwei TaiPrompt-Injection Detectors Model Auditing
School of Internet, Anhui University
Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics
Jun 2, 2026Xiaofei Wang, Mingliang Han, Tianyu Hao +3Diffusion-Based Vision-Language-Actions Realistic Threat Model
Department of Automation, University of Science and Technology of China, Hefei, China · SmartMore Corporation · Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou, China +2
AI Model Extraction Attacks: Bypassing Single-Client Assumptions in Defenses
Jun 2, 2026Maxime Schwarzer, Johannes F. Loevenich, Gustavo Sánchez +5Realistic Threat Model LLM Defense Mechanisms
CortAIx Labs, Thales Deutschland, Ditzingen, Germany · Department of Mathematics/Computer Science, University of Osnabrück, Osnabrück, Germany · Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany +2
A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning
May 28, 2026Ding Chen, Xinwen Cheng, Xuyang Zhong +3Membership Inference Attacks Model Evaluation
City University of Hong Kong · Shanghai Jiao Tong University · denotes the correspondence author.
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Muon
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2
When Agents Control Robots: A Zero Trust Policy Model for Agentic Cyber-Physical Systems
May 25, 2026Tharindu Ranathunga, Kavishka Fernando, Susan ReaCyber-Physical System Realistic Threat Model
Nimbus Centre, Munster Technological University, Cork, Ireland
Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions
May 24, 2026Wenjuan Li, Yitao Liu, Runze Chen +1Large Language Model Fine-Tuning Model Fine-Tuning
Hangzhou Normal University, No. 2318, Yuhangtang Rd, Hangzhou, 311121, Zhejiang, China · Zhejiang University, No. 866 Yuhangtang Rd, Hangzhou, 2026 310058, Zhejiang, China · China Mobile (Zhejiang) Innovation Research Institute Co., Ltd., No. 77 Gaoxin 7th Rd, China Mobile Zhejiang Information and Communication Industrial Park, Hangzhou, 311200, Zhejiang, China +1
Test-Time Training Undermines Safety Guardrails
May 21, 2026Simone Antonelli, Sadegh Akhondzadeh, Aleksandar BojchevskiInference-Time Defense Test-Time Training
CISPA Helmholtz Center for Information Security · University of Cologne
MV-Gate: Insider Threat Detection via Multi-View Behavioral Statistics and Semantic Modeling
May 18, 2026Kaichuan Kong, Dongjie Liu, Xiaobo Jin +1Intrusion Detection Realistic Threat Model
College of Cyber Security, Jinan University, Guangzhou, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China
STRIDE-AI: A Threat Modeling Framework for Generative AI Security Assessment
May 16, 2026Tsafac Nkombong Regine Cyrille, Franziska SchwarzArtificial Intelligence Risk Realistic Threat Model
SRH University of Applied Sciences Heidelberg School of Technology and Architecture CyberMACS (Applied Cybersecurity) Berlin, Germany · Universidad de Granada Facultad de Ciencias Econ´omicas y Empresariales Granada, Spain
Training on Documents About Monitoring Leads to CoT Obfuscation
May 14, 2026Reilly Haskins, Bilal Chughtai, Joshua EngelsSabotage Obfuscation
University of Canterbury · Google DeepMind · Pivotal Research
Watermarking Should Be Treated as a Monitoring Primitive
May 13, 2026Toluwani Aremu, Nils Lukas, Jie ZhangWatermarking Realistic Threat Model
1MBZUAI, UAE · 2A*STAR, Singapore
CoT-Guard: Small Models for Strong Monitoring
May 12, 2026Nirav Diwan, Han Wang, Berkcan Kapusuzoglu +6Realistic Threat Model LLM Defense Mechanisms
†University of Illinois Urbana-Champaign · ♢Capital One
MATRA: Modeling the Attack Surface of Agentic AI Systems -- OpenClaw Case Study
May 11, 2026Tim Van hamme, Thomas Vissers, Javier Carnerero-Cano +4Realistic Threat Model Artificial Intelligence Risk
DistriNet, KU Leuven, 3001 Leuven, Belgium · IBM Research · CISPA Helmholtz Center for Information Security +2
PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines
May 11, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleAttacker Large Language Model Realistic Threat Model
School of Computing and Electrical Engineering · Indian Institute of Technology, Mandi (IIT) · The Alan Turing Institute, London +1
Privacy-Preserving Distributed Learning in IoT Systems: A Unified Threat Model and Evaluation Framework
May 10, 2026John Cartmell, Alexander WilliamsPrivacy-Preserving Machine Learning Internet Of Thing
Department of Electrical Engineering and Computer Science
Methods for Formal Verification of Agent Skills: Three Layers Toward a Mechanically Checkable Capability-Containment Proof
May 9, 2026Alfredo MetereSkills Formal Verification
Metere Consulting, LLC
ORACLE: Anticipating Scams from Partial Trajectories in Streaming App Usage
May 9, 2026Wenbo Gao, Songbai Tan, Zhongan Wang +6Fraud Detection Threat Detection
Harbin Institute of Technology, Shenzhen · Shenzhen University · Zhejiang University +4
Defense effectiveness across architectural layers: a mechanistic evaluation of persistent memory attacks on stateful LLM agents
May 8, 2026Jun Wen LeongLLM Defense Mechanisms Realistic Threat Model
HBEE: Human Behavioral Entropy Engine -- Pre-Registered Multi-Agent LLM Simulation of Peer-Suspicion-Based Detection Inversion
May 8, 2026Vickson FerrelRealistic Threat Model Adversarial Robustness
Faculty of Computer Science & Information Technology Universiti Malaysia Sarawak (UNIMAS), Kota Samarahan, Malaysia · Founder & Lead Architect, Vixero Technology Enterprise, Kuching, Sarawak, Malaysia
Narrow Secret Loyalty Dodges Black-Box Audits
May 7, 2026Alfie Lamerton, Fabien RogerRealistic Threat Model Poisoning
Formation Research
Gray-Box Poisoning of Continuous Malware Ingestion Pipelines
May 6, 2026Jan Dolejš, Martin Jureček, Róbert LórenczMalware Poisoning
Faculty of Information Technology, Czech Technical University in Prague, Thákurova 9, Prague, 16000, Czech Republic
On the Privacy of LLMs: An Ablation Study
May 4, 2026Karima Makhlouf, Lamiaa Basyoni, Syed Khaderi +4Membership Inference Attacks Attacker Large Language Model
UDST, Doha, Qatar
Trojan Hippo Bench: A Dynamic Benchmark for Persistent Memory Attacks and Defenses in LLM Agents
May 3, 2026Debeshee Das, Julien Piet, Darya Kaviani +3Agentic Memory Realistic Threat Model
ETH Zürich · UC Berkeley · Snyk
From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systems
Apr 29, 2026Neha Nagaraja, Hayretdin Bahsi, Carlo R. da CunhaRealistic Threat Model Threat
School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia
Evaluation without Generation: Non-Generative Assessment of Harmful Model Specialization with Applications to CSAM
Apr 28, 2026Vinith M. Suriyakumar, Ayush Sekhari, Lena Stempfle +5Adversarial Evaluation Child Sexual Abuse Material
MIT · BU · Thorn
Risk Reporting for Developers' Internal AI Model Use
Apr 27, 2026Oscar Delaney, Sambhav Maheshwari, Joe O'Brien +2Artificial Intelligence Risk Pre-Deployment Safety Assessments
† Work done while at IAPS
Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines
Apr 26, 2026Mazal Bethany, Kim-Kwang Raymond Choo, Nishant Vishwamitra +1Attacker Large Language Model Evasion
Department of Information Systems and Cybersecurity, University of Texas at San Antonio, San Antonio, TX 78249 USA

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements

Jeremy Canale

Realistic Threat Model

Open paper AskPDF arXiv

Whether alignment gets easier or harder as models grow is often argued from isolated findings, as if alignment were one property. We treat it as a family of measurable scaling relations: for each risk category r, the alignment burden needed to hold a fixed safety target is modeled as B_r(N)=a_rN^alpha_r, with N a capability proxy; against a budget proportional to N, scaling helps if alpha_r<1, keeps pace if alpha_r~1, and accumulates alignment debt if alpha_r>1. We give three operationalizations of burden and distinguish observed, audited and true alignment. A toy model, in which corrections consume capability headroom, makes the consequences explicit. We prove that the largest exponent among corrected risks, not an average, sets the long-run regime; that above 1 any policy holding headroom above a floor must grow super-exponentially; that, for burdens that are positive mixtures of power laws, fits on small models underestimate large-scale exponents; and that an audit that uncovers hidden failures without false positives never underestimates true alignment. We propose a pre-registrable protocol and apply reduced versions of it twice. A preregistered reanalysis of public adversarial-training data for Pythia classifiers finds that the compute needed to bring attack success under 10% grows as N^0.60. A preregistered pilot on Qwen2.5 0.5B-72B finds exponents of -0.05 for truthfulness and 0.48 for stated dispositions (both scaling helps under its reduced rule, though local slopes approach 1 at the top; replicated on Qwen3 0.6B-14B), while sycophancy (0.89, or 0.83 with two seeds added at 72B) and a planted backdoor are undetermined: the backdoor is removed quickly when its trigger is known but survives blind safety training at four of five sizes. We release four browser games that play these laws (www.aisafety.fun). We make no claim about which regime holds for current frontier models.

Momentum

Related topics

Latest papers 63