cs.CRFeb 23, 2026

LLM-enabled Applications Require System-Level Threat Monitoring

Authors: Yedi Zhang, Haoyu Wang, Xianglin Yang, Jin Song Dong, Jun Sun

Organizations: Department of Computer Science, National University of Singapore, Singapore · School of Computing and Information Systems, Singapore Management University, Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM-enabled applications are rapidly reshaping the software ecosystem by using large language models as core reasoning components for complex task execution. This paradigm shift, however, introduces fundamentally new reliability challenges and significantly expands the security attack surface, due to the non-deterministic, learning-driven, and difficult-to-verify nature of LLM behavior. In light of these emerging and unavoidable safety challenges, we argue that such risks should be treated as expected operational conditions rather than exceptional events, necessitating a dedicated incident-response perspective. Consequently, the primary barrier to trustworthy deployment is not further improving model capability but establishing system-level threat monitoring mechanisms that can detect and contextualize security-relevant anomalies after deployment -- an aspect largely underexplored beyond testing or guardrail-based defenses. Accordingly, this position paper advocates systematic and comprehensive monitoring of security threats in LLM-enabled applications as a prerequisite for reliable operation and a foundation for dedicated incident-response frameworks.

Figures & tables

Figure 1: A respresentative LLM-enabled application workflow : The user submits an initial prompt (Stage 1); the client, responsible for orchestration, queries the MCP service for available tools (Stage 2) and forwards an integrated prompt to the LLM brain (Stage 3), which may interact with external resources such as vector databases, (Graph)-RAG systems, and memory (Stage 3*). The brain produces intermediate responses and tool plans (Stage 4); the client executes the selected tools via MCP and gathers results (Stage 5), assembles the final prompt (Stage 6), obtains the final response from the brain (Stage 7), and delivers it to the user (Stage 8).
Threat CategoryAttack Vector
(1) Prompt InjectionDirect prompt injection; Injected instructions in RAG; Service API outputs
(2) Adversarial InputsLexical obfuscation; Embedding-level attacks; Adversarial multi-modal inputs or intermediate outputs
(3) Response ManipulationPrompt chaining and contextual drift; Environment tampering and reuse; Feedback gaming
(4) DoS or Unbounded LoopsOversized requests; Recursive prompting or unbounded loops; Tool-call storms and expensive tools; Adversarial cache-bypass
(5) Live Data PoisoningFeedback API manipulation; Poisoned content ingestion; Telemetry tampering
(6) Live Model PoisoningCompromised model update plane; Runtime environment tampering; Model repository or registry compromise
Table 1: The threat categories and associated attack vectors (or attack surfaces) for LLM-based applications considered in this paper.
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
Figure 23
Figure 24
Figure 25
Figure 26
Figure 27
Figure 28
Figure 29
Figure 30
Figure 31
Figure 32
Figure 33
Figure 34
Figure 35
Figure 36
Figure 37
Figure 38
Figure 39
Figure 40
Figure 41
Figure 42
More in this topic

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 322 papers

Explore similar work

CardsList

  1. Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment

    Mar 8, 2026Neha Nagaraja, Hayretdin BahsiLLM SecurityCybersecurity

    School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia

  2. Online Safety Monitoring for LLMs

    Jul 2, 2026Mona Schirmer, Metod Jazbec, Alexander Timans +3Language Model Safety EvaluationLLM Safety Monitoring

    UvA Bosch-Delta Lab, University of Amsterdam · University of Wisconsin Madison · Johns Hopkins University

  3. Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

    Jun 9, 2026Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1Prompt InjectionLLM Agent Security

    State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany