Sentinel Agents for Secure and Trustworthy Agentic AI in Multi-Agent Systems
Organizations: Head of AI Tesisquare Voiceinteroperability.ai Initiative Member Linux Foundation AI & Data Torino, TO 10100, Italy · Principal Conversational Technologies Voiceinteroperability.ai Initiative Member Linux Foundation AI & Data Plymouth Meeting, Pennsylvania, USA
Abstract
Multi-agent systems (MAS) built on large language models (LLMs) increasingly share a common conversational space, in which a single adversarial or erroneous message can propagate from one agent to many. This paper proposes Sentinel Agents, a dedicated security layer that monitors inter-agent communication with rule-based checks, behavioral analysis and LLM-based semantic inspection, and reports to a Coordinator Agent that enforces policy, for example by quarantining misbehaving agents. We describe the architecture, its deployment patterns and monitoring layers, and we evaluate a continuous-listener prototype on a balanced corpus of 1,000 synthetic messages (500 attacks across five families and 500 benign messages) with two local LLMs. LLM inspection with Gemma 4 12B reached F1 0.86 at a false-positive rate of 0.11, against 0.38 for rules alone. Combining components by logical OR did not improve on it, while a cascade in which only high-precision rules act on their own matched it. Calibrating the thresholds of the more conservative Qwen3 8B on a development split raised its held-out F1 from 0.56 to 0.80, and on multi-turn conversations the behavioral check detected patterns the rules missed but could not tell legitimate repetition from probing. The value of combining detectors therefore depends on the model, on its calibration and on the aggregation rule, and the results yield practical guidance for deploying Sentinel Agents.
Figures & tables
| Aspect | Sidecar Pattern | Proxy / AI Gateway Pattern | Continuous Listener Pattern | Hybrid Pattern |
|---|---|---|---|---|
| Deployment Location | Co-located with each individual agent (same host/pod). | Centralized or distributed intermediary service through which all traffic flows. | Independent observer subscribing passively to the Shared Conversational Space. | Combination of sidecar, proxy, and continuous listener deployments. |
| Protection Strength | Strong for agent-specific threats (e.g., input validation, PII redaction). | Strong for cross-agent security, but may be bottlenecked if not optimized. | Reactive only — cannot block malicious content in real-time. | High overall resilience — combines proactive blocking and reactive observability. |
| Latency Impact | Low latency (local interception), but overhead for very “chatty” agents. | Potentially higher latency if centralized; mitigated with load-balancing. | Minimal latency (read-only subscription, no blocking). | Moderate — some overhead from layering, but balanced by distribution. |
| Scalability | Scales with number of agents but adds deployment overhead. | Scales with system traffic; requires careful optimization to avoid single point of failure. | Scales independently; low overhead, but only observational. | Scales flexibly by combining multiple deployment layers. |
| Aspect | Pattern A: Pre-validation (Blocking Mode) | Pattern B: Passive Listening (Reactive Flagging) | Pattern C: Hybrid Layering |
|---|---|---|---|
| Primary Role | Proactively blocks unsafe Interaction Messages before they reach the Coordinator Agent. | Reactively observes processed Interaction Messages and issues flags for anomalies or policy violations. | Combines proactive blocking with reactive flagging for comprehensive coverage. |
| Protection Strength | Maximum protection — malicious content is stopped before propagation. | Lower protection — threats are detected only after messages are processed. | Balanced — critical threats blocked early, less severe issues logged or flagged after processing. |
| Latency Impact | Higher latency due to pre-processing of every Interaction Message. | Minimal latency since no blocking occurs in real-time. | Moderate latency — pre-validation adds overhead but limited to high-risk checks. |
| Use Cases | High-security environments (finance, healthcare) where prevention is critical. | Environments prioritizing speed over strict prevention; useful for threat intelligence gathering. | General-purpose systems needing both robust protection and detailed observability. |
| Observability | Limited — mainly focused on block decisions, less on continuous monitoring. | High — provides detailed logging, anomaly detection, and post-hoc analysis. | Very high — combines proactive intervention with transparent auditing and continuous improvement. |
| Category | Easy | Medium | Hard | Total |
|---|---|---|---|---|
| Prompt injection | 80 | 70 | 50 | 200 |
| Data exfiltration | 40 | 35 | 25 | 100 |
| Hallucination | 30 | 40 | 30 | 100 |
| Tool abuse | 15 | 20 | 15 | 50 |
| Identity attack | 15 | 20 | 15 | 50 |
| Attacks | 180 | 185 | 135 | 500 |
| Configuration | TP | FP | TN | FN | Precision | Recall | F1 [95% CI] | FPR [95% CI] |
|---|---|---|---|---|---|---|---|---|
| R only | 130 | 50 | 450 | 370 | 0.722 | 0.260 | 0.382 [0.333, 0.434] | 0.100 [0.075, 0.126] |
| B only | 28 | 0 | 500 | 472 | 1.000 | 0.056 | 0.106 [0.021, 0.209] | 0.000 [0.000, 0.000] |
| R + B | 158 | 50 | 450 | 342 | 0.760 | 0.316 | 0.446 [0.386, 0.508] | 0.100 [0.075, 0.126] |
| L only | 419 | 57 | 443 | 81 | 0.880 | 0.838 | 0.859 [0.832, 0.885] | 0.114 [0.087, 0.145] |
| R + L | 419 | 84 | 416 | 81 | 0.833 | 0.838 | 0.835 [0.806, 0.863] | 0.168 [0.136, 0.202] |
| B + L | 419 | 57 | 443 | 81 | 0.880 | 0.838 | 0.859 [0.832, 0.885] | 0.114 [0.087, 0.145] |
| Strategy | TP | FP | Precision | Recall | F1 [95% CI] | FPR [95% CI] | Macro-F1 |
|---|---|---|---|---|---|---|---|
| Rules only | 130 | 50 | 0.722 | 0.260 | 0.382 [0.333, 0.434] | 0.100 [0.075, 0.126] | 0.283 |
| Gemma 4 12B alone | 419 | 57 | 0.880 | 0.838 | 0.859 [0.832, 0.885] | 0.114 [0.087, 0.145] | 0.718 |
| Sentinel + Gemma 4 | 419 | 84 | 0.833 | 0.838 | 0.835 [0.806, 0.863] | 0.168 [0.136, 0.202] | 0.707 |
| Qwen3 8B alone | 217 | 11 | 0.952 | 0.434 | 0.596 [0.546, 0.644] | 0.022 [0.010, 0.034] | 0.481 |
| Sentinel + Qwen3 | 276 | 58 | 0.826 | 0.552 | 0.662 [0.621, 0.704] | 0.116 [0.089, 0.145] | 0.494 |
| LLM | Policy | TP | FP | Precision | Recall | F1 [95% CI] | FPR [95% CI] |
|---|---|---|---|---|---|---|---|
| Gemma 4 | Logical OR | 419 | 84 | 0.833 | 0.838 | 0.835 [0.808, 0.862] | 0.168 [0.137, 0.202] |
| Gemma 4 | Rules AND LLM | 130 | 23 | 0.850 | 0.260 | 0.398 [0.349, 0.450] | 0.046 [0.028, 0.067] |
| Gemma 4 | Majority of R, B, L | 158 | 23 | 0.873 | 0.316 | 0.464 [0.407, 0.525] | 0.046 [0.028, 0.067] |
| Gemma 4 | Cascade | 419 | 57 | 0.880 | 0.838 | 0.859 [0.832, 0.882] | 0.114 [0.087, 0.143] |
| Qwen3 | Logical OR | 276 | 58 | 0.826 | 0.552 | 0.662 [0.621, 0.701] | 0.116 [0.091, 0.146] |
| Qwen3 | Rules AND LLM | 99 | 3 | 0.971 | 0.198 | 0.329 [0.279, 0.381] | 0.006 [0.000, 0.014] |
| LLM | TP | FP | Precision | Recall | F1 [95% CI] | FPR [95% CI] | |
|---|---|---|---|---|---|---|---|
| Gemma 4 | 1 (default) | 227 | 24 | 0.904 | 0.850 | 0.876 [0.843, 0.908] | 0.098 [0.063, 0.139] |
| Gemma 4 | 0.43 (development) | 239 | 34 | 0.875 | 0.895 | 0.885 [0.853, 0.915] | 0.139 [0.098, 0.185] |
| Qwen3 | 1 (default) | 106 | 4 | 0.964 | 0.397 | 0.562 [0.482, 0.641] | 0.016 [0.004, 0.034] |
| Qwen3 | 0.60 (development) | 184 | 10 | 0.948 | 0.689 | 0.798 [0.735, 0.856] | 0.041 [0.020, 0.068] |
| Level | Configuration | TP | FP | Precision | Recall | F1 [95% CI] | FPR [95% CI] |
|---|---|---|---|---|---|---|---|
| Conversation | R only | 23 | 0 | 1.000 | 0.460 | 0.630 [0.484, 0.750] | 0.000 [0.000, 0.000] |
| Conversation | B only | 40 | 25 | 0.615 | 0.800 | 0.696 [0.602, 0.786] | 0.500 [0.356, 0.648] |
| Conversation | R + B | 50 | 25 | 0.667 | 1.000 | 0.800 [0.710, 0.871] | 0.500 [0.356, 0.648] |
| Message | R only | 33 | 0 | 1.000 | 0.114 | 0.205 [0.128, 0.293] | 0.000 [0.000, 0.000] |
| Message | B only | 94 | 150 | 0.385 | 0.325 | 0.353 [0.289, 0.415] | 0.225 [0.149, 0.306] |
| Message | R + B | 114 | 150 | 0.432 | 0.394 | 0.412 [0.344, 0.472] | 0.225 [0.149, 0.306] |
| Attack vs. benign | Category (6 classes) | |||
| Pair | Agreement | Cohen’s | Agreement | Cohen’s |
| Annotator 1 vs. Annotator 2 | 97/100 | 0.87 | 92/100 | 0.90 |
| Annotator 1 vs. construction labels | 98/100 | 0.92 | 93/100 | 0.92 |
| Annotator 2 vs. construction labels | 99/100 | 0.96 | 99/100 | 0.99 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Item | Value |
|---|---|
| Dataset | evaluation_v2 , 1,000 messages, SHA-256 79ae058ee541b86e53d41eefdb454e698b36869c4cdaaa2cec71d124a4c864ff |
| Gemma 4 12B | gemma4:12b , digest 4eb23ef187e2c5462566d6a1d3bbbc2f1346d0b4327cbb66d58fffbcc9b2b05c , Q4_K_M, thinking disabled |
| Qwen3 8B | qwen3:8b , digest 500a1f067a9f782620b40bee6f7b0c89e17ae61f686b92c24933e4ca4b2b8b41 , Q4_K_M, thinking disabled |
| Classification prompt | sentinel-inspector-v2.0 , SHA-256 7f9d9dc204f6dfc6940a2fcab62fe028d1e8499d625b83b217c2fb9bb037b618 |
| Decoding | temperature 0, context 4,096 tokens, at most 400 output tokens, seed 20261031, timeout 120 s, two retries, one request at a time |
| Thresholds | prompt-injection risk ; data-exfiltration, hallucination, tool-abuse and identity-attack risk |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 351 papers