LatentTriad: Pre-Action Risk Representations for Malicious Agent Skill Detection
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Griffith University · Wake Forest University · George Mason University
Abstract
Large language model agents are increasingly extended with reusable skills, but untrusted skill content can induce operations outside the user's task. Detecting this contextual risk requires more than suspicious-pattern matching, while dynamic analysis incurs the cost of skill execution. We find that hidden states formed during prefill contain readable information about task-operation consistency, including across held-out expression forms in controlled evaluations. Matched Operation-state interventions also affect later Boundary states and shift fixed risk readouts toward the donor label. Based on these findings, we introduce LatentTriad, which combines contextual Boundary features, group-adjusted Operation directions, and departures from benign state relationships. All branches share one prefill: detection generates no output tokens, invokes no tools, and executes no skill code. Across 5,830 skill packages from five sources and seven backbones, LatentTriad achieves 99.0-99.2% pooled F1 and 93.0-94.6% source-macro F1. A separately measured compact implementation completes prefill and detection in a median of 259-284 ms. These results connect pre-action risk readability and functional influence to execution-free skill screening.
Figures & tables
| Approach | Method | Task- aware | Multi- file | Execution free | Decoding free | LLM pass |
| Content inspection | Cisco Skill Scanner ( Cisco, 2026 ) | |||||
| SkillFortify ( Bhardwaj, 2026 ) | ||||||
| SkillScan Security ( SkillScan, 2026 ) | ||||||
| SkillSieve ( Hou and Yang, 2026 ) | ||||||
| Sandbox | SkillDetonate ( Ji et al., 2026 ) | |||||
| State probing | RouteGuard ( Xiao et al., 2026 ) |
| Method | Backbone | FPR | FNR | Prec. | Recall | F1 | Src.-macro F1 [95% CI] | AUROC | AP |
|---|---|---|---|---|---|---|---|---|---|
| Input-level baselines | |||||||||
| Regex | Shared | 100.0 | 0.0 | 50.0 | 100.0 | 66.7 | 66.7 [66.7, 66.7] | 68.6 | 68.6 |
| SkillSieve L1 | Shared | 31.0 | 29.7 | 69.4 | 70.3 | 69.8 | 60.5 [53.8, 66.0] | 69.0 | 69.3 |
| Cisco AI Skill Scanner | Shared | 3.1 | 73.3 | 89.6 | 26.7 | 41.1 | 27.6 [21.6, 33.1] | 63.4 | 64.7 |
| LLM-as-judge | Mistral-7B-Instruct-v0.3 | 2.4 | 79.5 | 89.4 | 20.5 | 33.3 | 30.5 [25.0, 35.4] | 81.0 | 81.1 |
| Supervised Qwen text readout | Qwen2.5-7B | 7.5 | 37.3 | 89.3 | 62.7 | 73.7 | 58.4 [52.5, 63.4] | 88.3 | 90.2 |
| Configuration | AUROC | F1 | FPR | Recall |
|---|---|---|---|---|
| Full: | 99.02 | 93.91 | 9.51 | 96.73 |
| Boundary-only | 97.28 | 91.63 | 9.51 | 92.58 |
| Operation-only | 99.03 | 86.69 | 0.00 | 80.04 |
| Geometry-only | 99.68 | 95.85 | 0.00 | 92.42 |
| 99.15 | 94.08 | 9.51 | 97.05 | |
| Matched independent-state control | 98.95 | 93.89 | 9.51 | 96.68 |
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | Role | Layers | Sampled layer indices | Inference |
|---|---|---|---|---|
| Llama-3.1-8B-Instruct | All + exact/runtime | 32 | 3, 7, 11, 15, 19, 23, 27, 31 | BF16, SDPA |
| Qwen2.5-7B-Instruct | All + exact/runtime | 28 | 3, 6, 9, 13, 17, 20, 23, 27 | BF16, SDPA |
| Mistral-7B-Instruct-v0.3 | Detection/trajectory | 32 | 3, 7, 11, 15, 19, 23, 27, 31 | BF16, SDPA |
| InternLM2.5-7B | Detection/trajectory | 32 | 3, 7, 11, 15, 19, 23, 27, 31 | BF16, SDPA |
| Qwen2.5-14B | Detection/trajectory | 48 | 5, 11, 17, 23, 29, 35, 41, 47 | BF16, SDPA |
| Phi-3-medium-14B | Detection/trajectory | 40 | 4, 9, 14, 19, 24, 29, 34, 39 | BF16, SDPA |
| Dataset / panel | Support | Question tested |
|---|---|---|
| Task–operation compatibility | 800 examples/model | Cross-form readout and matched Operation intervention |
| Fixed-node composition | 4,608 presentations/model | Connection and authorization counterfactuals |
| Structural transfer | 2,304 inputs/model | Known-template panel; separate template holdouts in Appendix G.6 |
| Same-prompt diagnosis | questions/model | Generated judgments versus pre-action readouts |
| Large-ID | 2,915 matched pairs | Skill detection and final-configuration controls |
| CompoSkill | 690 matched path pairs | Whole-path detection; also the support for adaptive rewriting |
| Source | Pairs | Components | Packages | Share |
| MalSkillBench | 2,621 | 1,824 | 5,242 | 89.9% |
| SkillFortify | 180 | 39 | 360 | 6.2% |
| SkillsBench-1650 | 63 | 41 | 126 | 2.2% |
| Agent-skill malware | 29 | 29 | 58 | 1.0% |
| ATR Skill Benchmark | 22 | 21 | 44 | 0.8% |
| Total | 2,915 | 1,954 | 5,830 | 100.0% |
| Source | Representative positive behavior | Package presentation | Primary difficulty |
|---|---|---|---|
| MalSkillBench | Downloaders, substituted dependencies, protected archives, remote scripts, and implanted transaction utilities | A source-matched benign skill is paired with a malicious package derived from the same content family | Near-matched surfaces leave behavior-changing paths, prerequisites, or instructions as the discriminative evidence |
| SkillFortify | Prompt injection, unsafe tool delegation, exfiltration, and privilege misuse | Harmful requirements are embedded in otherwise functional SKILL.md packages | Semantic dilution and disagreement between stated purpose and requested operation |
| Agent-skill malware | Social-engineering instructions for fake prerequisite downloads, encoded shell commands, or credential exfiltration | Single SKILL.md documents; all 58 retained samples contain one file | Inferring requested external operations and their authorization from instructions |
| ATR Skill Benchmark | Exfiltration, reverse shells, droppers, typosquatting, and delayed triggers | The source taxonomy emphasizes relatively short and explicit attack artifacts | Only 22 matched malicious families remain in Large-ID, so uncertainty is dominated by family support |
| SkillsBench-1650 | Encoded or steganographic payloads, modular/RSA transforms, remote scripts, and binary execution | Related operations are expressed through different carriers and encodings | Representation opacity: pre-action states may not resolve latent payload semantics |
| Method | Input | Fit | Generation | Execution | Scope |
|---|---|---|---|---|---|
| Static regex | Text | No | No | CPU scan | Native baseline |
| SkillSieve L1 | Text + parser | No | No | CPU scanner | Official Layer 1 |
| Cisco scanner | Package + parser | No | No | CPU scanner | Default core policy |
| LLM-as-judge | Rendered package | No | No | Judge forward | Frozen Mistral |
| Supervised text | Qwen verbalizer scores | Train+development | No | Two model forwards | L2 logistic/0.5 |
| RouteGuard | Attention + hidden | Yes | No | Four probe prefills | Paper-spec reimplementation; fixed 0.5 |
| AgentLens (1P) | RouteGuard (4P) | LatentTriad (1P) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Backbone | TPR | FPR | TPR | TPR | FPR | TPR | TPR | FPR | TPR |
| Llama-3.1-8B | 84.1 | 17.5 | 83.9 | 94.2 | 8.3 | 94.1 | 93.0 | 4.3 | 92.9 |
| Qwen2.5-7B | 59.4 | 38.8 | 59.4 | 92.5 | 12.9 | 92.5 | 95.9 | 4.3 | 95.9 |
| Qwen2.5-14B | 71.2 | 29.4 | 71.0 | 91.8 | 11.2 | 91.7 | 96.4 | 4.1 | 96.3 |
| Qwen3-30B-A3B | 81.5 | 19.2 | 81.4 | 92.4 | 10.1 | 92.2 | 96.8 | 3.9 | 96.8 |
| Llama-3.3-70B-INT8 | 85.2 | 16.1 | 85.0 | 93.1 | 9.4 | 93.0 | 97.2 | 3.8 | 97.1 |
| Fixed | Nested development | ||||||
|---|---|---|---|---|---|---|---|
| Backbone | Mode | FPR | Recall | F1 | FPR | Recall | F1 |
| Llama-3.1-8B | Explicit | 19.8 | 92.4 | 87.1 | 8.7 | 86.3 | 88.5 |
| Implicit | 0.0 | 99.7 | 99.9 | 0.0 | 99.7 | 99.9 | |
| Qwen2.5-7B | Explicit | 13.7 | 95.3 | 91.2 | 7.6 | 92.7 | 92.6 |
| Implicit | 0.9 | 99.4 | 99.3 | 1.2 | 99.1 | 99.0 | |
| (a) Whole-path representation detectors complete task path | |||||||
| Method | Backbone | Explicit | Implicit | ||||
| FPR | Recall | F1 | FPR | Recall | F1 | ||
| AgentLens (1P) | Llama-3.1-8B | 15.7 | 69.2 | 74.8 | 19.4 | 98.8 | 90.6 |
| RouteGuard (4P) | Llama-3.1-8B | 16.3 | 90.1 | 87.3 | 0.3 | 98.3 | 99.0 |
| LatentTriad (1P) | Llama-3.1-8B | 8.7 | 86.3 | 88.5 | 0.0 | 99.7 | 99.9 |
| AgentLens (1P) | Qwen2.5-7B | 37.2 | 40.4 | 45.5 | 40.5 | 78.3 | 71.6 |
| Method | Build | Explicit | Implicit | ||||
|---|---|---|---|---|---|---|---|
| FPR | Recall | F1 | FPR | Recall | F1 | ||
| SkillsGuard | Pinned | 96.5 | 96.5 | 65.9 | 96.0 | 96.0 | 65.7 |
| Cisco independent | Pinned | 13.7 | 13.7 | 21.5 | 13.9 | 13.9 | 21.7 |
| SkillSpector | Pinned | 61.9 | 61.9 | 55.3 | 62.4 | 62.4 | 55.5 |
| (a) Frozen transfer to SCR-Bench 667 paired cases; no SCR label used for fitting | |||||
| Backbone | Capability flow | Trust transfer | Authorization | Holm significant | AgentLens control |
| Llama-3.1-8B | +0.145 ∗ | +0.002 ∗ | -0.027 | 2/3 | Opposite on flow |
| Qwen2.5-7B | -0.010 | +0.008 ∗ | +0.229 ∗ | 2/3 | Saturated at 1.0 |
| Source | Regex | SkillSieve | Cisco | Mistral judge | Sup. text | AgentLens | RouteGuard | LatentTriad |
|---|---|---|---|---|---|---|---|---|
| MalSkillBench | 66.7 | 70.3 | 43.2 | 32.2 | 73.3 | 71.2 [28.6, 98.6] | 85.0 [84.2, 86.3] | 99.4 [99.3, 99.6] |
| SkillFortify | 66.7 | 68.4 | 8.5 | 56.0 | 92.4 | 73.0 [35.7, 99.4] | 98.0 [97.2, 98.6] | 100.0 [100.0, 100.0] |
| SkillsBench-1650 | 66.7 | 61.8 | 42.9 | 6.0 | 35.4 | 54.0 [27.6, 67.9] | 55.4 [47.2, 60.5] | 90.9 [88.6, 93.9] |
| Agent-skill malware | 66.7 | 48.8 | 6.7 | 6.7 | 12.9 | 53.0 [25.6, 73.5] | 71.1 [56.5, 78.6] | 89.2 [83.6, 91.5] |
| ATR Skill Benchmark | 66.7 | 52.9 | 37.0 | 51.6 | 77.8 | 60.4 [26.7, 81.6] | 86.4 [82.9, 90.9] | 90.0 [85.7, 93.6] |
| Source | Benign | AgentLens | RouteGuard | LatentTriad |
|---|---|---|---|---|
| MalSkillBench | 2,621 | 0.84–40.02 | 16.06–22.51 | 0.53–0.76 |
| SkillFortify | 180 | 0.00–45.56 | 0.00–1.67 | 0.00–0.00 |
| SkillsBench-1650 | 63 | 19.05–82.54 | 23.81–34.92 | 11.11–23.81 |
| Agent-skill malware | 29 | 17.24–68.97 | 10.34–20.69 | 3.45–13.79 |
| ATR Skill Benchmark | 22 | 18.18–68.18 | 9.09–18.18 | 13.64–27.27 |
| Backbone | vs. RouteGuard | vs. AgentLens |
|---|---|---|
| Qwen2.5-7B | 13.5 [9.5, 18.1] | 50.0 [45.1, 58.2] |
| Llama-3.1-8B | 17.0 [13.1, 21.6] | 14.9 [12.0, 18.1] |
| Mistral-7B | 13.6 [9.6, 18.2] | 9.8 [6.3, 13.3] |
| InternLM2.5-7B | 15.6 [11.0, 20.8] | 18.4 [14.7, 22.3] |
| Qwen2.5-14B | 14.1 [10.8, 17.8] | 65.5 [59.0, 77.4] |
| Phi-3-medium-14B | 16.8 [12.4, 22.1] | 48.7 [42.7, 58.6] |
| Configuration | AUROC | AP | F1 | FPR | Recall |
|---|---|---|---|---|---|
| Full: | 99.02 | 99.18 | 93.91 | 9.51 | 96.73 |
| Boundary-only | 97.28 | 97.94 | 91.63 | 9.51 | 92.58 |
| Operation-only | 99.03 | 99.18 | 86.69 | 0.00 | 80.04 |
| Geometry-only | 99.68 | 99.78 | 95.85 | 0.00 | 92.42 |
| 99.15 | 99.29 | 94.08 | 9.51 | 97.05 | |
| 99.53 | 99.64 | 83.56 | 0.00 | 75.98 |
| Contrast | AUROC | AP | F1 |
|---|---|---|---|
| Full independent-state control | +0.076 [+0.027, +0.136] | +0.063 [+0.023, +0.112] | +0.026 [+0.001, +0.075] |
| Full single-member mean | +0.032 [-0.061, +0.144] | +0.029 [-0.041, +0.112] | +0.340 [-0.275, +0.917] |
| Full Boundary-only | +1.744 [+1.229, +2.328] | +1.240 [+0.874, +1.643] | +2.286 [+1.595, +3.131] |
| Full | -0.125 [-0.243, -0.027] | -0.106 [-0.194, -0.030] | -0.168 [-0.350, -0.036] |
| Backbone | Configuration | AUROC | AP | F1 | FPR | Recall |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | Full | 99.11 | 99.26 | 94.37 | 8.54 | 96.78 |
| Boundary-only | 97.63 | 98.19 | 92.52 | 8.54 | 93.37 | |
| Independent-state control | 99.11 | 99.26 | 94.37 | 8.54 | 96.78 | |
| Single-member mean | 98.95 | 99.12 | 93.73 | 8.88 | 96.00 | |
| Qwen2.5-7B | Full | 99.15 | 99.30 | 94.31 | 10.39 | 98.20 |
| Boundary-only | 97.01 | 97.79 | 91.95 | 10.39 | 93.90 |
| Method | Backbone | Recall | F1 [95% CI] | AUROC | AP |
|---|---|---|---|---|---|
| Regex | Shared | 100.0 | 66.7 [66.7, 66.7] | 68.6 | 68.6 |
| SkillSieve L1 | Shared | 53.0 | 60.5 [54.3, 65.9] | 69.0 | 69.3 |
| Cisco scanner | Shared | 17.5 | 27.6 [21.2, 33.1] | 63.4 | 64.7 |
| Mistral judge | Mistral-7B | 20.3 | 30.5 [24.7, 35.8] | 81.0 | 81.1 |
| Supervised Qwen text | Qwen2.5-7B | 49.6 | 58.7 [52.8, 64.0] | 88.1 | 90.0 |
| AgentLens | Llama-3.1-8B | 48.2 | 35.8 [34.1, 37.2] | 50.3 | 55.1 |
| Backbone | vs. RouteGuard | vs. AgentLens |
|---|---|---|
| Llama-3.1-8B | +16.1 [+10.0, +22.3] | +36.5 [+31.0, +40.9] |
| Qwen2.5-7B | +13.0 [+8.4, +18.5] | +45.6 [+42.9, +48.2] |
| Mistral-7B | +14.7 [+8.7, +20.8] | +10.6 [+4.7, +16.0] |
| InternLM2.5-7B | +15.7 [+9.7, +22.2] | +23.5 [+18.7, +28.1] |
| Qwen2.5-14B | +12.2 [+6.3, +18.3] | +35.3 [+31.8, +38.7] |
| Phi-3-medium-14B | +16.5 [+10.3, +22.7] | +45.6 [+41.4, +49.3] |
| Backbone | Rewrite | Clean-TP counts | Upper bound |
|---|---|---|---|
| Llama-3.1-8B | Padding | 23, 21, 70, 103, 31 | 12.48% |
| Qwen2.5-7B | Padding, Geometry | 23, 22, 70, 103, 31 | 12.32% |
| Mistral-7B | Padding, Geometry | 27, 21, 70, 103, 32 | 11.90% |
| InternLM2.5-7B | Padding, Geometry | 26, 21, 70, 103, 30 | 12.18% |
| Qwen2.5-14B | Padding | 22, 22, 69, 103, 31 | 12.48% |
| Backbone | Attack | TPR ret. | ASR [95% CI] | Benign FPR | [95% CI] | Pass | ||
|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | Region padding | 105.3 | 0.0 [0.0, 0.0] | 5.8 [5.1, 6.5] | 0.0021 | 0.0021 | yes | |
| Cross-file split | 92.8 | 11.0 [6.5, 16.3] | -0.6 [-1.3, 0.2] | 0.9179 | 0.0021 | yes | ||
| Geometry evasion | 101.6 | 3.1 [0.6, 6.1] | 3.7 [2.9, 4.5] | 0.0021 | 0.0021 | yes | ||
| Qwen2.5-7B | Region padding | 105.0 | 0.0 [0.0, 0.0] | 6.9 [6.2, 7.6] | 0.0021 | 0.7895 | yes | |
| Cross-file split | 92.3 | 9.0 [4.2, 14.0] | -3.7 [-4.3, -3.1] | 0.0021 | 0.0021 | yes | ||
| Geometry evasion | 105.0 | 0.0 [0.0, 0.0] | 7.5 [6.7, 8.4] | 0.0021 | 0.0021 | yes |
| Budget | Random | AgentLens | RouteGuard | LatentTriad | Oracle |
|---|---|---|---|---|---|
| 10% | 41.9/42.2 | 42.8/41.5 | 36.5/46.9 | 36.7/47.1 | 31.7/47.1 |
| 20% | 37.3/37.4 | 36.1/39.5 | 27.3/46.4 | 26.9/47.0 | 20.3/47.1 |
| 30% | 32.9/32.6 | 28.2/38.3 | 19.6/45.8 | 17.2/46.7 | 11.2/47.1 |
| Acting backbone | Budget | Valid | AgentLens | RouteGuard | LatentTriad |
|---|---|---|---|---|---|
| Qwen2.5-7B | 10% | 86.3 | 71.1/9.0 | 56.2/15.2 | 60.9/15.2 |
| 20% | 64.1/7.0 | 43.0/14.8 | 45.7/15.2 | ||
| 30% | 52.3/7.0 | 31.6/14.5 | 32.8/15.2 | ||
| Llama-3.1-8B | 10% | 99.0 | 61.7/21.1 | 63.7/21.1 | 61.3/21.1 |
| 20% | 46.9/20.7 | 48.8/21.1 | 45.7/21.1 | ||
| 30% | 35.5/19.5 | 32.8/21.1 | 28.9/21.1 |
| Backbone | Nat. Acc. | Nat. Macro-F1 [95% CI] | D/C AUROC | Ctrl. Macro-F1 | Blocked | Cond. exact | Joint |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | 85.7 | 86.2 [83.2, 88.8] | 0.998/0.992 | 90.9 | 375/400 | 92.0 | 85.7 |
| Qwen2.5-7B | 82.1 | 82.4 [79.6, 85.3] | 0.998/0.984 | 89.2 | 370/400 | 89.5 | 82.1 |
| Mistral-7B-v0.3 | 82.1 | 82.7 [79.7, 85.5] | 0.998/0.985 | 86.2 | 369/400 | 88.6 | 82.1 |
| InternLM2.5-7B | 79.7 | 80.7 [77.7, 83.5] | 0.996/0.981 | 86.2 | 363/400 | 89.5 | 79.7 |
| Qwen2.5-14B | 87.7 | 88.1 [85.5, 90.6] | 0.999/0.994 | 90.4 | 373/400 | 93.6 | 87.7 |
| Phi-3-medium-14B | 83.1 | 84.0 [81.0, 86.9] | 0.997/0.988 | 90.9 | 373/400 | 89.0 | 83.1 |
| Backbone | Med. ms | P95 ms | Prefill % | VRAM MB | Fwds. |
|---|---|---|---|---|---|
| Llama-3.1-8B | 1.8 | 3.1 | 1.26 | 14.2 | 1 |
| Qwen2.5-7B | 1.6 | 2.8 | 1.24 | 12.8 | 1 |
| Mistral-7B-v0.3 | 1.7 | 2.9 | 1.30 | 13.5 | 1 |
| InternLM2.5-7B | 1.7 | 3.0 | 1.26 | 13.1 | 1 |
| Qwen2.5-14B | 2.4 | 4.1 | 1.07 | 18.6 | 1 |
| Phi-3-medium-14B | 2.3 | 3.9 | 1.05 | 17.9 | 1 |
| Backbone | Natural PairAcc (%) | Boundary recovery | Boundary margin | Operation margin | Geometry margin | Fused margin |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | ||||||
| Qwen2.5-7B | ||||||
| Mistral-7B | ||||||
| InternLM2.5-7B | ||||||
| Qwen2.5-14B | ||||||
| Phi-3-medium-14B |
| Backbone | B active (%) | O active (%) | G active (%) | Eligible pairs | Verdict pp | |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | 33.0 | 51.0 | 16.0 | 54/100 | 53.4 [42.4, 64.3] | |
| Qwen2.5-7B | 22.0 | 48.0 | 30.0 | 58/100 | 39.9 [25.9, 53.1] | |
| Mistral-7B | 29.0 | 53.0 | 18.0 | 54/100 | 46.0 [34.5, 57.5] | |
| InternLM2.5-7B | 40.0 | 23.0 | 37.0 | 61/100 | 66.1 [55.1, 77.2] | |
| Qwen2.5-14B | 26.0 | 36.0 | 38.0 | 51/100 | 34.5 [29.3, 39.6] | |
| Phi-3-medium-14B | 30.0 | 38.0 | 32.0 | 60/100 | 47.5 [33.6, 62.9] |
| Study | Atoms | Readout | Evidence scope |
|---|---|---|---|
| Package detection | 8/256 | Concatenated logistic | Content-component-disjoint folds |
| White-box / broker | 8/256 | Fixed application probes | Matched-pair-disjoint folds; rewrites and request policies |
| Multi-depth intervention | 8/24 | Fixed application probes | Same fitted heads; compact extraction for intervention |
| Compact runtime | 8/24 | Member CDF scores | Llama-3.1-8B and Qwen2.5-7B timing |
| Backbone | Stage | Mean ms | Median ms | P95 ms | Std. ms |
|---|---|---|---|---|---|
| Llama-3.1-8B | Native prefill | 334.1 | 219.4 | 905.2 | 365.8 |
| LatentTriad incremental | 64.6 | 64.7 | 71.1 | 9.1 | |
| Prefill + detector | 398.7 | 283.9 | 1028.4 | 367.9 | |
| Qwen2.5-7B | Native prefill | 310.8 | 202.7 | 838.5 | 333.8 |
| LatentTriad incremental | 56.4 | 56.1 | 62.0 | 9.6 | |
| Prefill + detector | 367.2 | 258.7 | 896.5 | 334.5 |
| Model | Test form | B32 | O32 | D32 | Ind.192 | Rel.192 | TF-IDF | Length |
|---|---|---|---|---|---|---|---|---|
| Llama | All | 99.75 | 99.98 | 50.00 | 99.28 | 99.77 | 50.23 | 50.00 |
| Llama | NL | 90.09 | 90.92 | 50.00 | 89.26 | 59.98 | 50.00 | 50.00 |
| Llama | Python | 97.66 | 98.36 | 50.00 | 92.12 | 89.09 | 50.02 | 50.00 |
| Llama | JSON | 97.44 | 97.63 | 50.00 | 87.83 | 67.54 | 50.00 | 50.00 |
| Llama | Mixed | 94.33 | 98.81 | 50.00 | 89.33 | 84.02 | 50.70 | 50.00 |
| Qwen | All | 100.00 | 99.86 | 50.00 | 99.89 | 99.93 | 50.23 | 50.00 |
| Condition | Llama | Qwen | Mistral |
|---|---|---|---|
| Matched O | 2.376 [1.812, 2.988] | 2.622 [2.038, 3.353] | 2.909 [1.849, 4.110] |
| Matched O, B restored | 2.320 [1.768, 2.924] | 2.251 [1.640, 3.001] | 2.646 [1.701, 3.685] |
| Norm-matched random | 0.011 [0.007, 0.015] | 0.002 [-0.006, 0.008] | -0.004 [-0.011, 0.002] |
| Mismatched O | 0.971 [0.708, 1.251] | 0.957 [0.801, 1.146] | 1.097 [0.722, 1.540] |
| Unrelated D | 0.115 [-0.011, 0.260] | 0.158 [0.060, 0.243] | -0.019 [-0.053, 0.021] |
| Unpatched | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] |
| Readout | Llama | Qwen | Mistral |
|---|---|---|---|
| Full-path B32 | 50.83 | 54.69 | 52.77 |
| Task only | 50.00 | 50.00 | 50.00 |
| Workspace only | 50.00 | 50.00 | 50.00 |
| Task + workspace | 50.00 | 50.00 | 50.00 |
| Skills only | 50.00 | 50.00 | 50.00 |
| Node 1 | 50.00 | 50.00 | 50.00 |
| Model | Templates | Edit | Package | B | LT | Ind.L | Ind.S | Orth.S | Quad. |
|---|---|---|---|---|---|---|---|---|---|
| Llama | Known | 0 | Single | 51.01 | 52.31 | 53.91 | 54.02 | 53.47 | 52.93 |
| Llama | Known | 0 | Composed | 51.60 | 53.47 | 54.46 | 54.41 | 54.38 | 54.18 |
| Llama | Known | 1 | Single | 49.87 | 51.38 | 51.45 | 51.24 | 51.22 | 51.20 |
| Llama | Known | 1 | Composed | 49.98 | 50.05 | 50.93 | 50.90 | 50.88 | 51.03 |
| Llama | Held out | 0 | Single | 49.76 | 49.25 | 50.08 | 49.85 | 49.72 | 49.45 |
| Llama | Held out | 0 | Composed | 50.34 | 51.06 | 50.85 | 50.86 | 50.62 | 50.15 |
| Model | Object | Target | B32 | B128 | Bfull | Random | Text | Options |
|---|---|---|---|---|---|---|---|---|
| Llama | Inferred | actual | 55.47 | 60.16 | 61.46 | 47.92 | 50.00 | 50.00 |
| Llama | Inferred | risk | 49.83 | 51.82 | 53.39 | 50.69 | 50.00 | 50.00 |
| Llama | Supplied | actual | 67.19 | 78.12 | 75.52 | 47.31 | 50.00 | 50.00 |
| Llama | Supplied | risk | 59.72 | 60.94 | 59.90 | 49.74 | 50.00 | 50.00 |
| Qwen | Inferred | actual | 52.00 | 52.78 | 51.30 | 49.39 | 50.00 | 50.00 |
| Qwen | Inferred | risk | 51.65 | 54.51 | 53.12 | 48.96 | 50.00 | 50.00 |
| Model | Object | Target | 96 tokens | 256 tokens | 512 tokens |
|---|---|---|---|---|---|
| Llama | Inferred | actual | 55.47 (166) | 76.30 (42) | 77.60 (32) |
| Llama | Inferred | risk | 61.72 (77) | 76.56 (3) | 76.56 (3) |
| Llama | Supplied | actual | 59.90 (139) | 80.21 (15) | 80.21 (13) |
| Llama | Supplied | risk | 69.27 (87) | 84.64 (8) | 84.90 (4) |
| Qwen | Inferred | actual | 52.34 (176) | 93.23 (4) | 94.01 (1) |
| Qwen | Inferred | risk | 23.18 (286) | 92.71 (10) | 94.53 (2) |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 351 papers