ShadowBench-MCP: Measuring Shadow Invocations in MCP-Style Tool Routing
Organizations: University of Washington, School of Engineering and Technology
Abstract
LLM agents increasingly act as routers for tools, APIs, and software-supply-chain workflows, making an execution path - rather than generated text alone - a security-relevant model output. We study shadow invocation: a task-preserving request variant that causes an LLM router to select a riskier execution path, potentially skipping validation, provenance, staging, redaction, or audit controls even though the request does not ask for a malicious action. We introduce ShadowBench-MCP, a benchmark and metric suite that separates task preservation (TPSS), operational pressure (OPSS), and route-geometric drift (RGAS), and combines them in the pre-execution SODS score. In a focused policy-ablation study using a Groq-hosted qwen/qwen3-32b router, 120 routing decisions yield 96 original-versus-variant comparisons across six security-sensitive workflows. Efficiency-first routing produces a 29.2% shadow-invocation rate and 26.7% safety-tool skip rate. Policy-aware routing reduces these to 4.2% and 1.7%, respectively, corresponding to an 85.7% relative reduction in shadow invocations. Pre-execution signals distinguish shadow routes with ROC AUC up to 0.94. These results are early, single-router evidence of a policy-dependent routing-layer security effect, not a claim of universal vulnerability.
Figures & tables
| Policy | SIR | Skip | Risk | RGAS | SODS |
|---|---|---|---|---|---|
| Unprotected | 29.2% | 26.7% | .028 | .059 | .019 |
| Protected | 4.2% | 1.7% | .005 | .011 | .003 |
| Defense | Blocked | Review | Precision |
|---|---|---|---|
| Consensus replay | 57.1% | 16.7% | 100% |
| Policy-constrained | 100% | 43.8% | 66.7% |
LLM Agent Security
Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.
See all 351 papers