Beyond Plain MCP: Do Decision Models and Code Mode Make Tool Use Cheaper and Faster?
Organizations: Independent researcher
Abstract
Model Context Protocol (MCP) gateways manage tool catalogs for language model agents using two primary paradigms: code mode where agents write scripts against tool stubs and decision layers which are cheap routers that preselect tool subsets. Using a controlled harness across 12 servers, up to 1,958 distractor tools, and three open-weight agents including Qwen, Gemma, and Nemotron, we present a preregistered evaluation on a sealed test split to determine when each approach succeeds. Code mode substantially outperforms plain MCP, cutting token usage by 70 to 90 percent while significantly boosting utility. Declaring return-shape hints in code mode stubs is a first-order variable, swinging success by up to 39.5 points for lower-performing agents, an effect larger than adding a router. The central hypothesis that a decision layer in front of code mode beats code mode alone is not confirmed, showing minor, statistically non-significant improvements for weaker agents and a slight cost for the strongest. We publicly release our harness, a finetuned 4B tool selection model, and all evaluation traces to support robust community experimentation on local AI privacy and agent architectures.
Figures & tables
| Id | Tool choice | Call shape | Seen by the agent |
|---|---|---|---|
| A Plain MCP | agent, from all schemas | JSON calls | all schemas; every result |
| B Router + JSON | decision model (subset) | JSON calls | subset schemas; every result |
| C Code mode | agent via search() | script in sandbox | 2 meta-tools; final line |
| D Router + code | decision model (subset) | script; JSON fallback | subset stubs; final line |
| Agent | Config | Success | Tokens/task | Median latency (s) | |
|---|---|---|---|---|---|
| Qwen3.8-27B | A plain MCP | 0.857 | 26,602 | 4.9 | 0.576 |
| B router+JSON | 0.824 | 15,099 | 5.0 | 0.648 | |
| C code mode | 0.992 | 4,362 | 3.6 | 0.940 | |
| D router+code | 0.983 | 5,703 | 4.7 | 0.916 | |
| D-oracle (reference) | 1.000 | 3,083 | 3.1 | 0.962 | |
| Gemma-4-26B-A4B | A | 0.866 | 16,779 | 5.7 | 0.680 |
| Pre-registered (instances) | Resampling families | |||||
|---|---|---|---|---|---|---|
| Agent | Comparison | 95% CI | Holm | 95% CI | Holm | |
| Qwen | H1: C A | 0.0012 | 0.0012 | |||
| Qwen | H2: B A | 0.43 | 1.00 | |||
| Qwen | H3: D C | 0.43 | 1.00 | |||
| Qwen | H4: D B | 0.0012 | 0.042 | |||
| Gemma | H1 | 0.43 | 1.00 | |||
| Agent | Success, no hints | Success, hints | success [95% CI] | [95% CI] | Holm |
|---|---|---|---|---|---|
| Qwen3.8-27B | 0.975 | 0.992 | 0.20 | ||
| Gemma-4-26B-A4B | 0.664 | 0.891 | 0.0003 | ||
| Nemotron-3-Nano | 0.160 | 0.555 | 0.0003 |
| Agent | Sample | C success | D success | ( D C ) [95% CI] | Holm | |
|---|---|---|---|---|---|---|
| Qwen | seeds 3–5 | 119 | 1.000 | 0.983 | ||
| pooled | 238 | 0.996 | 0.983 | 0.15 | ||
| Gemma | seeds 3–5 | 119 | 0.866 | 0.958 | ||
| pooled | 238 | 0.878 | 0.945 | 0.21 | ||
| Nemotron | seeds 3–5 | 119 | 0.504 | 0.681 | ||
| pooled | 238 | 0.529 | 0.676 | 0.22 |
| Tools | Config | Success | Tokens/task | Latency mean / median / p90 (s) |
|---|---|---|---|---|
| 42 | A | 0.90 | 16,076 | 3.4 / 3.3 / 5.5 |
| 42 | C | 1.00 | 4,112 | 3.3 / 3.0 / 4.9 |
| 42 | D | 1.00 | 5,045 | 4.6 / 4.5 / 5.9 |
| 200 | A | 0.80 | 148,047 | 20.5 / 30.2 / 33.1 |
| 200 | C | 1.00 | 7,766 | 4.2 / 4.4 / 6.6 |
| 200 | D | 1.00 | 15,176 | 9.3 / 5.5 / 19.1 |
| Agent | Comparison | |||||
|---|---|---|---|---|---|---|
| Qwen | H1 C A | |||||
| Qwen | H3 D C | |||||
| Gemma | H1 | |||||
| Gemma | H3 | |||||
| Nemotron | H1 |
| Agent | Config | Success | Tokens/task | |
|---|---|---|---|---|
| Qwen3.8-27B | A plain MCP | 0.832 | 30,649 | 0.525 |
| B router+JSON (LLM router) | 0.857 | 10,815 | 0.749 | |
| C code mode | 0.975 | 4,474 | 0.930 | |
| D router+code (LLM router) | 0.966 | 4,576 | 0.921 | |
| Gemma-4-26B-A4B | A | 0.831 | 15,282 | 0.678 |
| B (Qwen router) | 0.839 | 7,886 | 0.760 |
| Agent | C unhinted | C hinted | success | A (reference) |
|---|---|---|---|---|
| Qwen3.8-27B | 0.950 | 0.975 | pts | 0.832 |
| Gemma-4-26B-A4B | 0.664 | 0.864 | pts | 0.831 |
| Nemotron-3-Nano | 0.185 | 0.504 | pts | 0.412 |
| Qwen3.8-27B | Gemma-4-26B | Nemotron-3-Nano | |
|---|---|---|---|
| succ / med s / p90 s | succ / med / p90 | succ / med / p90 | |
| A plain MCP | 0.82 / 5.0 / 15.6 | 0.82 / 5.8 / 21.0 | 0.38 / 6.4 / 22.9 |
| B router + JSON | 0.85 / 5.7 / 12.7 | 0.87 / 7.9 / 18.9 | 0.59 / 6.2 / 22.3 |
| C code mode | 0.97 / 3.7 / 5.9 | 0.79 / 6.6 / 33.9 | 0.56 / 4.5 / 29.3 |
| D router + code | 0.95 / 4.6 / 6.8 | 0.90 / 7.4 / 21.4 | 0.64 / 5.5 / 35.1 |
| D-oracle (free router) | 0.97 / 3.0 / 4.8 | 0.90 / 7.5 / 19.3 | 0.64 / 3.6 / 21.9 |
| Tools | Config | Success | Tokens/task | Latency mean / median / p90 (s) |
|---|---|---|---|---|
| 42 | A | 0.90 | 87,460 | 9.8 / 4.8 / 60.6 |
| 42 | C | 1.00 | 4,677 | 3.6 / 3.6 / 5.9 |
| 200 | A | 0.90 | 163,969 | 22.1 / 31.3 / 39.2 |
| 200 | C | 0.90 | 10,390 | 5.6 / 4.9 / 10.0 |
| 200 | D unbounded fallback | 0.80 | 66,008 | 24.4 / 6.4 / 98.9 |
| 200 | D bounded fallback | 0.90 | 14,961 | 9.1 / 6.1 / 22.1 |