Small Language Models for Inference-Capability Routing: Effects of Model Capacity and Prompt Optimization on Downstream Quality and Efficiency
Organizations: CORTEXA Innovations Pvt Ltd. Hyderabad, India
Abstract
Optional reasoning, retrieval, web search, and tool use can improve large language model responses but incur unnecessary computation when activated indiscriminately. This study investigates whether prompt optimization can compensate for limited router capacity and whether classification gains translate into downstream efficiency. The proposed framework uses prompt-driven small language models to make four independent capability decisions, allowing multiple capabilities to be selected per query. Qwen2.5 routers ranging from 1.5B to 14B parameters are compared across five prompting strategies, followed by an evaluation of capability-specific optimized configurations using a fixed GPT-OSS-120B downstream model on 127 questions. Under minimal prompting, increasing router capacity does not consistently improve classification performance. Prompt optimization narrows the capacity gap for reasoning and calculator routing, whereas RAG and web search retain larger benefits from higher capacity. Similar classification F1 scores do not necessarily imply equivalent downstream behavior: false-positive activations increase unnecessary computation, while missed capabilities can reduce answer quality. The 14B router achieves a mean BERTScore F1 of 0.8408, compared with 0.8418 when all capabilities are enabled, while reducing average downstream token consumption by 68.7% (8,837 to 2,764 tokens) and median downstream latency by 80.5% (9.79 to 1.91 seconds). These savings exclude router overhead, which is measured separately in a different inference environment and increases with router capacity. The findings demonstrate that the relationship between router capacity, prompt design, and downstream efficiency is capability-dependent. Consequently, router selection must consider error profiles and routing overhead alongside classification performance.
Figures & tables
| Capability | Total | Train | Validation | Test |
|---|---|---|---|---|
| Reasoning | 875 | 94 | 29 | 752 |
| RAG | 1,003 | 100 | 30 | 873 |
| Web Search | 642 | 100 | 30 | 512 |
| Calculator | 995 | 100 | 30 | 865 |
| Total | 3,515 | 394 | 119 | 3,002 |
| Capability | Overlap | Matching | Different | Agreement (%) |
|---|---|---|---|---|
| Calculator | 83 | 78 | 5 | 93.98 |
| RAG | 87 | 87 | 0 | 100.00 |
| Reasoning | 80 | 53 | 27 | 66.25 |
| Web Search | 87 | 61 | 26 | 70.11 |
| Capability | Required | Not Required |
|---|---|---|
| Reasoning | 83 | 44 |
| RAG | 97 | 30 |
| Web Search | 10 | 117 |
| Calculator | 29 | 98 |
| Size | Reasoning | RAG | Web | Calculator |
|---|---|---|---|---|
| 1.5B | Few-Shot | GEPA | GEPA | SME |
| 3B | Few-Shot | GEPA | GEPA | Minimal |
| 7B | Few-Shot | GEPA | GEPA | SME |
| 14B | Few-Shot | GEPA | GEPA | Few-Shot |
| Capability | Metric | 1.5B | 3B | 7B | 14B |
|---|---|---|---|---|---|
| Calculator | F1 | 0.644 | 0.928 | 0.917 | 0.888 |
| Balanced Accuracy | 0.644 | 0.936 | 0.924 | 0.903 | |
| RAG | F1 | 0.678 | 0.602 | 0.196 | 0.579 |
| Balanced Accuracy | 0.509 | 0.604 | 0.484 | 0.624 | |
| Reasoning | F1 | 0.745 | 0.771 | 0.809 | 0.768 |
| Balanced Accuracy | 0.500 | 0.569 | 0.663 | 0.559 |
| Capability | 1.5B | 3B | 7B | 14B |
|---|---|---|---|---|
| Reasoning | 0 | 0 | 0 | 0 |
| RAG | 0 | 0 | 0 | 0 |
| Web Search | 0 | 0 | 0 | 0 |
| Calculator | 0 | 0 | 0 | 24 |
| Capability | 1.5B | 3B | 7B | 14B |
|---|---|---|---|---|
| Calculator | 0.905 | 0.928 | 0.934 | 0.941 |
| RAG | 0.712 | 0.657 | 0.870 | 0.955 |
| Reasoning | 0.871 | 0.865 | 0.855 | 0.913 |
| Web Search | 0.651 | 0.639 | 0.834 | 0.868 |
| Method | 1.5B | 3B | 7B | 14B |
|---|---|---|---|---|
| Minimal Baseline | 0 | 0 | 0 | 24 |
| Descriptive SME | 0 | 0 | 0 | 6 |
| DSPy Baseline | 358 | 6 | 0 | 0 |
| DSPy Few-Shot | 53 | 6 | 0 | 1 |
| GEPA | 32 | 6 | 0 | 1 |
| Capability | Metric | 1.5B | 3B | 7B | 14B |
|---|---|---|---|---|---|
| Reasoning | F1 | 0.607 | 0.676 | 0.740 | 0.716 |
| BA | 0.556 | 0.698 | 0.684 | 0.755 | |
| RAG | F1 | 0.580 | 0.328 | 0.715 | 0.953 |
| BA | 0.661 | 0.598 | 0.778 | 0.908 | |
| Web Search | F1 | 0.243 | 0.143 | 0.118 | 0.194 |
| BA | 0.715 | 0.537 | 0.524 | 0.573 |
| Router | Capability | TP | TN | FP | FN | Unknown | Errors |
|---|---|---|---|---|---|---|---|
| 1.5B | Reasoning | 44 | 25 | 18 | 39 | 0 | 1 |
| RAG | 40 | 27 | 3 | 55 | 2 | 0 | |
| Web Search | 9 | 62 | 55 | 1 | 0 | 0 | |
| Calculator | 27 | 31 | 65 | 2 | 2 | 0 | |
| 3B | Reasoning | 46 | 37 | 7 | 37 | 0 | 0 |
| RAG | 19 | 30 | 0 | 78 | 0 | 0 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Configuration |
|---|---|
| Router models | qwen2.5:1.5b , qwen2.5:3b , qwen2.5:7b , and qwen2.5:14b ( Yang et al., 2024 ) , served through Ollama. |
| Ollama | Version 0.40.4; local API endpoint 127.0.0.1:11434 . |
| DSPy | Version 3.4.1 ( Khattab et al., 2024 ) . |
| Raw router inference | Temperature = 0.0; maximum generated tokens ( num_predict ) = 10. |
| DSPy router inference | Temperature = 0.0; maximum generated tokens = 64. |
| Few-Shot construction | Thirty examples were drawn from the 100-example training set to form the Few-Shot candidate pool. The DSPy configuration permitted up to 6 labeled demonstrations, up to 4 bootstrapped demonstrations, and 4 candidate programs. |
| Component | Configuration |
|---|---|
| Web Search | Tavily search with search_depth="advanced" , maximum 5 results, and include_answer=False . |
| Calculator | Structured calculator tool exposed to the downstream model through a JSON tool schema requiring a single arithmetic expression string. Expressions were evaluated with a restricted Python AST evaluator. |
| Downstream model | openai.gpt-oss-120b-1:0 through AWS Bedrock. |
| Downstream generation | Temperature = 0.0; maximum generated tokens = 2048 for the standard downstream configuration. |
| BERTScore | BERTScore ( Zhang et al., 2020 ) with roberta-large ; final scoring was executed on GPU. Additional options not explicitly retained in the experiment metadata are reported as unavailable. |
| Router hardware | Google Colab with an NVIDIA T4 GPU. Other Colab hardware characteristics were not fixed and are therefore not reported. |
LLM Routing
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
See all 383 papers