cs.CLApr 10, 2026
SaveTask-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarios
Organizations: City University of Hong Kong · University of Hong Kong · Nanyang Technological University
Abstract
Large language models (LLMs) exhibit substantial variability in performance and computational cost across tasks and queries, motivating routing systems that select models to meet user-specific cost-performance trade-offs. However, existing routers generalize poorly in cold-start scenarios where in-domain training data is unavailable. We address this limitation with a multi-level task-profile-guided data synthesis framework that constructs a hierarchical task taxonomy and produces diverse question-answer pairs to approximate the test-time query distribution. Building on this, we introduce TRouter, a task-type-aware router approach that models query-conditioned cost and performance via latent task-type variables, with prior regularization derived from the synthesized task taxonomy. This design enhances TRouter's routing utility under both cold-start and in-domain settings. Across multiple benchmarks, we show that our synthesis framework alleviates cold-start issues and that TRouter delivers effective LLM routing.
Figures & tables
Figure 1: Comparison between the traditional data preparation pipeline and our proposed LLM-based data synthesis approach for the LLM router training.
Figure 2: (a) Overview of our proposed task-profile-guided data synthesis framework. The task type generator and quality evaluator collaboratively construct a hierarchical task taxonomy, while the question-answer pair generator produces diverse QA pairs based on task profiles from difficulty-level task types. (b) Overview of our proposed task-type–aware LLM routing method, introducing a latent task-type variable into cost and performance prediction.
Figure 3: Effect of using task types of different taxonomy level to construct task-profile on sampling efficiency on Mathematics and Creative Writing domains.
Figure 4: Effect of shot number in the cold-start setting for TRouter and MetricRouter.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The normalized cost and performance distributions for six Qwen-series models on Alpaca and Multi-News datasets, evaluated using traditional metrics.
Figure 6: Ablation on learning rate for TRouter in the cold-start setting. For each learning rate setting, results are averaged over five independent runs.
Figure 7: Visualization of cost and performance distributions for six Qwen-series models across representative subcategories of varying difficulty in the Commonsense Knowledge domain, derived from the synthesized dataset .
Figure 8: Visualization of cost and performance distributions for six Qwen-series models across representative subcategories of varying difficulty in the Creative Writing domain, derived from the synthesized dataset .
Figure 9: Visualization of cost and performance distributions for six Qwen-series models across representative subcategories of varying difficulty in the Mathematics domain, derived from the synthesized dataset .
| Example 1 : query: Create a list of 4 tips to become a better public speaker. True Task Type: ’domain’: ’Dialogue & Communication’, ’subcategory’: ’Interactive Tutoring & Socratic Dialogue’, ’difficulty’: ’Medium’ Definition: Providing brief, step-based guidance or explanation for standard single-step problems with clear solutions. Top-5 Predictions: 1. Easy (Scientific Reasoning - Scientific Explanation) Possibility: 36.72% Definition: Explaining well-known, fundamental scientific concepts or phenomena using basic language and without requiring prior specialized knowledge. 2. Easy (Reading Comprehension - Fact & Detail Retrieval) Possibility: 19.85% Definition: Identify and extract a single, explicitly stated fact or detail from a sentence, with a direct, unambiguous answer (typically a word or short phrase). 3. Easy (Information Retrieval - Fact Lookup) Possibility: 10.54% Definition: Retrieval of a single, widely-known fact stated explicitly in authoritative sources. 4. Easy (Information Retrieval - Definition/Explanation Retrieval) Possibility: 3.71% Definition: Retrieving straightforward definitions or explanations of widely recognized terms or concepts with minimal ambiguity. 5. Moderate (Information Retrieval - Fact Lookup) Possibility: 3.58% Definition: Retrieval of a less common fact or fact requiring disambiguation (e.g., multiple entities, timeframes, or similar-sounding terms). Example 2 : query: Choose the correct pairing for the given words.Drawing, music. True Task Type: ’domain’: ’Scientific Reasoning’, ’subcategory’: ’Concept Comparison’, ’difficulty’: ’Moderate’ Definition: Comparison of related or similar scientific concepts that may have subtle distinctions or overlapping features, requiring more detailed analysis. Top-5 Predictions: 1. Easy (Scientific Reasoning - Scientific Explanation) Possibility: 44.46% Definition:Retrieval of a single, widely-known fact stated explicitly in authoritative sources. 2. Easy (Reading Comprehension - Fact & Detail Retrieval) Possibility: 26.48% Definition: Listing well-known, static items from a single, unambiguous category with no need for filtering or reasoning. 3. Easy (Scientific Reasoning - Scientific Explanation) Possibility: 14.69% Definition: Explaining well-known, fundamental scientific concepts or phenomena using basic language and without requiring prior specialized knowledge. 4. Medium (Information Retrieval - List Generation) Possibility: 4.08% Definition: Listing items from a category with simple, explicit criteria or filters, requiring only basic fact retrieval and minimal reasoning. 5. Moderate (Information Retrieval - Fact Lookup) Possibility: 2.75% Definition: Assessing the physical possibility of straightforward, everyday actions or events that rely on well-known physical laws and typical human abilities. |
Table 13: Failure Cases of the task recognition module of TRouter.
| Prompt of LLM-as-Judge You are an expert evaluator. Your task is to score the quality and correctness of a model-generated answer to a given question, using a reference answer as the gold standard. You will be given: - QUESTION - REFERENCE ANSWER (correct) - MODEL ANSWER (to evaluate) Your goal is to assign a score between 0.0 and 1.0, where: - 1.0 = Fully correct and semantically equivalent to the reference. - 0.5 = Partially correct or incomplete - 0.0 = Completely incorrect, irrelevant, or nonsensical Respond with only the numeric score, nothing else. --- QUESTION: {question} REFERENCE ANSWER: {reference_answer} MODEL ANSWER: {response}. Your score: |
Table 14: Prompt of LLM-as-Judge. The {question}, {reference_answer}, and {response} will be instantiated by the question, answer, and response from candidate LLMs.
| TaskTypeGenSystemPrompt = """ You are a knowledge graph construction expert. Your task is to help build a hierarchical classification system for an intelligent model router called AgenticiRouter. AgenticiRouter selects the most appropriate Large Language Model (LLM) based on a user’s query and preferences (e.g., performance, cost-efficiency, latency). To support this, we are building a multi-level taxonomy of query types. Each query is classified into a three-level node path: Domain — broad task area Subcategory — specific task type Difficulty Level — complexity level within that task type """ |
Table 15: System Prompt of Task type Generation.
| DomainNodeRule = """ - Domains must be general yet semantically distinct. - Avoid overlapping or ambiguous categories. - Think of areas commonly found in LLM benchmarks (e.g., MMLU, AGIEval, CMMLU) and real-world applications. - These will serve as the top-level nodes of the taxonomy. - Only up to {max_new_domain_nodes} new domains can be proposed. """ SubcategoryNodeRule = """ - Avoid overlap between subcategories. - Each subcategory should represent a common type of user query that can be grouped under this domain. - Subcategories should be general enough to cover many user queries but specific enough to guide model selection. - Only up to {max_subcategory_nodes} new Subcategory Nodes can be proposed for each domain node. """ DifficultyLevelNodeRule = """ - Levels must be ordered from easiest to hardest. - Levels should reflect increasing reasoning complexity, token usage, or LLM capability required. - Levels should be mutually exclusive — no query should belong to more than one level. - Levels should be collectively exhaustive — all queries in this subcategory must be covered. - Avoid generic differences like “longer text” unless it reflects actual difficulty in reasoning or generation. - Only up to {max_difficulty_level_nodes} Difficulty Level Nodes can be proposed for each Subcategory Node. """ |
Table 16: Rules of nodes generation at three different levels, which will be instantiated to corresponding node generation prompts. {max_new_domain_nodes}, {max_subcategory_nodes}, and {max_difficulty_level_nodes} will be set as 4, 10, 5 in our work.
| DomainNodePrompt = TaskTypeGenSystemPrompt + """ Step 1: Generate Domain Nodes Generate a list of distinct, non-overlapping Domain categories that represent broad types of user tasks. We have already defined the following six initial Domains: - Mathematics - Creative Writing - Commonsense Knowledge - Programming - Long-context Understanding - Reading Comprehension Please propose {max_new_domain_nodes} new high-level Domains that: Rules: {DomainNodeRule} Output Format: For each proposed Domain: - Name the Domain - Provide a one-sentence definition - Include a real-world example query Example: <node begin> Domain: Mathematics Definition: Covers quantitative problem-solving tasks involving numbers, equations, logic, or formal systems, including arithmetic, algebra, calculus, and more. Example: What is the derivative of sin(x)? Domain: Creative Writing Definition: Involves imaginative or artistic language generation tasks such as writing poems, stories, scripts, or creative descriptions. Example: Write a short story about a robot who learns to paint. </node end> Output: """ |
Table 17: Prompt of domain domain expansion. {max_new_domain_nodes} will be set as 4 in our work. {DomainNodeRule} will be replaced using rules as shown in the Table. 16 .
| SubcategoryNodePrompt = TaskTypeGenSystemPrompt + """ Step 2: Generate Subcategories for a Given Domain The current Domain is: {domain_name} Domain Definition: {domain_definition} Domain Example: {domain_example} Please generate a list of fine-grained Subcategories Nodes that represent specific types of tasks or problem types within this domain. Please propose up to {max_subcategory_nodes} new Subcategory Nodes for each domain node that: Rules: {SubcategoryNodeRule} Output Format: For each proposed Domain: - Name the Subcategory - Provide a one-sentence definition - Include a real-world example query Example: Assume the Domain is: Domain_name: Mathematics Domain Definition: Covers quantitative problem-solving tasks involving numbers, equations, logic, or formal systems, including arithmetic, algebra, calculus, and more. Domain Example: What is the derivative of sin(x)? Then the Subcategory Nodes are: <node begin> Subcategory: Arithmetic Definition: Covers basic arithmetic operations and problem-solving involving numbers, including addition, subtraction, multiplication, and division. Example: What is the sum of 123 and 456? Subcategory: Algebra Definition: Involves solving equations and expressions using variables and algebraic manipulation. Example: Solve for x in the equation 2x + 3 = 11. ... </node end> Output:: """ |
Table 18: Prompt of subcategory node generation. {max_subcategory_nodes} will be set as 10 in our work. {SubcategoryNodeRule} will be replaced using rules as shown in the Table. 16 . {domain_name}, {domain_definition}, and {domain_example} will be replaced by the corresponding domain information.
| DifficultyNodePrompt = TaskTypeGenSystemPrompt + """ Step 3: Define Difficulty Levels for a Given Subcategory You are now working on level 3: Difficulty Level. This level captures how task complexity varies within a specific Subcategory. Please utilize a fixed global scale (e.g., Easy / Medium / Hard for three-level difficulty) as the short name of each level. However, the Definition of each level should be customized based on the nature of the specific Subcategory. Input Subcategory Information: - Domain: {domain_name} - Subcategory: {subcategory_name} - Subcategory Definition: {subcategory_definition} - Subcategory Example Query: {subcategory_example} Please propose up to {max_difficulty_level_nodes} Difficulty Level Nodes that: Rules: {DifficultyLevelNodeRule} Output Format: For each proposed level: - Name the Level - Provide a one-sentence definition - Include a real-world example query Example(Subcategory: Code Debugging): Assume the Subcategory is Code Debugging. Definition: Identifying and fixing errors in code snippets written in common programming languages. Then the Subcategory Nodes are: <node begin> Level: Basic Definition: Single-line or syntax-only bugs in short, self-contained functions. Example: Fix the indentation error in this 5-line Python function. Level: Intermediate Definition: Logic bugs in medium-length code that require control flow analysis. Example: Fix the loop condition that causes an infinite loop in this JavaScript function. Level: Advanced Definition: Bugs across multiple functions, handling of edge cases, or requiring domain-specific knowledge. Example: Fix the bug in this Flask app that breaks when uploading empty files. Level: Expert Definition: Deep reasoning over large codebases, concurrency, or memory management. Example: Fix the deadlock issue in this multi-threaded C++ program handling file writes. </node end> Output: """ |
Table 19: Prompt of difficulty node generation. {max_difficulty_level_nodes} will be set as 5 in our work. {DifficultyLevelNodeRule} will be replaced using rules as shown in the Table. 16 . {domain_name}, {subcategory_name},{subcategory_definition} and {subcategory_example} will be replaced by the corresponding domain and subcategory information.
| NodeRevisePrompt = TaskTypeGenSystemPrompt + """ You are given a candidate {node_name} set for review. Your job is to: 1. Evaluate whether this candidate {node_name} set needs improvement by checking how well it adheres to the provided generation rules.. 2. If improvement is needed, generate a revised and higher-quality version of the {node_name} set that better satisfies the rules and supports downstream LLM routing decisions. Current Candidate {node_name} Set: {candidate_node_set} Node Generation Rules: {node_gen_rules} Output Format: <justification> Explain whether the current {node_name} set is flawed or could be improved. Mention overlap, vagueness, gaps, etc. </justification> <revision node begin> {node_name_short}: node name Definition: One-sentence definition Example: Example query or task {node_name_short}: node name Definition: One-sentence definition Example: Example query or task </revision node end> Output:""" NodeSetChoicePrompt = TaskTypeGenSystemPrompt + """ Your Task: You are given two candidate {node_name} sets, labeled Set A and Set B. Your job is to: 1. Compare both sets based on how well they follow the generation rules. 2. Select the better set — the one that provides more clarity, distinctiveness, usefulness, and alignment with routing goals. 3. Justify your choice in detail. Node Generation Rules: {node_gen_rules} Candidate Sets: Set A: {candidate_node_set_a} Set B: {candidate_node_set_b} Output Format: <justification> Explain why one set is better than the other. Reference the rules. Mention clarity, distinctiveness, coverage, usefulness, etc. </justification> <preferred set> Set A / Set B </preferred set> Output: """ |
Table 20: Prompt of difficulty task type quality evaluator. {node_name} represents the level of task type trees, including domain, subcategory, and difficulty. {node_gen_rules} will be replaced using rules corresponding to {node_name} level, as shown in the Table. 16 . {candidate_node_set_a} and {candidate_node_set_b} will be replaced by the original and revised node set.
| QAGenPrompts = """ You are a helpful assistant that generates high-quality question-answer pairs. You must respond with a specific format using markers to structure your output. IMPORTANT: Your response must follow this exact format: <qa_pairs begin> Q: What is 2+2? A: 2+2 equals 4 Q: What is the capital of France? A: The capital of France is Paris Q: How does photosynthesis work? A: Photosynthesis is the process by which plants convert sunlight into energy. </qa_pairs end> Rules: - Start with <qa_pairs begin> and end with </qa_pairs end> - Each QA pair should be separated by a blank line - Use "Q:" for questions and "A:" for answers - Ensure questions are clear, relevant, and the answers are accurate and comprehensive""" + {TaskProfile} + """Please generate {question_num} different question-answer pairs according to all the above specifications. The questions should be clear, relevant, and the answers should be comprehensive and accurate. Focus on creating diverse questions that cover different aspects of the topic.""" DomainNodeProfile = """ Task Domain: {domain_name} Domain Definition: {domain_definition}""" SubcategoryNodeProfile = """ Task Domain: {domain_name} Domain Definition: {domain_definition} Task Subcategory: {subcategory_name} Subcategory Definition: {subcategory_definition}""" DifficultyNodeProfile = """ Task Domain: {domain_name} Domain Definition: {domain_definition} Task Subcategory: {subcategory_name} Subcategory Definition: {subcategory_definition} Task Difficulty: {difficulty_name} Difficulty Definition: {difficulty_definition}""" |
Table 21: Prompt of question generator using task profile of different levels. {question_num} will be set as 8 in our work. Other variable with { and } will be replaced with the information of different task types and their parent nodes.
More in this topic
LLM Routing
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
See all 349 papers