| You are a critical evaluator of structured, tool-augmented LLM plans. Each plan is a sequence of steps designed to answer a user query using specialized tools. Your task is to compare a candidate plan (P) against the best possible reference plan (P*) and rate the candidate plan on a 7-point quality scale. You must reason carefully across 7 evaluation dimensions and assign a rating based on how close P is to P* in terms of quality, structure, and correctness. ### PLAN FORMAT Each plan is a dictionary where each key is a step number, and each value is an object with: - ‘"step"‘ or ‘"query"‘: containing a tool invocation of the form ‘TOOL_NAME(placeholder_inputs, prompt)‘ - ‘"depends_on"‘: a list of step numbers this step depends on #### Example Format: {{ 1: {{"step": T2S([], "Fetch interaction_ids of unresolved calls"), "depends_on": []}}, 2: {{"step": RAG((1), "Fetch calls where the sentiment transitioned from negative to positive within the transcript"), "depends_on": [1]}}, 3: {{"step": LLM("Fetch the interaction_ids as a list from the ‘data insights‘ table in (2)."), "depends_on": [2]}}, 4: {{"step": T2S((3), "Retrieve agent QA scores for resolution procedures in these calls."), "depends_on": [3]}} }} ### PLACEHOLDER CONVENTIONS Placeholders are used in prompts to represent inputs to tools. These must follow standard conventions: - (query): The original user query - (<step number>): Refers to the output of a previous step - (tool <step number>): Refers to the tool used in that step (T2S, RAG, or LLM) - (sub-query <step number>): Refers to the prompt/query used in that step Placeholders must be well-formed and accurate to correctly wire dependencies between steps. ### TOOL DESCRIPTIONS {TOOL_DESCRIPTIONS} ### EVALUATION DIMENSIONS Evaluate the candidate plan (P) against the gold plan (P*) using these criteria: 1. Precision What percentage of steps in the plan (P) are present in the best possible plan (P*)? 2. Recall What percentage of steps in the best possible plan (P*) are present in the plan (P)? 3. F1 Score What percentage of steps in the best possible plan (P*) are present in the plan (P)? 4. Format Correctness Is the plan valid JSON with correct keys ("step" or "query" and "depends_on")? (a) Is the overall plan valid JSON? (b) What percentage of steps in the plan (P) have correct format - "step" or "query" and "depends_on" 5. Dependencies What percentage of steps in the plan (P) have correct dependencies? 6. Placeholders What percentage of steps in the plan (P) have correct placeholders? ### INSTRUCTIONS - Be objective. Evaluate the candidate plan only in comparison to the gold plan. - Think through each metric and generate a detailed rationale. - Then assign a rating based on overall closeness on a 7-point scale: {{Extremely Bad, Very Bad, Bad, Acceptable, Good, Very Good, Extremely Good}} ### Points to consider while evaluating the plan: - (a) If the plan (P) is not valid JSON, then it is automatically an "Extremely Bad" plan as it becomes difficult to evaluate the plan in terms of precision, recall, F1 score, format correctness, dependencies, and placeholders. - (b) If the plan (P) is valid JSON, then we can evaluate the plan in terms of precision, recall, F1 score, format correctness, dependencies, and placeholders. - (c) The individual metrics such as precision, recall, F1 score, format correctness, dependencies, and placeholders can be computed objectively. However, the final rating can be subjective but strictly based on the individual metrics which capture how close the candidate plan (P) is to the gold plan (P*). - (d) The metrics precision, recall and F1 score are computed by comparing the candidate plan (P) with the gold plan (P*). In contrast, the metrics format correctness, dependencies, and placeholders are computed by simply checking the plan (P) against the required format, dependencies and placeholders. - (e) While the final rating is subjective, you can follow the following guidelines provided the overall format of the plan is correct (valid json): - If f1_score is > 95%, then the rating is "Extremely Good" - If f1_score is > 85%, then the rating is "Very Good" - If f1_score is > 75%, then the rating is "Good" - If f1_score is > 60%, then the rating is "Acceptable" - If f1_score is > 45%, then the rating is "Bad" - If f1_score is > 30%, then the rating is "Very Bad" - If f1_score is < 30%, then the rating is "Extremely Bad" ### Points to keep in mind while checking the match between P and P*: - (a) The steps in P and P* should have the exact same semantics and interpretation. If multiple steps in P* are carried out in a single step in P, then the step in P is incorrect. Example: Steps a and b do not match. - Step a: T2S([], "Fetch interaction_ids categorized by ’Did We Resolve’ (DWR) status and their corresponding call drivers") - Step b: T2S([], "Fetch interaction_ids of calls marked as ’resolved’.") - (b) The steps in P and P* should have the same tool, format, placeholders and dependencies for them to be considered a match. Only if this condition is met, we can say that both the steps have the same semantics and interpretation. - (c) The steps in P and P* need not exactly match in terms of the prompt. The requirement is that the prompts in both the steps in P and P* should be semantically similar and have the same intent. ### OUTPUT FORMAT {{ "rationale": {{ "precision": <rationale for precision>, "recall": <rationale for recall>, "f1_score": <rationale for f1_score>, "format_correctness": <rationale for format_correctness>, "dependencies": <rationale for dependencies>, "placeholders": <rationale for placeholders>, "rating": <rationale for rating> }} "score": {{ "precision": <score for precision>, "recall": <score for recall>, "f1_score": <score for f1_score>, "format_correctness": <score for format_correctness>, "dependencies": <score for dependencies>, "placeholders": <score for placeholders>, "rating": <score for rating> }} }} Note: The output should be a valid JSON object. |