Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition

Varun Nathan; Shreyas Guha; Ayush Kumar

cs.CLFeb 16, 2026

Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition

Authors: Varun Nathan, Shreyas Guha, Ayush Kumar

Organizations: Observe.AI Bangalore, India

Abstract

We present a domain-grounded framework and benchmark for tool-aware plan generation in contact centers, where answering a query for business insights, our target use case, requires decomposing it into executable steps over structured tools (Text2SQL (T2S)/Snowflake) and unstructured tools (RAG/transcripts) with explicit depends_on for parallelism. Our contributions are threefold: (i) a reference-based plan evaluation framework operating in two modes - a metric-wise evaluator spanning seven dimensions (e.g., tool-prompt alignment, query adherence) and a one-shot evaluator; (ii) a data curation methodology that iteratively refines plans via an evaluator->optimizer loop to produce high-quality plan lineages (ordered plan revisions) while reducing manual effort; and (iii) a large-scale study of 14 LLMs across sizes and families for their ability to decompose queries into step-by-step, executable, and tool-assigned plans, evaluated under prompts with and without lineage. Empirically, LLMs struggle on compound queries and on plans exceeding 4 steps (typically 5-15); the best total metric score reaches 84.8% (Claude-3-7-Sonnet), while the strongest one-shot match rate at the "A+" tier (Extremely Good, Very Good) is only 49.75% (o3-mini). Plan lineage yields mixed gains overall but benefits several top models and improves step executability for many. Our results highlight persistent gaps in tool-understanding, especially in tool-prompt alignment and tool-usage completeness, and show that shorter, simpler plans are markedly easier. The framework and findings provide a reproducible path for assessing and improving agentic planning with tools for answering data-analysis queries in contact-center settings.

Figures & tables

Panel of Figure 1 — Figure 1: Iterative step-wise evaluator $\rightarrow$ plan optimizer loop producing a plan lineage. For each pass, every step $i$ of the current plan $P$ is diagnosed by the step-wise evaluator and optionally edited by the plan optimizer. Any updated plan $P^{\prime}$ is appended to the lineage database. The loop stops when a full pass yields no changes or when the maximum number of passes is reached, and the final plan $P^{\star}$ is human-verified.

LLM	With Lineage			Without Lineage
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
o3-mini	43.15	53.30	72.59	49.75	65.99	80.20
gpt-4o	45.69	62.44	81.22	41.12	57.87	82.74
gpt-4o-mini	31.98	48.73	64.47	31.98	52.28	71.57
claude-3-5-haiku	24.87	45.69	62.44	30.46	47.72	72.59
claude-sonnet-4	30.46	52.79	73.60	29.95	49.75	74.11

Table 1: Plan generation quality comparison using prompts with and without lineage , evaluated with the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green; blue indicates better performance with lineage, and magenta indicates better performance without lineage.

LLM	Overall [0-100]	Format [0-20]	Tool Prompt Align. [0-20]	Step Exec. [0-15]	Query Adhr. [0-15]	Depend. [0-10]	Redund. [0-10]	Tool Usage Compl. [0-10]
claude-3-7-sonnet	84.8	18.46	15.32	12.61	12.6	9.78	8.97	7.07
llama4-maverick-17b-instruct	82.26	17.14	14	13.09	12.69	9.35	9.41	6.59
gpt-4o	81.47	16.42	13.77	13	12.85	9.66	8.58	7.2
claude-sonnet-4	79.9	12.89	14.68	13.18	12.35	9.68	8.59	8.54
llama3-3-70b-instruct	79.77	17.04	14.52	13.54	12.1	9.55	9.36	3.66
nova-pro	79.24	15.51	14.57	14.32	12.31	9.81	9.36	3.35

Table 2: Plan generation quality using prompts with lineage , evaluated with the metric-wise evaluator on test data ( 500 queries). The Normalized Average in the last row shows the average per metric normalized by that metric’s maximum score. Highest scores per metric are highlighted in blue, and the three metrics with the lowest normalized scores are highlighted in red.

LLM	Overall [0-100]	Format [0-20]	Tool Prompt Align. [0-20]	Step Exec. [0-15]	Query Adhr. [0-15]	Depend. [0-10]	Redund. [0-10]	Tool Usage Compl. [0-10]
claude-3-7-sonnet	83.33	17.36	15.48	11.46	13.37	9.63	9.08	6.95
llama4-maverick-17b-instruct	82.5	17.35	13.59	12.67	12.35	9.77	8.84	7.93
gpt-4o	82.82	15.97	14.6	12.84	12.78	9.75	8.59	8.29
claude-sonnet-4	77.98	10.91	16.67	11.78	12.91	9.7	8.7	7.32
llama3-3-70b-instruct	81.11	15.25	14.59	13.32	12.34	9.59	9.08	6.95
nova-pro	82.7	15.92	14.27	14.11	12.69	9.78	8.86	7.07

Table 3: Plan generation quality using prompts without lineage , evaluated with the metric-wise evaluator on test data ( 500 queries). The Normalized Average in the last row shows the average per metric normalized by that metric’s maximum score. Highest scores per metric are highlighted in blue, and the three metrics with the lowest normalized scores are highlighted in red.

Tag	Pre-Loop		Post-Loop
	#	%	#	%
Extremely Bad	28	5.60	23	4.60
Very Bad	127	25.40	109	21.80
Bad	107	21.40	122	24.40
Acceptable	66	13.20	48	9.60
Good	104	20.80	84	16.80

Table 4: Effectiveness of the iterative Evaluator

\rightarrow

Optimizer loop , measured using the one-shot evaluator on test data. Higher values between Pre-Loop and Post-Loop in the Good, Very Good, Extremely Good buckets are highlighted in green.

LLM	Objective Queries			Subjective Queries
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
gpt-4o	50.53	62.11	74.74	41.18	62.75	87.25
o3-mini	47.37	55.79	72.63	39.22	50.98	72.55
claude-3-7-sonnet	33.68	46.32	62.11	27.45	50.00	78.43
claude-sonnet-4	32.63	51.58	68.42	28.43	53.92	78.43
gpt-4o-mini	31.58	51.58	62.11	32.35	46.08	66.67

Table 5: Plan Generation quality comparison using prompts with lineage for objective and subjective queries, judged using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for both query categories; blue indicates better performance on Objective queries, and magenta indicates better performance on Subjective queries.

LLM	Objective Queries			Subjective Queries
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
o3-mini	55.79	67.37	80.00	44.12	64.71	80.39
gpt-4o	43.16	56.84	80.00	39.22	58.82	85.29
claude-3-5-haiku	33.68	49.47	69.47	27.45	46.08	75.49
gpt-4o-mini	32.63	44.21	58.95	31.37	59.80	83.33
claude-sonnet-4	27.37	41.05	65.26	32.35	57.84	82.35

Table 6: Plan Generation quality comparison using prompts without lineage for objective and subjective queries, judged using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for both query categories; blue indicates better performance on Objective queries, and magenta indicates better performance on Subjective queries.

LLM	Simple Queries			Compound Queries
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
o3-mini	45.28	51.89	69.81	40.66	54.95	75.82
gpt-4o	42.45	55.66	73.58	49.45	70.33	90.11
claude-3-7-sonnet	41.51	52.83	69.81	17.58	42.86	71.43
gpt-4o-mini	36.79	51.89	66.04	26.37	45.05	62.64
claude-sonnet-4	33.02	51.89	67.92	27.47	53.85	80.22

Table 7: Plan Generation quality comparison using prompts with lineage for Simple and Compound queries using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for both query categories; blue indicates better performance on Simple queries, and magenta indicates better performance on Compound queries.

LLM	Simple Queries			Compound Queries
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
o3-mini	47.17	63.21	75.47	52.75	69.23	85.71
gpt-4o	44.34	57.55	83.02	37.36	58.24	82.42
claude-sonnet-4	38.68	50.00	67.92	19.78	49.45	81.32
gpt-4o-mini	36.79	52.83	66.98	26.37	51.65	76.92
claude-3-5-haiku	33.96	50.94	66.98	26.37	43.96	79.12

Table 8: Plan Generation quality comparison using prompts without lineage for Simple and Compound queries using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for both query categories; blue indicates better performance on Simple queries, and magenta indicates better performance on Compound queries.

LLM	[1, 4] steps in Best Possible Plan			[5, 15] steps in Best Possible Plan
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
o3-mini	52.48	60.28	75.89	19.64	35.71	64.29
gpt-4o	48.23	62.41	78.72	39.29	62.50	87.50
gpt-4o-mini	38.30	54.61	68.09	16.07	33.93	55.36
claude-3-7-sonnet	37.59	54.61	72.34	12.50	32.14	66.07
claude-sonnet-4	32.62	52.48	70.21	25.00	53.57	82.14

Table 9: Plan Generation quality comparison using prompts with lineage for queries with 1 to 4 steps and 5 to 15 steps in the Best Possible Plan using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for both the groups; blue indicates better performance on queries with [1,4] steps in the Best Possible Plan, and magenta indicates better performance on queries with [5,15] steps in the Best Possible Plan.

LLM	[1, 4] steps in Best Possible Plan			[5, 15] steps in Best Possible Plan
LLM	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)	( A+ ) Extr. good, very good (%)	( A ) Extr. good, very good, good (%)	( B ) Extr. good, very good, good, acceptable (%)
o3-mini	54.61	71.63	80.85	37.50	51.79	78.57
gpt-4o	43.26	58.16	82.27	35.71	57.14	83.93
gpt-4o-mini	34.75	50.35	68.79	25.00	57.14	78.57
claude-sonnet-4	34.75	48.23	68.09	17.86	53.57	89.29
claude-3-5-haiku	31.91	51.06	70.92	26.79	39.29	76.79

Table 10: Plan Generation quality comparison using prompts without lineage for queries with 1 to 4 steps and 5 to 15 steps in the Best Possible Plan using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for both the groups; blue indicates better performance on queries with [1,4] steps in the Best Possible Plan, and magenta indicates better performance on queries with [5,15] steps in the Best Possible Plan.

LLM	Zero Hop	One Hop	Two Hop	Three-Plus Hop
LLM	( A+ ) Extr. good, very good (%)	( A+ ) Extr. good, very good (%)	( A+ ) Extr. good, very good (%)	( A+ ) Extr. good, very good (%)
claude-sonnet-4	25.00	31.40	38.89	17.24
llama4-maverick-17b-instruct	21.43	26.74	14.81	10.34
o3-mini	17.86	55.81	51.85	13.79
gpt-4o-mini	14.29	45.35	31.48	10.34
claude-3-7-sonnet	14.29	41.86	35.19	3.45

Table 11: Plan Generation quality comparison using prompts with lineage for Zero-Hop , One-Hop , Two-Hop and Three-Plus Hop queries using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for each group; The highest score across groups for each model is highlighted in blue.

LLM	Zero Hop	One Hop	Two Hop	Three-Plus Hop
LLM	( A+ ) Extr. good, very good (%)	( A+ ) Extr. good, very good (%)	( A+ ) Extr. good, very good (%)	( A+ ) Extr. good, very good (%)
nova-pro	14.29	23.26	27.78	0.00
llama4-maverick-17b-instruct	14.29	26.74	24.07	3.45
claude-sonnet-4	10.71	39.53	31.48	17.24
claude-3-7-sonnet	10.71	20.93	25.93	13.79
o3-mini	7.14	62.79	62.96	27.59

Table 12: Plan Generation quality comparison using prompts without lineage for Zero-Hop , One-Hop , Two-Hop and Three-Plus Hop queries using the one-shot evaluator on test data. The highest score in the Extremely Good, Very Good bucket is highlighted in green for each group; The highest score across groups for each model is highlighted in blue.

LLM	With Lineage		Without Lineage
	Objective Queries	Subjective Queries	Objective Queries	Subjective Queries
claude-3-7-sonnet	82.9	85.58	82.44	84.3
gpt-4o	81.94	82.02	83.49	83.44
claude-sonnet-4	80.85	80.09	76.1	79.61
llama4-maverick-17b-instruct	79.49	84.21	83.04	83.5
gpt-4o-mini	78.64	76.76	84.35	82.85

Table 13: Plan Generation quality comparison using prompts with and without lineage for Objective and Subjective queries, judged using the Overall Score of the metric-wise evaluator on test data ( 500 queries). For the two groups - With Lineage and Without Lineage - blue cells denote better performance on Objective queries, while magenta cells denote better performance on Subjective queries.

LLM	With Lineage		Without Lineage
	Simple Queries	Compound Queries	Simple Queries	Compound Queries
claude-3-7-sonnet	86.17	82.99	84.6	81.51
llama4-maverick-17b-instruct	85.01	78.83	82.39	82.53
gpt-4o	82.83	79.59	83.48	81.73
nova-pro	81.17	77	83.72	81.4
llama3-3-70b-instruct	80.36	78.95	82.42	79.36

Table 14: Plan Generation quality comparison using prompts with and without lineage for Simple and Compound queries, judged using the Overall Score of the metric-wise evaluator on test data ( 500 queries). For the two groups - With Lineage and Without Lineage - blue cells denote better performance on Simple queries, while magenta cells denote better performance on Compound queries.

LLM	With Lineage				Without Lineage
LLM	Zero Hop	One Hop	Two Hop	Three Plus Hop	Zero Hop	One Hop	Two Hop	Three Plus Hop
nova-pro	87.01	80.34	77.64	77.69	83.16	84.17	81.82	83.2
llama3-3-70b-instruct	85.4	81.23	79.98	76.22	82.27	82.56	80.95	78.19
claude-3-7-sonnet	85.36	86.53	84.05	83.49	82.52	84.81	83.49	82.77
gpt-4o	83.67	83.41	79.84	79.39	72.55	85	83.14	81.03
llama4-maverick-17b-instruct	83.52	85.1	79.31	82.6	80.71	84.43	82.73	83.17

Table 15: Plan Generation quality comparison using prompts with and without lineage grouped by Number of Hops , judged using the Overall Score of the metric-wise evaluator on test data ( 500 queries). For the two groups - With Lineage and Without Lineage - blue cells denote the best performance across categories of Number of Hops for each model.

LLM	With Lineage		Without Lineage
	[1, 4] steps	[5, 15] steps	[1, 4] steps	[5, 15] steps
claude-3-7-sonnet	86.07	82.98	84.29	82.46
nova-pro	83.16	80.56	82.85	82.07
gpt-4o-mini	82.92	79.07	83.78	81.37
gpt-4o	81.84	76.03	82.92	78.1
llama4-maverick-17b-instruct	80.63	76.67	83.7	81.12

Table 16: Plan Generation quality comparison using prompts with and without lineage grouped by Number of steps in the Best Possible Plan , judged using the Overall Score of the metric-wise evaluator on test data ( 500 queries). For the two groups - With Lineage and Without Lineage - blue cells denote better performance on queries with [1, 4] steps in the best possible plan, while magenta cells denote better performance on queries with [5, 15] steps in the best possible plan.

Tag	Simple Queries				Compound Queries
	Pre-Loop		Post-Loop		Pre-Loop		Post-Loop
	#	%	#	%	#	%	#	%
Extremely Bad	8	2.97	13	4.83	20	8.66	10	4.33
Very Bad	71	26.39	58	21.56	56	24.24	51	22.08
Bad	46	17.10	51	18.96	61	26.41	71	30.74
Acceptable	36	13.38	23	8.55	30	12.99	25	10.82

Table 17: Effectiveness of the iterative Evaluator

\rightarrow

Optimizer loop judged using one-shot evaluator broken down by simple and compound queries on test data. Higher values between Pre-Loop and Post-Loop in the Good, Very Good, Extremely Good buckets are highlighted in green for each group.

Tag	Objective Queries				Subjective Queries
	Pre-Loop		Post-Loop		Pre-Loop		Post-Loop
	#	%	#	%	#	%	#	%
Extremely Bad	13	5.39	10	4.15	15	5.79	13	5.02
Very Bad	86	35.68	66	27.39	41	15.83	43	16.60
Bad	58	24.07	66	27.39	48	18.53	56	21.62
Acceptable	23	9.54	18	7.47	43	16.60	30	11.58

Table 18: Effectiveness of the iterative Evaluator

\rightarrow

Optimizer loop judged using one-shot evaluator broken down by objective and subjective queries on test data. Higher values between Pre-Loop and Post-Loop in the Good, Very Good, Extremely Good buckets are highlighted in green for each group.

Tag	Zero-Hop				One-Hop
	Pre-Loop		Post-Loop		Pre-Loop		Post-Loop
	#	%	#	%	#	%	#	%
Extremely Bad	3	3.41	0	0.00	8	3.94	10	4.93
Very Bad	55	62.50	45	51.14	33	16.26	28	13.79
Bad	18	20.45	28	31.82	28	13.79	36	17.73
Acceptable	5	5.68	0	0.00	30	14.78	20	9.85

Table 19: Effectiveness of the iterative Evaluator

\rightarrow

Optimizer loop judged using one-shot evaluator broken down by number of hops (0-hops and 1-hop) on test data. Higher values between Pre-Loop and Post-Loop in the Good, Very Good, Extremely Good buckets are highlighted in green for each group.

Tag	Two-Hops				Three-Plus-Hops
	Pre-Loop		Post-Loop		Pre-Loop		Post-Loop
	#	%	#	%	#	%	#	%
Extremely Bad	15	10.71	5	3.57	3	4.35	8	11.59
Very Bad	13	9.29	20	14.29	26	37.68	15	21.74
Bad	43	30.71	38	27.14	17	24.64	20	28.99
Acceptable	20	14.29	23	16.43	10	14.49	5	7.25

Table 20: Effectiveness of the iterative Evaluator

\rightarrow

Optimizer loop judged using one-shot evaluator broken down by number of hops (2-hops and 3-plus-hops) on test data. Higher values between Pre-Loop and Post-Loop in the Good, Very Good, Extremely Good buckets are highlighted in green for each group.

Metric	Claude-Sonnet-4				GPT-5
	Reference-Free		Reference-Based		Reference-Based
	Single	Deconstructed	Single	Deconstructed	Deconstructed
	%	%	%	%	%
Dependency	62.74	96.00	67.81	97.3	93.38
Format	39.21	93.46	53.85	94.02	99.23
Step Executability	58.36	72.55	60.24	79.43	65.70

Table 21: Evaluation of Metric-Wise Evaluator : Triplet Ranking Agreement (Relaxed) Between Metric-Wise Evaluator and Human Annotations on the Validation Set (N=80) for Claude-Sonnet-4 and GPT-5 : Percent of Correct Inequalities per Metric. The best-performing setting is highlighted for every metric. The top-performing setting is highlighted for each metric: values

\geq 90\%

in blue,

\geq 80\%

in magenta, and

<80\%

in red.

Tag	# Annotator Assigned	# Evaluator Assigned	# Overlap	Precision	Recall	F1-Score
Extremely Bad	9	9	9	1.00	1.00	1.00
Very Bad	7	6	6	1.00	0.86	0.92
Bad	16	15	14	0.93	0.88	0.90
Acceptable	19	17	16	0.94	0.84	0.89
Good	9	9	9	1.00	1.00	1.00
Very Good	12	14	11	0.79	0.92	0.85

Table 22: Evaluation of Claude-Sonnet-4 as One-Shot Evaluator : Agreement in Assigning Tag Between One-Shot Evaluator and Human Annotations on the Validation Set (N=80). F1-Scores

\geq 90\%

in blue, and

\geq 80\%

in magenta.

Tag	# Annotator Assigned	# Evaluator Assigned	# Overlap	Precision	Recall	F1-Score
Extremely Bad	9	11	9	0.82	1.00	0.90
Very Bad	7	8	5	0.62	0.71	0.67
Bad	16	15	15	1.00	0.94	0.97
Acceptable	19	16	16	1.00	0.84	0.91
Good	9	8	8	1.00	0.89	0.94
Very Good	12	13	12	0.92	1.00	0.96

Table 23: Evaluation of GPT-5 as One-Shot Evaluator : Agreement in Assigning Tag Between One-Shot Evaluator and Human Annotations on the Validation Set (N=80). F1-Scores

\geq 90\%

in blue, and

\geq 80\%

in magenta.

Metric	$\rho$	$p-value$
Total metric-wise score	0.60	0.023
Oneshot overall quality	0.52	0.059
Dependency	0.84	0.0002
Format	0.58	0.029
Redundancy	0.55	0.043
Step Executability	0.54	0.047

Table 24: Spearman rank correlations between LLM rankings under Claude-Sonnet-4 and GPT-5 as Judges on a 50-query test subset (14 planners, prompt type = Without Lineage ).

\rho

< 0.5, and

p-value

> 0.1 are highlighted in red.

Tag	# Annotator Assigned	# Evaluator Assigned	# Overlap	Precision	Recall	F1-Score
No Change	66	70	51	0.77	0.73	0.75
Incorrect Tool	66	62	55	0.83	0.89	0.86
Incorrect Prompt	68	66	61	0.9	0.92	0.91
Complex Prompt	70	63	53	0.76	0.84	0.8
Repeated Detail	74	71	66	0.89	0.93	0.91
Multi-tool Prompt	56	68	50	0.89	0.74	0.81

Table 25: Evaluation of Step-Wise Evaluator : Agreement in Assigning Tag Between Step-Wise Evaluator and Human Annotations on the Validation Set (N=400). F1-Scores

\geq 90\%

in blue,

\geq 80\%

in magenta and

<80\%

in red.

Tag	#	%
Extremely Bad	2	1.25
Very Bad	8	5
Bad	14	8.75
Acceptable	17	10.63
Good	35	21.88
Very Good	39	24.38

Table 26: Evaluation of Plan Optimizer : Similarity Between Plans Generated By Plan Optimizer and Human Annotations (considered as reference) on the Validation Set (N=160) Computed Based on One-Shot Evaluator . The proportion of plans generated by the Plan Optimizer falling in the top 3 buckets - { Extremely Good, Very Good, Good } are highlighted in blue.

Module	Prompt
Query Generation	SYSTEM_PROMPT: You are an expert query generator. The queries you generate should mimick real-life user queries related to a call center with agents. The queries should aim to gain insight into the call center’s functioning and performance. Here are some reference examples. Generate 10 new queries. {QUERY_REFERENCE_EXAMPLES}
Plan Generation (Without Lineage)	SYSTEM_PROMPT: You are a high-level planner agent designed to orchestrate step-by-step solutions to business queries related to call center operations. Your role is to: - Analyze the given query and break it down into a coherent, step-by-step plan. - Ensure that all steps rely solely on the tools provided below — do not assume access to any external data, tools, or knowledge beyond what is defined. - Each step of the output plan should be simple and executable by specialized tool-aware agents. You have access to the following tools: {TOOL_DESCRIPTIONS} MAIN_PROMPT: Task: 1. You will be provided with a query to answer. 2. Generate a high-level, step-by-step plan to solve it. 3. Use only the tools available to you (described above). 4. Design the steps so that the plan is logically valid, realistically achievable, and contextually appropriate. 5. Clearly define dependencies between steps where intermediate results are needed. 6. Output each plan in the following format: <example number>: Query: <provided query> Plan: { <step number>: {"step": <tool name>("<input instruction to the tool>"), "depends_on": [<step numbers that this step depends on>]} .... } Guidelines: - Be structured: respect execution order and dependencies - For each step k, the depends_on list may only reference step indices in 1, …, k-1, and the overall dependency graph must remain acyclic (no cycles). - Be concise: focus each step on a specific actionable instruction - Return ONLY the plan Reference: Below are some real examples of business queries and their corresponding multi-step plans along with thought process behind the plans: {EXAMPLE_QUERY_PLANS_EXPLANATIONS} The input query to answer is: {QUERY}
Plan Generation (With Lineage)	SYSTEM_PROMPT: You are a high-level planner agent designed to orchestrate step-by-step solutions to business queries related to call center operations. Your role is to: - Analyze the given query and break it down into a coherent, step-by-step plan. - Ensure that all steps rely solely on the tools provided below do not assume access to any external data, tools, or knowledge beyond what is defined. - Each step of the output plan should be simple and executable by specialized tool-aware agents. You have access to the following tools: {TOOL_DESCRIPTIONS} MAIN_PROMPT: Task: 1. You will be provided with a query to answer. 2. Generate a high-level, step-by-step plan to solve it. 3. Use only the tools available to you (described above). 4. Design the steps so that the plan is logically valid, realistically achievable, and contextually appropriate. 5. Clearly define dependencies between steps where intermediate results are needed. 6. Output each plan in the following format: <example number>: Query: <provided query> Plan: { <step number>: {"step": <tool name>("<input instruction to the tool>"), "depends_on": [<step numbers that this step depends on>]} .... } Guidelines: - Be structured: respect execution order and dependencies - For each step k, the depends_on list may only reference step indices in 1, …, k-1, and the overall dependency graph must remain acyclic (no cycles). - Be concise: focus each step on a specific actionable instruction - Return ONLY the plan Reference: Below are examples of business queries paired with sequences of plans. Each sequence begins with a suboptimal plan and gradually improves through multiple iterations until reaching the best possible plan for the query. At each step, the plan is refined to better align with tool usage, reduce unnecessary complexity, and improve overall clarity. Use these plan lineages to identify common mistakes made by planners and to learn strategies for avoiding them. Given a query, generate an initial plan, then iteratively refine it, and return the full sequence of improved plans. {EXAMPLE_QUERY_PLANS_LINEAGES} The input query to answer is: {QUERY}

Table 27: Prompts Used For Generation of Queries and One-Shot Plans

Prompt

You are a Step Evaluator Agent responsible for carefully analyzing individual steps within a multi-step plan designed to answer a user query. Your mission is to evaluate each step against its dependent inputs and explain if there are any issues with the step, using the standardized error types and format. You need to carefully determine whether the step has errors and justify your evaluation based on the reference examples and the provided tool capabilities. ### Your Task: For a given step in a plan, you will evaluate and output the results as follows: 1. Analyze the step and its dependent steps to identify potential issues under the following error types: - INCORRECT PROMPT - COMPLEX PROMPT - INCORRECT TOOL - REPEATED DETAIL - MULTI-TOOL PROMPT - NO CHANGE 2. Follow this order of judgment for your evaluations: - Evaluate if the step has any of the following critical issues: INCORRECT PROMPT, COMPLEX PROMPT, or INCORRECT TOOL, comparing with the dependent step(s) if necessary. - If no critical issues are identified, assess the step for REPEATED DETAILS by comparing it to its dependent steps. - Check if the task in the step could also be performed by a different tool, and if it can then flag it as MULTI-TOOL PROMPT. - If none of the above issues are detected, then and only then, return NO CHANGE. 3. When identifying an error, include both: - A clear, concise reasoning for why that error applies. - The appropriate error tag corresponding to the error type. 4. Output Format: Use the following standardized output format for any identified issues: " <STEP NUMBER> 1. <REASONING BEHIND THE TYPE OF ERROR>: <ERROR TYPE> 2. <REASONING BEHIND THE TYPE OF ERROR>: <ERROR TYPE> ... " 5. Structure Of The Steps: <step_number>: {{"step": <Tool_Name>((placeholders), "<prompt>"), "depends_on": [dependent_steps]}} (when tool is RAG or T2S, placeholder is outside the prompt) <step_number>: {{"step": <Tool_Name>("<prompt>"), "depends_on": [<dependent_steps>]}} (when tool is LLM, placeholder is inside the prompt) ### When Evaluating a Point, Some Key Points to Consider: - Step Structure: Ensure the step matches the expected JSON format and adheres to placeholders (e.g., make sure any referenced dependent steps are properly reflected both in the ‘depends_on‘ array and the tool’s prompt using placeholders like ‘(1)‘ or ‘(2)‘). - Dependent Steps: Evaluate the dependent steps in relation to the current step. Any details already covered in the dependent steps should not be redundantly included in the current step’s prompt. - Tool Assignment: Each step must be handled by the appropriate tool. Use the tool capabilities section to determine whether the assigned tool is suitable for the task. A mismatch in tool functionality would lead to an INCORRECT TOOL error. ### Error Type Definitions: Here is the list of error types that you must use in your evaluations, along with their specific definitions: 1. INCORRECT PROMPT: - This error applies when the prompt given to the tool is incomplete or misaligned with the tool’s capabilities. This issue can lead to incorrect or incomplete outputs. - For example, if the query asks for calls, and the prompt to T2S extracts interaction_ids of all interactions, then that is an incorrect prompt that needs to be changed. If the placeholders are missing from an LLM step, that is an incorrect prompt and needs to be changed to include the placeholders. 2. COMPLEX PROMPT: - This error applies when the prompt is too complex to be executed by the tool in a single step. - Such prompts need to be broken down into simpler sub-steps. - This can happend when T2S is being asked to extract more than 1 thing at a time or when RAG is being asked to answer multiple questions at a time. If it can be sensibly broken down into simpler steps, it should be. 3. INCORRECT TOOL: - This error applies when the wrong tool is assigned to a particular task. - Look carefully at the tool description provided below, it explains in detail what kind of information can be answered by T2S and which can be answered by RAG. 4. REPEATED DETAIL: - This error applies if the step contains unnecessary redundancies, such as duplicating details already handled by a dependent step. 5. MULTI-TOOL PROMPT: - This error applies when the task assigned to a particular tool can also be accomplished with another available tool as well (e.g., if both T2S and RAG could perform the task). 6. NO CHANGE: - This result is valid when the step is completely correct, and none of the above points apply to it. --- ### Evaluation Order: When evaluating, follow this strict judgment sequence: 1. Look for critical issues in the step: - Does the step contain an INCORRECT PROMPT, a COMPLEX PROMPT, or an INCORRECT TOOL error? 2. Next, assess if there are REPEATED DETAILS by comparing the step to its dependent steps. 3. Check if the task described in the step can also be performed by another tool. If so, flag it as a MULTI-TOOL PROMPT. 4. If none of the above apply, conclude your evaluation with NO CHANGE. --- ### Tools Available: {Snowflake_tools_descriptions.TOOL_DESCRIPTIONS_ULTRASHORT} --- ### Reference Examples: {Evaluator_creator.REFERENCE_EXAMPLES} IMPORTANT REMINDER - Order of dependencies and placeholders does not matter. Dependencies are there to know which steps this step is dependent on. Placeholders in a prompt so that when that prompt is finally given to the respective tool, we know where to put the output of the dependent step into the prompt.

Table 28: System Prompt Used For Step-Wise Evaluator.

Prompt

#### ROLE: You are an Expert Plan Optimizer tasked with refining and updating multi-step plans designed to answer complex queries related to call centers. You will be provided with a plan, along with the output of a step-evaluator. This output contains a detailed analysis of a particular step of the plan. Based on the output of the step-evaluator, you will make changes to the plan and return the updated plan. --- ### GOAL: Produce a revised multi-step plan by: 1. Analyzing the output of the stepwise plan evaluator. 2. Based on the output of the step-evaluator, make changes to that particular step (CHANGE 0). 3. If necessary, make changes to the latter steps of the plan in order to maintain overall logical coherence (CHANGE 1). --- ### INPUTS: 1. Original Query: - The query from the user that the plan is attempting to answer. 2. Original Plan: - A JSON-formatted textual representation of the original multi-step plan. - Each step of the plan should strictly adhere to the following format: <step_number>: {{"step": <tool_name>("<prompt>"), "depends_on": [<dependent_step_numbers>]}} Key Details About Step Format: - It must be JSON parsable, ensuring that details like double quotes, braces, commas, etc., are accurate. - Referenced steps by number (e.g., "depends_on": [1, 2]) must be reflected as placeholders (e.g., "(1)", "(2)") in the corresponding ‘<prompt>‘, especially in LLM steps. 3. Step Evaluator Output: - Feedback related to a specific step in the plan. - Includes: - Step Number: The step under scrutiny. - Feedback: Specific insights into potential issues such as incorrect prompt/tool, overly complex prompt, repeated details, etc. - Error Type: The categorization of the issue (e.g., INCORRECT PROMPT, INCORRECT TOOL, COMPLEX PROMPT, etc.). --- ### EXPECTED OUTPUT: 1. CHANGE 0: - Update the problematic step directly based on the output of the step evaluator. Changes may include: - Prompt refinement. - Prompt breakdown resulting in the creation of new steps. - Tool reassignment. - Correction of repeated details. - Addition of new step based on "multi-tool prompt" output, if it is not already present. - Clearly and concisely explain what you changed and why. 2. CHANGE 1: - Address how the rest of the plan may need adjustments to ensure logical coherence based on CHANGE 0. - Must NOT include any plan optimization or improvement, only corrections to ensure that previous change does not break the entire plan. - Changes may include: - Updating indices of steps based on addition or removal of steps. - Updating dependencies/placeholders for downstream steps. - Adding additional steps for aggregation or reasoning. - Removing now-redundant or irrelevant steps. - Provide a detailed reasoning for any adjustments made. 3. Revised Plan: - A JSON-formatted version of the new, corrected plan. - Ensure correct syntax, adherence to format, and logical coherence throughout the revised plan. --- ### ERROR TYPES AND HOW TO HANDLE THEM: Here is a comprehensive list of potential issues the plan optimizer might encounter and the strategies to address them. {STEP_OPTIMIZER_ERROR_CATEGORIES_DESCRIBED} --- ### FLOW: Explain your analysis and actions in a two-stage process: #### CHANGE 0: - This stage addresses the feedback provided by the step evaluator for the particular step under consideration. - Describe the modifications made to this step and the rationale behind them explicitly. #### CHANGE 1: - This stage ensures coherence of the plan after implementing CHANGE 0. - Adjust dependencies, add/delete/update steps as necessary to ensure the plan works seamlessly after the initial change. - Your job is not to perform optimizations on the plan to improve it. It is simply to make any changes necessary to ensure that the rest of the plan isn’t wrong because of CHANGE 0. - An example of what you SHOULD do: If a step has been removed and that step was referenced in another step’s dependency, then your job is to correct all the step indices based on the removed step, and then to update the future step’s dependencies and placeholders to account for this change. - An example of what you SHOULD NOT do: If a step was detected to have repeated detail and CHANGE 1 removed that repeated detail, then it is NOT your job to find similar repeated details in future steps and change them as well. - This is a VERY important distinction and should be remembered at all times. --- ### FORMAT OF INPUT AND OUTPUT: INPUT FORMAT: - Query: <query> - Plan: <plan> - Step Evaluator Output: <step_evaluator_output> OUTPUT FORMAT: "CHANGE 0": <explanation of direct changes based on the step evaluator output> "CHANGE 1": <explanation of changes to the rest of the plan to ensure logical coherence> "NEW PLAN STARTS": <revised plan> --- ### TOOLS: {Snowflake_tools_descriptions.TOOL_DESCRIPTIONS_ULTRASHORT} --- ### REMINDER: - Follow the correct step format strictly. - Ensure all syntax, placeholders, and dependencies are accurate. - Changes must make the plan more logically connected and operationally sound with the tools provided. - Make sure that CHANGE 1 leads to correct and sequential indices in the new plan.

Table 29: System Prompt Used For Plan Optimizer

Module	Prompt
Dependency (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a plan and your task is to meticulously evaluate its dependencies and assign it a score. * Here is an example of a plan: { 1: {"step": T2S([], "Fetch call_ids of calls related to payment adjustment inquiries."), "depends_on": []}, 2: {"step": RAG((1), "What patterns of customer sentiment emerge in these calls?"), "depends_on": [1]} } Here are the things to keep in mind: * *The only part of this plan that is relevant for you is the list followed by "depends_on" in each step. That list contains the steps that the current step depends on. ** Your job is to find if for any step, that list of dependent steps is wrong which can only be in 1 of 2 cases. Either there is some dependent step that is missing from the "depends_on" list or there is some step in the "depends_on" list that shouldn’t be there. * Return the total steps - total number of steps with incorrect dependencies. Input Format: You will be simply provided with a plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explanation> \| <score> \| Here are some reference examples: {DEPENDENCY_REFERENCE_EXAMPLES}
Format (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a plan and your task is to meticulously evaluate its format and assign it a score. * Here is the correct format: Plan: {{ <step number>: {{"step/query": <tool name>("<input instruction to the tool>"), "depends_on": [<step numbers that this step depends on>]}} .... }} Here are the things to keep in mind: * Return the total steps - total number of format violations. * Make sure that the plan is json parsable. If that fails due to thing such as quadruple quotes instead of double quotes, mismatched paranthesis or some similar reason, then that is a violation. * Make sure that every step, that has a dependency on another step, has a valid placeholder. The placeholder is the dependent step number, surrounded by paranthesis, within the prompt. For example in step 2 of the above correct plan, "RAG((1), "Wha...)", here "(1)" is the placeholder. Similarly a placeholder MUST exist in every step with a dependency, ESPECIALLY in the steps with LLM. Each step with an absent placholder or an incorrect placeholder (e.g. (1, 2) instead of (1) and (2) etc.) is a format violation. * If a step, for e.g an LLM step has a mention of the query, for e.g. "Answer the question/query based on ...", then it MUST have a placeholder for the query (e.g. (query)) otherwise that is a format violation. Input Format: You will be simply provided with a plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explaination> \| <score> \| Here are some reference examples: {FORMAT_REFERENCE_EXAMPLES}
Redundancy (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a plan and your task is to meticulously evaluate its redundancy and assign it a score. * Here is the format of a plan: Plan: {{ <step number>: {{"step/query": <tool name>("<input instruction/prompt to the tool>"), "depends_on": [<step numbers that this step depends on>]}} .... }} There are only two types of redundancies that you need to look for: 1. T2S Step Redundancy: - In a T2S step, if the instruction/prompt provided contains some information that is already accounted for in its dependent step, then that is a redundancy. - The reason being that for T2S i.e. Snowflake structured database specifically, any additional detail in the prompt can lead to additional table joins or other things that are not needed. - Due to this reason, this step redundancy ONLY APPLIES TO T2S AND NOT TO RAG OR LLM STEPS. - Phrases like "in these calls" or "in these interactions" in a T2S step is NOT considered to be redundant. - Example for this will be provided in the reference examples. 2. Ignored Step Redundancy: - The final step of a plan is the overall output of the plan. - If there is a step or steps that do not contribute to the final step, then those steps are redundant. - This can be found by looking at the "depends_on" list of each step. - Example for this will be provided in the reference examples. Return the total steps - total number of redundancies/redundant steps. Input Format: You will be provided with a plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explanation> \| <score> \| Here are some reference examples: {REDUNDANCY_REFERENCE_EXAMPLES}

Table 30: System Prompts Used in Metric-Wise Evaluator (Dependency, Format and Redundancy)

Module	Prompt
Step-Executability (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a plan along with the best possible plan and your task is to meticulously evaluate its step-executability and assign it a score. * Here is the format of a plan: Plan: { <step number>: {"step/query": <tool name>("<input instruction/prompt to the tool>"), "depends_on": [<step numbers that this step depends on>]} .... } Here are the things to keep in mind: * The only thing that you need to look at is the tool and the corresponding instruction/prompt in each step. * Assume that everything that the prompt is asking from the tool is perfectly within the tool’s capabilities. * Under this assumption, judge whether the prompt is too complex to be executed by the tool in 1 go and needs to be broken down into multiple steps. * There are 2 tools that you need to check, RAG (which performs Retrieval-Augmented Generation on some unstructured data) and T2S (which performs Text-to-SQL on some structured data). * For RAG: The prompt should only ask the tool to do one thing at a time. Examples provided in the reference examples. * For T2S: For a T2S step, the number of trigger words/trigger phrases present in the prompt is directly proportional to the complexity of the step. Here is a list of trigger words to look for. Don’t look exactly for these trigger words, but look for similar looking phrases as well: For call drivers {call_drivers_op} For moments {moments_op} For qa {qa_op} Based on extracted data, T2S has the ability to perform simple mathematical operations such as finding mean etc. If a T2S step contains more than 3 trigger words/phrases, then it is not executable in 1 go. REMEMBER: "fetch interaction_id" or "fetch call_id" is NOT considered as a trigger phrase Return the total steps - total number of steps that are not executable in 1 go. Input Format: You will be provided with 2 plans, the plan in question and the best possible plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explanation> \| <score> \| Here are some reference examples: {STEP_ATOMICITY_REFERENCE_EXAMPLES}
Query Adherence (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a query, the corresponding plan and the best possible plan, and your task is to meticulously evaluate its query adherence and assign it a score. * Here is the format of a plan: Plan: { <step number>: {"step/query": "TOOL"("<input instruction to the tool>"), "depends_on": [<step numbers that this step depends on>]} .... } Here are the things to keep in mind: * Do not worry about things like which tool is used, format etc. Only look at the prompts/instructions given to the tools in each step and the step dependencies. * Assume that in each step, the instructions that are given to the tool in the prompt, run perfectly as intended and give the ideal output as per the prompt. * Under that assumption, judge whether the final output of the plan would answer the original query well or not. * Here are some possible scores to assign: * 0.0: If the plan is completely irrelevant to the query. * 0.5: If the plan partially answers the query but has some gaps/flaws. * 1.0: If the plan perfectly and completely addresses all aspects of the query. Additional Information: - Placeholders: There are 2 types of placeholders: - Step placeholder (for e.g (1), (2)) which gets replaced by the output of the corresponding step during execution. - Query placeholder i.e. (query) which gets replaced by the original query during execution. - So wherever in the plan you see something like (1), (2), (query) etc, assume that is replaced by their corresponding above mentioned values. - The terms "call_ids" and "interaction_ids" are interchangeable. What matters is what comes after it, i.e. "call_ids of calls" or "interaction_ids of calls" would both fetch the ids of CALLS. Similarly, "call_ids of interactions" and "interaction_ids of interactions" would both fetch the ids of interactions. - In the final step, if the prompt itself does a good job at describing what is asked in the query, then it is fine even if it doesn’t have the placeholder of the query. Input Format: You will be provided with a query and the corresponding plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explaination> \| <score> \| Here are some reference examples: {QUERY_ADHERENCE_EXAMPLES}

Table 31: System Prompts Used in Metric-Wise Evaluator (Step-Executability and Query-Adherence)

Module	Prompt
Tool-Usage Completeness (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a plan along with the best possible plan and your task is to meticulously evaluate its tool-usage completeness and assign it a score. * Here is the format of a plan: Plan: { <step number>: {"step/query": <tool name>("<input instruction/prompt to the tool>"), "depends_on": [<step numbers that this step depends on>]} .... } Here are the tools at your disposal: {TOOL_DESCRIPTIONS} ## TOOL DESCRIPTION ENDS HERE Here are the things to keep in mind: * Look at the best possible plan and see if it has two steps with same prompt but different tool. For e.g in the following plan { 1: {"step": T2S([], "Fetch call_ids of calls related to payment adjustment inquiries."), "depends_on": []}, 2: {"step": RAG((1), "What patterns of customer sentiment emerge in these calls?"), "depends_on": [1]}, 3: {"step": T2S((1), "What patterns of customer sentiment emerge in these calls?"), "depends_on": [1]}, 4: {"step": LLM("Synthesize and summarize your answer to the query (query) using outputs from the following: (sub-query 2) using (tool 2): (2), (sub-query 3) using (tool 3): (3)"), "depends_on": [2, 3]} } * If this kind of a thing does not exist in the best possible plan: - directly return total steps - 0 = total steps. * If this kind of thing IS present in the best possible plan, then: - If the exact same thing (same 2 steps with same prompt and different tools) is present in the plan to be evaluated: - that is NOT a tool-usage completeness violation. - If neither of the two steps is present in the plan to be evaluated: - that is NOT tool-usage completeness violation. - If only 1 of the 2 steps are present in the plan to be evaluated: - that is a tool-usage completeness violation. Return the total steps - total number of tool-usage completeness violations. Input Format: You will be provided with 2 plans, the plan to be evaluated and the best possible plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explaination> \| <score> \| Here are some reference examples: {TOOL_USAGE_COMPLETENESS_EXAMPLES_WR}
Tool-Prompt Alignment (Metric-wise Evaluator)	You are an expert plan evaluation agent You will be given a plan along with the best possible plan and your task is to meticulously evaluate its tool-prompt alignment and assign it a score. * Here is the format of a plan: Plan: { <step number>: {"step/query": <tool name>("<input instruction/prompt to the tool>"), "depends_on": [<step numbers that this step depends on>]} .... } Here are the tools at your disposal: {TOOL_DESCRIPTIONS} ## TOOL DESCRIPTION ENDS HERE Here are the things to keep in mind: * For each step, look at the tool and the prompt. * If the instruction/prompt in a step cannot be fulfilled/answered by the assigned tool, then that step has a tool-prompt alignment violation. * If the prompt is complex and asks the tool to do multiple things at the same time, if even one of those things is outside the scope of the provided tool, then that step has a tool-prompt alignment violation. * For example if the prompt asks how sentiment shifted during the call, or any other change during the call, that cannot be answered using T2S and can only be answered using RAG. Return the total steps - total number of steps that have tool-prompt violation. Input Format: You will be provided with 2 plans, the plan to be evaluated and the best possible plan. Output Format: You must provide a detailed explanation of your analysis related to that point, and then give the final score as per your verdict. Follow the below provided format: <explanation> \| <score> \| Here are some reference examples: {TOOL_PROMPT_ALIGNMENT_EXAMPLES}

Table 32: System Prompts Used in Metric-Wise Evaluator (Tool-Prompt Alignment and Tool-Usage Completeness)

Prompt

You are a critical evaluator of structured, tool-augmented LLM plans. Each plan is a sequence of steps designed to answer a user query using specialized tools. Your task is to compare a candidate plan (P) against the best possible reference plan (P*) and rate the candidate plan on a 7-point quality scale. You must reason carefully across 7 evaluation dimensions and assign a rating based on how close P is to P* in terms of quality, structure, and correctness. ### PLAN FORMAT Each plan is a dictionary where each key is a step number, and each value is an object with: - ‘"step"‘ or ‘"query"‘: containing a tool invocation of the form ‘TOOL_NAME(placeholder_inputs, prompt)‘ - ‘"depends_on"‘: a list of step numbers this step depends on #### Example Format: {{ 1: {{"step": T2S([], "Fetch interaction_ids of unresolved calls"), "depends_on": []}}, 2: {{"step": RAG((1), "Fetch calls where the sentiment transitioned from negative to positive within the transcript"), "depends_on": [1]}}, 3: {{"step": LLM("Fetch the interaction_ids as a list from the ‘data insights‘ table in (2)."), "depends_on": [2]}}, 4: {{"step": T2S((3), "Retrieve agent QA scores for resolution procedures in these calls."), "depends_on": [3]}} }} ### PLACEHOLDER CONVENTIONS Placeholders are used in prompts to represent inputs to tools. These must follow standard conventions: - (query): The original user query - (<step number>): Refers to the output of a previous step - (tool <step number>): Refers to the tool used in that step (T2S, RAG, or LLM) - (sub-query <step number>): Refers to the prompt/query used in that step Placeholders must be well-formed and accurate to correctly wire dependencies between steps. ### TOOL DESCRIPTIONS {TOOL_DESCRIPTIONS} ### EVALUATION DIMENSIONS Evaluate the candidate plan (P) against the gold plan (P*) using these criteria: 1. Precision What percentage of steps in the plan (P) are present in the best possible plan (P*)? 2. Recall What percentage of steps in the best possible plan (P*) are present in the plan (P)? 3. F1 Score What percentage of steps in the best possible plan (P*) are present in the plan (P)? 4. Format Correctness Is the plan valid JSON with correct keys ("step" or "query" and "depends_on")? (a) Is the overall plan valid JSON? (b) What percentage of steps in the plan (P) have correct format - "step" or "query" and "depends_on" 5. Dependencies What percentage of steps in the plan (P) have correct dependencies? 6. Placeholders What percentage of steps in the plan (P) have correct placeholders? ### INSTRUCTIONS - Be objective. Evaluate the candidate plan only in comparison to the gold plan. - Think through each metric and generate a detailed rationale. - Then assign a rating based on overall closeness on a 7-point scale: {{Extremely Bad, Very Bad, Bad, Acceptable, Good, Very Good, Extremely Good}} ### Points to consider while evaluating the plan: - (a) If the plan (P) is not valid JSON, then it is automatically an "Extremely Bad" plan as it becomes difficult to evaluate the plan in terms of precision, recall, F1 score, format correctness, dependencies, and placeholders. - (b) If the plan (P) is valid JSON, then we can evaluate the plan in terms of precision, recall, F1 score, format correctness, dependencies, and placeholders. - (c) The individual metrics such as precision, recall, F1 score, format correctness, dependencies, and placeholders can be computed objectively. However, the final rating can be subjective but strictly based on the individual metrics which capture how close the candidate plan (P) is to the gold plan (P*). - (d) The metrics precision, recall and F1 score are computed by comparing the candidate plan (P) with the gold plan (P*). In contrast, the metrics format correctness, dependencies, and placeholders are computed by simply checking the plan (P) against the required format, dependencies and placeholders. - (e) While the final rating is subjective, you can follow the following guidelines provided the overall format of the plan is correct (valid json): - If f1_score is > 95%, then the rating is "Extremely Good" - If f1_score is > 85%, then the rating is "Very Good" - If f1_score is > 75%, then the rating is "Good" - If f1_score is > 60%, then the rating is "Acceptable" - If f1_score is > 45%, then the rating is "Bad" - If f1_score is > 30%, then the rating is "Very Bad" - If f1_score is < 30%, then the rating is "Extremely Bad" ### Points to keep in mind while checking the match between P and P*: - (a) The steps in P and P* should have the exact same semantics and interpretation. If multiple steps in P* are carried out in a single step in P, then the step in P is incorrect. Example: Steps a and b do not match. - Step a: T2S([], "Fetch interaction_ids categorized by ’Did We Resolve’ (DWR) status and their corresponding call drivers") - Step b: T2S([], "Fetch interaction_ids of calls marked as ’resolved’.") - (b) The steps in P and P* should have the same tool, format, placeholders and dependencies for them to be considered a match. Only if this condition is met, we can say that both the steps have the same semantics and interpretation. - (c) The steps in P and P* need not exactly match in terms of the prompt. The requirement is that the prompts in both the steps in P and P* should be semantically similar and have the same intent. ### OUTPUT FORMAT {{ "rationale": {{ "precision": <rationale for precision>, "recall": <rationale for recall>, "f1_score": <rationale for f1_score>, "format_correctness": <rationale for format_correctness>, "dependencies": <rationale for dependencies>, "placeholders": <rationale for placeholders>, "rating": <rationale for rating> }} "score": {{ "precision": <score for precision>, "recall": <score for recall>, "f1_score": <score for f1_score>, "format_correctness": <score for format_correctness>, "dependencies": <score for dependencies>, "placeholders": <score for placeholders>, "rating": <score for rating> }} }} Note: The output should be a valid JSON object.

Table 33: System Prompt Used For One-Shot Evaluator

Prompt

You are an impartial Judge LLM. Evaluate three candidate responses (R1, R2, R3) to a given Query using the rubric below. Rely only on the provided text. Do not add external facts or assumptions. ## Rubric (score each response independently) (a) Validity (0/1): 0 = invalid (e.g., ‘‘cannot be answered due to no data,’’ system error messages, or refusal). 1 = valid (any other kind of substantive response). (b) Consistency (0/1): Is the response answering the given Query? A response cannot be consistent if it is invalid. Valid but off-topic

\implies

consistency = 0. Valid and on-topic

\implies

consistency = 1. (c) Completeness (1-7): How fully does the response address the Query with relevant details? Consider only details relevant to the Query. 1 = Extremely Bad, 7 = Extremely Good. (d) Redundancy (1-7): How free is the response from irrelevant or unrelated details? 7 = no fluff, to the point. The more irrelevant content, the lower the score. ### Gating rules - Only valid responses can be consistent or complete. - Only valid and consistent responses can be complete. - For invalid responses, set: consistency = 0, completeness = 1, redundancy = 7. ## Decision set (pick exactly one) {"Neither", "R1", "R2", "R3", "R1,R2", "R1,R3", "R2,R3", "All"} Use the following tie logic: - Disqualify any response with validity = 0 or consistency = 0. - If none remain, output "Neither". - Among remaining, compare by (completeness, redundancy) in lexicographic order (higher is better). - If all three tie on both metrics

\implies

"All". - If exactly two tie for best, then return that pair (e.g., "R1,R3"). - Otherwise return the single best (e.g., "R2"). ## Guidelines for scoring - Do not focus too much on the format of the response being scored. As long as the response is valid and consistent, the format can be a simple list or dictionary and not necessary textual. - Example: - Query: ‘‘What percentage of calls required a supervisor escalation, and what is the average QA score for such calls over the last year?’’ - Response 1: {’pct_escalated_calls’: Decimal(’12.326849’), ’avg_escalated_qa_score’: 0.9269446373201656} - This is a valid and consistent response. The format is a dictionary and not textual. Despite the format, the response is complete and accurate. - Response 2: ’’’ # Analysis of Supervisor Escalations and QA Scores Based on the analysis, I can provide some interesting insights about supervisor escalations and their relationship with QA scores: - Only 0.11% of all interactions required supervisor escalation over the past year - The average QA score for escalated calls was 94.00 - For comparison, non-escalated calls had an average QA score of 96.33 While escalated calls do show slightly lower QA scores, the difference is relatively small (only 2.33 points lower), suggesting that agents generally maintain good quality standards even in challenging situations that require escalation. ’’’ - This is a valid and consistent response. However, there’s a lot of fluff in the response. In addition, the response is redundant as it contains information about non-escalated calls which is not relevant to the Query. - Do not focus on the correctness of the numeric values such as counts, percentages, average scores etc. because you are not provided with the ground truth. Simply because there’s an alignment in the numeric values between two of the three responses does not mean that the third response is incorrect. ## Output format Return only a single JSON object (no prose, no markdown) with this exact schema and key order: {"rationale": "<2-4 sentence high-level justification that explains your reasoning for how the responses compare>", "R1": {"validity": <0|1>, "consistency": <0|1>, "completeness": <1-7>, "redundancy": <1-7>}, "R2": {"validity": <0|1>, "consistency": <0|1>, "completeness": <1-7>, "redundancy": <1-7>}, "R3": {"validity": <0|1>, "consistency": <0|1>, "completeness": <1-7>, "redundancy": <1-7>}, "overall_decision": "<one of the allowed strings>" } ## Process requirements - First, analyze the three responses and produce the rationale (2-4 sentences). - Then assign the four metric scores (validity, consistency, completeness, redundancy) for R1, R2, R3. - Finally, generate the overall_decision strictly from the allowed set.

Table 34: System prompt for the Judge LLM used to evaluate and compare R1/R2/R3 responses in the end-to-end QA study.

Stage	Generated Output	Annotated Output
Query Generated	What are customers most unhappy about in refund-related calls longer than 30 minutes?	N/A
Initial Plan Generated	{ 1: {"step": T2S([], "Fetch interaction_ids of interactions where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]} }	N/A
Metric-Wise and One-Shot Evaluator Score for Initial Plan:	1. Format: 20 2. Tool-Prompt Alignment: 15 3. Step-Atomicity: 7.5 4. Query-Adherence: 12 5. Dependencies: 10 6. Redundancies: 10 7. Tool-Usage Completeness: 5 8. Total (Metric-Wise): 79.5 9. One-Shot: 81.5 (Good)	N/A
Step-Evaluator Output: On 1st pass through feedback loop and step to be judged is Step 1	Step: 1 1. Nothing is wrong in this step and it can be executed normally : NO CHANGE	Step: 1 1. The query asks for "refund-related calls" specifically whereas the step filters for all interactions. So the "interaction_ids" needs to be changed to "call_ids" and "interactions" needs to be changed to "calls": INCORRECT PROMPT
Plan-Optimizer Output: On 1st pass through feedback loop. Recieves the input from previous step-evaluator output	"CHANGE 0": Based on the output of the step-wise evaluator, the step requires no change. "CHANGE 1": Since there is no change in change 0, it is clear that there won’t be any changes in the change 1 either and the plan remains unchanged. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch interaction_ids of interactions where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]} }	"CHANGE 0": Based on the output of the step-wise evaluator, the terms "interaction_ids" and "interactions" need to be changed "call_ids" and "calls". "CHANGE 1": The rest of the plan makes perfect sense with this change alone, so no additional change 1 needs to be made. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch call_ids of calls where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]} }
Step-Evaluator Output: On 2nd pass through feedback loop and step to be judged is Step 2	Step: 2. 1. Reasons for customer unhappiness can be found out using T2S as well. So this step should be performed by T2S along with RAG : MULTI-TOOL PROMPT	Step: 2. 1. The step itself is fine, but reasons for customer unhappiness can be found out using the call driver data in T2S as well. So this step should be performed by both the tools. : MULTI-TOOL PROMPT

Table 35: End-to-End Lineage Guided Decomposition Evaluation Trace: Stage-wise Generated Outputs vs. Human-Annotated Plans

Stage	Generated Output	Annotated Output
Plan-Optimizer Output: On 2nd pass through feedback loop. Recieves the input from previous step-evaluator output	"CHANGE 0": As per the step-evaluator output, an additional step needs to be added with the same prompt as that of step 3 but with T2S as the tool. "CHANGE 1": Since we are adding this new step with the same prompt but different tool, we also need to add an LLM aggregation step to combine the outputs from these 2 steps into the final output that the user will receive. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch interaction_ids of interactions where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }	"CHANGE 0": According to the step-evaluator’s feedback, an extra step should be inserted using the same prompt as step 3, but with T2S selected as the tool. "CHANGE 1": Since this new step uses the same prompt but a different tool, an additional LLM aggregation step must be included to merge the outputs from both steps into the final response that will be delivered to the user. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch call_ids of calls where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }
Step-Evaluator Output: On 3rd pass through feedback loop and step to be judged is Step 3	Step: 3 1. The step itself is fine, but reasons for customer unhappiness can be found out using RAG as well. So this step should be performed by both the tools. : MULTI-TOOL PROMPT	Step: 3 1. The step itself is appropriate, but since reasons for customer unhappiness can also be identified using RAG, this step should be executed by both tools. : MULTI-TOOL PROMPT
Plan-Optimizer Output: On 3rd pass through feedback loop. Receives the input from previous step-evaluator output	"CHANGE 0": Looking at the step-evaluator and the old plan, it seems that the step is already being performed by RAG in the previous step. So, nothing needs to be done to the plan and it can remain as it is. "CHANGE 1": Since there is no change 0, we don’t need any change 1 either. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch interaction_ids of interactions where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }	"CHANGE 0": Based on the step-evaluator output and the existing plan, this step is already being handled by RAG in the preceding step. Therefore, no modifications are required, and the plan can remain unchanged. "CHANGE 1": As no change 0 is needed, there is likewise no need for any change 1. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch call_ids of calls where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }

Table 36: End-to-End Lineage Guided Decomposition Evaluation Trace: Stage-wise Generated Outputs vs. Human-Annotated Plans (Contd.)

Stage	Generated Output	Annotated Output
Step-Evaluator Output: On 4th pass through feedback loop and step to be judged is Step 4	Step: 4 1. The step is appropriate and requires no additional changes. : NO CHANGE	Step: 4 1. The step is perfectly fine and executable as it is and does not require any further modifications. : NO CHANGE
Plan-Optimizer Output: On 4th pass through feedback loop. Recieves the input from previous step-evaluator output	"CHANGE 0": As per the step-evaluator output, no adjustment is required. "CHANGE 1": Since there is no change 0, we don’t need any change 1 either. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch interaction_ids of interactions where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }	"CHANGE 0": According to the step-evaluator output, no modifications are necessary. "CHANGE 1": As no change 0 is required, there is likewise no need for a change 1. "NEW PLAN STARTS": { 1: {"step": T2S([], "Fetch call_ids of calls where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }
Final Plan: Gotten upon doing Human Annotation on the feed-back loop output plan. In this case, "interaction_ids" was changed to "call_ids" in step 1 since the query specifically asks for calls. This is a mistake that the feedback-loop failed to fix.	{ 1: {"step": T2S([], "Fetch call_ids of calls where the call driver is related to refunds and duration is longer than 30 minutes."), "depends_on": []}, 2: {"step": RAG((1), "What are customers most unhappy about?"), "depends_on": [1]}, 3: {"step": T2S((1), "What are customers most unhappy about?"), "depends_on": [1]}, 4: {"step": LLM("Combine the following 2 outputs to answer the following query (query):First:(2)Second:(3)"), "depends_on": [2, 3]}, }	N/A
Metric-Wise and One-Shot Evaluator Score for Final Plan

Table 38

Tool-Use Planning

Tool-use planning enables AI agents to choose, order, and coordinate tools to complete tasks reliably and efficiently.

See all 18 papers

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Tool-Augmented Language Model Agents LLM Agent Evaluation
Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology +5
PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents
Sep 28, 2026Huayi Lai, Shichao Song, Qingchen Yu +4Long-Horizon Agent Evaluation AI Agent Benchmarks
Renmin University of China · Beihang University
Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
Oct 13, 2026Mingfei Zhang, Amit Surana, Ziauddin Majid +2Tool Discovery for LLM Agents LLM Tool Use
Apple

Jun 3, 2026J/K move · Enter open · S save

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

Haoyu Sun, Wenxuan Wang, Mingyang Song, Jujie He+4

Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology+5

Tool-Augmented Language Model Agents LLM Agent Evaluation AI Agent Benchmarks LLM Agent Planning LLM Agent Reliability

Open paper AskPDF arXiv

Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200

τ^2

-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in this URL.

Abstract

Tool-Use Planning

Explore similar work

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents

Plan or Retrieve First? A Scale Study of Tool Discovery Architectures