The Default Trap: Rethinking Plan Evaluation in Tool-Using LLM Agents

Xueqi Li; Jingjie Ning; Yibo Kong

cs.AISep 29, 2026

The Default Trap: Rethinking Plan Evaluation in Tool-Using LLM Agents

Authors: Xueqi Li, Jingjie Ning, Yibo Kong

Organizations: Carnegie Mellon University

Abstract

An executor can respond strongly to a change in a supplied plan's priority while showing a small change in the same information-selection probability when a default-aligned whole plan is removed. We call the risk of interpreting the latter as weak responsiveness to alternative priorities the default trap. We compare paired plans that prioritize different information targets with a shared no-plan reference. An accounting identity relates these distinct behavioral contrasts. Across 3,200 decision windows on 160 selected Retail, Airline, and AgentDojo tasks, switching priorities strongly redirects two models' choices, while the two plan-versus-default contrasts differ. In 2,160 additional windows, reversing account-list order shifts default target selection by 63.3-98.3 percentage points; priority-switching effects remain 96.7-100.0 points in either order. A separate 3,240-window component study finds strong control under single priority sentences, with effects of additional text varying by group and direction. Finally, 1,080 full-task episodes yield observed success differences of -19.4 to +8.3 points relative to no plan. All Retail and Airline success intervals include zero; AgentDojo results describe four fixed application worlds. These findings support joint reporting of priority responsiveness, presentation-dependent defaults, and task success and cost.

Figures & tables

Domain / model	n	D A	D B	R A	R B	N A	N B
Retail / Luna	96	90.63	92.97	5.99	91.41	4.69	5.73
Retail / Mini	96	76.82	79.17	5.73	76.30	13.02	13.54
Airline / Luna	28	99.11	99.11	9.82	100.00	1.79	0.00
Airline / Mini	28	92.86	91.96	24.11	91.96	8.93	14.29
AgentDojo / Luna	36	67.36	59.72	18.75	48.61	4.17	5.56
AgentDojo / Mini	36	68.75	65.97	20.83	57.64	5.56	18.06

Table 1: Joint measurements of plan influence.

n

counts tasks.

D

measures priority switching,

R

plan-minus-no-plan contrasts, and

N

repeated-run disagreement; subscripts identify the branch. Effects are percentage points and disagreement a percentage. All endpoint categories remain in the denominator.

		Success (%)			Tool calls
Domain / model	D A	None	A plan	B plan	None	A plan	B plan
Retail / Luna	93.3	80.0	77.8	75.6	6.6	6.7	6.4
Retail / Mini	82.2	75.6	75.6	68.9	6.6	6.5	6.6
Retail / Gemini	97.8	84.4	84.4	80.0	8.1	8.0	7.7
Airline / Luna	94.9	61.5	66.7	59.0	8.4	9.3	8.8
Airline / Mini	94.9	61.5	59.0	64.1	8.5	8.3	7.7

Table 2: Local control and task outcomes in the same episode collection.

D_{A}

is the A-selection probability under A-first minus B-first, in percentage points. None means no plan; success is a percentage; cost is mean attempted tool calls, including failed episodes. Retail, Airline, and AgentDojo contain 15, 13, and 12 tasks, with three repeats per condition. AgentDojo describes four fixed worlds.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Domain / model	p A (P A )	p B (P A )	p A (P B )	p B (P B )	p A (none)	p B (none)
Retail / Luna	90.89	1.04	0.26	94.01	84.90	2.60
Retail / Mini	81.25	2.34	4.43	81.51	75.52	5.21
Airline / Luna	99.11	0.89	0.00	100.00	89.29	0.00
Airline / Mini	95.54	0.00	2.68	91.96	71.43	0.00
AgentDojo / Luna	85.42	6.94	18.06	66.67	66.67	18.06
AgentDojo / Mini	87.50	1.39	18.75	67.36	66.67	9.72

Appendix

Table 3: Measured branch probabilities in percent. The subscript identifies the selected branch;

P_{A}

and

P_{B}

prioritize A and B, and none means no plan. The remaining probability belongs to OTHER, BOTH, ERROR, YIELD, WRITE, or BUDGET.

Domain / model	K	D A range	R A range	Min. G A
Retail / Luna	5	[90.63, 94.01]	[5.84, 7.36]	84.64
Retail / Mini	5	[76.82, 80.31]	[3.17, 5.91]	71.09
Airline / Luna	5	[98.75, 99.11]	[9.82, 13.75]	85.00
Airline / Mini	5	[92.59, 94.58]	[23.15, 26.04]	67.26
AgentDojo / Luna	2	[64.91, 67.36]	[18.75, 20.98]	43.93
AgentDojo / Mini	2	[65.45, 68.75]	[20.83, 24.55]	40.89

Appendix

Table 4: Sensitivity to weighting. Ranges span point estimates in percentage points over

K

schemes defined in Appendix B . The minimum

G_{A}=D_{A}-R_{A}

compares effects under the same weights.

Domain	Model	Default change [95% CI]	D A original / reversed	Interaction [95% CI]
Retail	Luna	96.7 [90.0, 100.0]	100.0 / 100.0	0.0 [0.0, 0.0]
	Mini	63.3 [46.6, 80.0]	100.0 / 100.0	0.0 [0.0, 0.0]
	Gemini	81.7 [68.3, 93.3]	100.0 / 100.0	0.0 [0.0, 0.0]
Airline	Luna	80.0 [62.9, 93.9]	100.0 / 100.0	0.0 [0.0, 0.0]
	Mini	63.3 [44.4, 81.5]	96.7 / 96.7	0.0 [-6.9, 8.8]
	Gemini	98.3 [94.4, 100.0]	100.0 / 100.0	0.0 [0.0, 0.0]

Appendix

Table 5: Paired list-order effects in percentage points. Default change is original minus reversed A selection with no plan. Directional estimates and their order interaction use the same paired tasks. Brackets show nominal 95% customer-cluster confidence intervals (CI); zero-width empirical intervals do not establish population invariance.

Domain / model	Success: B minus A [CI]
Retail / Luna	-2.2 [-6.7, 0.0]
Retail / Mini	-6.7 [-22.2, 6.7]
Retail / Gemini	-4.4 [-13.3, 4.4]
Airline / Luna	-7.7 [-17.9, 2.6]
Airline / Mini	5.1 [-7.7, 15.4]
Airline / Gemini	2.6 [-7.7, 12.8]

Appendix

Table 6: Native success under a change in persistent priority text.

P_{B}

minus

P_{A}

, in percentage points, with nominal 95% paired customer intervals. The Retail/Airline samples contain 15/13 customers; AgentDojo describes 12 tasks in four fixed worlds. Plans remain visible throughout each episode.

Study	Audited	Initial	Adjudicated	Uncertain
Components (E1)	145	140	142	7
List order (E2)	78	77	78	0
Full-task local endpoint (E3)	0	—	—	—

Appendix

Table 7: Scope of model-based local endpoint review. Counts describe an enriched audit sample. Initial and adjudicated columns count agreement with the retained endpoint label; uncertainty is a separate flag.

Domain / model	Full	Sentence	Full minus sentence [CI]
Retail / Luna	86.7	91.7	-5.0 [-13.3, 0.0]
Retail / Mini	80.0	86.7	-6.7 [-15.0, 0.0]
Retail / Gemini	98.3	100.0	-1.7 [-5.0, 0.0]
Airline / Luna	98.3	88.3	10.0 [0.0, 23.3]
Airline / Mini	96.7	88.3	8.3 [-1.7, 21.7]
Airline / Gemini	100.0	100.0	0.0 [0.0, 0.0]

Appendix

Table 8: Component contrasts for

D_{A}

. Values are percentage points; Full and Sentence prioritize the same targets. Each group contains 20 tasks with three repeats per condition. Brackets give nominal 95% paired customer-cluster intervals; AgentDojo is a fixed-world description. All endpoints remain in the denominator.

Domain / model	Full	Sentence	Full minus sentence [CI]
Retail / Luna	86.7	95.0	-8.3 [-18.3, 0.0]
Retail / Mini	80.0	90.0	-10.0 [-20.0, -1.7]
Retail / Gemini	100.0	100.0	0.0 [0.0, 0.0]
Airline / Luna	98.3	93.3	5.0 [-3.3, 15.0]
Airline / Mini	91.7	81.7	10.0 [-3.3, 25.0]
Airline / Gemini	100.0	100.0	0.0 [0.0, 0.0]

Appendix

Table 9: Component contrasts for

D_{B}

Domain / model	A selection change [CI]	B selection change [CI]
Retail / Luna	-6.7 [-18.3, 0.0]	5.0 [0.0, 15.0]
Retail / Mini	1.7 [-5.0, 8.3]	-1.7 [-10.0, 8.3]
Retail / Gemini	8.3 [0.0, 18.3]	-8.3 [-18.3, 0.0]
Airline / Luna	10.0 [-5.0, 26.7]	3.3 [0.0, 10.0]
Airline / Mini	26.7 [8.3, 46.7]	0.0 [0.0, 0.0]
Airline / Gemini	5.0 [0.0, 15.0]	0.0 [0.0, 0.0]

Appendix

Table 10: Neutral-plan package minus no plan. A/B selection changes in percentage points, with nominal 95% customer intervals. The neutral package changes shared guidance, reminders, and length together. AgentDojo has no cross-world interval.

Explore similar work

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
Sep 29, 2026Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera +2Large Language Model Planning Large Language Model Agents
ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg +2
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
May 28, 2026Alejandra Zambrano, Sara Vera Marjanovic, Imene Kerboua +2Large Language Model Planning Web Agents
Concordia University · Mila - Quebec AI Institute · University of Copenhagen +2
PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents
Sep 28, 2026Huayi Lai, Shichao Song, Qingchen Yu +4Large Language Model Agents
Renmin University of China · Beihang University

Sep 29, 2026, cs.AIJ/K move · Enter open · S save

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera, Marcos López de Prado+1

ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg+2

Large Language Model Planning Large Language Model Agents Planning Long-Horizon Task Planning

Open paper AskPDF arXiv

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner--executor systems can fail at either stage, while final task success alone cannot distinguish selection from execution failures. We therefore study the Plan Declaration--Execution Gap and introduce Planning-as-Routing, where an LLM declares one of four planning modes: Predefined, Sequential, Hierarchical, or Search, and a deterministic router dispatches the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve declared planning structure, especially for longer plans: across three benchmarks, only (22)--(45%) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from execution: pattern-specific executors improve task success from (0.48) to (0.92) on ALFWorld and from (0.36) to (0.44) on SWE-bench Verified over Plan+ReAct. Current LLMs, however, do not reliably select the strongest mode for each task, although few-shot examples improve selection in some benchmark--model combinations. Overall, reliable agent planning requires both effective mode selection and faithful execution: routing substantially closes the execution gap, while task-specific mode selection remains open.

Abstract

Explore similar work

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents

PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents