Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Subba Reddy Oota; Francisco Herrera; Jordi Cabot Sagrera; Marcos López de Prado; Shadab Khan

cs.AISep 29, 2026

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Authors: Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera, Marcos López de Prado, Shadab Khan

Organizations: ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg · Cornell University, Ithaca, USA · Lawrence Berkeley National Laboratory, Berkeley, CA

Abstract

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner--executor systems can fail at either stage, while final task success alone cannot distinguish selection from execution failures. We therefore study the Plan Declaration--Execution Gap and introduce Planning-as-Routing, where an LLM declares one of four planning modes: Predefined, Sequential, Hierarchical, or Search, and a deterministic router dispatches the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve declared planning structure, especially for longer plans: across three benchmarks, only (22)--(45%) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from execution: pattern-specific executors improve task success from (0.48) to (0.92) on ALFWorld and from (0.36) to (0.44) on SWE-bench Verified over Plan+ReAct. Current LLMs, however, do not reliably select the strongest mode for each task, although few-shot examples improve selection in some benchmark--model combinations. Overall, reliable agent planning requires both effective mode selection and faithful execution: routing substantially closes the execution gap, while task-specific mode selection remains open.

Figures & tables

Figure 1: Overview of the three execution conditions. (a) Flat ReAct : a single ReAct-style executor selects actions from environment feedback without an explicit plan. (b) Planner–Executor (Plan+ReAct) : a planner first selects a planning mode and produces a plan, which is then passed to a generic ReAct executor; however, the declared plan is not structurally enforced during execution. (c) Planning-as-Routing : the LLM first declares a planning mode, and a deterministic router dispatches the task to a matching pattern-specific executor: predefined, sequential, hierarchical, or search. A verifier checks whether execution preserves the declared plan structure.

Benchmark	Model	Overall	Avg. steps	Structure maintained by declared plan length
				$\mathbf{\leq 3}$	4–5	6–7	8–10	$\mathbf{>10}$
ALFWorld	DeepSeek-V4	$27.4{\pm}5.1$	$6.2{\pm}0.3$	$53.3{\pm}37.7$	$36.7{\pm}7.0$	$21.4{\pm}3.0$	$0.0{\pm}0.0^{\ddagger}$	$0.0{\pm}0.0^{\ddagger}$
	Qwen3.6-35B	$23.7{\pm}2.7$	$6.1{\pm}0.3$	$66.7{\pm}27.2$	$36.5{\pm}1.3$	$9.5{\pm}3.9$	$1.2{\pm}1.7$	$0.0{\pm}0.0^{\ddagger}$
	Gemma-4-26B	$21.5{\pm}1.9$	$7.6{\pm}0.2$	$91.7{\pm}8.3$	$49.6{\pm}2.2$	$4.8{\pm}4.8$	$0.0{\pm}0.0$	$0.0{\pm}0.0$
Mind2Web	DeepSeek-V4	$44.6{\pm}0.9$	$4.8{\pm}0.0$	$73.3{\pm}0.5$	$39.2{\pm}1.4$	$31.2{\pm}1.7$	$10.0{\pm}3.4$	$5.9{\pm}3.9$
	Qwen3.6-35B	$36.1{\pm}0.9$	$4.8{\pm}0.0$	$62.4{\pm}1.8$	$31.5{\pm}0.3$	$17.9{\pm}1.1$	$8.6{\pm}1.4$	$4.2{\pm}3.1$

Table 1: Plan-structure maintenance under generic Plan+ReAct, overall and by declared plan length. Overall is the percentage of trajectories preserving the declared structure; Avg. steps is the mean number of declared plan units. Pattern-specific executors are omitted because they enforce their execution structure by design. Values are mean percentages

\pm

SD across seeds 7, 13, and 42. ‡ Sparse bins should be interpreted cautiously.

Figure 2: Qualitative examples of plan-structure preservation. (a) A Plan+ReAct trajectory that maintains the declared hierarchical structure: declared units are reached in order despite intervening actions. (b) A Plan+ReAct trajectory that does not maintain the declared predefined structure, skipping two plan units before continuing with later ones. (c) A routed pattern-specific hierarchical executor, where the declared tree directly determines the dispatch sequence. Green denotes maintained/dispatched plan units and red denotes declared units that are not reached.

Pattern	Model	Planned / Task	Completed / Task	Adherence	Full Plan Completed (%)
Predefined	DeepSeek-V4	$5.917{\pm}0.023$	$5.167{\pm}0.040$	$0.871{\pm}0.009$	$58.7{\pm}1.1$
	Qwen3.6-35B	$5.867{\pm}0.132$	$5.053{\pm}0.235$	$0.864{\pm}0.018$	$54.0{\pm}3.5$
	Gemma-4-26B	$6.243{\pm}0.040$	$5.797{\pm}0.121$	$0.930{\pm}0.023$	$72.9{\pm}7.5$
Sequential	DeepSeek-V4	$4.070{\pm}0.056$	$4.070{\pm}0.056$	$1.000{\pm}0.000$	$100.0{\pm}0.0$
	Qwen3.6-35B	$4.023{\pm}0.057$	$4.023{\pm}0.057$	$1.000{\pm}0.000$	$100.0{\pm}0.0$
	Gemma-4-26B	$3.810{\pm}0.108$	$3.810{\pm}0.108$	$1.000{\pm}0.000$	$100.0{\pm}0.0$

Table 2: Plan adherence on ALFWorld under the fixed execution budget, evaluated on 134 tasks with three random seeds per model. Values are mean

\pm

standard deviation across seeds. Adherence is the fraction of declared executable plan units completed within the allocated budget; Full Plan is the percentage of tasks for which all declared units were completed. Hierarchical planning is evaluated over executable leaf nodes.

Bench	Model	$N$	SEQ	PRED	HIER	SEARCH	DSR max	$H$	DSR max w/o Search	$H$ w/o Search
ALFWorld	DeepSeek-V4	134	$0.633{\pm}0.020$	$0.556{\pm}0.008$	$0.840{\pm}0.009$	$\mathbf{0.918{\pm}0.026}$	$0.953{\pm}0.013$	$0.035{\pm}0.019$	$0.898\pm 0.004$	$0.057\pm 0.013$
	Qwen3.6-35B	134	$0.662{\pm}0.027$	$0.550{\pm}0.023$	$0.714{\pm}0.037$	$\mathbf{0.796{\pm}0.004}$	$0.925{\pm}0.016$	$0.129{\pm}0.020$	$0.843\pm 0.030$	$0.129\pm 0.007$
	Gemma-4-26B	134	$0.550{\pm}0.014$	$0.515{\pm}0.016$	$\mathbf{0.600{\pm}0.013}$	$0.495{\pm}0.058$	$0.841{\pm}0.007$	$0.241{\pm}0.013$	$0.781\pm 0.009$	$0.182\pm 0.021$
Mind2Web	DeepSeek-V4	1341	$0.053{\pm}0.005$	$0.048{\pm}0.003$	$0.041{\pm}0.001$	$\mathbf{0.057{\pm}0.006}$	$0.097{\pm}0.003$	$0.037{\pm}0.005$	$0.087{\pm}0.007$	$0.034{\pm}0.003$
	Qwen3.6-35B	1341	$0.032{\pm}0.001$	$0.036{\pm}0.002$	$0.029{\pm}0.002$	$\mathbf{0.052{\pm}0.001}$	$0.084{\pm}0.004$	$0.031{\pm}0.004$	$0.064{\pm}0.003$	$0.028{\pm}0.002$
	Gemma-4-26B	1341	$0.114{\pm}0.006$	$\mathbf{0.147{\pm}0.005}$	$0.083{\pm}0.002$	$0.086{\pm}0.006$	$0.282{\pm}0.002$	$0.135{\pm}0.006$	$0.253{\pm}0.002$	$0.106{\pm}0.002$

Table 3: Pattern-ceiling analysis across benchmarks.

S(p)

is task success under forced execution of pattern

p

;

\mathrm{DSR}_{\max}

is the per-task oracle; and

H=\mathrm{DSR}_{\max}-\max_{p}S(p)

is the headroom beyond the strongest fixed pattern. Values are mean

\pm

SD across seeds where available.

Mode	Model	PQ	TSR	PQ–success $r$
Predefined	DeepSeek-V4	$2.75{\pm}0.01$	$0.55{\pm}0.00$	$-0.031,\,-0.005$
Predefined	Qwen3.6-35B	$2.44{\pm}0.05$	$0.56{\pm}0.03$	$+0.045,\,+0.095$
Sequential	DeepSeek-V4	$\mathbf{2.84{\pm}0.03}$	$0.64{\pm}0.02$	$-0.025,\,-0.061$
Sequential	Qwen3.6-35B	$2.66{\pm}0.03$	$0.68{\pm}0.01$	$+0.090,\,-0.040$
Hierarchical	DeepSeek-V4	$2.72{\pm}0.02$	$0.85{\pm}0.00$	$-0.060,\,-0.035$
Hierarchical	Qwen3.6-35B	$\mathbf{2.82{\pm}0.00}$	$0.74{\pm}0.03$	$+0.140,\,-0.031$

Table 4: Plan quality (PQ) and task success on ALFWorld. Plans are evaluated by a Gemma-4-26B judge. Forced-pattern results report the mean

\pm

half-range across seeds (

n=134

tasks per seed). The final column reports the task-level Pearson correlation between plan quality and task success separately for seeds 13 and 7. † Plan+ReAct is reported for one seed. Bold indicates the highest PQ or TSR for each model.

Benchmark	Model	Metric	Flat ReAct	Plan+ReAct	Best Fixed $S^{\star}$	Routing@1	Oracle
ALFWorld	DeepSeek-V4	TSR	$0.440{\pm}0.012$	$0.480{\pm}0.009$	$\mathbf{0.918{\pm}0.026}$ (SEARCH)	$0.721{\pm}0.029$	$0.953{\pm}0.013$
	Qwen3.6-35B	TSR	$0.535{\pm}0.007$	$0.540{\pm}0.037$	$\mathbf{0.796{\pm}0.004}$ (SEARCH)	$0.667{\pm}0.049$	$0.925{\pm}0.016$
	Gemma-4-26B	TSR	$0.112{\pm}0.040$	$0.153{\pm}0.026$	$\mathbf{0.600{\pm}0.013}$ (HIER)	$0.560{\pm}0.016$	$0.841{\pm}0.007$
Mind2Web	DeepSeek-V4	TSR	$\mathbf{0.058{\pm}0.003}$	$0.038{\pm}0.005$	$0.057{\pm}0.006$ (SEARCH)	$0.051{\pm}0.004$	$0.097{\pm}0.003$
	DeepSeek-V4	SSR	$\mathbf{0.434{\pm}0.001}$	$0.407{\pm}0.002$	$\mathbf{0.449{\pm}0.000}$ (SEARCH)	$0.425{\pm}0.007$	$0.550{\pm}0.001$
	Qwen3.6-35B	TSR	$0.042{\pm}0.002$	$0.036{\pm}0.004$	$\mathbf{0.052{\pm}0.001}$ (SEARCH)	$0.034{\pm}0.001$	$0.084{\pm}0.004$

Table 5: End-to-end comparison of planning and execution strategies. Best Fixed (

S^{\star}

) is the strongest single planning pattern applied to all tasks; Routing@1 executes the top-1 declared pattern with its corresponding executor; and Oracle is the per-task forced-pattern ceiling. Bold denotes the strongest non-oracle method. Routing@1 uses thinking disabled.

Figure 3: Task success as a function of task length under generic and pattern-specific execution. Pattern-specific execution provides the largest advantage on longer tasks, where Plan+ReAct and Flat ReAct degrade more rapidly. Error bars denote variation across seeds.

Benchmark	Model	Metric	Think	@1	@2	@3	Few-shot@1	$\Delta_{\mathrm{FS}}$	Oracle
ALFWorld	DeepSeek-V4	TSR	Off	$0.721{\pm}0.028$	$0.893{\pm}0.001$	$0.948{\pm}0.011$	$\mathbf{0.793{\pm}0.083}$	$\mathbf{+0.073}$	$0.953{\pm}0.013$
	DeepSeek-V4	TSR	On	$0.656{\pm}0.019$	$0.863{\pm}0.007$	$0.908{\pm}0.012$	$\mathbf{0.812{\pm}0.047}$	$\mathbf{+0.156}$	$0.953{\pm}0.013$
	Qwen3.6	TSR	Off	$0.667{\pm}0.049$	$0.804{\pm}0.028$	$0.873{\pm}0.016$	$\mathbf{0.706{\pm}0.034}$	$\mathbf{+0.040}$	$0.925{\pm}0.016$
	Qwen3.6	TSR	On	$0.674{\pm}0.021$	$0.769{\pm}0.010$	$0.853{\pm}0.031$	$\mathbf{0.704{\pm}0.069}$	$\mathbf{+0.030}$	$0.925{\pm}0.016$
	Gemma-4-26B	TSR	Off	$0.560{\pm}0.016$	$0.744{\pm}0.015$	$0.873{\pm}0.016$	$\mathbf{0.635{\pm}0.015}$	$\mathbf{+0.075}$	$0.841{\pm}0.007$
	Gemma-4-26B	TSR	On	$0.597{\pm}0.012$	$0.744{\pm}0.015$	$0.813{\pm}0.012$	$\mathbf{0.647{\pm}0.060}$	$\mathbf{+0.050}$	$0.841{\pm}0.007$

Table 6: Planning-pattern selection and improvement. @1–@3 report ranked fallback success when declared modes are attempted in order up to rank

k

. Few-shot@1 uses task–pattern demonstrations, and

\Delta

is the absolute gain over the original top-1 declaration. Oracle is the per-task forced-pattern ceiling. Mind2Web reports both TSR and SSR; results are averaged across seeds.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Illustration of three planning executor patterns. (a) Predefined planning : the planner generates a fixed ordered plan and the executor follows the steps without replanning. (b) Sequential planning : the executor runs the current step, observes the result, and invokes a replanner to revise the remaining plan when needed. (c) Hierarchical planning : an orchestrator decomposes the task into subgoals, dispatches them to workers, and synthesizes the final result.

Figure 5: Illustration of search planning pattern. (a) Search-based planning : The planner generates multiple competing candidate plans, and each candidate is executed independently from a reset environment. (b) Rubric judge : A task-specific rubric judge evaluates the resulting trajectory summaries against a rubric generated from the task specification and selects the highest-scoring candidate. The judge has no access to benchmark ground-truth success labels or hidden evaluation scores.

Benchmark	Environment	$N$	Task type	Evaluation
Mind2Web	Web interaction	1,341	Multi-step website interaction	Benchmark task success
WebArena	Web interaction	204	Navigation, retrieval, and website interaction	Benchmark task success
SWE-bench Verified	Software engineering	500	GitHub issue resolution and code modification	Patch-based evaluation
ALFWorld	Embodied environment	134	Multi-step household tasks	Goal completion

Appendix

Table 7: Agent benchmarks used in our experiments.

N

denotes the number of tasks evaluated in this work.

Model	Role	Thinking
Qwen3.6-35B-A3B	Declaration and execution	Enabled
DeepSeek-V4-Flash	Declaration and execution	Enabled
Gemma-4-26B-A4B-it	Declaration and execution	Enabled

Appendix

Table 8: Backbone LLMs used in our experiments.

Benchmark	Group	Tasks	Auxiliary statistic	Additional information
ALFWorld	Pick & Place	24	Mean expert steps: 4.58
	Examine in Light	18	Mean expert steps: 3.78
	Clean & Place	31	Mean expert steps: 6.32
	Heat & Place	23	Mean expert steps: 6.04
	Cool & Place	21	Mean expert steps: 6.10
	Pick Two & Place	17	Mean expert steps: 8.65

Appendix

Table 9: Composition of the evaluation sets used in our experiments. ALFWorld tasks are grouped by task type, with the mean number of expert demonstration steps shown for each category. Mind2Web tasks are grouped by the benchmark’s three non-overlapping generalization splits; together they account for all 1,341 evaluation tasks. SWE-bench Verified instances are grouped by the dataset’s time-to-fix difficulty label.

Model	Temp.	Top- $p$	Top- $k$	Pres. Pen.	Rep. Pen.	Thinking
Qwen3.6-35B-A3B	1.0	0.95	20	1.5	1.0	Yes
Gemma4-26B-A4B-it	1.0	0.95	64	0.0	1.0	Yes
DeepSeek-V4-Flash	1.0	0.95	Disabled	0.0	1.0	Yes

Appendix

Table 10: Sampling parameters used for each backbone LLM. Disabled settings are not sent to the backend.

Model	Model Budget	Default Profile	Hard Profile	Tool Choice
Qwen3.6-35B-A3B	81,920	32,768	81,920	Required
Gemma-4-26B-A4B-it	131,072	32,768	81,920	Required
DeepSeek-V4-Flash	81,920	32,768	81,920	Required

Appendix

Table 11: Per-turn generation budgets and tool-calling configuration. The effective budget is the minimum of the model budget and the selected profile budget, subject to the minimum thinking floor.

Benchmark	Model	Episode mean	Across seeds	Range
ALFWorld	DeepSeek-V4	$3.33\pm 0.47$	$3.34\pm 0.05$	3–4
	Qwen3.6-35B	$3.22\pm 0.45$	$3.22\pm 0.06$	0–4
	Gemma-4-26B	$2.99\pm 0.11$	$2.99\pm 0.01$	2–4
Mind2Web	DeepSeek-V4	$3.47\pm 0.50$	$3.47\pm 0.01$	2–4
	Qwen3.6-35B	$3.35\pm 0.48$	$3.35\pm 0.02$	0–4
	Gemma-4-26B	$3.00\pm 0.03$	$3.00\pm 0.00$	2–3

Appendix

Table 12: Realized number of Search candidates per task. “Episode mean” reports mean

\pm

SD across individual Search episodes, while “Across seeds” reports mean

\pm

SD of the seed-level means. Search permits at most four candidate plans per task.

Benchmark	Max env. steps	max_steps	max_branch	max_depth
ALFWorld	$24$	8	4	3
Mind2Web	$24$	8	4	3
WebArena	$24$	8	4	3
SWE-bench	$24$	8	4	3

Appendix

Table 13: Execution and planning-structure budgets used in each benchmark. All planning-structure limits are fixed across models and random seeds.

Benchmark	Model	Declared mode	$n$	Structure maintained	Steps declared
ALFWorld	DeepSeek-V4	Sequential	$107$ /seed	$30.5{\pm}5.0\%$	$5.3{\pm}0.1$
	DeepSeek-V4	Predefined ‡	$2$	$0.0\%$	$5.5$
	DeepSeek-V4	Hierarchical ‡	$5$ /seed	$11.4{\pm}8.4\%$	$9.5{\pm}0.3$
	DeepSeek-V4	Search ‡	$5$ /seed	$0.0{\pm}0.0\%$	$16.8{\pm}0.9$
	Qwen3.6-35B	Sequential	$89$ /seed	$30.5{\pm}2.2\%$	$5.0{\pm}0.1$
	Qwen3.6-35B	Hierarchical	$28$ /seed	$4.6{\pm}0.8\%$	$9.1{\pm}0.4$

Appendix

Table 14: Plan+ReAct structure preservation by declared planning mode. Hierarchical declarations exhibit consistently low structure maintenance across benchmarks, while Search is also difficult to preserve where enough declarations are available.

n

denotes the smallest per-seed count when multiple seeds are available; values are mean

\pm

standard deviation across seeds. ‡ indicates cells with fewer than 15 declarations in at least one seed and should be interpreted cautiously. Search is included for completeness, but its competing-candidate structure is not directly comparable to a single ordered Sequential, Predefined, or Hierarchical plan.

Model	Planning mode	Planned/task	Completed/task	Adherence	Full plan
DeepSeek-V4	Predefined	$4.597{\pm}0.023$	$2.590{\pm}0.000$	$0.558{\pm}0.002$	$9.2{\pm}0.1\%$
	Sequential	$4.443{\pm}0.012$	$3.543{\pm}0.012$	$0.800{\pm}0.003$	$40.0{\pm}0.7\%$
	Hierarchical	$4.800{\pm}0.010$	$2.637{\pm}0.006$	$0.542{\pm}0.002$	$1.2{\pm}0.4\%$
	Search	$3.467{\pm}0.015$	$3.433{\pm}0.015$	$0.989{\pm}0.001$	$96.5{\pm}0.2\%$
Qwen3.6-35B	Predefined	$4.987{\pm}0.006$	$2.597{\pm}0.015$	$0.521{\pm}0.003$	$6.8{\pm}0.9\%$
	Sequential	$4.540{\pm}0.050$	$3.603{\pm}0.021$	$0.797{\pm}0.006$	$42.5{\pm}2.0\%$

Appendix

Table 15: Mind2Web: Plan adherence under pattern-specific execution. Planned/task and Completed/task report the mean number of needed plan units generated and completed per task, respectively. Adherence measures the fraction of needed plan units completed within the execution budget, while Full plan reports the fraction of trajectories completing all needed units. Values are mean

\pm

standard deviation across seeds 7, 13, and 42. Search operates over competing candidate strategies and its completion values are therefore not directly comparable to the ordered planning patterns.

Model	Planning mode	Planned/task	Completed/task	Adherence	Full plan
DeepSeek-V4	Predefined	$4.790$	$1.410$	$0.297$	$1.0\%$
	Sequential §	$3.770$	$3.770$	$1.000$	$100.0\%$
	Hierarchical	$13.650$	$2.040$	$0.163$	$0.5\%$
	Search	$3.260$	$3.020$	$0.927$	$83.0\%$
Qwen3.6-35B	Predefined	$5.490$	$1.700$	$0.311$	$1.0\%$
	Sequential §	$3.750$	$3.750$	$1.000$	$100.0\%$

Appendix

Table 16: SWE-bench Verified: plan adherence under pattern-specific execution. Columns as in the Mind2Web table. Both models ran at seed 13 only, so no row carries a spread. Hierarchical declares by far the most units (

13.7

and

9.4

per task against

3.3

–

5.5

for the other patterns) and completes the smallest fraction of them. § marks the same Sequential caveat as above.

Figure 6: Distribution of declaration ranks assigned to each planning mode on ALFWolrd, Mind2Web, SWE-Bench Verified and WebArena (right), averaged across three seeds. Each stacked bar shows the fraction of tasks for which a planning mode is assigned Rank 1–4. Results are shown for DeepSeek-V4 and Qwen3.6 with thinking enabled and disabled. The strongly non-uniform distributions reveal systematic mode preferences in the declaration policy.

Benchmark	Model	Metric	Flat ReAct	$3\times$ Flat ReAct + Judge	Flat ReAct Pass@3	Search
ALFWorld	DeepSeek-V4	TSR	$0.440{\pm}0.012$	$0.526{\pm}0.011$	$0.600{\pm}0.015$	$\mathbf{0.918{\pm}0.026}$
ALFWorld	Qwen3.6-35B	TSR	$0.535{\pm}0.007$	$0.664{\pm}0.015$	$0.758{\pm}0.004$	$\mathbf{0.796{\pm}0.004}$
Mind2Web	DeepSeek-V4	TSR	$\mathbf{0.058{\pm}0.003}$	$\mathbf{0.059{\pm}0.004}$	$0.077{\pm}0.002$	$\mathbf{0.057{\pm}0.006}$
	DeepSeek-V4	SSR	$\mathbf{0.434{\pm}0.001}$	$0.448{\pm}0.005$	$0.504{\pm}0.001$	$\mathbf{0.449{\pm}0.000}$
	Qwen3.6-35B	TSR	$0.042{\pm}0.002$	$0.044{\pm}0.000$	$0.069{\pm}0.001$	$\mathbf{0.052{\pm}0.001}$
	Qwen3.6-35B	SSR	$0.366{\pm}0.003$	$0.385{\pm}0.002$	$0.466{\pm}0.002$	$\mathbf{0.411{\pm}0.003}$

Appendix

Table 17: Repeated-rollout ReAct control for Search.

3\times

Flat ReAct executes three independent trajectories from a reset environment and applies the same rubric-based judge used by Search to select one trajectory. Search dynamically chooses between 0 and 4 candidate plans per task, with four as the configured maximum. The realized candidate count averages

3.35\pm 0.49

across Search episodes (Appendix D , Table 12 ). Results in this table report mean

\pm

standard deviation across seeds 7, 13, and 42. This control tests how much of Search’s performance can be reproduced by repeated generic ReAct execution and trajectory selection.

Benchmark	Model	Metric	$n$	$S(p^{\star})$ (Pattern)	$\mathrm{DSR}_{\max}$	$H$	pass@2	pass@3	$H_{\mathrm{retry}}@3$
ALFWorld	DeepSeek-V4	TSR	134	$0.918$ (SEARCH)	$0.953$	$+0.035$	$0.963$	$\mathbf{0.978}$	$\mathbf{-0.025}$
	Qwen3.6-35B	TSR	134	$0.796$ (SEARCH)	$0.925$	$+0.129$	$0.896$	$\mathbf{0.940}$	$\mathbf{-0.015}$
	Gemma-4-26B	TSR	134	$0.600$ (HIER)	$0.841$	$+0.241$	$0.754$	$0.813$	$\mathbf{+0.027}$
Mind2Web	DeepSeek-V4	TSR	1341	$0.057$ (SEARCH)	$0.097$	$+0.040$	$0.073$	$0.083$	$\mathbf{+0.014}$
	DeepSeek-V4	SSR	1341	$0.449$ (SEARCH)	$0.550$	$+0.101$	$0.500$	$0.526$	$\mathbf{+0.024}$
	Qwen3.6-35B	TSR	1341	$0.053$ (SEARCH)	$0.084$	$+0.031$	$0.071$	$0.083$	$\mathbf{+0.001}$

Appendix

Table 18: Retry control for the forced-pattern ceiling. For each benchmark–model pair,

p^{\star}

denotes the strongest fixed planning pattern under the primary task-level metric, and

S(p^{\star})

denotes its corresponding score.

\mathrm{DSR}_{\max}

denotes the per-task forced-pattern oracle over the evaluated planning patterns, and

H=\mathrm{DSR}_{\max}-S(p^{\star})

is the original oracle gap. pass@

k

reports performance when the same fixed pattern

p^{\star}

is executed independently

k

times, with a task counted as successful if any of the

k

executions succeeds.

H_{\mathrm{retry}}@3=\mathrm{DSR}_{\max}-\mathrm{pass@3}(p^{\star})

denotes the residual oracle gap after three executions of the same fixed pattern. Negative values indicate that repeated execution of the fixed pattern exceeds the forced-pattern oracle. For Mind2Web, we report both task success rate (TSR) and step success rate (SSR). For Gemma-4-26B,

p^{\star}

is Predefined because it is the strongest fixed pattern under TSR; the corresponding SSR values therefore use the same Predefined pattern rather than the SSR-maximizing Search pattern.

Benchmark	Model	$n$	$\mathrm{TSR}(\pi)$	$\mathrm{TSR}_{\mathrm{null}}$	$\Delta(\pi)$	95% null interval	$p$ ( $\Delta>0$ )	$p$ (two-sided)	Holm
ALFWorld	DeepSeek-V4	134	$0.721{\pm}0.029$	$0.745{\pm}0.011$	$\mathbf{-0.024{\pm}0.035}$	$[-0.027,+0.026]$	$0.972$	$0.062$	$1.000$
	Qwen3.6-35B	134	$0.667{\pm}0.049$	$0.659{\pm}0.028$	$+0.008{\pm}0.021$	$[-0.017,+0.018]$	$0.226$	$0.406$	$1.000$
	Gemma-4-26B	134	$0.560{\pm}0.016$	$0.553{\pm}0.012$	$+0.007{\pm}0.005$	$[-0.013,+0.014]$	$0.217$	$0.364$	$1.000$
Mind2Web	DeepSeek-V4	1341	$0.053{\pm}0.003$	$0.051{\pm}0.001$	$+0.003{\pm}0.002$	$[-0.004,+0.004]$	$0.094$	$0.180$	$0.944$
	Qwen3.6-35B	1341	$0.036{\pm}0.001$	$0.034{\pm}0.001$	$+0.002{\pm}0.000$	$[-0.003,+0.003]$	$0.126$	$0.251$	$1.000$
	Gemma-4-26B	1341	$0.113{\pm}0.006$	$0.113{\pm}0.007$	$+0.001{\pm}0.000$	$[-0.001,+0.002]$	$0.707$	$0.715$	$1.000$

Appendix

Table 19: Task-blind declaration null.

\mathrm{TSR}(\pi)

denotes task success under the model’s top-1 declared planning mode.

\mathrm{TSR}_{\mathrm{null}}

preserves each model’s overall declaration frequencies while randomly reassigning declarations across tasks. We report

\Delta(\pi)=\mathrm{TSR}(\pi)-\mathrm{TSR}_{\mathrm{null}}

, the 95% interval under 5,000 permutations, one-sided permutation

p

-values for

\Delta>0

, two-sided

p

-values, and Holm-corrected

p

-values.

Figure 7: Inference-token usage across execution patterns on ALFWorld, Mind2Web, SWE-bench, and WebArena. Bars report the mean number of generated tokens per task, separated into thinking tokens (solid) and answer/content tokens (hatched); error bars denote standard deviation across available seeds. Percentages above each bar indicate the fraction of generated tokens used for thinking. Hierarchical and Search generally consume more aggregate tokens, although greater token usage does not consistently correspond to higher task success.

Figure 8: LLM calls per task across execution patterns on ALFWorld, Mind2Web, SWE-bench, and WebArena. Bars report the mean number of LLM invocations per task and error bars denote standard deviation across available seeds. Labels above the bars report the mean generated tokens per LLM call. Hierarchical and Search generally require more model invocations because of decomposition, coordination, and candidate exploration, while the average length of each call does not increase proportionally.

Explore similar work

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Large Language Model Planning Agentic Benchmarks
Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology +5
GATS: Graph-Augmented Tree Search with Layered World Models for Efficient Agent Planning
Jul 9, 2026Maureese Williams, Dymitr NowickiLarge Language Model Planning Classical Planning
Institute for Cybernetics of NAS of Ukraine
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
May 28, 2026Alejandra Zambrano, Sara Vera Marjanovic, Imene Kerboua +2Large Language Model Planning Web Agents
Concordia University · Mila - Quebec AI Institute · University of Copenhagen +2

Jun 3, 2026, cs.CLJ/K move · Enter open · S save

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

Haoyu Sun, Wenxuan Wang, Mingyang Song, Jujie He+4

Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology+5

Large Language Model Planning Agentic Benchmarks Large Language Model Agents Planning Diagnostic Benchmark Diagnostic Framework

Open paper AskPDF arXiv

Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200

τ^2

-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in this URL.

Abstract

Explore similar work

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

GATS: Graph-Augmented Tree Search with Layered World Models for Efficient Agent Planning

Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents