From Plan to Action: How Well Do Agents Follow the Plan?

Shuyang Liu; Saman Dehghan; Jatin Ganhotra; Martin Hirzel; Reyhaneh Jabbarvand

cs.SEApr 13, 2026

From Plan to Action: How Well Do Agents Follow the Plan?

Authors: Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, Reyhaneh Jabbarvand

Organizations: University of Illinois Urbana-Champaign Urbana, USA · IBM New York, USA

Abstract

Agents are commonly instructed to follow a task-specific plan for guidance. However, it is unknown to what extent agents actually follow instructed plans. Without such an analysis, determining the extent agents comply with a given plan, it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliance in programming agents, examining 21,120 trajectories from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on internalized workflows during training, which are often incomplete, overfit, or inconsistently applied. Providing the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and improve task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, inserting additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model's internal problem-solving strategy. These findings call for fine-tuning paradigms that teach models to follow instructed plans, rather than encoding task-specific plans in them, so that they reason and act adaptively, rather than memorizing workflows.

Figures & tables

Figure 1. Illustrative examples of agent trajectories and their corresponding Graphectory and Langutory representations. Plan:

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V}

Plan Setting	Plan Formulation	Plan Variation	Plan Description
Standard (Default) Plan	$\langle$ $N$ , $R$ , $P$ , $V$ $\rangle$	Baseline	Standard Navigation-Reproduction-Patch-Validation plan
No Plan	—	Reduction	Plan removed entirely from the system prompt
Default Plan - Reproduction	$\langle$ $N$ , $\neg{\text{R}}$ , $P$ , $V$ $\rangle$	Reduction	Reproduction phase removed
Default Plan - Validation	$\langle$ $N$ , $R$ , $P$ , $\neg{\text{V}}$ $\rangle$	Reduction	Validation (after patching) phase removed
Default Plan + Regression Test Execution	$\langle$ $R_{G}$ , $N$ , $R$ $P$ , $V$ , $V_{G}$ $\rangle$	Augmentation	Regression test execution phases added
Default Plan + Summary of Changes	$\langle$ $N$ , $R$ , $P$ , $V$ , $S$ $\rangle$	Augmentation	Summarizing changes before submission added

Table 1. Summary of studied plan settings, their corresponding formulation, and type of plan variation.

Figure 2. Standard plan compliance metrics and success rate for studied trajectories across all models.

Figure 3. Phase flow analysis under Standard plan (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V}

). Flow thickness notes the proportion of trajectories going from one phase to another. The black bar indicates trajectory termination, and gray flows represent out-of-plan phases.

Figure 4. Phase flow analysis under No Plan setting. Agents still exhibit traces of Standard plan phases (

\Phi=\{\hbox{\pagecolor[HTML]{CCC7E6}N},\hbox{\pagecolor[HTML]{EEC7D4}R},\hbox{\pagecolor[HTML]{FFED99}P},\hbox{\pagecolor[HTML]{D9EDCC}V}\}

Figure 5. Impact of No Plan setting on the success rate.

Figure 6. Phase flow analysis under No Reproduction (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V}

) and No Validation plan setting (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}

). Trajectories still show traces of removed phases from the Standard plan.

Figure 7. Impact of No Reproduction plan (a) and No Validation plan (b) on the success rate.

Figure 8. No Reproduction plan compliance metrics.

Figure 9. No Validation plan compliance metrics.

Figure 10. Phase flow analysis under Regression Testing (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{B4DDF7}

R_{G}

}\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V}\hbox{\pagecolor[HTML]{E1B28D}

V_{G}

}

) and Summary plan setting (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V}\hbox{\pagecolor[HTML]{808000}S}

Figure 11. Impact of Regression Testing (a) and Summary plan (b) on the success rate.

Figure 12. Regression Testing plan compliance metrics.

Figure 13. Summary plan compliance metrics.

Figure 14. Phase flow analysis under Reordered (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{D9EDCC}V}

) and Reminded setting (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V}

Figure 15. Impact of Reordered plan (a) and Reminded plan (b) on the success rate.

Figure 16. Reordered plan compliance metrics.

Figure 17. Reminded plan compliance metrics.

Figure 18. Phase flow analysis under Standard plan for SWE-bench Pro (

\mathcal{L}^{\star}(\Phi)=\hbox{\pagecolor[HTML]{CCC7E6}N}\hbox{\pagecolor[HTML]{EEC7D4}R}\hbox{\pagecolor[HTML]{FFED99}P}\hbox{\pagecolor[HTML]{D9EDCC}V})

Figure 19. Compliance metrics on SWE-bench Verified deterministic (a) and SWE-bench Pro (b)

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 344 papers

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
Sep 29, 2026Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera +2LLM Agents Long-Horizon LLM Agents
ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg +2
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
May 28, 2026Alejandra Zambrano, Sara Vera Marjanovic, Imene Kerboua +2LLM Agent Evaluation LLM Agent Planning
Concordia University · Mila - Quebec AI Institute · University of Copenhagen +2
Testing the Planning Claims of LLM-Based Agents against Formal Semantics
Oct 15, 2026Oriol Miro-Lopez-Feliu, Oscar Miró López-Feliu, Victor Gimenez-Abalos +3LLM Planning LLM Agent Evaluation
Barcelona Supercomputing Center, Barcelona, Spain · University of Amsterdam, Amsterdam, The Netherlands · Safe AI Netherlands, The Netherlands +1

Sep 29, 2026J/K move · Enter open · S save

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera, Marcos López de Prado+1

ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg+2

LLM Agents Long-Horizon LLM Agents LLM Agent Planning LLM Agent Routing

Open paper AskPDF arXiv

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner--executor systems can fail at either stage, while final task success alone cannot distinguish selection from execution failures. We therefore study the Plan Declaration--Execution Gap and introduce Planning-as-Routing, where an LLM declares one of four planning modes: Predefined, Sequential, Hierarchical, or Search, and a deterministic router dispatches the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve declared planning structure, especially for longer plans: across three benchmarks, only (22)--(45%) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from execution: pattern-specific executors improve task success from (0.48) to (0.92) on ALFWorld and from (0.36) to (0.44) on SWE-bench Verified over Plan+ReAct. Current LLMs, however, do not reliably select the strongest mode for each task, although few-shot examples improve selection in some benchmark--model combinations. Overall, reliable agent planning requires both effective mode selection and faithful execution: routing substantially closes the execution gap, while task-specific mode selection remains open.

Abstract

Software Engineering Agents

Explore similar work

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents

Testing the Planning Claims of LLM-Based Agents against Formal Semantics