Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Tomoya Kawabe; Rin Takano

cs.ROFeb 25, 2026

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Authors: Tomoya Kawabe, Rin Takano

Organizations: Data Science Laboratories, NEC Corporation, 1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan.

Abstract

Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot teams. Conventional Planning Domain Definition Language (PDDL) planners provide rigorous guarantees but struggle to handle ambiguous or long-horizon missions, while large language models (LLMs) can interpret instructions and propose plans but may hallucinate or produce infeasible actions. We present a hierarchical multi-agent LLM-based planner with prompt optimization: an upper layer decomposes tasks and assigns them to lower-layer agents, which generate PDDL problems solved by a classical planner. When plans fail, the system applies TextGrad-inspired textual-gradient updates to optimize each agent's prompt and thereby improve planning accuracy. In addition, meta-prompts are learned and shared across agents within the same layer, enabling efficient prompt optimization in multi-agent settings. On the MAT-THOR benchmark, our planner achieves success rates of 0.95 on compound tasks, 0.84 on complex tasks, and 0.60 on vague tasks, improving over the previous state-of-the-art LaMMA-P by 2, 7, and 15 percentage points respectively. An ablation study shows that the hierarchical structure, prompt optimization, and meta-prompt sharing contribute roughly +59, +37, and +4 percentage points to the overall success rate.

Figures & tables

Fig. 1 : Multi-robot task planning in home environments. Forward step : Decomposing long-horizon tasks using a hierarchical LLM agent structure. Feedback step : Optimizing LLM agent prompts based on results verified by the planner.

Fig. 2 : Overview of the hierarchical MAP framework. The red flows indicate prompt optimization for each agent , while the blue flows denote meta-prompt optimization across layers .

Step	Description
Inputs	Task instruction $u$ ; agents $\mathcal{E}$ with prompts $\theta_{E_{l,i}}$ and meta-prompts $\hat{\theta}_{l}$ ; maximum iterations $K_{\max}$ .
Data structures	Plan list $\Phi$ , task map $\Psi$ , set of sub-plans (PDDL specifications).
Initialization	Set iteration counter $k\!\leftarrow\!0$ , $\Phi\!\leftarrow\![E_{0,0}]$ , $\Psi(E_{0,0})\!\leftarrow\!u$ .
Top-down reasoning	For each layer $l=0$ to $L-1$ : each agent $E_{l,i}$ decomposes its task into subtasks; leaf agents generate PDDL problems.
Classical planning	Each PDDL problem is solved with a planner; success/failure recorded.
Replanning	Failed agents climb hierarchy by LLM decision (“self” vs. “parent”).

TABLE I : Algorithm overview

Method	Compound				Complex				Vague
Method	SR	GCR	RU	Eff	SR	GCR	RU	Eff	SR	GCR	RU	Eff
CoT [ 23 ] (GPT-4o)	0.32	0.40	0.72	0.59	0.00	0.12	0.47	0.38	0.00	0.00	0.00	0.00
SMART-LLM [ 6 ] (GPT-4o)	0.70	0.82	0.78	0.64	0.20	0.33	0.65	0.56	0.06	0.42	0.68	0.42
LaMMA-P [ 13 ] (GPT-4o)	0.93	0.94	0.91	0.90	0.77	0.83	0.87	0.67	0.45	0.48	0.71	0.65
Ours (GPT-4o)	0.95	0.95	1.00	0.90	0.84	0.84	1.00	0.75	0.60	0.60	1.00	0.75

TABLE II : Performance comparison on MAT-THOR. Higher values are better for all metrics.

Variant	SR	Time
Full method (H+P+M)	0.84	173 s
–H (single-LLM agent)	0.25 (-59.3%)	140 s (-18.9%)
–M (no meta-prompt optimization)	0.80 (-4.2%)	116 s (-33.1%)
–(P, M) (no prompt optimization)	0.47 (-37.0%)	32 s (-81.3%)

TABLE III : Ablation study on our method (GPT-4o). Values are averages over the entire dataset. Both SR (success rate) and Time are reported relative to the full method (H+P+M).

Explore similar work

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams
Mar 9, 2026Piyush Gupta, Sangjae Bae, Jiachen Li +1Large Language Model-Based Robot Planning Symbolic Planning
Honda Research Institute, San Jose, CA 95134, USA. · Georgia Institute of Technology, Atlanta, GA 30332, USA.
HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning
Jun 24, 2026Junho Lee, Seabin Lee, Wonjong Lee +3Large Language Model-Based Robot Planning Robot Task Planning
Dept. of Electronic Engineering, Sogang University, Seoul, Korea
Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation
Mar 6, 2026Kai Göbel, Pierrick Lorang, Patrik Zips +1LLM Planning LLM Agent Planning
AIT Austrian Institute of Technology GmbH, Vienna, Austria

Mar 9, 2026J/K move · Enter open · S save

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams

Piyush Gupta, Sangjae Bae, Jiachen Li, David Isele

Honda Research Institute, San Jose, CA 95134, USA. · Georgia Institute of Technology, Atlanta, GA 30332, USA.

Large Language Model-Based Robot Planning Symbolic Planning Multi-Robot Task Allocation Multi-Robot Motion Planning

Open paper AskPDF arXiv

Long-horizon task planning for heterogeneous multi-robot systems is essential for deploying collaborative teams in real-world environments; yet, it remains challenging due to the large volume of perceptual information, much of which is irrelevant to task objectives and burdens planning. Traditional symbolic planners rely on manually constructed problem specifications, limiting scalability and adaptability, while recent large language model (LLM)-based approaches often suffer from hallucinations and weak grounding-i.e., poor alignment between generated plans and actual environmental objects and constraints-in object-rich settings. We present Scale-Plan, a scalable LLM-assisted framework that generates compact, task-relevant problem representations from natural language instructions. Given a PDDL domain specification, Scale-Plan constructs an action graph capturing domain structure and uses shallow LLM reasoning to guide a structured graph search that identifies a minimal subset of relevant actions and objects. By filtering irrelevant information prior to planning, Scale-Plan enables efficient decomposition, allocation, and long-horizon plan generation. We evaluate our approach on complex multi-agent tasks and introduce MAT2-THOR, a cleaned benchmark built on AI2-THOR for reliable evaluation of multi-robot planning systems. Scale-Plan outperforms pure LLM and hybrid LLM-PDDL baselines across all metrics, improving scalability and reliability. Project website: https://github.com/honda-research-institute/Scale_Plan

Abstract

Explore similar work

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams

HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning

Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation