Persistent LLM Orchestration of Heterogeneous Robot Teams via Partial Delegation to Multi-Agent Protocols

Maria G. Mendoza; Victoria M. Tuck; Zachary Ravichandran; Shankar Sastry; George J. Pappas

cs.MAOct 10, 2026

Persistent LLM Orchestration of Heterogeneous Robot Teams via Partial Delegation to Multi-Agent Protocols

Authors: Maria G. Mendoza, Victoria M. Tuck, Zachary Ravichandran, Shankar Sastry, George J. Pappas

Organizations: Department of Mechanical Engineering (M.G.M.) and Department of Electrical Engineering and Computer Sciences (S.S.), University of California, Berkeley, CA, USA. · Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA.

Abstract

Natural language specifications let non-expert users task robot teams, and large language models (LLMs) have recently proven effective at grounding such semantic specifications into plans for heterogeneous teams. Existing approaches assume missions that terminate and environments that, while unknown, do not change, but both assumptions fail in many emerging applications. For example, a disaster-response mission does not terminate, and its environment changes continuously as hazards spread and evacuees move. We identify three ways an LLM orchestrator breaks in this setting: its prompt grows without bound, its reasoning degrades as agents and targets accumulate, and it stops actively monitoring after covering the search space once. We build on the successful framework of grounded generative intelligence, and address each issue with a classical idea. A Markovian state summary, rebuilt from the semantic map at every invocation, replaces the growing interaction history. Partial delegation lets the orchestrator assign a subset of robots to an autonomous multi-agent protocol and reserve its reasoning for generating subtasks that require detailed parameters and careful ordering. Lastly, observation staleness, surfaced explicitly in the prompt, together with task-and event-triggered re-prompting, keeps the team vigilant. In a simulated urban search-and-rescue mission with spreading fire and moving evacuees, our framework rescues 98% of civilians versus 89% for the strongest baseline while using six times fewer tokens. Additional ablations isolate the contribution of each mechanism.

Figures & tables

Fig. 1: System architecture. The mission statement, team specification, including task primitives and multi-agent protocols, and a partial semantic map are provided as inputs. The LLM is prompted with these inputs and the prompt information of Sec. IV-D to delegate a subset of the robots to the multi-agent protocols and generate subtasks for the rest according to the system needs. This plan is validated, and the LLM is provided feedback if there are issues. The actual task assignment is done symbolically with a linear program (LP) assignment. Multi-agent Protocols (green) are shown to be continuously operating with the self-loop or similar to tasks in that they have an endpoint. Asymmetric observations from all operations are constantly fused into the map. The feedback mechanism is triggered on task completion (solid lines) or on relevant map update events (dashed lines). Feedback on the map and observation age (orange) is compiled with a state summary of the history (blue) and provided in the new prompt to the LLM.

Failure mode	Mechanism
Time-varying environment Exogenous, endogenous, and epistemic changes	Invocation triggered by changes to objects of interest (Sec. IV-F )
Memory Accumulated history grows and describes outdated states, distracting the orchestrator	Markovian input, rebuilt from the current map at every invocation (Sec. IV-D )
Dynamic objects Asymmetric observation aliases distinct objects, making data association ambiguous	Pending tasks and failures trigger mechanism (Sec. IV-F )
Scale Planning degrades as team, targets, and map size grow	Delegation to a multi-robot protocol (Sec. IV-C ); assignment by linear program (Sec. IV-E )
Mission persistence The mission has no completion signal, and the orchestrator has no reason to revisit regions	Observation age summarized in the state: mean age and the regions observed least recently (Sec. IV-G )

TABLE I: Failure modes and the proposed mechanisms

Fig. 4: A $10\times 10$ urban environment for disaster response, shown at selected steps. Dark regions are city blocks, which ground robots cannot enter; the remaining regions form the street network. Fire ignites at two regions (red blocks with stars) and spreads to neighbors over the course of the mission. Groups of people appear at street regions, vary in density, and move between steps; they must be escorted to the safe zone in the upper left (green). Robots are omitted for clarity.

Vehicle type	Movement	Speed (m/s)	Sensor Range (m)
Aerial (UAV)	survey	4.2	150
	go_to, local_search	2.9	50
	escort	1.4	50
Ground (UGV)	go_to, local_search	2.9	50
	escort	1.4	50

TABLE II: Robot properties.

		Configuration			Results
	Name	History Repr.	Multi-Agent Protocols	Re-prompt Trigger	Time to rescue (k) $\downarrow$	Rescues (%) $\uparrow$	# Orchestrator calls $\downarrow$	Total tokens ( $\times 10^{6})$ $\downarrow$
Comparisons	SPINE-HT [ 3 ]	Full	No	Task	$106.9\pm 23.8$	$89.4\pm 12.8$	$33.6\pm 12.5$	$2.40\pm 1.25$
	CoMuRoS [ 21 ]	Full	No	Event	$90.1\pm 35.8$	$32.5\pm 19.0$	$14.2\pm 6.1$	$0.45\pm 0.35$
	Ours	Summ.	Yes	All	$102.3\pm 7.7$	$98.1\pm 2.8$	$32.4\pm 6.1$	$0.41\pm 0.07$
Ablations	History	Full	Yes	All	$105.2\pm 15.0$	$83.1\pm 16.3$	$25.8\pm 3.6$	$1.16\pm 0.27$
	Re-Prompt	Summ.	Yes	Cadence	$137.3\pm 17.4$	$95.6\pm 4.7$	$10.0\pm 0.0$	$0.14\pm 0.01$

TABLE III: Experimental results across configurations. Values are mean

\pm

std.

Fig. 5: The orchestrator reasoning about moving humans under partial information. Left: at $k=34$ people are reported in r6_c0 and r7_c0 , with an escort under way to r7_c0 . Right: by $k=41$ both regions have been re-observed clear and two civilians are reported in r6_c1 . The orchestrator treats the re-observation as evidence the pickups are stale, cancels the escort, issues a local_search around each vacated region to recover the missing group, and escorts the live report at r6_c1 .

Fig. 6: Total tokens used across ablations. We show that for the same triggering mechanism, the stateless prompt (which does not re-send the prompt history) uses fewer overall tokens though more output tokens. These output tokens come from additional ”thinking” tokens the LLM needs.

Fig. 7: Effect of exposing observation staleness to the orchestrator. With staleness in the prompt (green), the orchestrator sees $\max\text{age}=202$ and a median of 83, assigns all three aerial robots on coverage, and re-observes the region later in the run, recovering the last group of humans. Without it (blue), coverage looks complete ( $\text{unobserved}=0$ ), the mandatory-H constraint is dropped, aerial robots are released to other work, and the group is never found and the map is only covered once.

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 462 papers

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning
Feb 25, 2026Tomoya Kawabe, Rin TakanoLarge Language Model-Based Robot Planning Multi-Agent LLM Systems
Data Science Laboratories, NEC Corporation, 1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan.
Prompting Robot Teams with Natural Language
Sep 29, 2025Eduardo Sebastián, Nicolas Pfitzer, Ajay Shankar +1Large Language Model-Based Robot Planning Multi-Robot Coordination
Department of Computer Science and Technology, University of Cambridge, UK · Department of Mechanical and Process Engineering, ETH Zurich, Switzerland
DynaHMRC: Decentralized Heterogeneous Multi-Robot Collaboration for Dynamic Tasks with Large Language Models
Jun 12, 2026Wenhao Yu, Yu'ang Xie, Yifan Duan +5Multi-Agent LLM Systems Multi-Robot Coordination
University of Science and Technology of China (USTC) · University of Macau (UM)

Feb 25, 2026J/K move · Enter open · S save

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Tomoya Kawabe, Rin Takano

Data Science Laboratories, NEC Corporation, 1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan.

Large Language Model-Based Robot Planning Multi-Agent LLM Systems Robotics Prompt Optimization Multi-Robot Motion Planning

Open paper AskPDF arXiv

Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot teams. Conventional Planning Domain Definition Language (PDDL) planners provide rigorous guarantees but struggle to handle ambiguous or long-horizon missions, while large language models (LLMs) can interpret instructions and propose plans but may hallucinate or produce infeasible actions. We present a hierarchical multi-agent LLM-based planner with prompt optimization: an upper layer decomposes tasks and assigns them to lower-layer agents, which generate PDDL problems solved by a classical planner. When plans fail, the system applies TextGrad-inspired textual-gradient updates to optimize each agent's prompt and thereby improve planning accuracy. In addition, meta-prompts are learned and shared across agents within the same layer, enabling efficient prompt optimization in multi-agent settings. On the MAT-THOR benchmark, our planner achieves success rates of 0.95 on compound tasks, 0.84 on complex tasks, and 0.60 on vague tasks, improving over the previous state-of-the-art LaMMA-P by 2, 7, and 15 percentage points respectively. An ablation study shows that the hierarchical structure, prompt optimization, and meta-prompt sharing contribute roughly +59, +37, and +4 percentage points to the overall success rate.

Abstract

LLM Agent Orchestration

Explore similar work

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Prompting Robot Teams with Natural Language

DynaHMRC: Decentralized Heterogeneous Multi-Robot Collaboration for Dynamic Tasks with Large Language Models