From Dialogue to Execution: Mixture-of-Agents Assisted Interactive Planning for Behavior Tree-Based Long-Horizon Robot Execution

Kanata Suzuki; Kazuki Hori; Haruka Miyoshi; Shuhei Kurita; Tetsuya Ogata

cs.ROMar 1, 2026

From Dialogue to Execution: Mixture-of-Agents Assisted Interactive Planning for Behavior Tree-Based Long-Horizon Robot Execution

Authors: Kanata Suzuki, Kazuki Hori, Haruka Miyoshi, Shuhei Kurita, Tetsuya Ogata

Organizations: Waseda University, Tokyo 169-8050, Japan. · NII Research and Development Center for Large Language Models. · The Graduate University for Advanced Studies (SOKENDAI), Kanagawa 240-0193, Japan. · National Institute of Informatics (NII), Tokyo 101-8430, Japan.

Abstract

Interactive task planning with large language models (LLMs) lets robots generate high-level action plans from natural language, but over long horizons it asks many questions, and tabular plan representations become hard to manage. We propose a framework that integrates Mixture-of-Agents (MoA)-based proxy answering into interactive planning and generates Behavior Trees (BTs) for structured long-term execution. We formulate the MoA as an abstention-based delegation cascade: each expert agent answers only the questions entailed by its own prerequisite description, forwards the rest unchanged, and the human user acts as the terminal fallback. The question set is thus partitioned disjointly, so no answer fusion or arbitration is required while every question is still resolved. The BT represents task logic hierarchically and enables retry and dynamic switching among robot policies. Experiments on a cocktail-making task show that the method removes approximately 27% of the human responses while keeping the generated BTs within the baseline generator's own variance. Real-robot experiments on a smoothie-making task further demonstrate successful long-horizon execution with adaptive policy switching and recovery from action failures. We further analyze the failure modes of the framework and show that its applicability boundary is set by the reliability of the weakest action node rather than by the planner. These results indicate that MoA-assisted interactive planning improves dialogue efficiency while preserving execution quality in real-world robotic tasks.

Figures & tables

Fig. 1: Overview of the proposed framework. A task instruction is turned into a BT, whose uncertainty the LLM analyzes to produce clarification questions. The three expert boxes form the cascade of Sec. III : agent $e_{k}$ holds a prerequisite description $P_{k}$ —the robot’s functional specification, the task-domain procedure, and everyday knowledge, respectively—and answers only what $P_{k}$ entails, forwarding the rest to the next box and finally to the human, the terminal fallback. Each action node of the refined BT is then bound to a learned policy and the tree is executed on a real robot.

TABLE I: Design comparison with multi-agent LLM architectures
Architecture	Aggregation of agent outputs	Abstention	LLM calls per turn	Human role
Self-consistency [ 22 ]	Majority vote over samples	No	$N$ samples	None
Multi-agent debate [ 21 ]	Iterative mutual critique	No	$N\times$ rounds	None
Layered MoA [ 20 ]	Aggregator LLM synthesizes proposals	No	proposers $\times$ layers $+\,1$	None
Role-based agents [ 29 ] [ 30 ]	Role-specific message passing	No	Task dependent	None
Ours	Not required (disjoint partition)	Yes	$\leq K$ (early exit)	Terminal fallback

Table 2

Fig. 2: Prompt design for the three expert agents, with the wording of the deployed prompts left as issued. The #Prerequisites block ( #Bartender Manual for the Task Domain Expert) is the prerequisite description

P_{k}

that the answerability predicate

\alpha_{k}

is evaluated against: each agent answers only what its own

P_{k}

entails and forwards the rest. Sentences common to all three prompts are shown once at the top, and the two long domain blocks are truncated at […] .

TABLE II: Normalized TED against BTs w/o MoA ( $\downarrow$ )
		A	B	C	Ave.
w/o MoA	A	-	0.807	0.817
	B	0.807	-	0.776	0.800
	C	0.817	0.776	-
w/ MoA	A’	0.813	0.774	0.808
	B’	0.828	0.750	0.762	0.812

Table 4

TABLE III: Node similarity against BTs w/o MoA ( $\uparrow$ )
		A	B	C	Ave.
w/o MoA	A	-	0.640	0.595
	B	0.736	-	0.618	0.664
	C	0.731	0.666	-
w/ MoA	A’	0.757	0.726	0.664
	B’	0.724	0.677	0.674	0.697

Table 5

Fig. 4: BT generated through MoA-assisted interactive planning for the smoothie-making task, with example question–answer interactions. Robot-related questions are answered by the Robot Expert, user-preference questions by the human, and action nodes are assigned either Diffusion Policy or

\pi_{0.5}

Fig. 5: Real-robot execution sequence guided by the generated BT, switching between $\pi_{0.5}$ and Diffusion Policy according to the action node.

TABLE IV: Success rates of actions (10 trials)
	(a) DP ∗	(b) $\pi_{0.5}$	(c) $\pi_{0.5}$
Action		subtask (i)	all subtasks
Insert strawberry	0/10	3/10	0/10
Insert banana	0/10	6/10	6/10
Insert kiwi	0/10	6/10	2/10
Close blender lid	7/10	—	0/10

Table 8

TABLE V: Failure modes, applicability boundaries, and mitigation directions
Failure mode	Evidence	Boundary	Mitigation direction
Grasp failure on variable objects	Strawberry 3/10 with best policy; DP 0/10 on all fruits	High pose and shape variability	Allocate demonstrations by object variance
Cross-task interference	Jointly trained $\pi_{0.5}$ : lid 0/10, switches 1–2/10	One policy over heterogeneous subtasks	Widen scope only with a matching demo budget
Horizon amplification	Estimated end-to-end success $\approx 0.34$	Long sequences with a weak node	Estimate at planning time; escalate weak nodes
Unchecked proxy answer	One agent per question; no second opinion	Stale or incorrect prerequisite $P_{k}$	Ground $P_{k}$ in a machine-readable platform spec
Over-answering preference questions	Not observed in Exp. 2	Factual and preference questions overlap	Batch confirmation before BT generation

Table 9

Mixture-of-Agents

See all 9 papers

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
Sep 29, 2026Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera +2LLM Agents Long-Horizon LLM Agents
ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg +2
Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework
Sep 11, 2026Kazi Abrar Mahmud, Nilotpaul Kundu Dhurubo, Tamal Kirttonia +2Long-Horizon LLM Agents LLM Agent Reliability
Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1000, Bangladesh · University of Massachusetts Amherst, 300 Massachusetts Ave, Amherst, MA 01003, United States
Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning
Feb 25, 2026Tomoya Kawabe, Rin TakanoLarge Language Model-Based Robot Planning Multi-Agent LLM Systems
Data Science Laboratories, NEC Corporation, 1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan.

Sep 29, 2026J/K move · Enter open · S save

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera, Marcos López de Prado+1

ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg+2

LLM Agents Long-Horizon LLM Agents LLM Agent Planning LLM Agent Routing

Open paper AskPDF arXiv

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner--executor systems can fail at either stage, while final task success alone cannot distinguish selection from execution failures. We therefore study the Plan Declaration--Execution Gap and introduce Planning-as-Routing, where an LLM declares one of four planning modes: Predefined, Sequential, Hierarchical, or Search, and a deterministic router dispatches the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve declared planning structure, especially for longer plans: across three benchmarks, only (22)--(45%) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from execution: pattern-specific executors improve task success from (0.48) to (0.92) on ALFWorld and from (0.36) to (0.44) on SWE-bench Verified over Plan+ReAct. Current LLMs, however, do not reliably select the strongest mode for each task, although few-shot examples improve selection in some benchmark--model combinations. Overall, reliable agent planning requires both effective mode selection and faithful execution: routing substantially closes the execution gap, while task-specific mode selection remains open.

Abstract

Mixture-of-Agents

Explore similar work

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning