Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework

Kazi Abrar Mahmud; Nilotpaul Kundu Dhurubo; Tamal Kirttonia; Sabbir Hossain Ujjal; Mohammad Ariful Haque

cs.ROSep 11, 2026

Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework

Authors: Kazi Abrar Mahmud, Nilotpaul Kundu Dhurubo, Tamal Kirttonia, Sabbir Hossain Ujjal, Mohammad Ariful Haque

Organizations: Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1000, Bangladesh · University of Massachusetts Amherst, 300 Massachusetts Ave, Amherst, MA 01003, United States

Abstract

Large Language Models (LLMs) have enabled more natural human-robot interaction, but open-source models often exhibit unstable long-horizon reasoning and inefficient action execution when deployed in agentic robotic frameworks. This paper presents an enhanced ROS-Agent based architecture that improves task reliability and execution efficiency for agentic robotic systems using open-source LLMs. The proposed system introduces a novel intermediate mechanism, termed the MetaTool, which enforces structured planning prior to action execution. Given a natural-language command, the MetaTool induces the LLM to generate a pseudo-code plan of intended tool invocations, which is stored in the ROS-Agent's scratchpad and persists throughout execution. By explicitly separating planning from execution, the proposed approach reduces execution loops and improves deterministic behavior. The architecture is validated on a custom mobile robotic platform with multimodal perception and motion control capabilities. Experimental results on real-world interactive tasks demonstrate improved task completion and contextual consistency, with up to ~24% gains on complex tasks compared to the baseline framework.

Figures & tables

Fig. 1: Overview of the custom-designed robotic platform developed for system evaluation. The platform integrates perception modules, facial emotion display, articulated arm gestures, and mobility components, all managed through a ROS-based control stack.

System	Explicit Plan Buffer	Execution Constraint	ROS-native	Open-LLM Focus	Real Robot
ReAct [ 10 ]	$\times$	$\times$	$\times$	$\times$	$\times$
SayPlan [ 11 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$
RoboGPT [ 2 ]	$\checkmark$	$\sim$	$\times$	$\checkmark$	$\times$
CaP [ 12 ]	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$
MetaTool-ROS-Agent (ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE I: Feature comparison of LLM-based robotic planning systems.

Fig. 2: Functional architecture of the proposed MetaTool-enhanced ROS-Agentic framework.

Tool	Functionality
describe_image	Vision–language understanding from live camera input
RAG_query	Knowledge retrieval via vector database querying
face_expression	Affective state publishing for social interaction
follow_me	Human-following behavior using pose keypoints
get_goal_odometry	Goal-conditioned odometry estimation
set_goal	Goal definition and storage for navigation tasks

TABLE II: Task-Oriented Tools Integrated into the MetaTool-Enhanced ROS Agent

Fig. 3: Operational overview of the proposed MetaTool integrated with the existing ROS agentic architecture, illustrating intermediate task planning and structured tool execution via scratchpad memory.

ID	Configuration	Task	GPT-4o [ 33 ]	GPT-OSS-20B [ 32 ]	GPT-OSS-120B [ 32 ]	Kimi-K2 [ 34 ]
C1	ROS-Agent + Adaptive Prompt	T1	15/20 (75%)	11/20 (55%)	11/20 (55%)	15/20 (75%)
		T2	10/20 (50%)	8/20 (40%)	15/20 (75%)	17/20 (80%)
		T3	11/20 (55%)	6/20 (30%)	8/20 (40%)	10/20 (50%)
C2	ROS-Agent + Prompt-Based Planner (Non-Persistent)	T1	17/20 (85%)	12/20 (60%)	15/20 (75%)	15/20 (75%)
		T2	17/20 (85%)	12/20 (60%)	15/20 (75%)	17/20 (80%)
		T3	15/20 (75%)	7/20 (35%)	8/20 (40%)	13/20 (65%)

TABLE III: Component-Wise Ablation Results Across Tasks. Task IDs are defined in Section IV-A .

Fig. 4: Gazebo-based simulation and testing across diverse environments to ensure the robustness of agentic responses: (A) large office space; (B) hospital floor environment; (C) factory floor with scattered obstacles; and (D) standard urban home setting.

ID	Configuration	Description
C1	ROS-Agent + Adaptive Prompt	Natural-language task execution guided solely by adaptive prompting, without explicit planning or persistent state.
C2	ROS-Agent + Prompt-Based Planner (Non-Persistent)	Prompt-based task decomposition generated at runtime; intermediate plans are not stored persistently and may be regenerated.
C3	ROS-Agent + MetaTool (Persistent Scratchpad, No Execution Constraint)	Explicit task plan stored in persistent scratchpad memory; tool selection is not constrained by the stored plan.
C4	ROS-Agent + MetaTool (Persistent Scratchpad + Execution Constraint)	Persistent task plan maintained and used to explicitly condition tool invocation, enforcing adherence throughout execution.

TABLE IV: MetaTool Component Ablation Configurations

Panel of Figure 9 — Fig. 5: Comparative Performance Analysis of Baseline ROSA vs. MetaTool-Enabled ROS Agent. The proposed framework demonstrates a higher Success Rate (indicated by the opacity) across all task complexities. The Token Usage Distribution highlights significantly lower variance in the proposed method, indicating that the MetaTool constraint prevents infinite loops and reasoning instability common in the open-source model.

	Repeated Tool Calls (%)	Invalid/Failed Tool Calls (%)	Execution Time (s)
ROS-Agent	19.0 ± 5.7	21.7 ± 6.5	89.51 ± 16.48
MetaTool (ours)	6.1 ± 2.2	8.1 ± 2.9	84.23 ± 9.71

TABLE V: Hardware evaluation of execution efficiency and stability across 40 trials using GPT-OSS-120B.All metrics are reported as mean

\pm

standard deviation.

Note: Failure categories are non-exclusive. Individual failed trials may encounter multiple failure modes, so total category counts may exceed total failed trials.
Method	Incorrect / Hallucinated Plan	Navigation Failure	Tool Unavailable / Timeout	Successful Trials
ReAct [ 10 ]	17/40	13/40	10/40	17/40
CaP [ 12 ]	13/40	10/40	13/40	20/40
ROSA [ 4 ]	17/40	10/40	17/40	23/40
MetaTool (ours)	10/40	7/40	3/40	30/40

TABLE VI: Quantitative comparison of failure modes and overall task success rates across evaluated planning frameworks.

Fig. 6: Real-world implementation and experimental evaluation of the proposed architecture on a custom robotic platform: (A) vLM invocation for visual understanding; (B) autonomous navigation within an indoor laboratory environment; (C) visual memorization using agentic vLM calls for future spatial recollection; (D) chauffeuring the user to a requested destination; (E) spontaneous user greeting based on predicted mood and perceived direction of communication; and (F) navigation through a narrow corridor while describing the surroundings to maintain user engagement.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 429 papers

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning
Feb 25, 2026Tomoya Kawabe, Rin TakanoLarge Language Model-Based Robot Planning Multi-Agent LLM Systems
Data Science Laboratories, NEC Corporation, 1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan.
HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning
Jun 24, 2026Junho Lee, Seabin Lee, Wonjong Lee +3Large Language Model-Based Robot Planning Robot Task Planning
Dept. of Electronic Engineering, Sogang University, Seoul, Korea
ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration
Apr 6, 2026Rongfeng Zhao, Xuanhao Zhang, Zhaochen Guo +4VLMs for Robotics Cross-Embodiment Robot Learning
Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai 201210, China · National Key Laboratory of Autonomous Intelligent Unmanned Systems (Tongji University), Shanghai 201210, China · Frontiers Science Center for Intelligent Autonomous Systems, Shanghai 201210, China +1

Feb 25, 2026J/K move · Enter open · S save

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

Tomoya Kawabe, Rin Takano

Data Science Laboratories, NEC Corporation, 1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan.

Large Language Model-Based Robot Planning Multi-Agent LLM Systems Robotics Prompt Optimization Multi-Robot Motion Planning

Open paper AskPDF arXiv

Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot teams. Conventional Planning Domain Definition Language (PDDL) planners provide rigorous guarantees but struggle to handle ambiguous or long-horizon missions, while large language models (LLMs) can interpret instructions and propose plans but may hallucinate or produce infeasible actions. We present a hierarchical multi-agent LLM-based planner with prompt optimization: an upper layer decomposes tasks and assigns them to lower-layer agents, which generate PDDL problems solved by a classical planner. When plans fail, the system applies TextGrad-inspired textual-gradient updates to optimize each agent's prompt and thereby improve planning accuracy. In addition, meta-prompts are learned and shared across agents within the same layer, enabling efficient prompt optimization in multi-agent settings. On the MAT-THOR benchmark, our planner achieves success rates of 0.95 on compound tasks, 0.84 on complex tasks, and 0.60 on vague tasks, improving over the previous state-of-the-art LaMMA-P by 2, 7, and 15 percentage points respectively. An ablation study shows that the hierarchical structure, prompt optimization, and meta-prompt sharing contribute roughly +59, +37, and +4 percentage points to the overall success rate.

Abstract

Long-Horizon LLM Agents

Explore similar work

Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning

HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning

ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration