MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robots

Xiao Wang; Lu Dong; Jingchen Sun; Ifeoma Nwogu; Srirangaraj Setlur; Venu Govindaraju

cs.ROMar 4, 2026

MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robots

Authors: Xiao Wang, Lu Dong, Jingchen Sun, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju

Organizations: State University of New York at Buffalo, Buffalo, NY 14260, USA

Abstract

With the availability of open APIs in social robots, it has become easier to customize general-purpose tools to meet users' needs. However, interpreting high-level user instructions, selecting and configuring appropriate tools, and executing them reliably remain challenging for users without programming experience. To address these challenges, we introduce MistyPilot, an agentic LLM-driven framework for autonomous tool selection, orchestration, and parameter configuration. MistyPilot comprises two core components: a Physically Interactive Agent (PIA) and a Socially Intelligent Agent (SIA). The PIA enables robust sensor-triggered and tool-driven task execution, while the SIA generates socially intelligent and emotionally aligned dialogue. MistyPilot further integrates a fast-slow thinking paradigm to capture user preferences, reduce latency, and improve task efficiency. To comprehensively evaluate MistyPilot, we contribute five benchmark datasets. Extensive experiments demonstrate the effectiveness of our framework in routing correctness, task completeness, fast-slow thinking retrieval efficiency, tool scalability,and emotion alignment. All code, datasets, and experimental videos will be made publicly available on the project webpage.

Figures & tables

Fig. 1: Overview of MistyPilot workflow for interpreting high-level human instructions. Instead of requiring professionals to hand-code robot behaviors and deploy features on the Misty robot, MistyPilot parses natural-language instructions, analyzes the task, selects and parameterizes tools from its library, and executes them on Misty Robot.

Fig. 2: Overview of the MistyPilot framework. MistyPilot interprets high-level user instructions through a Task Router that dispatches tasks to either a Physically Interactive Agent (PIA) or a Socially Intelligent Agent (SIA) for automated tool utilization and parameter adaptation. The PIA oversees a Sensor & Tool Manager to dynamically orchestrate sensor-related and tool-dependent tasks, while the SIA maintains a Task Status Manager to track the current task state. Fast Thinking accelerates inference by memory hitting for stored knowledge and preferences, while Slow Thinking employs a Script Writer to interpret the task, align responses with fine-grained emotional expression, and deliver them via speaking and movement modules.

Task State	System Control
$\textbf{NEW}\langle\textit{main\_task},\textit{details}\rangle$	UPGRADE
$\textbf{UPDATE}\langle\textit{details}\rangle$	DOWNGRADE
$\textbf{DELETE}\langle\textit{details}\rangle$	MEMORY

TABLE I: Keyword set

\mathcal{A}

for task state parsing

User Task	“Hi Misty, I’d like to plan a day trip to New York City for tomorrow. Please create an itinerary that allows me to enjoy the city and return to my hotel by 7:00 PM.”
	TSM
$\mathit{main\_task}$	Plan a day trip to New York City
$\mathit{details}$	Date tomorrow; return by 7:00 PM; goal enjoy the city

TABLE II: An example task status managed by the TSM

Prompt	”Tell me the story of Three Little Pig”
Utterance	Text	Emotion
1	Three little pigs left home to build their own houses.	Neutral
2	The first pig quickly built a house of straw.	Contempt
3	The second pig put up a house of sticks with little effort.	Contempt
4	The third pig worked diligently to lay strong bricks for a sturdy home.	Happiness
…

TABLE III: Emotional attunement example by the Script Writer

Command	Description
BIND $\langle s,k\rangle$	Bind skill $k$ to sensor $s$ .
UPDATE $\langle s,k\rangle$	Update skill $k$ to sensor. $s$
UNBIND $\langle s\rangle$	Unbind sensor $s$ from all associated skills.
INVOKE $\langle k\rangle$	Directly invoke skill $k$ .

TABLE IV: Command Set for Sensor & Tool Management

Sensor	PID	Bound Skill	Status
Head Touch	1023	Greeting	Active
Scruff Touch	1098	DailyMemoReport	Active
Front Left Bumper	1120	NodHead	Inactive

TABLE V: Example of STM process table entries

Condition	MistyPilot (Multi-Agent)	Single-Agent
SIA – Easy	$100\%\pm 0.00\%$	$99.2\%\pm 1.79\%$
SIA – Hard	$100\%\pm 0.00\%$	$100\%\pm 0.00\%$
PIA – Easy	$100\%\pm 0.00\%$	$100\%\pm 0.00\%$
PIA – Hard	$96.2\%\pm 5.66\%$	$90.4\%\pm 15.65\%$

TABLE VI: Task Routing Correctness on MistyPilot-Route100 Dataset.(mean

\pm

std over 5 runs).

Subset	MistyPilot (Multi-Agent)	Single-Agent
Easy	$100.00\%\pm 0.00\%$	$100.00\%\pm 0.00\%$
Hard	$100.00\%\pm 0.00\%$	$81.00\%\pm 4.18\%$

TABLE VII: PIA Performance Evaluation on the MistyPilot-SensorBind40 Dataset (mean

\pm

std over 5 runs).

Subset	MistyPilot (Multi-Agent)	Single-Agent
Easy	$99.29\%\pm 1.60\%$	$91.43\%\pm 13.27\%$
Hard	$96.75\%\pm 1.15\%$	$93.50\%\pm 5.36\%$

TABLE VIII: SIA Performance Evaluation on MistyPilot-TaskParser256 Dataset (mean

\pm

std over 5 runs).

Data	Model	Top-1 (%)	Rank1 Dist.	Rank2 Dist.
			Mean (Std)	Mean (Std)
Raw Text	TE3-L	67.83%	0.365 (0.070)	0.464 (0.071)
Raw Text	TE3-S	72.61%	0.342 (0.082)	0.453 (0.066)
Raw Text	MiniLM	58.70%	0.397 (0.127)	0.570 (0.066)
Fast Path	TE3-L	100.00%	0.003 (0.015)	0.392 (0.216)
Fast Path	TE3-S	100.00%	0.003 (0.015)	0.394 (0.211)

TABLE IX: Fast-Thinking Retrieval Evaluation on MistyPilot-FastThinking230 Dataset: Raw text vs. Fast Path .

Scale	30	50	70	100
Success rate	100.00%	100.00%	100.00%	100.00%

TABLE X: Tool Extensibility Evaluation on MistyPilot-ToolExtension30/50/70/100 Dataset

Evaluation Dimension and Questionnaire Item	Mean	Std
Interaction Naturalness: Communicating with the robot through natural language to perform specific tasks was intuitive and straightforward.	4.75	0.45
Comprehension Accuracy: The robot accurately understood my intentions and provided responses that met my expectations.	4.42	0.67
Emotional Expressiveness: The robot’s speech and reactions were emotionally expressive, giving it a highly anthropomorphic and human-like presence.	4.75	0.45
System Responsiveness: The overall response latency was within an acceptable range.	4.67	0.49
User Satisfaction: Overall, I am satisfied with the multimodal interaction experience provided by the MistyPilot system.	4.75	0.45

TABLE XI: Subjective Evaluation Results of the MistyPilot System

Agentic AI for Robotics

See all 39 papers

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration
Aug 16, 2026Xiao Wang, Lu Dong, Ifeoma Nwogu +2Agentic AI for Robotics LLM Agent Orchestration
State University of New York at Buffalo
Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework
Sep 11, 2026Kazi Abrar Mahmud, Nilotpaul Kundu Dhurubo, Tamal Kirttonia +2Long-Horizon LLM Agents LLM Agent Reliability
Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1000, Bangladesh · University of Massachusetts Amherst, 300 Massachusetts Ave, Amherst, MA 01003, United States
Botson: An Accessible and Low-Cost Platform for Social Robotics Research
Feb 23, 2026Samuel Bellaire, Abdalmalek Abu-raddaha, Natalie Kim +3Social Robotics Human-Robot Interaction
University of Michigan-Dearborn

Aug 16, 2026J/K move · Enter open · S save

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

Xiao Wang, Lu Dong, Ifeoma Nwogu, Srirangaraj Setlur+1

State University of New York at Buffalo

Agentic AI for Robotics LLM Agent Orchestration Social Robotics Human-Robot Interaction Multi-Agent LLM Systems

Open paper AskPDF arXiv

Programming small social robots from natural-language instructions requires more than invoking isolated APIs. Interactive tasks combine reactive physical behaviors with stateful social behaviors, while existing interfaces often require developers to manually compose APIs into skills, configure their parameters, bind sensor events to skills, and manage task states at runtime. We present MistyPilot, a multi-agent LLM framework that interprets high-level natural-language instructions and orchestrates the corresponding skills on the Misty social robot. A Task Router dispatches each instruction to one of two specialized agents: a Physically Interactive Agent for sensor-triggered robot control and direct skill invocation, and a Social Interaction Agent for dialogue-oriented task-state management and context-dependent multimodal response generation. To improve efficiency, the Social Interaction Agent reuses previously generated results when applicable and invokes full generation otherwise. We evaluate MistyPilot on five component-level suites, with sensor bindings and skill invocations executed on the physical Misty robot, and a preliminary user study with 12 participants. MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills, and lower variance than an otherwise identical single-agent baseline, while participants report positive perceptions of usability and interaction quality. The code will be made publicly available via the project page.

Abstract

Agentic AI for Robotics

Explore similar work

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework

Botson: An Accessible and Low-Cost Platform for Social Robotics Research