cs.HCOct 5, 2026

MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality

Authors: Xiaoran Yang, Xun Qian, Yang Zhan, Nathan Tran, Ziyi Liu, Qiao Jin

Organizations: North Carolina State University Raleigh, North Carolina, USA · Google San Jose, California, USA · Purdue University West Lafayette, Indiana, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) let users direct heterogeneous multi-robot systems (MRS) through natural language, but make task interpretation, robot assignment, and coordination difficult to inspect and change. Based on a formative study with 12 non-expert users, we developed MRPilot, a mixed reality system organized around four stages of supervision and intervention. MRPilot represents robot-team plans and execution states as structured commitments shared across synchronized situated and overview views. Across four stages, it helps users resolve ambiguous references (Forming), review plans before execution (Reviewing), monitor distributed execution (Following), and make robot-level or team-level changes when problems arise (Repairing). In a within-subjects study with 20 participants in a virtual reality-simulated home, MRPilot reduced workload, increased situational awareness, transparency, trust, and perceived control compared with a conventional LLM-based conversational interface using the same LLM planner and robot capabilities. We provide design implications for multi-scale intervention, adaptive supervision, and calibrated reliance in LLM-based MRS.

Figures & tables

Figure 1. Overview of MRPilot. MRPilot is a MR system for supervising LLM coordinated multi-robot systems (MRS), enabling rapid intervention in MRS tasks through spatial interaction. In a task execution workflow, the user first issues an instruction in natural language. (a) MRPilot detects ambiguities in the instruction, highlights non-unique elements in the environment, and displays them in the interaction panel. (b) MRPilot visualizes the current candidate plans through the MR view and Mini-Map for user selection. (c) The MR view and Mini-Map continuously update robot positions and plan information during execution. (d) When unexpected events requiring intervention occur during execution, the user can quickly inspect relevant information and intervene through the MR view.
Figure 2. MRPilot system Front-end and back-end communication structure. Frontend consists of the user interface, UI translation, and action translation, which translate the commitments generated by the backend into information that can be understood by both the robot team and the user. The backend consists of an LLM processor, which includes an LLM planner together with prompts describing spatial information and robot capabilities. Communication between the frontend and backend is based on structured commitments as the basic data unit, which are transmitted in JSON files.
Figure 3. 1: Forming stage UI. 1-A: Users select an ambiguous object through the chat panel. 1-B: In the MR view, spatially ambiguous objects are highlighted with bounding boxes, labeled with their names and relative distances. 2: Reviewing stage UI. 2-A: Plan preview in the Mini Map. Each robot has a marker, colored numbers 1 and 2 indicate the execution order, and the stationary robot arm is shown in white. 2-B: During plan preview, the MR view shows each robot with a bounding box and label. 2-C: White lines show the robot team’s predicted execution paths.3: Following stage MR view. Each robot’s label shows its execution progress, the spatial card shows the current task content, and bounding boxes highlight all robots and objects involved in the task. Execution paths update in real time with robot positions. 3-A: When an event requiring user interaction falls outside the user’s field of view, a green guidance arrow at the edge of the view points toward its source.4:Baseline-Processing Status interface, which is similar to the ChatGPT chat interface. Users control the robot team through voice input using the recording button. 5: Baseline- Plan Preview interface. The upper section uses text to show dependencies among multiple robots and task details. The lower section shows a layout map with the room layout and object names.
Figure 4. 1 Mini Map during task execution, with markers and paths updated in real time. 2-A: In the Mini Map, tapping a robot marker opens its action bar (2-A). 2-B: In the MR view, clicking the robot label opens the same action bar. The action bar supports per-robot adjustments: replanning, removing the robot from the plan, and reverting it to its previous position.
#Intervention EventMessage Presented to ParticipantRobotIntervention TypeInitiator
1Ambiguous object referenceWhich item do you want to choose? Select one highlighted option in the space [Glasses 1] [Glasses 2]Robot ArmInformation ClarificationAgent
2Route preferenceChoose a route preference: Default route is safer but slower; the Fast route is quicker but riskier.DronePlan Selection or OptimizationAgent
3Potentially unstable graspOops! I noticed that the robotic arm is pinching the glasses by the temple, so they may slip. I need to adjust the grasp so it holds the frame securely.Robot TeamPre-Failure Action CorrectionHuman
4Delivery robot blockedExecution failure: The robot is blocked. Assign another robot to take over or roll the delivery robot back to its previous checkpoint and replan a route.Delivery RobotFailure RecoveryAgent
5Private-room accessThe spaceguest room is marked PRIVATE. Allow the drone to enter?DroneAuthorization or AccessAgent
6Recipient relocationI just realized that the Jim has moved to the studio. I have to pick the glasses back up from the bedside table and deliver them to the desk in the studio.Robot TeamTask or Goal UpdateHuman
Table 1. Intervention events in Task A, including the messages presented to participants, affected robots, intervention types, and initiation modes. Human-initiated events were cued by condition-independent text-and-speech updates on a separate VR panel, whereas agent-initiated events requested input through a spatial label in MRPilot or a text message in the Conversational Interface.
Figure 5. The Results of Intervention Type Complete Time, Note: Throughout all figures, significance levels are indicated as follows: * p<.05p<.05 , ** p<.01p<.01 , *** p<.001p<.001
Figure 6. The Result of Prompt issued and Prompt Length
Figure 7. NASA TLX Questionnaire Results
Figure 8. The Results of SART, MDMT v1, TOROS and self report questionnaires
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

 1234567
 Strongly disagreeDisagreeSomewhat disagreeNeither agree nor disagreeSomewhat agreeAgreeStrongly agree
I was able to make the robot team do what I intended.◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc
When I wanted to change the robot team’s plan, I was able to change it.◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc
I was in control of how the task was carried out.◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc
Appendix
Table 10
#Scenario / MessageRobotTypeInterventionInitiator
1Which room should the drone begin in? Select one highlighted option in the space [Room1] [Room2]DroneT1Information ClarificationAgent
2Choose a route preference: Default route is safer but slower; the Fast route is quicker but riskier.DroneT2Plan Selection or OptimizationAgent
3The room is passcode-protected. Enter the passcode.DroneT3Authorization or AccessAgent
4Oops! The drone is unstable and the bottle may fall during the flight. I need to regrab the bottel before it continues. T5Pre-Failure Action CorrectionHuman
5The drone is out of battery and has landed. Delete the drone and use other robots to continue the task.DroneT6Failure RecoveryAgent
6Please place the medicine bottle on the delivery robot.Delivery RobotT7Capability-Gap AssistanceAgent
Appendix
Table 2. Task Bintervention events and corresponding intervention types.
Intervention TypeDescriptionInclude
Task or Goal UpdateThe human changes what the agent is trying to achieve after execution has begun—adding, revising, narrowing, or withdrawing requirements—and the agent must re-plan from its current state rather than restart. Distinct from plan editing (the goal is stable) and from norm adjustment (the goal is stable but the acceptable way of reaching it changes).( Mozannar et al., 2025 ; Huq et al., 2026 ; Feng et al., 2026 ; Zou et al., 2026 ; Gao et al., 2025 ; Liu et al., 2025b ; Sharma et al., 2022 ; Thierauf et al., 2024 )
Plan Selection or OptimizationBefore execution, or between steps, the agent exposes its intended plan or a set of candidate next steps and the human approves, chooses among alternatives, or edits, re-orders, or optimises steps. The goal is unchanged; the human shapes the how . Typically agent-proposed, human-decided.( Mozannar et al., 2025 ; Feng et al., 2026 ; Dhanorkar et al., 2026 ; Zhang et al., 2024b ; Ren et al., 2023 ; Takerngsaksiri et al., 2025 ; He et al., 2025 ; Kim et al., 2025 ; He et al., 2026 ; Peng et al., 2025 )
Information ClarificationThe agent (occasionally the human) resolves ambiguity, underspecification, or missing information in the task input—such as referents, parameters, facts, or intent—by asking and answering questions before or during execution. The human supplies information the agent lacks ; the goal, plan, permissions, and norms are otherwise unchanged. The intervention is almost always agent-initiated; the central design problem is whether, what, and when to ask while keeping human burden low.( Vijayvargiya et al., 2026b ; Su and Cardie, 2026 ; Wu et al., 2026 ; Wang et al., 2025 ; Qian et al., 2024 ; Chen et al., 2024a ; Huq et al., 2026 ; Suri et al., 2026 ; Lin et al., 2025 ; Doğan et al., 2022 ; Ramrakhya et al., 2025 ; Zhang et al., 2024b ; Fang et al., 2025b ; Edwards and Schuster, 2026 ; Shi et al., 2022 ; Zhang et al., 2023 ; Testoni and Fernández, 2024 ; Ren et al., 2023 ; Vijayvargiya et al., 2026a ; Deng et al., 2026 ; Gulati et al., 2026 ; Wu et al., 2025 ; Sun et al., 2025 ; Saracay et al., 2026 )
Pre-Failure Action CorrectionWhile the agent is acting, the human (or a pre-emptive monitor that alerts the human) sees that the next or current action is wrong, premature, or risky and corrects or blocks it before it commits or before error compounds. Fine-grained, real-time, and usually human-initiated; control is normally handed back.( Mozannar et al., 2025 ; Huq et al., 2026 ; Huq et al., 2025 ; Dhanorkar et al., 2026 ; Gao et al., 2025 ; Liu et al., 2025b ; Fang et al., 2025a ; Shi et al., 2024 ; Liu et al., 2023b ; Kelly et al., 2019 ; Spencer et al., 2020 ; Saunders et al., 2017 ; He et al., 2025 ; Epperson et al., 2025 ; Zhang et al., 2025a )
Appendix
Table 3. Intervention types synthesised from the literature.
Table 3 continued
Intervention TypeDescriptionInclude
Failure RecoveryAfter the agent has failed, stalled, or looped, the human diagnoses, un-sticks, rolls back, or completes the failed step so the agent can resume or the task can be salvaged. Often agent-initiated through a targeted help request, but also human-initiated when the agent does not notice its own failure.( Huq et al., 2026 ; Dhanorkar et al., 2026 ; Gao et al., 2025 ; Liu et al., 2025b ; Zhou et al., 2026 ; Shi et al., 2024 ; Liu et al., 2023b ; Sharma et al., 2022 ; Knepper et al., 2015 ; Ahn et al., 2024 ; Das et al., 2021 ; Epperson et al., 2025 ; Zhang et al., 2025a ; Lu et al., 2026 ; Piao et al., 2025 )
Authorization or AccessThe agent (or a policy layer) stops at an action boundary and the human grants or denies permission for a consequential action, supplies credentials or access, or sets permission scope in advance. The human decision is binary or scoped; the plan and goal are otherwise unchanged.( Mozannar et al., 2025 ; Huq et al., 2025 ; Dhanorkar et al., 2026 ; Ruan et al., 2024 ; Zhang et al., 2025c ; Weng, 2026 ; Luo et al., 2026 ; Jia et al., 2026 ; Korbak et al., 2025 ; Zhou et al., 2026 ; Takerngsaksiri et al., 2025 ; Peng et al., 2025 )
Capability-Gap AssistanceThe agent reaches a step it cannot perform—physically, perceptually, or because of a technical or policy barrier—and the human performs that step or supplies the missing resource before returning control. Unlike failure recovery, nothing has necessarily gone wrong; the step lies outside the agent’s capability envelope.( Mozannar et al., 2025 ; Huq et al., 2026 ; Huq et al., 2025 ; Feng et al., 2024 ; Knepper et al., 2015 ; Rosenthal et al., 2010 ; Ahn et al., 2024 ; Peng et al., 2025 ; Piao et al., 2025 )
Contextual or Norm AdjustmentThe human supplies or changes situational context, preferences, constraints, or norms that govern how the agent should act without changing the goal or a specific action, so that behaviour is appropriate to the setting or person. Such information may be agent-elicited or human-volunteered and may persist beyond the current task.( Mozannar et al., 2025 ; Huq et al., 2026 ; Shao et al., 2026 ; Wang et al., 2025 ; Zhang et al., 2024b ; Ramrakhya et al., 2025 ; Ren et al., 2023 ; Glaze and Inclezan, 2025 ; Wu et al., 2023 ; Shankar et al., 2026 ; Jacniacki and Bilski, 2026 )
Appendix
Table 13

Explore similar work

CardsList
Jun 5, 2026cs.RO

A Multi-Operator Mixed-Reality Interface for Multi-Robot Control and Coordination: Co-Located and Private Workspace Collaboration

Multi-operator control of robot teams requires not only access to the same mission information, but also mechanisms for maintaining shared awareness and preventing conflicting interventions. Building on our previous HORUS interface (Holistic Operational Reality for Unified Systems) we present a mixed-reality interface that extends single-operator multi-robot supervision to collaborative multi-operator use. The system supports two complementary modes: a co-located shared workspace, in which operators observe and manipulate the same mini-map in the same physical location, and a private-workspace mode, in which operators work on the same mission through independently placed local workspaces. The architecture combines registration-driven scene construction, lightweight shared-session synchronization, and per-robot control leases to support collaborative monitoring, tasking, and teleoperation while preventing conflicting commands. We evaluated the approach in a human-subject study with 36 participants (18 pairs) controlling three Nova Carter mobile robots in two search environments. The performance of the objective task was comparable across the two modes, indicating that both modes supported effective mission execution. However, the co-located shared workspace significantly improved perceived collaboration, shared understanding, and handoff clarity, and was the preferred collaborative mode. These results indicate that physically co-locating the MR workspace improves how operators coordinate even when the underlying robot-control tools remain unchanged.

Omotoye Shamsudeen Adekoya, Antonio Sgorbissa, Carmine Tommaso Recchiuto

DIBRIS Department, RICE Laboratory, University of Genoa, Italy

Multi-Robot SystemsTeleoperationCoordinationHorizon
AskPDF
Aug 7, 2026cs.RO

When Coordination Becomes a Threat: Communication Attacks in LLM-Controlled Multi-Robot Systems

Large Language Models (LLMs) are increasingly used as high-level planners in embodied multi-robot systems, enabling robots to interpret natural language instructions and coordinate executable actions. Yet, this growing reliance on LLM planners also raises security concerns. Prior work has focused mainly on individual robots, while communication risks in multi-robot collaboration remain insufficiently understood. Existing multi-robot studies are further limited to preliminary analysis under the Decentralized Multi-agent System (DMAS) architecture, so it remains unclear whether these risks persist across other common communication architectures and how attacker access settings shape their propagation. To fill this gap, we formulate two communication attacks corresponding to distinct attacker access settings: the External Entry Point Attack and the Privileged In-System Attack. We evaluate both attacks across DMAS, HMAS-1, and HMAS-2 using three LLMs and five embodied multi-robot tasks. Results show that unsafe information can turn into unsafe actions across all three architectures: DMAS reaches a 96.7% entry endorsement rate and a 100% post endorsement activation rate, HMAS-1 reaches a 97.8% unsafe action success rate, and HMAS-2 triggers 88.3% of task defined unsafe action slots. To mitigate risks from trusted information flow, we introduce the Claim Provenance and Verification (CPV) Gate, which verifies communicated claims before downstream reuse and reduces the violation rate from 70.0% to 36.6%.

Zhen Huang, Zhihuang Liu, Weijia Shi +3

College of Computer Science and Technology, National University of Defense Technology, Changsha, China

Multi-Robot SystemsMulti-Agent Large Language Model SystemsModel-Based Multi-Agent SystemsThreatCoordination
AskPDF
May 14, 2026cs.RO

MR-SLAM: Immersive Spatial Supervision for Multi-Robot Mapping via Mixed Reality

Operating a multi-robot fleet for simultaneous localization and mapping (SLAM) in applications such as building inspection or warehouse-aisle monitoring requires the operator to maintain spatial awareness of each robot's position and mapping state, a task that scales poorly on conventional 2D interfaces. We present MR-SLAM, a mixed reality (MR) system in which an operator wearing a Meta Quest 3 headset teleoperates three simulated TurtleBot3 robots through a passthrough view with real-world occlusion, while spatially anchored dashboard panels report mapping progress in situ. Each robot runs an independent SLAM Toolbox instance whose occupancy grid is merged in real time on a Robot Operating System 2 (ROS 2) back end. Across five 9-minute evaluation sessions, the system delivered scans at 8.83 +/- 0.16 Hz, mapped 17.9 +/- 0.8 m^2 of merged occupancy, and reached 94.7 +/- 0.5% cross-instance occupancy consistency across robot pairs. An additional session recorded 6.3 ms median transform jitter and 26.7 m^2 coverage of a 41 m^2 grid. We position MR-SLAM as a reference implementation for combining passthrough mixed reality supervision with multi-robot SLAM on consumer hardware.

Prakash Aryan, Cem Erdogdu, Kavinaya Kumarchokkappan +2

University of Bern, Bern, Switzerland · AI4I – The Italian Institute of Artificial Intelligence, Turin, Italy

Simultaneous Localization And MappingMulti-Robot SystemsSpatial SupervisionSpatial Reasoning
AskPDF