Advancing Multi-Robot Networks via MLLM-Driven Sensing, Communication, and Computation: A Comprehensive Survey

Hyun Jong Yang; Howon Lee; Kyuhong Shim; Jeongho Kwak; Hyunsoo Kim; Donghoon Kim; Khoa Anh Ngo; Sehyun Ryu

cs.ROMar 31, 2026

Advancing Multi-Robot Networks via MLLM-Driven Sensing, Communication, and Computation: A Comprehensive Survey

Authors: Hyun Jong Yang, Howon Lee, Kyuhong Shim, Jeongho Kwak, Hyunsoo Kim, Donghoon Kim, Khoa Anh Ngo, Sehyun Ryu,

Organizations: Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon, South Korea · Department of Intelligent Software, College of Computing and Informatics, Sungkyunkwan University, Suwon, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea ·

Abstract

Imagine advanced humanoid robots, powered by multimodal large language models (MLLMs), coordinating missions across industries like warehouse logistics, manufacturing, and safety rescue. While individual robots show local autonomy, realistic tasks demand coordination among multiple agents sharing vast streams of sensor data. Communication is indispensable, yet transmitting comprehensive data can overwhelm networks, especially when a system-level orchestrator or cloud-based MLLM fuses multimodal inputs for route planning or anomaly detection. These tasks are often initiated by high-level natural language instructions. This intent serves as a filter for resource optimization: by understanding the goal via MLLMs, the system can selectively activate relevant sensing modalities, dynamically allocate bandwidth, and determine computation placement. Thus, R2X is fundamentally an intent-to-resource orchestration problem where sensing, communication, and computation are jointly optimized to maximize task-level success under resource constraints. This survey examines how integrated design paves the way for multi-robot coordination under MLLM guidance. We review state-of-the-art sensing modalities, communication strategies, and computing approaches, highlighting how reasoning is split between on-device models and powerful edge/cloud servers. We present four end-to-end demonstrations (sense -> communicate -> compute -> act): (i) digital-twin warehouse navigation with predictive link context, (ii) mobility-driven proactive MCS control, (iii) a FollowMe robot with a semantic-sensing switch, and (iv) real-hardware open-vocabulary trash sorting via edge-assisted MLLM grounding. We emphasize system-level metrics -- payload, latency, and success -- to show why R2X orchestration outperforms purely on-device baselines.

Figures & tables

Fig. 1 : A scenario comparing (left) individually planned routes vs. (right) communication-based collaboration. The lower timeline highlights that the gain is conditional: “with communications” outperforms when $T_{\mathrm{uplink}}+T_{\mathrm{edge}}+T_{\mathrm{downlink}}<T_{\mathrm{wait}}$ , where $T_{\mathrm{wait}}$ is the expected collision-induced waiting time without coordination.

Paradigm	Defining scope and features	R2R comm. protocol	Ultra-low latency support	Consideration of multiple robot collab.	Task-central comm. reqs. design	Integrated sensing/ computing/ comm. design	Inclusion of LLM-Based AI synergy	Relevant surveys
Robot collaboration	Emphasizes multi-robot synergy through shared perception and distributed planning, yet rarely integrates underlying communication protocols or edge/cloud offloading mechanisms.	X	X	O	X	X	X	[ 9 , 10 , 11 , 12 ]
V2X	Concentrates on vehicle-to-everything links for low-latency road safety and traffic coordination, lacking a broader focus on multi-robot contexts or advanced AI-driven orchestration.	X	O	X	X	X	X	[ 5 ]
M2M	A device-centric communication protocol for automated IoT interactions, including R2R, yet generally omits goal-oriented design or large-model–assisted collaboration.	O	X	X	X	X	X	[ 3 , 4 ]
URLLC	Promises ultra-reliable, sub-ms network links for mission-critical 5G/6G, but seldom integrates multi-robot perception or large-scale AI inference into the communication structure.	X	O	X	X	X	X	[ 6 , 7 , 8 ]
R2X (Proposed)	Offers a full-stack paradigm coupling LLM-based feature vector encoding, multimodal sensing (incl. emergency audio/visual cues), edge/cloud resource management, and multi-robot collaboration from physical-layer protocols to high-level orchestration, enabling intent-to-resource orchestration from language goals to sensing/communication/computation decisions.	O	O	O	O	O	O	This work.

TABLE I : Comparison of robot collaboration, V2X, M2M, URLLC, and the proposed R2X Paradigm.

Fig. 2 : High-level overview of this paper’s key concept, four end-to-end demonstrations (Demo I–IV), and overall organization.

Fig. 3 : Common block diagram of MLLMs.

Topic	Method	Ref.	Key Contribution
Multi-modal Sensing	MUTEX	[ 60 ]	LMM equipped with a Transformer-based cross-modal fusion module capable of processing six modalities. Introduces a two-stage training procedure and a multi-modal training dataset collected from both simulated and real-world environments.
	Matcha	[ 61 ]	Robotic planning system that actively collects information from multi-modal sensors (vision, sound, haptics) by performing epistemic actions. The LMM reasons over the information obtained from such actions and plans the main task execution.
	REFLECT	[ 59 ]	Robot failure explanation system based on the summarization of the robot experiences, represented as hierarchically structured multi-sensory data. The summarization is used to develop a natural language behavior explanation system and to improve the model’s correction ability.
	MultiPLY	[ 62 ]	Embodied LMM that utilizes encoded sensor representations obtained from diverse interactive 3D environments. Sensors include visual, audio, tactile, and thermal information. A large-scale multi-sensor dataset, Multisensory Universe , is also introduced.
Language-based Task & Env. Underst.	PaLM-E	[ 54 ]	Embodied LMM that takes text-based task instructions for various robot tasks, including perception, visually-grounded question-answering, and planning. PaLM-E leverages a pre-trained LLM as the initial state and adapts the model using robot-aware training data.
	CLIPORT	[ 57 ]	CLIP-based language conditioning model for vision-based robot control. The model utilizes semantic (RGB) and spatial (RGB-D) visual data using separate submodules, both conditioned on the CLIP text embedding.

TABLE II : LLM/MLLM for robotics.

	Robot Operating System (ROS)	Vision Language Action Model (VLA)
Purpose	• Middleware framework for robotic system development. • System-level integration of hardware and software for robots.	• AI-driven reasoning and perception. • High-level reasoning, task abstraction, and perception-to-action mappings.
Example	• ROS1 [ 70 ] , ROS2 [ 76 ]	• SayCan [ 55 ] , PaLM-E [ 54 ] , RT-1 [ 14 ] , RT-2 [ 15 ] , OpenVLA [ 65 ] , LLARVA [ 49 ] , $\pi_{0}$ [ 50 ]
Communication & Computing Aspect	• Publish-subscribe model, standardized messaging, widely used. • Distributed processes, moderate hardware requirements.	• High-bandwidth data handling for large AI models, integrates advanced inference frameworks. • Transformer-based deep learning, requires substantial computational resources.
Strength	• Robustness in hardware interfacing and real-world deployment. • Modular design and large ecosystem of tools/libraries.	• Superior performance in high-level tasks and generalization across different task domains. • Leverages state-of-the-art AI advancements.
Limitation	• Communication overhead in large-scale robot systems. • Limited AI/ML capabilities.	• Extremely high computational cost. • Dependency on data and compute infrastructure for training and inference.
Application	• Autonomous navigation, robotic arm control, sensor fusion, multi-robot systems.	• Task planning using natural language, multi-modal reasoning, human-robot interaction.

TABLE III : Comparison of ROS and VLA.

	Principle	Advantages	Disadvantages
Camera	Recognizes surrounding objects using photosensitive sensor	Cost-effectiveness, measurements of texture, color, and contrast	Sensitive to weather conditions, short detection range
Radar	Measures distance and speed using electromagnetic wave	Long-distance object detection, weather resistance, and material penetration detection	Difficult to identify the shape of objects, and low resolution
LiDAR	Measures distance, shape, and speed using laser	High resolution detection compared to radar	High cost, cannot penetrate objects, shorter detection range compared to radar
IMU	Measures motion and orientation using accelerometers and gyroscopes	Compact, lightweight, and capable of high-frequency measurements	Accumulates drift over time, requiring correction from other sensors
Microphone	Converts sound waves into electrical signals	Enables voice recognition, sound localization, and interaction with humans	Sensitive to background noise, limited to auditory applications
Ultrasonic sensor	Measures distance using high-frequency sound waves	Extremely low cost	Very limited range

TABLE IV : Comparison of multimodal sensors.

Sensor type	Sensing frequency	Resolution	Sensitivity	Data volume	Applications by robot type
Camera	30–60 fps	High (e.g., 4K)	High	Very large	Mobile robots: visual SLAM [ 78 , 79 ] Autonomous vehicles: lane detection [ 80 ] , object recognition [ 81 ] Humanoid robots: Human interaction / medical triage / security alert
Radar	20–100 Hz	Medium	Moderate	Moderate	Autonomous vehicles: Obstacle detection in adverse weather [ 82 , 83 ] Industrial robots: Machinery monitoring [ 84 ]
LiDAR	10 Hz	High (mm-level)	High	Large	Autonomous vehicles: navigation and obstacle avoidance [ 85 ] Agriculture robots: Plant detection and mapping [ 86 , 87 ]
IMU	50-1000 Hz	Medium	Low	Low	Drones: Flight stabilization [ 88 ] Humanoid: Balance and motion tracking [ 89 ] Autonomous vehicles: Dead reckoning [ 90 ]
Microphone	16–48 kHz	Medium	Moderate	Moderate	Service robots: Voice interaction [ 91 ] Surveillance robots: Sound localization [ 92 ]
Ultrasonic Sensor	10–100 Hz	Low to medium	Moderate	Low	Mobile robots: Obstacle avoidance Drones: Altitude sensing Warehouse robots: Proximity detection

TABLE V : Sensor characteristics and applications in robotics.

Fig. 4 : A taxonomy of R2X communications.

Ref.	Resources	Architecture	Application	AI model	Control strategy	Features in robotics
[ 123 ]	Processing & networking	Mobile-cloud	cloud storage, offloading	None	Distributed	Energy efficiency
[ 124 ]	Processing & networking	Mobile-edge-cloud	IoT applications	None	Centralized & distributed	Load balancing
[ 125 ]	Processing & networking	Mobile-SBS-MBS	Hierarchical edge computing	None	Distributed	Hybrid control
[ 126 ]	Storage & networking	Mobile-edge	Contents/service caching	None	Centralized & distributed	AI model caching
[ 127 ]	Storage & networking	Mobile-BS	Video streaming	None	Centralized & distributed	Robot-human interaction
[ 128 ]	Storage & networking	Mobile-edge-cloud	Cloud gaming	None	Centralized	Low latency

TABLE VI : Summary of resource management schemes in different network architectures.

Metric	Cloud LLM	Edge LLM	On-Device LLM
Processor	A100 GPU	A10 GPU	Octa-core ARM CPU
Model	ChatGPT-4o	LLaMA 2	TinyLLaMA
Model size	100B+	13–30B	1–7B
Prefill throughput	500–1000 tokens/s	80–250 tokens/s	10–30 tokens/s
Decode throughput	200–400 tokens/s	30–100 tokens/s	3–10 tokens/s
Connectivity	Internet	Local	Not required

TABLE VII : Comparison of estimated throughput between cloud, edge, and on-device LLMs.

Fig. 5 : End-to-end architecture of Demo I (warehouse digital twin): robots uplink semantic visual observations and positions; the server orchestrator fuses sensing and predicted link states to trigger global replanning and proactive uplink configuration, then downlinks updated commands.

Panel of Figure 13 — Fig. 6 : Demo I digital-twin and perception snapshots: (a)–(c) multiple viewpoints of the Isaac Sim warehouse scene and (d) an example robot-view frame with the on-device detection trigger used to request server-side replanning.

Panel of Figure 14 — Fig. 7 : Demo I link-aware planning context: (a) grid-based mobility abstraction and (b)–(c) ray-tracing-derived radio context consumed by the orchestrator for proactive link-aware control.

Uplink sensing type	Payload (per update)	Notes
Raw image streaming (bit-based)	$\approx 1.31$ MB	8-bit quantized images (LORC-P)
Semantic features (proposed)	$\approx 1280$ real values	e.g., $256\times 5$ features (LORC-SC/LORC-SC-P)

TABLE VIII : Demo I sensing payload comparison (uplink).

JSON field	Meaning (objective/constraint/parameter)
pp_config.objective	Planning objective type: makespan (minimize $\max_{i}T_{i}$ ) or safety_first (tighten safety constraints around predicted humans and conflicts).
pp_config.priority_robot	Conflict priority policy: robot_1 / robot_2 / none . When a conflict is detected, the lower-priority robot is constrained (rerouted / delayed) first.
pp_config.min_time_gap_at_conflict	Temporal separation constraint (integer time steps) imposed at shared/conflicting cells to reduce close encounters, especially under safety-first operation.
ra_config.fairness	Wireless fairness policy: max_min (protect worst-case robot) or proportional (maximize weighted $\sum_{i}\log(\cdot)$ style utility).
ra_config.priority_weights	Per-robot weights (e.g., $[w_{1},w_{2}]$ , $w_{1}+w_{2}=1$ ) used to reflect task criticality in the chosen fairness policy.

TABLE IX : Demo I: LLM-configurable knobs for path planning and resource/power configuration (JSON fields and meanings).

Method	Semantic sensing	Predictive link context	Central replanning
Stop-and-Go	–	–	–
LORC-P	– (raw images)	✓	✓
LORC-SC	✓	– (reactive pilots)	✓
LORC-SC-P	✓	✓	✓

TABLE X : Baseline variants in Demo I (which orchestration components are enabled).

Experimental Setup
CPU	AMD Ryzen 5950X (16C/32T)
GPU	2 $\times$ NVIDIA RTX 3090 (24 GB VRAM)
GPU usage	GPU 0: semantic encoder + YOLOv8n, GPU 1: LLaMA 3 8B
Latency Breakdown
Stage	Role in loop	Latency
Object detection (YOLOv8n)	on-robot trigger	$96.17$ ms

TABLE XI : Demo I experimental setup and latency breakdown (representative values).

Method	Scenario 1 (Short)	Scenario 2 (Medium)	Scenario 3 (Long)	Scenario 4 (Long+Obstacles)
Stop-and-Go	72.0	151.0	209.0	209.5
LORC-SC-P	66.0	77.5	108.0	117.0
LORC-P	110.0	127.0	179.5	189.5
LORC-SC	78.5	93.5	128.5	138.0

TABLE XII : Demo I: Task completion time (seconds) for two robots in four warehouse navigation scenarios.

Panel of Figure 20 — Fig. 8 : Demo I (Scenario 1): Qualitative trajectory comparison across baselines.

Panel of Figure 21 — Fig. 9 : Demo I (Scenario 2): Qualitative trajectory comparison across baselines.

Panel of Figure 22 — Fig. 10 : Demo I (Scenario 3): Qualitative trajectory comparison across baselines.

Panel of Figure 23 — Fig. 11 : Demo I (Scenario 4): Qualitative trajectory comparison across baselines. The orchestrated design (LORC-SC-P) reduces waiting/halts by combining semantic sensing with predictive link-aware control.

Fig. 12 : End-to-end architecture of Demo II (mobility scenario): local agents transmit compact semantic features; the Orchestrator MLLM predicts near-future link states and proactively configures MCS (and other network/robot control signals) to stabilize reliability and latency under mobility.

Fig. 13 : Demo II testbed snapshots: urban simulation environment (left) and Sionna 1.2.1-based path-loss heatmap (right).

Panel of Figure 26 — Fig. 14 : Demo II temporal performance: (a) throughput, (b) latency, and (c) BLER. The proposed MLLM-driven MCS control tracks near-ideal behavior and mitigates BLER spikes caused by delayed feedback.

Panel of Figure 27 — Fig. 15 : Demo II SNR-based performance: (a) throughput, (b) latency, and (c) BLER distribution. The proposed MLLM-driven controller consistently outperforms delayed-feedback baselines.

Baseline	TP gain (%)	Latency reduction (%)
Ideal MCS policy ( $\mathrm{BLER}\leq 0.1$ )	-14.31	-39.95
Baseline t–1	2.83	-0.03
Baseline t–3	14.14	11.62
Baseline t–5	26.43	20.23
Baseline t–10	44.77	29.15
Baseline t–20	102.68	40.89

TABLE XIII : Demo II: Aggregate performance gains of the proposed MLLM-driven MCS control over baselines.

Fig. 16 : Demo III real-hardware FollowMe pipeline. The orchestrator switches between JPEG image streaming and compact ViT embeddings based on WiFi state to stabilize closed-loop tracking and control.

Panel of Figure 30 — Fig. 17 : Demo III prototype photos. The system performs human tracking at the edge and sends velocity commands to the robot; uplink sensing is adaptively compressed based on WiFi state.

Mode	Enc.	Comm.	Dec.	Total	Payload	MSE
	(ms)	(ms)	(ms)	(ms)	(KB)	diff
JPEG (Q=95)	6.5	22.34 $\pm$ 20.26	1.6	30.44	80.00	0
JPEG (Q=80)	5.6	22.32 $\pm$ 19.90	1.5	29.42	33.38	0.00017
JPEG (Q=60)	5.4	21.68 $\pm$ 12.26	1.5	28.58	22.28	0.00025
ViT-small (grid $1\times 3$ )	76.4	14.01 $\pm$ 7.85	3.6	94.1	5.16	0.00225
ViT-small (grid $1\times 2$ )	54.5	12.28 $\pm$ 6.42	2.4	69.25	3.44	0.00279

TABLE XIV : Demo III codec-level microbenchmark (5 m LOS, 5 GHz WiFi; representative single-frame measurement).

Method	Sensing policy	Avg. payload (KB)	CTA mean (ms)	CTA 95% (ms)	UTFR (%)	Success ratio (%)
JPEG-only (HQ)	Fixed JPEG streaming (Q=95), reliable QoS	420.0	118.75 $\pm$ 188.83	514.77	63.73	75
JPEG-only (LQ)	Fixed JPEG streaming (Q=60), best-effort QoS	120.0	108.10 $\pm$ 157.82	363.87	26.47	75
ViT-only	Fixed ViT embedding ( $1\times 1$ ), best-effort QoS	1.72	7.99 $\pm$ 6.86	17.93	25.37	75
Orchestrated	LLM-powered switch: JPEG(Q=80) $\rightarrow$ ViT( $1\times 3$ or $1\times 1$ ) + QoS adaptation	15.04	19.60 $\pm$ 9.69	32.62	8.47	100

TABLE XV : Demo III end-to-end performance. Metrics averaged over repeated corridor runs with varying WiFi quality.

Fig. 18 : Demo IV orchestration overview: multi-view sensing (robot camera + corridor camera) is uplinked to an edge server. The edge MLLM performs open-vocabulary grounding (returns bounding boxes) from text instructions, then downlinks grasp/navigation goals. The robot grasps the item and navigates (SLAM) to the correct bin for disposal.

Fig. 19 : Demo IV prototype mobile manipulation platform (mobile base + robot arm).

Fig. 20 : Demo IV task concept: open-vocabulary pick & dispose from a text instruction.

Fig. 21 : Demo IV room-to-corridor navigation: representative SLAM/map view used for bin approach.

Fig. 22 : Demo IV decision logic: closed-set on-device detection when labels exist; otherwise, switch to edge MLLM grounding with multi-view sensing (robot + corridor camera).

Fig. 23 : Open-vocabulary grounding example on a reconstructed image after semantic communication with a 1/12 compression ratio. Given a text query describing a target object that is classifiable by YOLO (e.g., a cup or a person), the MLLM localizes the corresponding region in the image and returns its bounding box coordinates.

Fig. 24 : Open-vocabulary grounding example on a reconstructed image after semantic communication with a 1/12 compression ratio. Given a text query describing a target object that is not classifiable by YOLO (e.g., sneakers, headphones, or a trash bin), the MLLM localizes the corresponding region in the image and returns its bounding box coordinates.

Panel of Figure 40 — Fig. 25 : Baseline illustration: closed-set detection is fast when labels exist, but extending to new targets (e.g., bins) often requires additional labeling and retraining.

Method	Perception stack	Corridor cam	Open-vocab	Item det. success (%)	Bin det. success (%)	Task time (s)	Success ratio (%)
Baseline A	On-device closed-set detector + exploration	–	No	40–50	0	Almost Fail	0
Baseline B	On-device closed-set retrained for bins/items	–	No	50–70	70–90	$70\pm 15$ (Seldom Fail)	40–50
Proposed	Edge MLLM grounding + multi-view (robot + corridor)	✓	Yes	90–95	90–100	$55\pm 5$	90–100

TABLE XVI : Demo IV End-to-end performance in the room-to-corridor trash sorting task.

Demo	Task + environment	Orchestrated knobs (sense/comm/compute)	Measured (or reported) outcomes
Demo I	Warehouse navigation in a ray-tracing digital twin with dynamic humans; two robots must reach goals safely under link variability.	Sense: raw RGB-D vs. compact semantic features; Comm: predictive path-gain-aware adaptation; Compute: server-side global replanning parameterized by schema-valid JSON.	Task completion time across scenarios; end-to-end RTT feasibility relative to grid-transition time; qualitative trajectory stability (halt/wait reduction).
Demo II	Mobility scenario with fast channel variation and delayed feedback; proactive MCS control under reliability constraint.	Sense: uplink semantic features as link-prediction context; Comm: predictive MCS selection with target BLER; Compute: centralized MLLM infers near-future link proxy under delay.	Throughput/latency/BLER time-series stability and SNR-conditioned performance; aggregate gains under delayed feedback.
Demo III	Real FollowMe robot in an indoor corridor over WiFi; maintain smooth tracking/control under interference and range variation.	Sense: JPEG streaming vs. ViT embedding switch; Comm: practical QoS choice (reliable vs. best-effort); Compute: edge tracking/control with closed-loop delay sensitivity.	Codec microbenchmark (payload/latency/MSE); CTA/UTFR and run success.
Demo IV	Real mobile manipulation: open-vocabulary trash sorting across room (item) and corridor (bins) using multi-view sensing.	Sense: robot camera + corridor camera; semantic-sensing switch; Comm: multi-view uplink + bbox/goal downlink; Compute: edge MLLM grounding + goal parsing; Act: grasp + SLAM navigation + disposal.	Open-vocabulary detection success (IoU-based); time-to-first item/bin localization; end-to-end task completion time; success ratio.

TABLE XVII : Cross-demo synthesis: tasks, orchestrated knobs, and measurable outcomes across Demos I–IV.

Signal / loop (examples)	Typical rate	Typical payload	What to report (task-facing, measurable)
Safety interlock (e-stop, proximity alert, speed limit)	event + 10–100 Hz beacons	10–200 B	Tail deadline reliability, e.g., $99.999\%$ delivery within $5$ – $10$ ms; bounded jitter; verified safe fallback if deadline missed.
Coordination state (pose, grasp state, handoff intent)	30–100 Hz	0.2–2 KB	Age-of-information (AoI) distribution and sync error (ms); $99$ th% AoI within $\sim\!20$ – $50$ ms to avoid stale handoff decisions.
Semantic perception (tracks, keypoints, embeddings, scene graph updates)	10–30 Hz	1–50 KB	End-to-end tail delay (e.g., $95/99$ th% within $50$ – $150$ ms) and its impact on cycle time / error rate under contention.
High-fidelity video / logging / remote QA	5–30 Hz	0.1–3 MB	Sustained goodput (Mbps) and graceful degradation (best-effort acceptable) without harming safety/coordination traffic.

TABLE XVIII : Representative quantitative KPIs for collaborative humanoid workcells (illustrative orders of magnitude).

Multi-Agent Orchestration

Multi-agent orchestration coordinates specialized AI agents, tools, and workflows to solve complex tasks efficiently and reliably.

See all 206 papers

IndoorR2X: Indoor Robot-to-Everything Coordination with LLM-Driven Planning
Mar 20, 2026Fan Yang, Soumya Teotia, Shaunak A. Mehta +8World Models for Robotics Large Language Model-Based Robot Planning
1Fujitsu Research · 2Carnegie Mellon University
DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication
Oct 1, 2026Hanchu Zhou, Dechen Gao, Hang Wang +5Semantic Communication Multi-Robot Coordination
University of California, Davis · Microsoft Research · Analog Devices
OSDAG: Online Scheduling for Efficient Multi-Robot Collaboration
Jun 13, 2026Thanh Nguyen Canh, Thang Tran Viet, Phuc Van Dinh +2Multi-Robot Systems Large Language Model-Based Robot Planning
School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea.

Mar 20, 2026J/K move · Enter open · S save

IndoorR2X: Indoor Robot-to-Everything Coordination with LLM-Driven Planning

Fan Yang, Soumya Teotia, Shaunak A. Mehta, Prajit KrisshnaKumar+7

1Fujitsu Research · 2Carnegie Mellon University

World Models for Robotics Large Language Model-Based Robot Planning Multi-Robot Coordination

Open paper AskPDF arXiv

Although robot-to-robot (R2R) communication improves indoor scene understanding beyond what a single robot can achieve, R2R alone cannot overcome partial observability without substantial exploration overhead or scaling team size. In contrast, many indoor environments already include low-cost Internet of Things (IoT) sensors (e.g., cameras) that provide persistent, building-wide context beyond onboard perception. We therefore introduce IndoorR2X, a benchmark and simulation framework for Large Language Model (LLM)-driven multi-robot task planning with Robot-to-Everything (R2X) perception and communication in indoor environments. IndoorR2X integrates observations from mobile robots and static IoT devices to construct a global semantic state that supports scalable scene understanding, reduces redundant exploration, and enables high-level coordination through LLM-based planning. IndoorR2X provides configurable simulation environments, sensor layouts, robot teams, and task suites to systematically evaluate semantic-level coordination strategies. Extensive experiments across diverse settings demonstrate that IoT-augmented world modeling improves multi-robot efficiency and reliability, and we highlight key insights and failure modes for advancing LLM-based collaboration between robot teams and indoor IoT sensors. Project page: https://fandulu.github.io/IndoorR2X_project_page/.

Abstract

Multi-Agent Orchestration

Explore similar work

IndoorR2X: Indoor Robot-to-Everything Coordination with LLM-Driven Planning

DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication

OSDAG: Online Scheduling for Efficient Multi-Robot Collaboration