Safe Multi-Robot Coordination via VLM-LLM Reasoning and Reachability Analysis

Mohamed Dwedar; Ahmad Hafez; Alexander Jesser; Amr Alanwar

cs.ROAug 18, 2026

Safe Multi-Robot Coordination via VLM-LLM Reasoning and Reachability Analysis

Authors: Mohamed Dwedar, Ahmad Hafez, Alexander Jesser, Amr Alanwar

Organizations: TUM School of Computation, Information, and Technology (CIT), Technical University of Munich (TUM), Heilbronn, Germany. · Institute for Intelligent Cyber-Physical Systems (ICPS), Heilbronn University of Applied Science, Heilbronn, Germany.

Abstract

Safe coordination in heterogeneous machine-to-machine (M2M) robotic systems is challenging when robots differ in sensing capabilities, environmental awareness, and motion execution roles. This paper presents a centralized safety-aware M2M framework for cooperative goal-directed navigation in a heterogeneous mobile robot team comprising a vision-capable quadruped and a camera-less robotic vehicle. The objective is to guide both platforms toward a goal region while avoiding static and dynamic obstacles and preventing unsafe inter-robot interactions. Under the principle of shared perception, the vision-capable robot provides semantic environmental awareness through a centralized server over an MQTT broker, enabling the camera-less platform to navigate using this shared scene representation alongside its own odometry, IMU, and state feedback. A vision-language model (VLM) interprets the visual stream, and the extracted semantic data is mapped into conservative metric geometric constraints, including inflated obstacle sets, safe corridors, and goal regions. A large language model (LLM) proposes high-level task allocations, while physical command authority is restricted to a robot-specific zonotope reachability gate. This verification engine propagates independent reachable tubes to evaluate obstacle avoidance, safe-corridor containment, and inter-robot separation predicates before approving commands. Online experiments across clear-path and dynamic-obstacle scenarios show that the pipeline reliably approves safe motion, triggers conservative replanning or holding maneuvers upon constraint violation, and enforces a strict architectural separation between advisory semantic reasoning and formally verified motor execution.

Figures & tables

Fig. 1 : Simplified view of the proposed method. Shaded blocks are gates; light blocks are advisory and carry no command authority. Semantic evidence produced by the Go2 propagates to SVEA, but command authority does not: Gate 4 is evaluated independently for each robot.

		Capability axis
System	Category	SEM	SYM	GATE	MR	XFER	INT	Safety mechanism reported
Language-model planning and vision–language–action models
SayCan [ 6 ]	LLM planning	✓	✓	✗	✗	✗	✗	Affordance scoring; no kinodynamic verification.
PaLM-E [ 11 ]	VLA model	✓	✓	✗	✗	✗	✗	End-to-end; safety implicit in training data.
RT-2 [ 40 ]	VLA model	✓	(✓)	✗	✗	✗	✗	Co-trained action tokens; no runtime gate.
Code-as-Policies [ 23 ]	LLM code generation	✓	✓	✗	✗	✗	✗	Programmatic guards written by the model itself.

TABLE I : Capability comparison against representative systems from the three literatures the proposed pipeline draws on. ✓ provided; (✓) partial, offline, or informal; ✗ not provided. The final row tallies how many of the twelve prior systems cover each axis.

Fig. 2 : Centralized M2M multi-robot safety decision pipeline.

Component	Pipeline Role	Produced Information	Command Authority
Unitree Go2	Vision-capable robot; publishes camera frames, odometry, IMU, and robot-state data over MQTT.	ROS 2 image topics; odometry, IMU, and state messages.	None; receives approved commands only.
SVEA robot car	Camera-less robot; publishes odometry, IMU, and state; executes independently verified robot-specific commands.	Odometry, IMU, and state messages; used for robot-specific reachability evaluation.	None; receives approved commands only.
Mosquitto MQTT	Asynchronous M2M transport for telemetry, status messages, and approved command packets.	Routed state messages, server status, and verified commands.	None; transport layer only.
Qwen2.5-VL via Ollama	Semantic interpretation of Go2 camera frames.	Schema-validated JSON scene output with labels, regions, and confidence values.	None; perception evidence only.
Qwen3 via Ollama	Symbolic task-level reasoning from validated scene context and robot states.	Symbolic task proposal for downstream reachability verification.	None; symbolic guidance only.
Python zonotope module	CORA-style set propagation; robot-specific predicate evaluation.	Safety flag $\sigma_{k,i}$ and approved command $u_{k,i}^{\mathrm{safe}}$ .	Sole command-approval authority.

TABLE II : System components, pipeline roles, and command authority.

Fig. 3 : Online clear-path validation from the Go2 camera stream.

Fig. 4 : Corridor-containment in the local workspace.

Fig. 5 : Online dynamic-obstacle validation from the Go2 camera stream.

Fig. 6 : Reachability under dynamic-obstacle interaction.

Fig. 7 : Reconstructed Go2 and SVEA planar trajectories.

Fig. 8 : Confidence, reprojection error, and safety-radius surface.

Fig. 9 : VLM confidence versus conservative safety radius.

ID	Scenario	Cycles	Duration (s)	Objective	Predicate stressed
S1	Office room, goal approach	720	24.0	Establish nominal acceptance: verify the gate approves motion when a traversable corridor and a goal cue are both present and no dynamic obstacle is in the workspace.	$\eta^{\mathrm{obs}}$ , $\eta^{\mathrm{cor}}$
S2	Stairwell, non-traversable hazard	361	12.0	Establish maximal rejection: verify that a region which is geometrically open but not traversable is excluded by corridor containment even though no obstacle intersection occurs.	$\eta^{\mathrm{cor}}$ (isolated)
S3	Corridor, pedestrian entry	421	14.0	Verify dynamic-obstacle handling: compound inflation of a moving human footprint and redirection of the candidate path through the remaining traversable corridor.	$\eta^{\mathrm{obs}}$
S4	Sustained human tracking	451	15.0	Verify sustained operation under continuous re-planning at close range, including decision stability and clearance maintenance.	$\eta^{\mathrm{obs}}$ , $\eta^{\mathrm{cor}}$
Total		1,953	65.1	$17{,}577$ verified candidate arcs, $N=15$ steps each

TABLE III : Test matrix: evaluation scenarios, objectives, and the safety predicate each is designed to exercise.

ID	Scenario	Cycles	Arcs	Approved	Withheld	Safe	Warning	Unsafe	Mean risk
		Volume		Gate outcome (%)		Internal safety state (%)
S1	Office, goal approach	720	6,480	56.4	43.6	92.2	7.8	0.0	$0.187\pm 0.092$
S2	Stairwell hazard	361	3,249	0.0	100.0	48.5	51.5	0.0	$0.341\pm 0.126$
S3	Corridor, pedestrian	421	3,789	47.7	52.3	78.9	17.1	4.0	$0.272\pm 0.186$
S4	Sustained tracking	451	4,059	37.9	62.1	18.8	12.6	68.5	$0.678\pm 0.230$
All scenarios		1,953	17,577	39.8	60.2	64.3	19.0	16.7	$\mathbf{0.347\pm 0.248}$

TABLE IV : Gate behavior per evaluation scenario. Approval and withholding are reported as fractions of decision cycles; the internal safety state is the fraction of cycles spent in each condition; risk is the mean cycle-level score

\pm

one standard deviation.

Motion approved
Decision	Cycles	Share	Mean risk	Clearance (m)
PROCEED	247	$12.6\%$	$0.155$	$2.43$
AVOID $+$ PROCEED	201	$10.3\%$	$0.737$	$0.92$
TURN LEFT	170	$8.7\%$	$0.300$	$1.70$
TURN RIGHT	160	$8.2\%$	$0.167$	$1.56$
Motion withheld

TABLE V : Statistics conditioned on the decision issued by the gate. Clearance is the mean distance to the closest detected obstacle in cycles carrying that decision.

Unresolved confidence: 89 ( $9.1\%$ ), treated as maximum uncertainty.
Class	Detections	Confidence	Range (m)	Min (m)
Person	456	$0.870\pm 0.068$	$0.91\pm 0.26$	$0.51$
Chair	404	$0.842\pm 0.083$	$3.19\pm 2.94$	$0.99$
Cabinet	113	$0.613\pm 0.086$	$1.00\pm 0.42$	$0.64$
All	973	$0.844\pm 0.095$	$1.86\pm 2.21$	$0.51$
Dynamic: 456 ( $46.9\%$ ); static: 517 ( $53.1\%$ ).

TABLE VI : Perception statistics over the

973

detections admitted to geometric conversion.

Condition	Active Predicate(s)	Decision	Status
All predicates satisfied	$\eta^{\mathrm{obs}}\land\eta^{\mathrm{cor}}\land\eta^{\mathrm{int}}$	PROCEED	Approved
Obstacle present; alt. path available	$\eta^{\mathrm{obs}}\land\eta^{\mathrm{cor}}\land\eta^{\mathrm{int}}$ (after redirect)	AVOID+PROCEED	Approved
Obstacle-free; corridor violated	$\eta^{\mathrm{obs}}\land\neg\eta^{\mathrm{cor}}\land\eta^{\mathrm{int}}$	REPLAN	Withheld
Obstacle intersection	$\neg\eta^{\mathrm{obs}}$	STOP/HOLD	Withheld
Inter-robot overlap	$\neg\eta^{\mathrm{int}}$	HOLD	Withheld
Synchronization failure	Gate 1 (Eq. ( 2 ))	HOLD/STOP	Withheld

TABLE VII : Reachability gate decision logic.

Pipeline Stage	Symbol	Range (ms)	Budget (ms)	Rationale
MQTT transport	$T_{\mathrm{MQTT}}$	5–25	15	Local LAN or loopback broker; single-hop, low contention.
Synchronization & validation	$T_{\mathrm{sync}}$	1–8	5	Timestamp-pair checking, alignment, duplicate and out-of-order rejection.
VLM inference & JSON validation	$T_{\mathrm{VLM}}$	350–1200	750	Qwen2.5-VL via Ollama; range depends on GPU availability, image resolution, and prompt complexity.
Semantic-to-geometric conversion	$T_{\mathrm{geo}}$	5–35	20	Homography projection, Minkowski inflation, corridor and goal-region construction.
LLM symbolic reasoning	$T_{\mathrm{LLM}}$	120–600	300	Qwen3 via Ollama processing compact JSON scene context and robot states.
Reachability propagation ( ${\times}2$ )	$T_{\mathrm{reach}}$	20–160	100	Short-horizon zonotope propagation and three-predicate evaluation per robot; $50\,\mathrm{ms}{\times}2$ .

TABLE VIII : Representative per-cycle design-time latency budget.

ID	Condition	Pipeline Modification	Isolated Contribution
B1	No reachability gate	Reachability module disabled; LLM proposals published without geometric verification.	Role of the reachability gate in preventing unsafe command execution.
B2	No VLM perception	VLM removed; geometric conversion receives no semantic scene evidence.	Role of online semantic perception in constructing obstacle sets, safe corridors, and goal regions.
B3	VLM $+$ LLM without reachability	Both language models retained; reachability gate removed.	Whether language-model outputs alone suffice for safe command publication.
B4	No shared perception for SVEA	SVEA does not receive Go2-derived scene evidence and relies only on its own state feedback.	Role of shared perception for the camera-less robot’s safety constraint construction.
A1	Obstacle inflation sweep	$r_{\mathrm{safe}}$ varied; remaining pipeline unchanged.	Sensitivity of command approval rate to obstacle buffering.
A2	Perception uncertainty sweep	$r_{\mathrm{perc}}$ varied with VLM confidence.	Sensitivity to semantic uncertainty in the safety-margin model.

TABLE IX : Baseline and ablation evaluation conditions.

Safe Robot Navigation

Safe robot navigation develops methods for reaching goals while avoiding collisions and managing uncertainty, dynamic obstacles, and environmental hazards.

See all 141 papers

Verifier-Mediated Multi-LLM Coordination for Multi-Robot Path Planning
Oct 8, 2026Vivek Khatana, Zijian Song, Naira Hovakimyan +1Multi-Robot Coordination Large Language Model-Based Robot Planning
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA
D-VLC: Decentralized Vision-Language Collaboration for Heterogeneous Embodied Multi-Robot Systems in Unknown Environments
Jul 31, 2026Yuan Zhou, Ruitong Lin, Shen Wang +5Multi-Agent LLM Systems Multi-Robot Systems
Zhejiang University, Hangzhou, China · Differential Robotics, Hangzhou, China · Beihang University, Beijing, China
CoPlanVLM: Coordinated Path Planning for a Robot Team using Vision Language Models
Oct 9, 2026David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque +4Robot Task Planning Multi-Robot Motion Planning
Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Mechanical and Aerospace Engineering at the University of Florida · Torch Technologies, LLC in Shalimar, FL +1

Oct 8, 2026J/K move · Enter open · S save

Verifier-Mediated Multi-LLM Coordination for Multi-Robot Path Planning

Vivek Khatana, Zijian Song, Naira Hovakimyan, Petros Voulgaris

Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA

Multi-Robot Coordination Large Language Model-Based Robot Planning Multi-LLM Collaboration Multi-Robot Motion Planning

Open paper AskPDF arXiv

Foundation models are increasingly used for high-level multi-robot task planning, yet independently generated symbolic plans can become jointly infeasible after geometric grounding because of shared spatial and temporal constraints. We study execution-aware coordination of large-language-model (LLM) planners for multi-robot navigation. We compare a centralized architecture, in which one LLM jointly reasons over the robot team, with a decentralized multi-LLM architecture in which each robot independently proposes a task plan. A geometric verifier grounds the candidate plans into robot trajectories, detects predicted inter-robot conflicts, and converts conflict information into structured feedback for targeted replanning. This verifier-mediated architecture separates semantic task reasoning from execution-level feasibility while enabling iterative coordination among LLM planners. Experiments in CARLA and on Crazyflie aerial robots evaluate coordination performance as task complexity increases and characterize the trade-offs between centralized reasoning, decentralized verifier-mediated coordination, and physical executability.

Abstract

Safe Robot Navigation

Explore similar work

Verifier-Mediated Multi-LLM Coordination for Multi-Robot Path Planning

D-VLC: Decentralized Vision-Language Collaboration for Heterogeneous Embodied Multi-Robot Systems in Unknown Environments

CoPlanVLM: Coordinated Path Planning for a Robot Team using Vision Language Models