CoPlanVLM: Coordinated Path Planning for a Robot Team using Vision Language Models
Authors: David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque, Pramod Khargonekar, Kyle Volle, J. Humberto Ramos, Scott Nivison
Organizations: Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Mechanical and Aerospace Engineering at the University of Florida · Torch Technologies, LLC in Shalimar, FL · Air Force Research Laboratory, Air Warfare Directorate, Eglin AFB Okaloosa County, Fl
We present CoPlanVLM, which uses a Vision Language Model (VLM) to translate a single natural-language instruction and image into coordinated paths for a team of two robots. While VLMs offer a flexible, natural-language interface for commanding robot teams, their weak spatial reasoning has greatly limited VLM-based path planning. CoPlanVLM improves the VLM's spatial reasoning and path planning abilities through three components: (1) an image overlay marking both free space and obstacles, (2) task classification that selects the most appropriate low-level planner and reasoning prompt for each instruction, and (3) task-specific chain-of-thought reasoning that prompts the VLM to identify key objects, allocate roles, and verify routes are collision-free before committing to a plan. Together, these components enable the team to complete complex coordinated maneuvers requiring both robots and divide large tasks between robots to increase efficiency. CoPlanVLM also supports detailed maneuvers involving specific routes, loops, and coverage of large areas, as well as tasks requiring directional reasoning such as north/south or left/right. We validate CoPlanVLM in both Gazebo simulation and on physical Turtlebot 4 hardware. CoPlanVLM attains an average success rate of 84.4% compared to 50.0% for the closest baseline. On trials where both methods succeed, CoPlanVLM's paths are 27% shorter on average. Project code is available at https://github.com/AICPS/CoPlanVLM.
Figures & tables
Fig. 1: Paths planned by CoPlanVLM on six different tasks in simulation and hardware experiments. Two tasks are shown from each of the task categories Nav2Point , Coverage , and Maneuver . (Top) Offline testing results on three prompts in the Gazebo environment. Dotted lines indicate planned paths and blue dots with alphanumeric labels mark the waypoints selected by the VLM. (Bottom) Hardware test results on three prompts. Dotted lines indicate planned paths, and solid lines mark the true path followed by the robots with magenta and blue paths corresponding to Raph and Donnie respectively. The robots’ start locations are labeled in green, with stop locations in red.
Fig. 2: CoPlanVLM methodology overview. First an overhead image is processed by CLIPSeg [ 22 ] to detect obstacles and create the occupancy map. A classifier LLM classifies each instruction into one of three categories (Nav2Point, Coverage, Maneuver) which determines the corresponding executive VLM prompt and low-level path planning algorithm. The executive VLM selects marked regions on the image which are used as inputs to the low-level path planning algorithms. These paths are then processed by low level controllers to send commands to the robots.
Fig. 3: Image preprocessing. (Top) Occupancy map visualization with obstacles detected by CLIPSeg shown in red, and obstacle inflation and the map-edge buffer in yellow. (Bottom) Marked image passed to the executive VLM, with free space marked by blue dots, obstacles by red “x” marks, and robots labeled with their names (“raph” and “donnie”).
Prompt
Category
Env.
SR (%)
MS (m)
Prompt
1
Nav2Point
Sim
100.0
14.6
“Surround the person in the purple shirt by sending robots to the left and right sides of the person.”
2
Nav2Point
Sim
80.0
3.6
“Have each robot hide on the left side of the nearest obstacle.”
3
Nav2Point
Sim
80.0
8.0
“Visit each of the four colored boxes.”
4
Nav2Point
AVL
100.0
3.7
“Send both robots to the white X marked in tape.”
5
Nav2Point
AVL
100.0
2.9
“Move one robot to the north of the boxes and one to the south of the boxes.”
6
Nav2Point
AVL
50.0
2.8
“Visit each object in the scene.”
TABLE I: CoPlanVLM success rates and average makespan for all evaluation prompts (10 trials per prompt).
Fig. 4: Measured success rates for CoPlanVLM and compared methods on all 18 prompts.
Method
Nav2Point
Coverage
Maneuver
All
Average Success Rate (%) ↑
CoPlanVLM
85.0
80.0
88.3
84.4
Pixel Selection [ 9 , 31 ]
26.7
25.0
43.3
31.7
Grid Overlay [ 17 ]
40.0
25.0
51.7
38.9
CoNVOI Marking [ 7 ]
45.0
45.0
60.0
50.0
Average Normalized Makespan ↓
TABLE II: Comparison with baselines, averaged by task category.
Method
Nav2Point
Coverage
Maneuver
All
Average Success Rate (%) ↑
CoPlanVLM
85.0
80.0
88.3
84.4
CoPlanVLM w/o Marked Obs.
45.0
95.0
70.0
70.0
CoPlanVLM Maneuver Only
71.7
25.0
88.3
61.7
CoPlanVLM w/o CoT
53.3
71.7
21.7
48.9
Average Normalized Makespan ↓
TABLE III: Ablation study of CoPlanVLM , averaged by task category.
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA