CoPlanVLM: Coordinated Path Planning for a Robot Team using Vision Language Models
Authors: David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque, Pramod Khargonekar, Kyle Volle, J. Humberto Ramos, Scott Nivison
Organizations: Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Mechanical and Aerospace Engineering at the University of Florida · Torch Technologies, LLC in Shalimar, FL · Air Force Research Laboratory, Air Warfare Directorate, Eglin AFB Okaloosa County, Fl
We present CoPlanVLM, which uses a Vision Language Model (VLM) to translate a single natural-language instruction and image into coordinated paths for a team of two robots. While VLMs offer a flexible, natural-language interface for commanding robot teams, their weak spatial reasoning has greatly limited VLM-based path planning. CoPlanVLM improves the VLM's spatial reasoning and path planning abilities through three components: (1) an image overlay marking both free space and obstacles, (2) task classification that selects the most appropriate low-level planner and reasoning prompt for each instruction, and (3) task-specific chain-of-thought reasoning that prompts the VLM to identify key objects, allocate roles, and verify routes are collision-free before committing to a plan. Together, these components enable the team to complete complex coordinated maneuvers requiring both robots and divide large tasks between robots to increase efficiency. CoPlanVLM also supports detailed maneuvers involving specific routes, loops, and coverage of large areas, as well as tasks requiring directional reasoning such as north/south or left/right. We validate CoPlanVLM in both Gazebo simulation and on physical Turtlebot 4 hardware. CoPlanVLM attains an average success rate of 84.4% compared to 50.0% for the closest baseline. On trials where both methods succeed, CoPlanVLM's paths are 27% shorter on average. Project code is available at https://github.com/AICPS/CoPlanVLM.
Figures & tables
Fig. 1: Paths planned by CoPlanVLM on six different tasks in simulation and hardware experiments. Two tasks are shown from each of the task categories Nav2Point , Coverage , and Maneuver . (Top) Offline testing results on three prompts in the Gazebo environment. Dotted lines indicate planned paths and blue dots with alphanumeric labels mark the waypoints selected by the VLM. (Bottom) Hardware test results on three prompts. Dotted lines indicate planned paths, and solid lines mark the true path followed by the robots with magenta and blue paths corresponding to Raph and Donnie respectively. The robots’ start locations are labeled in green, with stop locations in red.
Fig. 2: CoPlanVLM methodology overview. First an overhead image is processed by CLIPSeg [ 22 ] to detect obstacles and create the occupancy map. A classifier LLM classifies each instruction into one of three categories (Nav2Point, Coverage, Maneuver) which determines the corresponding executive VLM prompt and low-level path planning algorithm. The executive VLM selects marked regions on the image which are used as inputs to the low-level path planning algorithms. These paths are then processed by low level controllers to send commands to the robots.
Fig. 3: Image preprocessing. (Top) Occupancy map visualization with obstacles detected by CLIPSeg shown in red, and obstacle inflation and the map-edge buffer in yellow. (Bottom) Marked image passed to the executive VLM, with free space marked by blue dots, obstacles by red “x” marks, and robots labeled with their names (“raph” and “donnie”).
Prompt
Category
Env.
SR (%)
MS (m)
Prompt
1
Nav2Point
Sim
100.0
14.6
“Surround the person in the purple shirt by sending robots to the left and right sides of the person.”
2
Nav2Point
Sim
80.0
3.6
“Have each robot hide on the left side of the nearest obstacle.”
3
Nav2Point
Sim
80.0
8.0
“Visit each of the four colored boxes.”
4
Nav2Point
AVL
100.0
3.7
“Send both robots to the white X marked in tape.”
5
Nav2Point
AVL
100.0
2.9
“Move one robot to the north of the boxes and one to the south of the boxes.”
6
Nav2Point
AVL
50.0
2.8
“Visit each object in the scene.”
TABLE I: CoPlanVLM success rates and average makespan for all evaluation prompts (10 trials per prompt).
Fig. 4: Measured success rates for CoPlanVLM and compared methods on all 18 prompts.
Method
Nav2Point
Coverage
Maneuver
All
Average Success Rate (%) ↑
CoPlanVLM
85.0
80.0
88.3
84.4
Pixel Selection [ 9 , 31 ]
26.7
25.0
43.3
31.7
Grid Overlay [ 17 ]
40.0
25.0
51.7
38.9
CoNVOI Marking [ 7 ]
45.0
45.0
60.0
50.0
Average Normalized Makespan ↓
TABLE II: Comparison with baselines, averaged by task category.
Method
Nav2Point
Coverage
Maneuver
All
Average Success Rate (%) ↑
CoPlanVLM
85.0
80.0
88.3
84.4
CoPlanVLM w/o Marked Obs.
45.0
95.0
70.0
70.0
CoPlanVLM Maneuver Only
71.7
25.0
88.3
61.7
CoPlanVLM w/o CoT
53.3
71.7
21.7
48.9
Average Normalized Makespan ↓
TABLE III: Ablation study of CoPlanVLM , averaged by task category.
Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility. To successfully execute a task, a robot must decompose goals, select task-relevant objects, and sequence actions, while ensuring that plans satisfy spatial constraints such as limited free space and object collisions. In this work, we propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning. APIVOT learns to leverage language for semantic reasoning, while using visual thoughts as imagined future states for internal verification of geometric feasibility. On long-horizon kitchen tasks, APIVOT outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings. We find that APIVOT learns meaningful modality selection behavior, demonstrating that adaptive interleaving of vision-language thoughts improves both planning success and reasoning efficiency.
Understanding user instructions and object spatial relations in surrounding environments is crucial for intelligent robot systems to assist humans in various tasks. The natural language and spatial reasoning capabilities of Vision-Language Models (VLMs) have the potential to enhance the generalization of robot planners on new tasks, objects, and motion specifications. While foundation models have been applied to task planning, it is still unclear the degree to which they have the capability of spatial reasoning required to enforce user preferences or constraints on motion, such as desired distances from objects, topological properties, or motion style preferences. In this paper, we evaluate the capability of four state-of-the-art VLMs at spatial reasoning over robot motion, using four different querying methods. Our results show that, with the highest-performing querying method, Qwen2.5-VL achieves 71.4% accuracy zero-shot and 75% on a smaller model after fine-tuning, and GPT-4o leads to lower performance. We evaluate two types of motion preferences (object-proximity and path-style), and we also analyze the trade-off between accuracy and computation cost in number of tokens. This work shows some promise in the potential of VLM integration with robot motion planning pipelines.
Foundation models are increasingly used for high-level multi-robot task planning, yet independently generated symbolic plans can become jointly infeasible after geometric grounding because of shared spatial and temporal constraints. We study execution-aware coordination of large-language-model (LLM) planners for multi-robot navigation. We compare a centralized architecture, in which one LLM jointly reasons over the robot team, with a decentralized multi-LLM architecture in which each robot independently proposes a task plan. A geometric verifier grounds the candidate plans into robot trajectories, detects predicted inter-robot conflicts, and converts conflict information into structured feedback for targeted replanning. This verifier-mediated architecture separates semantic task reasoning from execution-level feasibility while enabling iterative coordination among LLM planners. Experiments in CARLA and on Crazyflie aerial robots evaluate coordination performance as task complexity increases and characterize the trade-offs between centralized reasoning, decentralized verifier-mediated coordination, and physical executability.
Vivek Khatana, Zijian Song, Naira Hovakimyan +1
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA