Successful cooperation among decentralized agents requires each agent to quickly adapt its plan to the behavior of other agents. In scenarios where agents cannot confidently predict one another's intentions and plans, language communication can be crucial for ensuring safety. In this work, we focus on path-level cooperation in which agents must adapt their paths to one another in order to avoid collisions or perform physical collaboration such as joint carrying. In particular, we propose a safe and interpretable multimodal path planning method, CaPE (Code as Path Editor), which generates and updates path plans for an agent based on the environment and language communication from other agents. CaPE leverages a vision-language model (VLM) to synthesize a path editing program verified by a model-based planner, grounding communication to path plan updates in a safe and interpretable way. We evaluate our approach in diverse simulated and real-world scenarios, including multi-robot and human-robot cooperation in autonomous driving, household, and joint carrying tasks. Experimental results demonstrate that CaPE can be integrated into different robotic systems as a plug-and-play module, greatly enhancing a robot's ability to align its plan to language communication from other robots or humans. We also show that the combination of the VLM-based path editing program synthesis and model-based planning safety enables robots to achieve open-ended cooperation while maintaining safety and interpretability.
Figures & tables
Fig. 2: Illustration of how CaPE works. The example here shows the target agent i ’s planning process given language communication from another agent j , which can be generalized to scenarios with more agents. From the environment, CaPE takes the current state, the communication, and agent j ’s goal. It runs the joint planner to plan candidate paths for the target agent and predict a path for the other agent. With the multimodal context, the VLM writes a path edit program, and a verifier checks each line to obtain the final path.
DSL operation
Functionality
select_path(path_index, agent)
select the indexed path
modify_translation(step, dx, dy, agent)
change the translation of the waypoint at the corresponding step from (x,y) to (x+dx,y+dy)
modify_rotation(step, d θ , agent)
change the rotation of the waypoint at the corresponding step from θ to θ+dθ
wait(step, t, agent)
stop for time t at the location for the corresponding step
insert_waypoint(step, x, y, θ , agent)
after the corresponding step insert a waypoint with translation and rotation (x, y, θ )
TABLE I: Our DSL definition.
Fig. 3: Representative examples of multi-agent cooperation scenarios and the corresponding CaPE integration. ( A ) In multi-robot path coordination, it can take in the communication from the negotiation among multiple robots and edit the path for all robots accordingly. ( B ) In human-robot teaming for household object rearrangement, it can adjust the robot’s low-level path given high-level task assignments for the human and the robot, as well as the human instructions. ( C ) In human-robot teaming for joint carrying, it can align the robot’s path with the one intended by the human instructions.
Models
SR ↑
SEL ↑
Time(s) ↓
Token ↓
Planner-only
23.3
15.1
0
0
RoboPoint
0
0
32.8
2266
VLM Agents
w/ Gemini 3.0 pro
0
0
44.1
w/ Claude Opus 4.5
0
0
12.7
2807
w/ GPT 5.2
10.0
0.6
56.5
2828
TABLE II: Multi-robot coordination results with 2 robots (IV.A). CaPE achieves significantly higher success rates than planner-only and VLM-based baselines, enabling effective multi-agent coordination where conventional planning and waypoint generation fail.
Models
SR ↑
SEL ↑
Time(s) ↓
Token ↓
Planner-only
30.0
25.9
0
0
VLM Agents (w/ Gemini 3 pro)
0
0
33.9
4125
Waypoint (w/ Gemini 3 pro)
0
0
14.8
3666
CaPE (w/ Gemini 3 pro)
60.0
8.9
43.5
1426
CaPE (w/ Finetuned Qwen 2.5 7B)
50.0
18.8
5.9
1257
TABLE III: Multi-robot coordination with 3 robots (IV.A). CaPE maintains strong performance as the number of robots increases, achieving up to 60.0% success rate, while VLM agent and waypoint baselines fail on all tasks.
Models
SR ↑
SEL ↑
Time(s) ↓
Token ↓
Planner-only
8.3
4.9
3.2
0
RoboPoint
3.3
0.8
2.1
507
VLM Agents
w/ Gemini 3.0 pro
0.0
0.0
17.0
4238
w/ Claude Opus 4.5
0.0
0.0
13.7
2378
w/ GPT5.2
0.0
0.0
2.2
2195
TABLE IV: Human-robot teaming in VirtualHome household environments (IV.B). CaPE significantly improves success rate and SEL compared to planner-only, RoboPoint, and VLM-based baselines. The best-performing model (Gemini 3.0 pro) achieves 80.0% success rate, while VLM agent baselines fail and waypoint baselines achieve at most 8.3% success rate.
Fig. 4: Four maps we designed for human and robot joint carrying experiments. The maps are used with common office objects in an open space. For each map, the human and the robot start at the right of the map and need to get to the goals in the left region.
Models
SR ↑
Time(s) ↓
Token ↓
Planner-only
20
9.4
0
Waypoint (w/Gemini 3 pro)
10
5.8
1962
CaPE (w/ Gemini 3 pro)
70
32.7
7694
CaPE (w/ Finetuned Qwen 2.5 7B)
70
14.3
7281
TABLE V: Real-world human-robot joint carrying (IV.C). Both variants of CaPE achieve a higher success rate (70%) than planner-only (20%) and waypoint baseline (10%). The finetuned model achieves lower runtime than the Gemini-based variant while maintaining the same success rate.
Fig. 5: Participants’ subjective ratings for the models. Participants rated how well the robot comprehended their instructions, how efficient the robot’s selected path was, and how safe the overall execution was for both the robot and the human. They also ranked their collaboration with the four models. The numbers indicate means; the error bars indicate standard errors.
Fig. 6: Qualitative examples of CaPE path modification effect. (A) In a real-world joint-carrying task, the path editing program selects the human’s intended path and avoids colliding with obstacles. (B) In a household rearrangement task, when the robot and the human are both approaching the fridge, the robot waits for the human to get to the fridge first. (C) In a household rearrangement task, when the robot and the human are passing through a narrow door, the robot moves away to let the human in first. The path from the joint planner is showing candidate paths for the robot and the predicted path for the human.
Models
SR ↑
SEL ↑
Time(s) ↓
Token ↓
Multi-Robot Coordination
w/ Homotopy-Aware Planner
55.0
4.9
41.0
1552
w/ Single-Path Planner
53.4
6.7
37.2
1504
Human-Robot Teaming
w/ Homotopy-Aware Planner
80.0
22.5
8.8
2711
w/ Single-Path Planner
63.3
17.0
5.0
2249
TABLE VI: Results of CaPE using single path planner.
Models
SR ↑
SEL ↑
Time(s) ↓
Token ↓
Multi-Robot Coordination
w/ Safety Check
55.0
4.9
41.0
1552
w/o Safety Check
53.4
7.0
42.7
1538
Human-Robot Teaming
w/ Safety Check
80.0
22.5
8.8
2711
w/o Safety Check
60.0
14.7
7.3
2440
TABLE VII: Results of CaPE without feasibility checks for the edits
Metrics
Ours
IC-MPC
Object length
0.85m
0.914m
Area size
36 m2
15.68m2
Map number
4
1
Obstacle number
4-7
1
TABLE VIII: Comparing our real experiments setup with IC-MPC
We present CoPlanVLM, which uses a Vision Language Model (VLM) to translate a single natural-language instruction and image into coordinated paths for a team of two robots. While VLMs offer a flexible, natural-language interface for commanding robot teams, their weak spatial reasoning has greatly limited VLM-based path planning. CoPlanVLM improves the VLM's spatial reasoning and path planning abilities through three components: (1) an image overlay marking both free space and obstacles, (2) task classification that selects the most appropriate low-level planner and reasoning prompt for each instruction, and (3) task-specific chain-of-thought reasoning that prompts the VLM to identify key objects, allocate roles, and verify routes are collision-free before committing to a plan. Together, these components enable the team to complete complex coordinated maneuvers requiring both robots and divide large tasks between robots to increase efficiency. CoPlanVLM also supports detailed maneuvers involving specific routes, loops, and coverage of large areas, as well as tasks requiring directional reasoning such as north/south or left/right. We validate CoPlanVLM in both Gazebo simulation and on physical Turtlebot 4 hardware. CoPlanVLM attains an average success rate of 84.4% compared to 50.0% for the closest baseline. On trials where both methods succeed, CoPlanVLM's paths are 27% shorter on average. Project code is available at https://github.com/AICPS/CoPlanVLM.
David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque +4
Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Mechanical and Aerospace Engineering at the University of Florida · Torch Technologies, LLC in Shalimar, FL +1
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
Foundation models are increasingly used for high-level multi-robot task planning, yet independently generated symbolic plans can become jointly infeasible after geometric grounding because of shared spatial and temporal constraints. We study execution-aware coordination of large-language-model (LLM) planners for multi-robot navigation. We compare a centralized architecture, in which one LLM jointly reasons over the robot team, with a decentralized multi-LLM architecture in which each robot independently proposes a task plan. A geometric verifier grounds the candidate plans into robot trajectories, detects predicted inter-robot conflicts, and converts conflict information into structured feedback for targeted replanning. This verifier-mediated architecture separates semantic task reasoning from execution-level feasibility while enabling iterative coordination among LLM planners. Experiments in CARLA and on Crazyflie aerial robots evaluate coordination performance as task complexity increases and characterize the trade-offs between centralized reasoning, decentralized verifier-mediated coordination, and physical executability.
Vivek Khatana, Zijian Song, Naira Hovakimyan +1
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA