Organizations: Department of Electrical and Systems Engineering, University of Pennsylvania, USA · School of Arts and Sciences, University of Pennsylvania, USA · GRASP Laboratory, University of Pennsylvania, USA
Large Language Models (LLMs) introduce an exciting new paradigm for planning and navigation in robotics, but fail on even simple multi-robot tasks as team sizes grow. We propose COMPASS, a scalable, decentralized multi-robot architecture for controlling large collectives of agentic robots with reasoning space feedback control. Feedback is generated locally on each robot by a spatial transformer which aggregates multi-hop messages across the fleet into a learned feedback token. Our experiments find that collectives of language models demonstrate performance gains from structured diversity of the input command, which can cancel biases; an advantage that is held across scale. Compared against a centralized frontier LLM policy and a language-only communication ablation, we find that the coupled design of COMPASS decisively produces cohesive flocking formations that accurately fly the commanded intent. We show that reasoning feedback works best when composed with a compact learned token. Our ablations show that hand engineered feedback with raw state appearing in the language channel obliterates cohesion. COMPASS generalizes zero-shot to unseen instructions of ambiguous meaning while commanding flocks up to 16 times its training scale, flying up to 1024 robots under natural language commands.
Figures & tables
Fig. 1: Left: “Fly toward twelve o’clock” – the N=1024 decentralized swarm individually decodes the language instruction and mobilizes north-bound at 5ms−1 , shown at 0 , 15 , and 30s (agents black, communication graph at rc=4\text{,}\mathrm{m}$$ ). The center of mass is drawn as a red throughline. Top Right: the speed of every agent converges to the commanded 5ms−1 over the rollout. Bottom Right: the coefficient of variation of the robots’ nearest-neighbor distances converges as the flock settles into an even lattice.
Fig. 2: COMPASS uses local LMs to update the reasoning state z(t) from both the command intent and the learned feedback token (orange) produced by MAST from neighborhood information. COMPASS is deployed identically on each robot despite being trained centralized with spatial masking. Neighboring robots exchange reasoning over the communication network and store received communications in buffers. The same MAST output used to produce feedback is also projected into the action space for execution of low-level control (purple). MAST incorporates low-level state feedback into its input features.
Class
Example
Prompts
Surfaces
Templates
Dirs.
Tr/Va/Te [%]
Plain
“steer North”
1920
64
30
8
45/10/45
Abbreviation
“go N”
1920
64
30
8
45/10/45
Opposite
“the opposite of South”
1920
64
30
8
45/10/45
Map
“toward the top of the map”
1920
64
30
8
45/10/45
Clock
“toward twelve o’clock with twelve at the top”
1920
64
30
8
45/10/45
Numeric
“travel 0.0”
5760
720
8
720
50/10/40
TABLE I: Training and Evaluation Corpus
Fig. 3: Scalability evaluation for the prompt “go east” for N=[8,16,32,64,128,256,512,1024] and 30s rollouts. Our model, COMPASS (green) is compared against a centralized controller with hand engineered feedback using API calls to Claude Opus 4.8 (purple) [ 26 ] and the ablated language-only communication method (red). Left: Velocity variance (consensus) and heading error are shown in log-scale and fragmentation measured during the last 15s of the rollout at each N . Right: Rollouts for each policy shown at the largest evaluated scale. The trajectory of the center of mass (COM) overlays the migrating flock, marked with an × at each of the captured snapshots in time.
Fig. 4: Performance of Claude Opus 4.8 tasked to flock a team of robots over growing scales in a centralized scenario. The baseline prompt (yellow) requested flocking with uniform distance between agents. The no-uniform prompt (brown) limited the task to . The image prompt (blue) incorporates vision for flocking with uniform distance. We evaluate COMPASS on the same initial conditions and demonstrate that it maintains near-uniform nearest-neighbor spacing and latency as the fleet grows. We use the coefficient of variation of each robot’s nearest-neighbor distance, averaged over the rollout, to evaluate the spatial distribution of the fleet across scales.
Fig. 5: Left: Example initialization where four different phrasings are distributed over the team. A 1/4 of the team receives a simpler command for the LM to reason over. Middle: Median COM speed averaged over the trajectory against team size N on a selection of cardinal vague prompts. Right: Median settled heading error over N .
Engine
ℓ
dLM
Err. (v. ref)
Latency / VRAM
Qwen3-0.6B (ref)
28
1024
1.00
3.7 s / 1.2 GB
Qwen3-1.7B
28
2048
0.80
4.0 s / 3.3 GB
Qwen3-4B
36
2560
0.65
5.0 s / 7.7 GB
TABLE II: Reasoning-engine tradeoff on zero-shot vague decoding.
Fig. 6: Zero-shot generalization to new prompt classes ( N=64 , cardinal directions). Left: The box plot shows the settled heading error per evaluation category; strip plot points are colored by the commanded direction. Each class has 16 flights: 4 per cardinal direction. Right: the settled headings for the held-out concept commands; arrows are colored according to the commanded direction. Bottom: The COMPASS flies the command “go east” for 15s and then the command switches to “stop” with thinking regenerated at the switch. The trace shows the absolute difference ∣ COMPASS − expert ∣ in mean fleet speed over time and over 10 trials with N=64 .
Fig. 7: Settled COM headings (red) for the eight held-out secondary intercardinals, N=64 , 60s rollouts. The shaded sector is the commanded 16-wind sector; each panel reports its mean ± std error.
Direction family
Trials
In sector
Med. err. [ ∘ ]
Cardinals (trained)
20
100%
1.1
Intercardinals (trained)
20
100%
1.2
Sec. intercardinals (zero-shot)
24
75%
7.7
Numeric bearings (zero-shot)
16
N/A
2.0
TABLE III: Generalization to the 16-point rose and numeric bearings.
Large Language Models (LLMs) have shown strong capabilities in reasoning and planning, prompting researchers to explore their use as decentralized decision-makers for multi-robot formation control. However, directly applying LLMs to such tasks often leads to unstable behavior, in which robots may collapse to the centroid of their positions or diverge entirely, due to hallucinated reasoning and limited coordination awareness. We propose LLM-Flock, a decentralized multi-robot flocking framework that addresses these challenges through two key mechanisms: (1) an influence-based consensus protocol, where robots iteratively refine individually generated formation plans by adopting those of more connected neighbors, driving the team toward a unified plan without centralized control, and (2) a two-layer collision avoidance strategy that pairs LLM-based urgency reasoning with a programmatic collision checker and re-prompting mechanism. We evaluate LLM-Flock across five state-of-the-art LLMs, including both closed-source (GPT-5, Claude 4.5) and open-source (Llama3.1-405b, Qwen-Max, DeepSeek-R1) models, demonstrating notable improvements in formation stability and convergence over prior LLM-based methods. Real-world experiments with Crazyflie drones further validate the framework's applicability to physical multi-robot systems.
Foundation models are increasingly used for high-level multi-robot task planning, yet independently generated symbolic plans can become jointly infeasible after geometric grounding because of shared spatial and temporal constraints. We study execution-aware coordination of large-language-model (LLM) planners for multi-robot navigation. We compare a centralized architecture, in which one LLM jointly reasons over the robot team, with a decentralized multi-LLM architecture in which each robot independently proposes a task plan. A geometric verifier grounds the candidate plans into robot trajectories, detects predicted inter-robot conflicts, and converts conflict information into structured feedback for targeted replanning. This verifier-mediated architecture separates semantic task reasoning from execution-level feasibility while enabling iterative coordination among LLM planners. Experiments in CARLA and on Crazyflie aerial robots evaluate coordination performance as task complexity increases and characterize the trade-offs between centralized reasoning, decentralized verifier-mediated coordination, and physical executability.
Vivek Khatana, Zijian Song, Naira Hovakimyan +1
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA
This paper presents a framework to prompt multi-robot teams with high-level tasks using natural language expressions. Our objective is to use the reasoning capabilities of language models in understanding and decomposing multi-robot collaboration and decision-making tasks, but in settings where such models cannot be called at deployment time. However, it is hard to specify the behavior of an individual robot from a team instruction, and have it continuously adapt to actions from other robots. This necessitates a framework with the representational capacity required by the logic and semantics of a task, and yet supports decentralized, real-time operation. We solve this dilemma by recognizing that a task can be represented as a deterministic finite automaton, and that recurrent neural networks (RNNs) can encode numerous automata. This allows us to distill the logic and sequential decompositions of sub-tasks obtained from a language model into an RNN, and align its internal states with the semantics of a given task. This leads to a tiny model that encapsulates the reasoning of the language model and can be implemented onboard. To interpret the internal state of the RNN for a decentralized execution, we train a graph neural network control policy conditioned on the hidden states of the RNN and the language embeddings. We present evaluations on simulated and real-world multi-robot tasks that require sequential and collaborative behavior by the team, demonstrating scalable, robust, real-time performance -- sites.google.com/view/prompting-teams.
Eduardo Sebastián, Nicolas Pfitzer, Ajay Shankar +1
Department of Computer Science and Technology, University of Cambridge, UK · Department of Mechanical and Process Engineering, ETH Zurich, Switzerland