Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers

Frederic Vatnsdal; Roshan Gopal; Romina Garcia Camargo; Vijay Kumar; Alejandro Ribeiro

cs.ROSep 23, 2026

Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers

Authors: Frederic Vatnsdal, Roshan Gopal, Romina Garcia Camargo, Vijay Kumar, Alejandro Ribeiro

Organizations: Department of Electrical and Systems Engineering, University of Pennsylvania, USA · School of Arts and Sciences, University of Pennsylvania, USA · GRASP Laboratory, University of Pennsylvania, USA

Abstract

Large Language Models (LLMs) introduce an exciting new paradigm for planning and navigation in robotics, but fail on even simple multi-robot tasks as team sizes grow. We propose COMPASS, a scalable, decentralized multi-robot architecture for controlling large collectives of agentic robots with reasoning space feedback control. Feedback is generated locally on each robot by a spatial transformer which aggregates multi-hop messages across the fleet into a learned feedback token. Our experiments find that collectives of language models demonstrate performance gains from structured diversity of the input command, which can cancel biases; an advantage that is held across scale. Compared against a centralized frontier LLM policy and a language-only communication ablation, we find that the coupled design of COMPASS decisively produces cohesive flocking formations that accurately fly the commanded intent. We show that reasoning feedback works best when composed with a compact learned token. Our ablations show that hand engineered feedback with raw state appearing in the language channel obliterates cohesion. COMPASS generalizes zero-shot to unseen instructions of ambiguous meaning while commanding flocks up to 16 times its training scale, flying up to 1024 robots under natural language commands.

Figures & tables

Fig. 1: Left: “Fly toward twelve o’clock” – the $N=1024$ decentralized swarm individually decodes the language instruction and mobilizes north-bound at $5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ , shown at $0$ , $15$ , and $30\text{\,}\mathrm{s}$ (agents black, communication graph at $r_{c}=$4\text{\,}\mathrm{m}$$ ). The center of mass is drawn as a red throughline. Top Right: the speed of every agent converges to the commanded $5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ over the rollout. Bottom Right: the coefficient of variation of the robots’ nearest-neighbor distances converges as the flock settles into an even lattice. — Fig. 1: Left: “Fly toward twelve o’clock” – the $N=1024$ decentralized swarm individually decodes the language instruction and mobilizes north-bound at $5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ , shown at $0$ , $15$ , and $30\text{\,}\mathrm{s}$ (agents black, communication graph at $r_{c}=$ 4\text{,}\mathrm{m}$$ ). The center of mass is drawn as a red throughline. Top Right: the speed of every agent converges to the commanded $5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ over the rollout. Bottom Right: the coefficient of variation of the robots’ nearest-neighbor distances converges as the flock settles into an even lattice.

Fig. 2: COMPASS uses local LMs to update the reasoning state

\bf{z}(t)

from both the command intent and the learned feedback token (orange) produced by MAST from neighborhood information. COMPASS is deployed identically on each robot despite being trained centralized with spatial masking. Neighboring robots exchange reasoning over the communication network and store received communications in buffers. The same MAST output used to produce feedback is also projected into the action space for execution of low-level control (purple). MAST incorporates low-level state feedback into its input features.

Class	Example	Prompts	Surfaces	Templates	Dirs.	Tr/Va/Te [%]
Plain	“steer North”	1920	64	30	8	45/10/45
Abbreviation	“go N”	1920	64	30	8	45/10/45
Opposite	“the opposite of South”	1920	64	30	8	45/10/45
Map	“toward the top of the map”	1920	64	30	8	45/10/45
Clock	“toward twelve o’clock with twelve at the top”	1920	64	30	8	45/10/45
Numeric	“travel 0.0”	5760	720	8	720	50/10/40

TABLE I: Training and Evaluation Corpus

Fig. 3: Scalability evaluation for the prompt “go east” for

N=[8,16,32,64,128,256,512,1024]

and

30\text{\,}\mathrm{s}

rollouts. Our model, COMPASS (green) is compared against a centralized controller with hand engineered feedback using API calls to Claude Opus 4.8 (purple) [ 26 ] and the ablated language-only communication method (red). Left: Velocity variance (consensus) and heading error are shown in log-scale and fragmentation measured during the last

15\text{\,}\mathrm{s}

of the rollout at each

N

. Right: Rollouts for each policy shown at the largest evaluated scale. The trajectory of the center of mass (COM) overlays the migrating flock, marked with an

\times

at each of the captured snapshots in time.

Fig. 4: Performance of Claude Opus 4.8 tasked to flock a team of robots over growing scales in a centralized scenario. The baseline prompt (yellow) requested flocking with uniform distance between agents. The no-uniform prompt (brown) limited the task to . The image prompt (blue) incorporates vision for flocking with uniform distance. We evaluate COMPASS on the same initial conditions and demonstrate that it maintains near-uniform nearest-neighbor spacing and latency as the fleet grows. We use the coefficient of variation of each robot’s nearest-neighbor distance, averaged over the rollout, to evaluate the spatial distribution of the fleet across scales.

Fig. 5: Left: Example initialization where four different phrasings are distributed over the team. A 1/4 of the team receives a simpler command for the LM to reason over. Middle: Median COM speed averaged over the trajectory against team size

N

on a selection of cardinal vague prompts. Right: Median settled heading error over

N

Engine	$\ell$	$d_{\mathrm{LM}}$	Err. (v. ref)	Latency / VRAM
Qwen3-0.6B (ref)	28	1024	1.00	3.7 s / 1.2 GB
Qwen3-1.7B	28	2048	0.80	4.0 s / 3.3 GB
Qwen3-4B	36	2560	0.65	5.0 s / 7.7 GB

TABLE II: Reasoning-engine tradeoff on zero-shot vague decoding.

Fig. 6: Zero-shot generalization to new prompt classes (

N=64

, cardinal directions). Left: The box plot shows the settled heading error per evaluation category; strip plot points are colored by the commanded direction. Each class has

16

flights:

4

per cardinal direction. Right: the settled headings for the held-out concept commands; arrows are colored according to the commanded direction. Bottom: The COMPASS flies the command “go east” for

15\text{\,}\mathrm{s}

and then the command switches to “stop” with thinking regenerated at the switch. The trace shows the absolute difference

|

COMPASS

-

expert

|

in mean fleet speed over time and over

10

trials with

N=64

Fig. 7: Settled COM headings (red) for the eight held-out secondary intercardinals,

N=64

60\text{\,}\mathrm{s}

rollouts. The shaded sector is the commanded 16-wind sector; each panel reports its mean

\pm

std error.

Direction family	Trials	In sector	Med. err. [ ∘ ]
Cardinals (trained)	20	100%	1.1
Intercardinals (trained)	20	100%	1.2
Sec. intercardinals (zero-shot)	24	75%	7.7
Numeric bearings (zero-shot)	16	N/A	2.0

TABLE III: Generalization to the 16-point rose and numeric bearings.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,001 papers

LLM-Flock: LLM-Driven Decentralized Multi-Robot Flocking via Influence-Based Consensus with Collision Avoidance
May 10, 2025Peihan Li, Lifeng ZhouMulti-Robot Formation Control Swarm Robotics
Drexel University, United States
Verifier-Mediated Multi-LLM Coordination for Multi-Robot Path Planning
Oct 8, 2026Vivek Khatana, Zijian Song, Naira Hovakimyan +1Multi-Robot Coordination Large Language Model-Based Robot Planning
Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Mechanical Engineering, University of Nevada, Reno, NV, USA
Prompting Robot Teams with Natural Language
Sep 29, 2025Eduardo Sebastián, Nicolas Pfitzer, Ajay Shankar +1Large Language Model-Based Robot Planning Multi-Robot Coordination
Department of Computer Science and Technology, University of Cambridge, UK · Department of Mechanical and Process Engineering, ETH Zurich, Switzerland

May 10, 2025J/K move · Enter open · S save

LLM-Flock: LLM-Driven Decentralized Multi-Robot Flocking via Influence-Based Consensus with Collision Avoidance

Peihan Li, Lifeng Zhou

Drexel University, United States

Multi-Robot Formation Control Swarm Robotics Language Model-Based Control Multi-Agent Consensus

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown strong capabilities in reasoning and planning, prompting researchers to explore their use as decentralized decision-makers for multi-robot formation control. However, directly applying LLMs to such tasks often leads to unstable behavior, in which robots may collapse to the centroid of their positions or diverge entirely, due to hallucinated reasoning and limited coordination awareness. We propose LLM-Flock, a decentralized multi-robot flocking framework that addresses these challenges through two key mechanisms: (1) an influence-based consensus protocol, where robots iteratively refine individually generated formation plans by adopting those of more connected neighbors, driving the team toward a unified plan without centralized control, and (2) a two-layer collision avoidance strategy that pairs LLM-based urgency reasoning with a programmatic collision checker and re-prompting mechanism. We evaluate LLM-Flock across five state-of-the-art LLMs, including both closed-source (GPT-5, Claude 4.5) and open-source (Llama3.1-405b, Qwen-Max, DeepSeek-R1) models, demonstrating notable improvements in formation stability and convergence over prior LLM-based methods. Real-world experiments with Crazyflie drones further validate the framework's applicability to physical multi-robot systems.

Abstract

Multi-Agent LLM Systems

Explore similar work

LLM-Flock: LLM-Driven Decentralized Multi-Robot Flocking via Influence-Based Consensus with Collision Avoidance

Verifier-Mediated Multi-LLM Coordination for Multi-Robot Path Planning

Prompting Robot Teams with Natural Language