Latent Telepathy: Multi-Robot Communication with Self-Supervised Perceptual Latents

Howard Wang; Han Zheng; Cathy Wu

cs.ROSep 20, 2026

Latent Telepathy: Multi-Robot Communication with Self-Supervised Perceptual Latents

Authors: Howard Wang, Han Zheng, Cathy Wu

Organizations: Columbia University, New York, NY, USA · Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA

Abstract

In a decentralized multi-robot team under partial observability, the fact that decides a robot's next action is often visible only to a teammate. Existing decentralized methods communicate kinematic information, such as position or planned trajectory, which cannot convey what the teammate perceives. Learned communication in multi-agent reinforcement learning (MARL) can carry perceptual content, but the resulting messages are task-coupled and opaque. We propose Latent Telepathy. Each robot broadcasts the perceptual latent vector it already computes for its own use, the output of an encoder trained with a self-supervised joint-embedding predictive objective, frozen, and shared across the team. A teammate learns to act on it from task reward alone. Because the encoder already runs for perception, the message costs no additional computation and a single compact vector of bandwidth. Because the encoder is frozen before any policy is trained, the message means the same thing to every robot, and the receiving robot is never told what it means. We evaluate Latent Telepathy with a content-controlled protocol in which bandwidth, latency, topology and receiver are held fixed and only the message content varies. Broadcasting the latent lets a navigator avoid an occluded hazard in 99.7% of episodes, matching a noiseless hand-designed message. Position and trajectory messages remain at chance, and the raw camera image, 186 times wider, is less reliable than the compressed latent. The result holds from a discrete gridworld to rendered pixels under continuous velocity control, and the encoder decodes the hazard from a physical robot's camera in 102 of 102 live decisions. We also identify a requirement for porting MARL communication results to continuous control, that the decision a message informs must remain reachable by exploration, and show how to restore it.

Figures & tables

Fig. 1: Latent Telepathy: broadcast what you see. (a) The physical arena of the hardware test (Section VI-E ): outer walls and a central divider forming one corridor each side, the red hazard slab, both robots, and the goal pad. (b) The simulated arena: a barrier pierced by two corridors, a slab in one of them, a stationary scout that can see it and a navigator that cannot. (c) At the route decision the navigator’s camera contains $0$ of $262{,}144$ hazard pixels under either slab placement; (d) the scout’s post contains $89{,}475$ . The decisive bit exists only on the scout’s side of the channel. Broadcasting the scout’s frozen self-supervised latent lets the navigator choose the clear corridor in $99.7\%$ of episodes; broadcasting its position or trajectory leaves the navigator at chance.

Fig. 2: The pipeline. Stage 1 trains the encoder $E$ on rendered frames with a joint-embedding predictive objective, using no labels and no reward, then freezes it; both robots receive the same weights. Stage 2 learns from reward alone. The scout’s latent and a two-float position anchor form the message; the only trainable module is the part of the navigator that reads it, here a route head that commits to one categorical route for a frozen executor to drive. The scalar episode reward (red) is the only gradient, and it reaches the head alone.

lateral $\sigma$	$\tau$	coverage	success
0.55	0	0.00	0.68
0.55	30	0.00	0.61
1.00	0	0.00	0.58
1.00	30	0.01	0.31
1.00	100	0.02	0.22
1.65	30	0.16	0.11

TABLE I: Coverage (alternative-corridor entry) and task success under a swept exploration distribution,

128

episodes per row, one pretrained navigator, message removed, no learning.

\sigma

is the imposed lateral noise (the trained value is

0.601

);

\tau

is the AR(1) correlation time in control steps,

\tau=0

being iid.

Fig. 3: The point-mass study. (a) Fraction of seeds solved against

z=d\sqrt{N}/\sigma

, computed before training from task geometry alone, five seeds per setting; marker shape encodes window length

N

, and a sixfold range of

N

collapses onto one curve. The dotted line marks our scene. The categorical arm is flat at

1.0

; the no-signal control never solves. (b) The same runs against coverage realized during training; outcomes are bimodal with complete separation.

role	condition	content $f(o^{i}_{t})$	width
lower bound	floor	anchor + zeros	66
kinematic	position	anchor + $(x,y)$ , zero-padded	66
kinematic (strong)	trajectory	anchor + $(x,y)$ + const.-vel. path	66
ours (C1)	latent $z_{t}$	anchor + $E(o^{i}_{t})$	66
ours (C2)	predicted $\hat{z}_{t+1}$	anchor + $P(E(o^{i}_{t}),a)$	66
upper bound	raw observation	anchor + $64{\times}64{\times}3$ frame	12,290

TABLE II: Message contents compared. Every row shares the anchor, delivery, latency, receiver and training budget; only the content differs. Widths are for the pixel experiments (in the gridworld the raw observation is a

225

-cell egocentric patch).

message content	$n$	route opt.	task success
latent $z_{t}$	5	$\mathbf{0.997\pm 0.006}$	$0.992\pm 0.007$
predicted $\hat{z}_{t+1}$	3	$0.995\pm 0.005$	$0.991\pm 0.006$
oracle (noiseless bit)	5	$0.988\pm 0.006$	$0.984\pm 0.003$
raw observation ( $12{,}290$ -d)	3	$0.819\pm 0.230$	$0.815\pm 0.228$
floor (anchor + zeros)	5	$0.496\pm 0.020$	$0.496\pm 0.021$
floor, whole wire zeroed	3	$0.500\pm 0.016$	$0.499\pm 0.015$

TABLE III: Pixel experiments. Route optimality and task success over the last

500

6{,}000

training episodes, the head’s sampled choice, mean

\pm

s.d. over seeds; executor obedience is

1.000

throughout. Lower block: the same head, wire and executor with the route redrawn every

k

steps (Section V ).

Fig. 4: Route optimality on pixels under continuous velocity control, seven message contents, per-seed dots, dotted line at chance. Position and trajectory sit on the floor with the empty wire; the

66

-float latent matches the noiseless oracle; the raw image,

186\times

wider, loses a seed to premature entropy collapse.

wire at evaluation	$z_{t}$ mean	$z_{t}$ range	oracle
intact	$\mathbf{1.000}$	$1.000$ – $1.000$	$1.000$
content zeroed	$0.507$	$0.496$ – $0.527$	$0.547$
whole wire zeroed	$0.520$	$0.496$ – $0.555$	$0.488$
sender shuffled	$0.492$	$0.445$ – $0.520$	$0.547$
matched-scale noise	$0.458$	$0.410$ – $0.484$	$0.555$

TABLE IV: Frozen trained heads re-evaluated greedily under wire corruption,

256

episodes per cell: route optimality of three

z_{t}

heads (mean and range) and of the oracle head.

Multi-Robot Systems

Multi-robot systems coordinate teams of robots to plan, perceive, communicate, and act together in shared tasks and environments.

See all 205 papers

May 26, 2026

SCALE-COMM: Shared, Contrastively-Aligned Latent Embeddings for MARL Communication

Emergent communication enables partially observant Autonomous Mobile Robots (AMRs) to coordinate effectively in decentralized multi-agent reinforcement learning (MARL) settings. However, existing approaches often struggle with unstable communication protocols, ungrounded message semantics, and interference between communication learning and policy optimization, leading to degraded coordination over time. We propose SCALE-COMM (Shared, Contrastively-Aligned Latent Embeddings for COMMunication), a self-supervised framework for learning compact, stable, and policy-relevant communication representations. SCALE-COMM decouples communication learning from policy optimization by training low-dimensional latent messages that capture task-relevant planning and traffic information, while enforcing consistency across agents and time. Across standard MARL benchmarks and a realistic warehouse coordination task, SCALE-COMM consistently outperforms existing communication frameworks in both representation quality and task performance. The learned communication space yields improved stability, sample efficiency, and throughput under policy fine-tuning, demonstrating the effectiveness of representation-driven communication for scalable multi-agent coordination.

Mahmoud Abouelyazid, Eman Hammad

Electrical and Computer Engineering Department Texas A&M University, College Station, TX, USA

Multi-Robot Coordination Decentralized MARL Multi-Agent Communication Emergent Communication

Ask PDF

Jun 10, 2026

CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy

Multi-robot collaboration allows robots to efficiently take on a wide range of tasks, from moving a couch through a doorway to assembling structures on a construction site. However, achieving such coordination in mobile multi-robot settings remains challenging: centralized methods conditioned on the combined observations of a team scale poorly with team size, and decentralized methods that train one policy per robot often require explicit alignment procedures or information sharing at inference time to overcome partial observability. Our key insight is that the visuomotor priors of pretrained vision-language-action (VLA) models should enable reactive, decentralized collaboration from each robot's local observations alone, without these inference-time assumptions. We propose CHORUS, a framework that adapts a single VLA backbone to control diverse, multi-robot teams. At inference time, each robot runs an independent copy of CHORUS, conditioned only on its own observations and a robot-identifying prompt. In real-world experiments including mobile tape measurement, library book handovers, and laundry basket lifting, CHORUS achieves a 64% point improvement over decentralized, from-scratch models, improves reactivity to teammate behavior by 40% points, and outperforms centralized baselines. Together, these results show that a shared VLA backbone is capable of achieving decentralized multi-robot collaboration, without per-robot policies or inter-robot communication at inference.

Ria Doshi, Tian Gao, Annie Chen +2

Stanford University

Mobile Robotics Cross-Embodiment Robot Learning Multi-Robot Coordination Vision-Language-Action Models

Ask PDF

May 25, 2026

Acting on the Unseen: Communication-Free Collaborative Filtering for Decentralized Multi-Robot Task Allocation

Multi-robot task allocation usually assumes some combination of communication, known task models, or a coordinator. We study the opposite extreme, a regime common in practice but overlooked in theory, which we name Zero-Knowledge MRTA (ZK-MRTA): a robot team with no prior knowledge (no task models, not even the latent rank), no communication (no messages, no parameter sharing, no coordinator), and only a partial and privately-noisy view of a public stream of teammates' outcomes. A hidden low-rank structure governs which robot suits which task, and there are far more tasks than rounds, so most (robot, task) pairs are never attempted. Yet each robot can act well on tasks it never attempted, and onboard new tasks, by running online low-rank collaborative filtering over the broadcast (SwarmCF). The advantage over any structure-free learner is categorical, not a constant factor: a structure-free learner is provably at the prior-mean error floor on unseen pairs. We prove a matching per-robot sample complexity (Θ(d) versus Θ(n), in the rank d and the task count n), an anytime (cumulative-reward) separation under task scarcity, and a deterministic condition under which decentralized recovery from the masked broadcast is exact (validated empirically). Experiments quantify the value of the broadcast, a positive scaling law (per-robot unseen-pair skill rises with team size), and the strongest masking-robustness and anytime profile among low-rank methods, recovering most (about 80% on earned skill) of a centralized full-communication ceiling, and holding under capacity-1 contention and in a robotics-grounded sensing instance.

Alexander Apartsin, Yigal Meshulam, Yehudit Aperstein

Holon Institute of Technology, Holon, Israel. · Afeka Tel Aviv Academic College of Engineering, Tel Aviv, Israel

Robust Matrix Completion Multi-Robot Task Allocation Collaborative Filtering

Ask PDF

Abstract

Multi-Robot Systems

Explore similar work

SCALE-COMM: Shared, Contrastively-Aligned Latent Embeddings for MARL Communication

CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy

Acting on the Unseen: Communication-Free Collaborative Filtering for Decentralized Multi-Robot Task Allocation