Latent Telepathy: Multi-Robot Communication with Self-Supervised Perceptual Latents
Authors: Howard Wang, Han Zheng, Cathy Wu
Organizations: Columbia University, New York, NY, USA · Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA
In a decentralized multi-robot team under partial observability, the fact that decides a robot's next action is often visible only to a teammate. Existing decentralized methods communicate kinematic information, such as position or planned trajectory, which cannot convey what the teammate perceives. Learned communication in multi-agent reinforcement learning (MARL) can carry perceptual content, but the resulting messages are task-coupled and opaque. We propose Latent Telepathy. Each robot broadcasts the perceptual latent vector it already computes for its own use, the output of an encoder trained with a self-supervised joint-embedding predictive objective, frozen, and shared across the team. A teammate learns to act on it from task reward alone. Because the encoder already runs for perception, the message costs no additional computation and a single compact vector of bandwidth. Because the encoder is frozen before any policy is trained, the message means the same thing to every robot, and the receiving robot is never told what it means. We evaluate Latent Telepathy with a content-controlled protocol in which bandwidth, latency, topology and receiver are held fixed and only the message content varies. Broadcasting the latent lets a navigator avoid an occluded hazard in 99.7% of episodes, matching a noiseless hand-designed message. Position and trajectory messages remain at chance, and the raw camera image, 186 times wider, is less reliable than the compressed latent. The result holds from a discrete gridworld to rendered pixels under continuous velocity control, and the encoder decodes the hazard from a physical robot's camera in 102 of 102 live decisions. We also identify a requirement for porting MARL communication results to continuous control, that the decision a message informs must remain reachable by exploration, and show how to restore it.
Figures & tables
Fig. 1: Latent Telepathy: broadcast what you see. (a) The physical arena of the hardware test (Section VI-E ): outer walls and a central divider forming one corridor each side, the red hazard slab, both robots, and the goal pad. (b) The simulated arena: a barrier pierced by two corridors, a slab in one of them, a stationary scout that can see it and a navigator that cannot. (c) At the route decision the navigator’s camera contains 0 of 262,144 hazard pixels under either slab placement; (d) the scout’s post contains 89,475 . The decisive bit exists only on the scout’s side of the channel. Broadcasting the scout’s frozen self-supervised latent lets the navigator choose the clear corridor in 99.7% of episodes; broadcasting its position or trajectory leaves the navigator at chance.
Fig. 2: The pipeline. Stage 1 trains the encoder E on rendered frames with a joint-embedding predictive objective, using no labels and no reward, then freezes it; both robots receive the same weights. Stage 2 learns from reward alone. The scout’s latent and a two-float position anchor form the message; the only trainable module is the part of the navigator that reads it, here a route head that commits to one categorical route for a frozen executor to drive. The scalar episode reward (red) is the only gradient, and it reaches the head alone.
lateral σ
τ
coverage
success
0.55
0
0.00
0.68
0.55
30
0.00
0.61
1.00
0
0.00
0.58
1.00
30
0.01
0.31
1.00
100
0.02
0.22
1.65
30
0.16
0.11
TABLE I: Coverage (alternative-corridor entry) and task success under a swept exploration distribution, 128 episodes per row, one pretrained navigator, message removed, no learning. σ is the imposed lateral noise (the trained value is 0.601 ); τ is the AR(1) correlation time in control steps, τ=0 being iid.
Fig. 3: The point-mass study. (a) Fraction of seeds solved against z=dN/σ , computed before training from task geometry alone, five seeds per setting; marker shape encodes window length N , and a sixfold range of N collapses onto one curve. The dotted line marks our scene. The categorical arm is flat at 1.0 ; the no-signal control never solves. (b) The same runs against coverage realized during training; outcomes are bimodal with complete separation.
role
condition
content f(oti)
width
lower bound
floor
anchor + zeros
66
kinematic
position
anchor + (x,y) , zero-padded
66
kinematic (strong)
trajectory
anchor + (x,y) + const.-vel. path
66
ours (C1)
latent zt
anchor + E(oti)
66
ours (C2)
predicted z^t+1
anchor + P(E(oti),a)
66
upper bound
raw observation
anchor + 64×64×3 frame
12,290
TABLE II: Message contents compared. Every row shares the anchor, delivery, latency, receiver and training budget; only the content differs. Widths are for the pixel experiments (in the gridworld the raw observation is a 225 -cell egocentric patch).
message content
n
route opt.
task success
latent zt
5
0.997±0.006
0.992±0.007
predicted z^t+1
3
0.995±0.005
0.991±0.006
oracle (noiseless bit)
5
0.988±0.006
0.984±0.003
raw observation ( 12,290 -d)
3
0.819±0.230
0.815±0.228
floor (anchor + zeros)
5
0.496±0.020
0.496±0.021
floor, whole wire zeroed
3
0.500±0.016
0.499±0.015
TABLE III: Pixel experiments. Route optimality and task success over the last 500 of 6,000 training episodes, the head’s sampled choice, mean ± s.d. over seeds; executor obedience is 1.000 throughout. Lower block: the same head, wire and executor with the route redrawn every k steps (Section V ).
Fig. 4: Route optimality on pixels under continuous velocity control, seven message contents, per-seed dots, dotted line at chance. Position and trajectory sit on the floor with the empty wire; the 66 -float latent matches the noiseless oracle; the raw image, 186× wider, loses a seed to premature entropy collapse.
wire at evaluation
zt mean
zt range
oracle
intact
1.000
1.000 – 1.000
1.000
content zeroed
0.507
0.496 – 0.527
0.547
whole wire zeroed
0.520
0.496 – 0.555
0.488
sender shuffled
0.492
0.445 – 0.520
0.547
matched-scale noise
0.458
0.410 – 0.484
0.555
TABLE IV: Frozen trained heads re-evaluated greedily under wire corruption, 256 episodes per cell: route optimality of three zt heads (mean and range) and of the oracle head.