Emergent Compositional Communication for Latent World Properties

Tomek Kaszyński

cs.MAMar 18, 2026

Emergent Compositional Communication for Latent World Properties

Authors: Tomek Kaszyński

Organizations: Independent Researcher, Amsterdam, Netherlands

Abstract

Can multi-agent communication pressure extract discrete, compositional representations of invisible physical properties from frozen video features? We show that agents communicating through a Gumbel-Softmax bottleneck with iterated learning develop positionally disentangled protocols for latent properties (elasticity, friction, mass ratio) without property labels or supervision on message structure. With 4 agents, 100% of 80 seeds converge to near-perfect compositionality (PosDis=0.999, holdout 98.3%). Controls confirm multi-agent structure -- not bandwidth or temporal coverage -- drives this effect. Causal intervention shows surgical property disruption (~15% drop on targeted property, <3% on others). A controlled backbone comparison reveals that the perceptual prior determines what is communicable: DINOv2 dominates on spatially-visible ramp physics (98.3% vs 95.1%), while V-JEPA 2 dominates on dynamics-only collision physics (87.4% vs 77.7%, d=2.74). Scale-matched (d=3.37) and frame-matched (d=6.53) controls attribute this gap entirely to video-native pretraining. The frozen protocol supports action-conditioned planning (91.5%) with counterfactual velocity reasoning (r=0.780). Validation on Physics 101 real camera footage confirms 85.6% mass-comparison accuracy on unseen objects, temporal dynamics contributing +11.2% beyond static appearance, agent-scaling compositionality replicating at 90% for 4 agents, and causal intervention extending to real video (d=1.87, p=0.022).

Figures & tables

Group	$N$	Holdout (both)	PosDis	Comp. seeds
$2{\times}5$ compositional (PosDis $>0.4$ )	43	79.3% $\pm$ 5.2%	0.617 $\pm$ 0.15	43/80 (54%)
$2{\times}5$ holistic (PosDis $\leq 0.4$ )	37	75.9% $\pm$ 6.1%	0.244 $\pm$ 0.09	—
$1{\times}25$ control	20	71.2% $\pm$ 3.5%	0.000	0/20 (0%)

Table 1: Two-property communication on ramp physics (80 seeds, 400 epochs). Holdout accuracy on unseen elasticity-friction combinations. Compositional seeds (PosDis

>0.4

) generalize significantly better (

p{=}0.011

, Cohen’s

d{=}0.59

Figure 1: Mutual information between message positions and physical properties. Compositional agents (left) show clean diagonal specialization; holistic agents (right) encode both properties in a single undifferentiated symbol.

Encoder ablation (2 agents, IL + pop., 400 epochs):
Configuration	Epochs	Train	Holdout	PosDis	Comp. %
LazImpa (no IL, lazy+impatient)	400	—	70.3%	0.165	0%
No iterated learning	400	93.6%	77.7%	0.311	20%
IL + population (simultaneous)	200	—	74.5%	0.295	40%
IL + pop. + extended (2 agents)	400	—	79.3%	0.444	54%
IL + pop. + extended (4 agents)	400	—	98.3%	0.999	100%

Table 2: Ablation of training components (all using DINOv2,

2{\times}5

vocabulary). Train-holdout gap reveals that iterated learning regularizes: no-IL overfits heavily while IL sacrifices train accuracy for compositional structure. LazImpa ( Rita et al.,, 2020 ) , designed for efficiency rather than compositionality, does not produce disentangled protocols on this task. Scaling to 4 agents makes compositionality reliable across all seeds. End-to-end fine-tuning of the encoder degrades communication despite improving feature quality (

R^{2}=0.988

vs.

0.953

Figure 2: PosDis distribution across 20 seeds (representative subset; full 80-seed characterization in Table 1 ). Compositional seeds (PosDis

>0.4

) generalize significantly better than holistic seeds. The distribution is bimodal with mass above 0.4 and below 0.3; the same pattern holds across all 80 seeds (43/80 compositional).

Condition	Holdout	Std
Frozen compositional sender	93.8%	0.7%
Freshly-trained $2{\times}5$	92.2%	2.4%
Freshly-trained $1{\times}25$ holistic	87.5%	9.5%

Table 3: Cross-property reasoning (20 seeds). The frozen compositional sender outperforms both freshly-trained conditions on a novel task.

Zeroed Position	Accuracy Drop	Relevance
A position 0 (elasticity of A)	$-$ 14.7% $\pm$ 1.6%	Relevant
A position 1 (friction of A)	$-$ 0.4% $\pm$ 1.9%	Irrelevant
B position 0 (elasticity of B)	$-$ 2.9% $\pm$ 1.7%	Irrelevant
B position 1 (friction of B)	$-$ 15.2% $\pm$ 0.8%	Relevant

Table 4: Position ablation for the frozen compositional sender on cross-property task. Zeroing relevant positions causes large drops; irrelevant positions cause negligible drops.

Figure 3: Causal intervention on message positions during cross-property reasoning. The receiver selectively reads elasticity from position 0 of message A (

-

14.7%) and friction from position 1 of message B (

-

15.2%), while irrelevant positions cause negligible drops.

Domain	Both Acc.	Oracle	Spec. Ratio	Info Structure
Spring-mass (2-prop)	93.8% $\pm$ 1.2%	95.7%	0.946	Temporal, clean separation
Ramp physics (2-prop)	88.4% $\pm$ 1.6%	$\sim$ 90%	0.688	Temporal, partial separation
Abstract scenes (2-prop)	50.4% $\pm$ 4.4%	94.7%	0.582	Spatial, uniform
Visual attributes (2-prop)	75.9% $\pm$ 1.0%	75.7%	0.500	Static, uniform

Table 5: Cross-domain results with identical communication architecture. Encoders differ per domain (DINOv2 for ramp/vision; frozen random MLPs for spring/scenes); the communication module is unchanged.

Figure 4: Agents allocate communication bandwidth proportional to property extractability across both vision (6 properties) and physics (3 properties) domains. Correlations are strong but based on small

n

and should be interpreted as suggestive of information-allocation principles.

Task	Test Accuracy	Chance	Outcome
Categorical identity	27.5%	20%	Memorizes class codes
Continuous 2-prop	75.9%	50%	Compositional encoding
Continuous 6-prop	40.5%	1.6%	Compositional encoding
NN baseline (no communication)	50.2%	20%	DINOv2 clusters by class

Table 6: Categorical vs. continuous tasks on the same CIFAR-100 images with the same DINOv2 features.

Condition	Agents	Message	Holdout	PosDis	Comp. %
$2{\times}5$ sequential	2	$2{\times}5$	76.7% $\pm$ 6.5%	0.486	54%
$4{\times}5$ matched bandwidth	2	$4{\times}5$	87.1% $\pm$ 4.4%	0.345	35%
$2{\times}5$ sequential	3	$3{\times}(2{\times}5)$	98.1% $\pm$ 0.9%	0.998	100%
$2{\times}5$ sequential	4	$4{\times}(2{\times}5)$	98.3% $\pm$ 1.6%	0.999	100%
$2{\times}5$ random frames	4	$4{\times}(2{\times}5)$	97.3% $\pm$ 2.1%	0.998	100%

Table 7: Controls separating agent count, bandwidth, and temporal structure. All conditions use DINOv2 ramp features, identical IL recipe, 20 seeds each (80 seeds for 2-agent

2{\times}5

and 4-agent

2{\times}5

sequential).

Setting	Compositional	Holistic	$\Delta$
Vision, 2 properties	75.9%	74.9%	+1.0pp
Vision, 6 properties	40.5%	35.5%	+5.0pp
Ramp physics, 2 properties	76.7%	71.2%	+5.5pp

Table 8: Compositional advantage scales with number of properties on CIFAR-100 vision.

Dataset	Backbone	Agents	Oracle	Holdout	PosDis	Comp. %
Ramp	DINOv2 (ViT-S)	2	$\sim$ 90%	76.7% $\pm$ 6.5%	0.486	43/80 †
	V-JEPA 2 (ViT-L)	2	86.5%	74.8% $\pm$ 6.8%	0.402	10/20
	DINOv2 (ViT-S)	4	—	98.3% $\pm$ 1.6%	0.999	80/80
	V-JEPA 2 (ViT-L)	4	—	95.1% $\pm$ 3.3%	0.999	20/20
Collision	DINOv2 (ViT-S)	2	78.7%	73.4% $\pm$ 3.3%	0.439	9/20
	V-JEPA 2 (ViT-L)	2	88.0%	75.0% $\pm$ 3.0%	0.456	10/20

Table 9: Controlled backbone

\times

dataset

\times

agent-count comparison, including scale-matched (DINOv2 ViT-L) and frame-matched (DINOv2 48-frame) controls. At 2 agents, the discrete bottleneck equalizes performance. At 4 agents, backbone differences emerge: DINOv2 dominates on spatially-visible physics (ramp); V-JEPA 2 dominates on dynamics-only physics (collision). The V-JEPA 2 advantage survives both scale and frame-count matching.

Condition	Input dim	Holdout accuracy
V-JEPA 2 messages	40	88.7% $\pm$ 0.5%
DINOv2 messages	40	78.5% $\pm$ 1.3%
V-JEPA 2 raw features	1024	94.6% $\pm$ 0.7%

Table 10: Outcome prediction from frozen messages vs. raw features. V-JEPA 2 messages retain 94% of raw-feature performance despite 25

\times

compression, and beat DINOv2 messages by 10.2pp (

p{<}0.0001

d{=}10.50

). Messages trained for property comparison transfer to outcome prediction without retraining the sender.

Scenario	V-JEPA 2	DINOv2 (static)	Volume only	Gap
Spring	0.905	0.819	0.761	+0.086
Fall	0.657	0.718	0.531	$-$ 0.061
Ramp	0.727	0.817	0.733	$-$ 0.090

Table 11: Probe gate: mass comparison AUC on held-out objects across Physics 101 scenarios. V-JEPA 2 temporal features outperform both static appearance (DINOv2) and volume-only baselines on the spring scenario, where mass is dynamically visible through spring extension (

F{=}kx{=}mg

). On ramp and fall, appearance dominates—consistent with mass canceling in ramp sliding dynamics (

a=g(\sin\theta-\mu\cos\theta)

Agents	Both-property acc.	PosDis	Comp. rate
1	50.8%	0.294	2/10 (20%)
2	55.0%	0.325	3/10 (30%)
4	52.8%	0.483	9/10 (90%)

Table 12: Agent scaling on real video (Physics 101, mass + restitution). Compositionality rate = fraction of seeds with PosDis

>0.4

. Multi-agent pressure induces compositionality on real camera footage, replicating the synthetic scaling result (Table 8 ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
DINOv2 model	ViT-S/14 (frozen)
V-JEPA 2 model	ViT-L/16 (frozen)
Frames per scene (ramp)	8 (evenly spaced from 24)
Frames per scene (collision)	24 (evenly spaced from 48)
Hidden dimension	128
Temporal encoder (DINOv2)	Conv1D( $384{\to}256{\to}128$ ) + AdaptiveAvgPool

Appendix

Table 13: Hyperparameters used across all experiments unless otherwise noted.

	Elasticity	Friction	Damping
Position 0	0.336	0.276	0.531
Position 1	0.289	0.283	0.574
Position 2	0.260	0.276	0.655

Appendix

Table 14: Average MI between message positions and physical properties (3-property ramp, 20 seeds). Damping dominates every position.

Domain	Both Acc.	Mean Spec.	Key Pattern
Spring-mass (2-prop)	93.8%	0.946	Agent 0 $\to$ damping ( $\gamma=b/2m$ )
Ramp physics (2-prop)	88.4%	0.688	Agent 0 dead; 1 $\to$ f; 2,3 $\to$ e
Abstract scenes (2-prop)	50.4%	0.582	Weak, distributed
Vision (2-prop)	75.9%	0.500	Moderate
Vision (6-prop)	40.5%	0.196	Fully distributed

Appendix

Table 15: Mean agent specialization ratio across all tested domains.

Emergent Communication

Emergent communication studies how agents develop shared signals or languages through interaction to coordinate, convey information, and build conventions.

See all 30 papers

Emergence, Not Bandwidth: Physical Coupling and the Limits of Learned Multi-Agent Communication
Sep 28, 2026Mihir Chauhan, Aniket BeraSequential MARL Partially Observable Markov Decision Processes
IDEAS Lab, Department of Computer Science Purdue University
LatCom: Cross-Agent Latent Compression for Efficient Multi-Agent Collaboration
Sep 29, 2026Shinan Zhang, Tao Zhang, Qihui Zhu +7Multi-Agent LLM Systems Multi-Agent Collaboration
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems
Apr 23, 2026Ye Yu, Heming Liu, Haibo Jin +3Multi-Agent System Optimization LLM Agent Training
University of Illinois Urbana-Champaign, IL, USA

Sep 28, 2026J/K move · Enter open · S save

Emergence, Not Bandwidth: Physical Coupling and the Limits of Learned Multi-Agent Communication

Mihir Chauhan, Aniket Bera

IDEAS Lab, Department of Computer Science Purdue University

Sequential MARL Partially Observable Markov Decision Processes Multi-Agent Communication Cooperative MARL Emergent Communication

Open paper AskPDF arXiv

Rate-limited multi-agent teams raise three questions the emergent-communication literature has answered only empirically: what an optimal message should encode, what compression costs over a horizon, and when a learned protocol is unique enough for a teammate to read. We answer them for rate-limited Dec-POMDPs, then measure how far reinforcement learning falls short of the optimum. Our theorems fix what is achievable independently of any learner, so a gap between an engineered and a learned sender at the same bit budget is an optimization fact, not an information-theoretic one. We instantiate this on three MuJoCo arenas spanning zero, partial and rigid physical coupling, charging every condition exactly 2 bits per decision, and create the discriminating regime by closing a physical side channel within one arena, holding bodies, task and reward fixed. Communication value is governed by coupling: under rigid coupling through a shared object, no channel beats silence (+0.001 +/- 0.001, p = 0.982, n = 25), since proprioception already carries that information; without coupling, every condition solves the task; under partial coupling, the engineered 2-bit sender reaches an interquartile mean of 1.000 but the learned one reaches 0.482, indistinguishable from silence (p = 0.400, n = 25). With a shared alphabet, bandwidth cannot explain the gap. Warm-starting from an engineered receiver localizes the failure: the same channel reaches 0.857 versus 0.562 cold-started (p < 0.001), so it is neither representational nor one of maintenance; reinforcement learning fails to discover the protocol. Cross-play shows learned protocols are individually meaningful but mutually unintelligible: self-play 0.980 collapses to 0.144 across seeds, and our best constructed alignment leaves at least 77% of that gap. All headline results use 25 seeds per arena and seven published baselines at matched rate.

Abstract

Emergent Communication

Explore similar work

Emergence, Not Bandwidth: Physical Coupling and the Limits of Learned Multi-Agent Communication

LatCom: Cross-Agent Latent Compression for Efficient Multi-Agent Collaboration

Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems