Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing

Adam Deng; Shicheng Liu; Yada Zhu

cs.LGOct 15, 2026

Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing

Authors: Adam Deng, Shicheng Liu, Yada Zhu

Organizations: MIT ORC · MIT CSAIL · IBM Research

Abstract

Multi-agent systems built from frozen language models communicate through text, which discards the internal state the sender already computed and makes the receiver pay to rebuild it. Work on internal channels avoids that cost, framing the problem as transfer: agent A sends information to agent B. Every agent pair that must communicate needs its own transfer bridge, and every agent added to a system needs a new bridge fitted against each previous agent. We reframe this problem as sharing: an agent publishes its internal state once into a pool, and every peer reads that pool through a single mechanism fitted to no sender in particular, so adding an agent adds a publisher and no new bridge. Whereas the carrier of the internal state and the fusion mechanism are usually chosen independently in the literature, our main finding is that these two choices are coupled; the deciding factor is where the carrier enters the receiver's computation. Consumability is a property of the receiving computation rather than of the representation. We find that the key-value (KV) cache must be augmented rather than swapped in: the receiver cross-attends over a peer's cache while keeping its own, and with only a small query-side adapter trained, matches the accuracy it gets from reading the source document. Activations invert this. They are best written straight into the receiver, with no bridge, no adapter and no training, which beats exposing them alongside the receiver's own, though that advantage rests on knowing where in the receiver the state belongs. The rule is augment the KV cache, replace the activation, established under one protocol with a shared control ladder. One adapter, trained once, also serves publishers it never trained on, and a pool of several at once, but only those whose internal state lies close to the receiver's own.

Figures & tables

Figure 1: The two carriers take opposite mechanisms. (a) The KV cache cannot be substituted, even at the fidelity annotated on the bar, and must be augmented instead. (b) Activations invert this: direct replacement works, training-free. Floors and native references are per carrier.

	$\alpha{=}0.0$	$0.1$	$0.25$	$0.5$	$0.75$	$1.0$
$k{=}1$	0.180	0.185	0.110	0.005	0.000	0.000
$k{=}2$	0.180	0.180	0.110	0.020	0.010	0.005
$k{=}4$	0.180	0.205	0.120	0.010	0.005	0.000
$k{=}8$	0.180	0.175	0.120	0.005	0.000	0.000
$k{=}16$	0.180	0.145	0.135	0.020	0.015	0.010

Table 1: Oracle substitution: the receiver’s key is interpolated toward the

k

-th nearest key the receiver itself produced elsewhere, so every candidate is on its own manifold. No neighbor rank recovers a graded tolerance. Qwen2.5-1.5B,

n{=}200

, floor

0.18

, ceiling

0.665

Setting	Adapted	Shared	Control	$\Delta$ vs. raw	vs. ceiling
same model	none (raw)	0.555	0.185	—	$-0.110$
	$q$	0.600	0.155	$+0.045$	$-0.065$
	$k,v$	0.640	0.090	$+0.085$	$-0.025$
	$q,k,v$	0.695	0.200	$+0.140$	$+0.030$
cross family	none (raw)	0.010	0.005	—	—
	$q$	0.010	0.005	$+0.000$	—

Table 2: KV augmentation. The receiver keeps its own cache and cross-attends over the holder’s verbatim cache; only the receiver’s projections adapt. The control feeds a holder cache built on the answer-masked passage. Same-model,

n{=}200

Inject layer	Shared fact	Cond. acc.	Span-copy	Mismatch ctrl	No-op ctrl
$L{=}4$	0.533	0.708	0.167	0.067	0.200
$L{=}8$	0.550	0.719	0.167	0.058	0.200
$L{=}12$	0.567	0.742	0.175	0.100	0.200
$L{=}16$	0.567	0.708	0.167	0.408	0.200
$L{=}20$	0.575	0.708	0.183	0.425	0.200
all of the above	0.517	0.685	0.142	0.067	0.200

Table 3: Activation replacement, training-free. The receiver’s hidden states at the fact region are overwritten with the holder’s. Qwen2.5-1.5B,

n{=}120

(

\mathrm{cond}\,n{=}89

), floor

0.20

, ceiling

0.742

. The mismatch control is clean at

L\leq 12

but not at

L\geq 16

(shaded reading in the text).

Figure 2: Carrier ordering across injection depth: activations dominate values, which dominate keys, and keys never leave the floor band. The clean operating band is shallower than the peak (Section 4.3 ).

Mechanism	KV cache ( $\mathbf{K},\mathbf{V}$ )	Activations ( $\mathbf{h}$ )
Substitution / replacement	$0.000$ at floor, dead	$\mathbf{0.742}$ at ceiling
Augmentation	$\mathbf{0.695}$ at ceiling	$0.438$ above floor, below replacement
Winning mechanism	augment	replace
Tolerable displacement	$\cos\approx 0.995$	$\cos\approx 0.90$
Consumption point	through the softmax	before the softmax
Cross-family	collapses	collapses

Table 4: The asymmetry. Best result per (carrier, mechanism) cell, each against its own probe’s anchors. Cache numbers are shared-fact accuracy at

n{=}200

(floor

0.18

, ceiling

0.665

); activation numbers are conditional accuracy at

n{=}120

(floor

0.20

, ceiling

0.742

). Cells are not comparable across columns in absolute terms, only relative to their anchors.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Multi-source KV augmentation over a pool of $N$ holders, one of which read the answer-bearing passage. (a) Raw cross-attention collapses below the floor by $N{=}4$ ; the trained query-side bridge decays gracefully and remains above floor at $N{=}8$ . Dotted lines are the corresponding controls, which themselves lift above floor at large $N$ . (b) Lift over the strongest control at each pool size: positive throughout, eroding with $N$ .

sender	cosine	rel. diff.	raw lift	bridge shared	bridge lift	$95\%$ interval
self (seen)	—	—	$+0.070$	$0.550$	$+0.310$	$[+0.219,+0.401]$
instruct (held out)	$0.9994$	$0.033$	$+0.070$	$0.550$	$+0.325$	$[+0.235,+0.415]$
coder (held out)	$0.9933$	$0.170$	$+0.000$	$0.130$	$+0.030$	$[-0.032,+0.092]$
math (held out)	$0.9810$	$0.433$	$+0.000$	$0.100$	$-0.005$	$[-0.064,+0.054]$

Appendix

Table 5: Held-out senders. One rank-16 query-side bridge, trained on pools built from the receiver’s own weights only, evaluated against senders it never saw. Senders are Qwen2.5-1.5B checkpoints with the receiver’s geometry and a bit-identical tokeniser, ordered by the cosine between their cache and the receiver’s. We report relative Frobenius difference alongside cosine: the two rank the senders identically, so the ordering is not an artifact of one metric, though two metrics over three checkpoints cannot settle which is the right notion of compatibility. Lift is shared accuracy minus the stronger of the two controls; intervals are

95\%

normal-approximation intervals on that difference. Pool size

N{=}4

n{=}200

, floor

0.18

, native ceiling

0.665

. The self row reproduces the

N{=}4

point of Figure 3 exactly.

Figure 4: Held-out senders. (a) The bridge, trained on the receiver’s own pools only, reaches the same accuracy on the nearest held-out sender as on the sender it trained on, and falls to its control on the two more distant senders. (b) Control-subtracted lift against the cosine between the sender’s cache and the receiver’s own. The effect is present at $0.9994$ and absent by $0.9933$ , which measures where the effect falls on the sender axis. Three checkpoints locate a transition, not a radius.

receiver	$L$	$G$	floor	identity	nearest genuine
Pythia-70M	$6$	$1$	$0.000$	$0.025$	$0.000$
GPT-2	$12$	$1$	$0.033$	$0.233$	$0.000$
Pythia-410M	$24$	$1$	$0.067$	$0.375$	$0.000$
Pythia-1.4B	$24$	$1$	$0.092$	$0.517$	$0.000$
Qwen2.5-0.5B	$24$	$7$	$0.092$	$0.633$	$0.008$
Qwen2.5-1.5B	$28$	$6$	$0.200$	$0.742$	$0.000$

Appendix

Table 6: The substitution bound across architectures. Each receiver is probed with its own bank of genuine keys and its own anchors. Identity hands the receiver its own exact keys and recovers that receiver’s ceiling in every case, which verifies the harness on each architecture. Nearest genuine substitutes the closest key the receiver itself produced elsewhere. Depth spans

L{=}6

L{=}36

and the grouped-query factor spans

G{=}1

(multi-head) to

G{=}8

n{=}120

Figure 5: Why the two carriers differ. (a) The measured readout Jacobian gain is $O(1)$ both on and off manifold, with a ratio of only $1.47\times$ : the wall is not built by amplification, so no single layer and no stacking of layers blows up a tolerable perturbation. (b) Tolerable displacement by carrier: activations remain usable to cosine $0.90$ , keys only to $\approx 0.995$ , about $20\times$ less headroom. Same receiver, same depth; the difference is the consumption point.

Figure 6: The boundaries. (a) Cross-family KV augmentation collapses at every rung of the adaptation ladder, while the same ladder within one model reaches the native ceiling. (b) On the activation carrier, the apparently strongest trained arm fails a mis-source control ( $0.300$ against a floor of $0.19$ ); the additive arm passes it ( $0.069$ ) at a lower but genuine $0.454$ .

Multi-Agent Communication

Multi-agent communication studies how autonomous agents exchange information to coordinate decisions, share knowledge, and act together under constraints.

See all 375 papers

May 19, 2026

Latent Cache Flow: Model-to-Model Communication Without Text

LLM agents today communicate via text, which incurs considerable latency and information loss due to the need to autoregressively decode the sharer model's state and encode at the receiver model. Recent work such as Cache-to-Cache (C2C; Fu et al., 2026) seeks to exchange KV caches by learning adapters that translate sharer KV matrices to the receiver model. However, the adapters are large and expensive to train, and translate individual tokens, which requires the target context to be identical. This is unsuitable for agent communication, where the LLMs have differing context. We introduce Latent Cache Flow (LCF). To address efficiency, we observe that keys and values can be jointly translated and compressed, reducing the adapter to about 4% of C2C's size. To address differing context, we design the adapter to transmit a summary of new information that the target model does not have. Our early experiments show that a pruned 13 MB LCF adapter can be more accurate than C2C at 956 MB in shared-context settings; for different contexts, LCF improves F1 by 7.5% and Exact Match by 23% while 8.5 times faster than text-based communication.

Maximillian Rossi, Prajwal Raghunath, Eugene Wu

Columbia University

KV Caching KV-Cache Compression Multi-Agent LLMs

Ask PDF

Jun 11, 2026

See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

Language-model agents build internal representations of the information they observe and the reasoning they perform. Sharing these representations offers a way to communicate both source information and reasoning across agents. For agents built from different models, this requires aligning their representations while preserving information useful to the receiver. We study this problem through KV-cache communication, examining how an agent uses internal states shared by other agents, with or without direct access to the information that other agents observed. A controlled self-communication study shows that cache pruning causes substantially greater degradation when the receiving agent lacks access to that information. We use this finding to guide dense cross-model cache alignment, combining positional disentanglement and KV-group transformations with reconstruction followed by generation training. Across six directed Qwen3 pairs, aligned caches improve in-domain accuracy over text communication when both agents observe the same input, with fewer estimated inference FLOPs. Experiments with three-agent document sharing and Mistral-to-Qwen transfer further demonstrate that aligned caches can carry information across both multiple separate observations and different model families.

Siyi Chen, Xiaoyan Zhang, Meng Wu +7

University of Michigan · NVIDIA · University of Pennsylvania +2

Multi-Agent Communication Multi-Agent LLMs

Ask PDF

Aug 5, 2026

When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs

Multi-agent LLM systems relay key--value caches instead of text and credit their gains to exchanged ``latent thoughts''. That credit is a claim about which example's cache is relayed, not merely that one is. We audit it causally in released systems. The cache is replaced with deranged (mismatched-example), zeroed, and moment-matched random counterparts, under two regimes defined by whether the receiver needs the sender's private information. Where it does, the battery reads ceiling: 100% against 23--25% for answer-irrelevant relays on the primary backbone, a contrast replicated across three families, five checkpoints, and a prose document-QA surface. Where it does not, a pre-registered five-seed protocol establishes equivalence within 2.8 points, a margin anchored to the audited system's reported gain, under Holm-corrected TOST on GSM8K and ARC-Challenge across three Qwen3 scales and on MedQA at 8B (one cell shows a small detected advantage inside the margin); a second family shows no detected advantage. A large cache effect need not be a pairing effect. In one natural cell, zeroing the relay costs 14.7 points; a mismatched cache, 0.4. Nor is need sufficient: under the same test, delivered channels span ceiling (LatentMAS's native relay), partial (KVComm's layer subset), and no detected example-specific transfer (C2C's released projector). Benchmark deltas do not by themselves establish latent-thought transmission; establishing it takes a mismatched-cache audit, which we release.

Jiaming Cheng, Subhransu Das, Rajiv Ramnath

Independent Researcher · The Ohio State University

KV Caching LLM Agent Evaluation Multi-Agent Communication Multi-Agent LLMs

Ask PDF

Abstract

Multi-Agent Communication

Explore similar work

Latent Cache Flow: Model-to-Model Communication Without Text

See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs