Receiver-Conditioned Latent Communication gives 94% CacheBack

Maximillian Rossi; Prajwal Raghunath; Haoqing Xuan; Yusen Zhang; Eugene Wu

cs.AISep 25, 2026

Receiver-Conditioned Latent Communication gives 94% CacheBack

Authors: Maximillian Rossi, Prajwal Raghunath, Haoqing Xuan, Yusen Zhang, Eugene Wu

Organizations: Columbia University · Amazon Web Services

Abstract

Multi-agent systems distribute large contexts across agents that communicate to solve a task. Text messages are compact but require decoding and may omit evidence the receiving agent needs. Recent latent communication instead transfers KV caches. This avoids text generation and can improve accuracy and latency. However, a full KV cache grows linearly with both the context an individual agent processes, and the number of agents that coordinate together. This raises memory and context costs, often far exceeding available GPU resources and context window sizes. Our key observation is that agents need only send what the receiving agent requires for its local task -- which we call receiver-conditioned communication. The receiver agent passes the sender a small description of its information needs, which serves to filter and compress the sender agent's KV cache. CacheBack is a simple, robust, training-free instance of receiver conditioning based on the sender's attention weights. On FanOutQA, CacheBack with Qwen 3 removes 75% of the state the agent would otherwise receive, improving accuracy by 14.7 percentage points and reducing median task-completion latency by 3.2x relative to text communication. We show comparable improvements across model families that span dense Transformers, Mamba-attention hybrids, and sliding-window attention.

Figures & tables

Figure 1: Left: highest-accuracy CacheBack setting per family versus same-size text on FanOutQA;

n=50

concurrent tasks on

8\times

H100. Right: the receiver query guides selection of sender KV states.

Figure 2: Receiver-conditioned communication limits context accumulation. Full-cache messages accumulate all agents’ positions at the receiver. Receiver-conditioned messages retain selected source and latent positions, represented as token IDs and continuous inputs for receiver prefill.

Figure 3: Averaging can hide query-specific support for A and choose irrelevant token B .

Figure 4: Results from diagnostic tests. (a) CacheBack retains high evidence recall even at

16\times

compression compared to mean query attention and receiver-independent baselines. (b) CacheBack generally retains more evidence as model size increases. (c) The largest gains from CacheBack over mean query attention shift toward higher compression ratios with model size. (d) Mean query attention ranks gold tokens higher as model size increases.

Figure 5: CacheBack improves accuracy–latency tradeoffs on FanOutQA (strict accuracy) and LongBench v2 (answer accuracy).

rX

X

-fold compression;

x

K: fixed budget of

x

thousand positions. Faded points are off-channel frontiers; LongBench labels mark frontier points only.

Model (ratio)	Text	CB	Gain (pp)	Text	CB	Speedup	Text	CB	Gain (pp)
	Strict accuracy (%)			Median TTEOA (s)			Message recall (%)
Qwen 3 8B ( $4\times$ )	40.7	55.3	+14.7	612	192	$\mathbf{3.2\times}$	89.5	95.8	+6.3
Ministral 3 14B ( $32\times$ )	38.7	59.3	+20.7	1,537	193	$\mathbf{8.0\times}$	87.5	91.5	+4.0
Gemma 4 12B ( $4\times$ )	52.0	59.3	+7.3	603	201	$\mathbf{3.0\times}$	91.1	95.8	+4.7
Nemotron Nano 2 12B ( $8\times$ )	38.7	50.0	+11.3	131	102	$\mathbf{1.3\times}$	84.8	95.6	+10.8

Table 1: Selected FanOutQA operating points versus same-size text. CB: CacheBack.

Figure 6: FanOutQA mean TTEOA by stage: queueing, agent processing, message preparation and transfer, receiver prefill, and generation. Breakdowns beyond

16\times

are omitted as they change little.

Figure 7: Effect of the receiver query

q

on FanOutQA for Qwen 3 8B. The unrelated query is “Why was Le Chaton Fat denied a small-business loan?”

Model	Setting	Text	CB	Gain (pp)	Text	CB	Speedup
		Accuracy (%)			Median TTEOA (s)
Qwen 3 8B	Relative $4\times$	39.3	54.0	+14.7	884	237	$\mathbf{3.7\times}$
	Fixed 64K	39.3	46.7	+7.3	884	355	$\mathbf{2.5\times}$
Nemotron Nano 2 12B	Relative $8\times$	38.7	48.0	+9.3	329	133	$\mathbf{2.5\times}$
	Fixed 32K	38.7	44.7	+6.0	329	173	$\mathbf{1.9\times}$

Table 2: Selected operating points on LongBench v2 Easy versus same-size text. CB: CacheBack.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	OBF evaluation	Our evaluation
Information	Same question at every agent; no private source documents	Agents receive distinct evidence unavailable to the receiver
Selection signal	Attention from current latent reasoning states	Attention from the receiver’s query
Topology	Sequential reasoning chain	Sequential (LongBench v2) and parallel (FanOutQA)
Total input	496–1,012 prompt tokens across three reasoning agents	FanOutQA: 122k–150k evidence tokens; LongBench v2: 100k–246k source tokens
Context growth	Inherited state is preserved; retained prompt KV and new thoughts accumulate	Relative budgets grow with added context; fixed budgets cap the message size. Fan-in combines compressed agent messages

Appendix

Table 3: Information access and context growth. Shared-question reasoning versus communication of private evidence.

Figure 8: Continuous-row versus token-ID encoding with latent vectors. Left: payload per sender. Right: bandwidth-only transfer time. Both panels use log scales.

Figure 9: Mean evidence recall by selector across Qwen 3 models (

n=13

$\alpha$	$p$	$2\times$	$4\times$	$8\times$	$16\times$	$32\times$	$64\times$	$128\times$
Mean query attention
$0$	$-$	95	81	58	33	12	3	1
CacheBack
$0.25$	$1.5$	96	86	67	41	15	3	1
$0.5$		96	86	70	50	21	9	1
$1$		95	90	78	57	33	16	4

Appendix

Table 4: Evidence recall (%) over 13 tasks and four Qwen 3 sizes. Shading marks

p=\alpha=2

; bold marks the best value at each ratio.

Text channels
Setting	Qwen 3	Ministral 3	Gemma 4	Nemotron
Small	72.1	82.6	86.7	74.5
Medium	86.7	90.5	87.1	77.6
Same-size	89.5	87.5	91.1	84.8
Latent channels (CacheBack)
Full rows	–	96.4	96.4	–

Appendix

Table 5: FanOutQA message recall over 50 questions. CacheBack recall has a 96.4% ceiling.

Figure 10: FanOutQA message recall over 50 questions and 495 reference groups.

Figure 11: FanOutQA strict accuracy and median TTEOA for all evaluated settings.

rX

denotes CacheBack compression ratio

X

	Qwen 3 [-0.15em]8B receiver					Ministral 3 [-0.15em]14B receiver
	Accuracy $\uparrow$		TTEOA			Accuracy $\uparrow$		TTEOA
Setting	Loose	Strict	p50 (s)	p95 (s)	Speedup	Loose	Strict	p50 (s)	p95 (s)	Speedup
Question only	57	19	19	62	–	53	21	53	309	–
Text channels
Small	53	13	183	515	3.3 $\times$	63	25	274	709	5.6 $\times$
Medium	73	35	424	661	1.4 $\times$	69	34	280	552	5.5 $\times$

Appendix

Table 6: Full FanOutQA results for global-attention Transformers. Loose and strict accuracy (%) over 50 questions and three receiver draws per setting.

	Gemma 4 [-0.15em]Global + sliding-window attention [-0.15em]12B receiver					Nemotron [-0.15em]Mamba + attention [-0.15em]12B receiver
	Accuracy $\uparrow$		TTEOA			Accuracy $\uparrow$		TTEOA
Setting	Loose	Strict	p50 (s)	p95 (s)	Speedup	Loose	Strict	p50 (s)	p95 (s)	Speedup
Question only	3	1	27	70	–	60	25	8	13	–
Text channels
Small	75	41	123	272	4.9 $\times$	59	22	63	110	2.1 $\times$
Medium	80	53	138	228	4.4 $\times$	66	24	105	331	1.3 $\times$

Appendix

Table 7: Full FanOutQA results for hybrid architectures. Loose and strict accuracy (%) over 50 questions and three receiver draws per setting.

Model	Text draws	CacheBack draws	Gain (pp)	95% CI (pp)
Qwen 3 8B	42 / 40 / 40	56 / 52 / 58	+14.7	[1.3, 28.0]
Ministral 3 14B	36 / 36 / 44	62 / 56 / 60	+20.7	[8.7, 32.7]
Gemma 4 12B	54 / 50 / 52	58 / 56 / 64	+7.3	[-3.3, 18.0]
Nemotron Nano 2 12B	44 / 34 / 38	48 / 46 / 56	+11.3	[-0.7, 24.0]
Macro-average	44 / 40 / 44	56 / 53 / 60	+13.5	[6.5, 20.3]

Appendix

Table 8: Receiver sampling on FanOutQA. Strict accuracy (%) over 50 questions and three receiver draws per setting.

Figure 12: LongBench v2 accuracy and median TTEOA for all evaluated settings.

rX

denotes relative compression

X

;

x

K denotes a fixed budget of

x

thousand positions.

Model	Setting	Text draws	CacheBack draws	Gain (pp) [95% CI]
Qwen 3 8B	Relative $4\times$	36 / 42 / 40	56 / 54 / 52	+14.7 [0.0, 29.3]
	Fixed 64K	36 / 42 / 40	50 / 44 / 46	+7.3 [-8.0, 22.7]
Nemotron Nano 2 12B	Relative $8\times$	40 / 38 / 38	44 / 48 / 52	+9.3 [-5.3, 23.4]
	Fixed 32K	40 / 38 / 38	38 / 46 / 50	+6.0 [-8.7, 20.7]

Appendix

Table 9: LongBench v2 receiver sampling at selected settings.

Figure 13: LongBench v2 mean TTEOA by stage at selected settings.

	Qwen 3 8B				Nemotron Nano 2 12B
	Accuracy $\uparrow$	TTEOA			Accuracy $\uparrow$	TTEOA
Setting	(%)	p50 (s)	p95 (s)	Speedup	(%)	p50 (s)	p95 (s)	Speedup
Question only	34	20	74	–	26	9	20	–
Text
Small	26	558	1014	1.6 $\times$	27	107	434	3.1 $\times$
Medium	36	642	1137	1.4 $\times$	37	214	409	1.5 $\times$

Appendix

Table 10: LongBench v2 results on the 50-question Easy panel.

	Qwen 3	Ministral 3	Gemma 4	Nemotron
	8B	14B	12B	Nano 2 12B
Global / total layers	36/36	40/40	8/48	6/62
vLLM	0.11.1	0.26.0	0.26.0	0.26.0
Receiver window	131,072	180,224	262,144	131,072
Temperature	0.6	0.7	1.0	0.6 †
Top- $k$	20	Off	64	Off

Appendix

Table 11: Model and serving settings.

Setting	Sender GPUs	Receiver GPUs
CacheBack, $2\times$	4	4
CacheBack, $4\times$ – $128\times$	5	3
Text, smaller senders	6	2
Text, same-size senders	8 fused GPUs
Question only	0	8
Full rows, Gemma and Ministral	4	4

Appendix

Table 12: GPU allocation by communication setting.

Family	Ratio	Before	After	Change	Before	After	Change
		Text			CacheBack
Qwen	$4\times$	30.0	40.7	$+10.7$	41.3	55.3	$+14.0$
Ministral	$32\times$	32.0	38.7	$+6.7$	46.7	59.3	$+12.7$
Gemma	$4\times$	36.7	52.0	$+15.3$	46.0	59.3	$+13.3$
Nemotron	$8\times$	32.0	38.7	$+6.7$	38.7	50.0	$+11.3$

Appendix

Table 13: FanOutQA strict accuracy (%) before and after the reference audit.

Multi-Agent Communication

Multi-agent communication studies how autonomous agents exchange information to coordinate decisions, share knowledge, and act together under constraints.

See all 380 papers

May 19, 2026

Latent Cache Flow: Model-to-Model Communication Without Text

LLM agents today communicate via text, which incurs considerable latency and information loss due to the need to autoregressively decode the sharer model's state and encode at the receiver model. Recent work such as Cache-to-Cache (C2C; Fu et al., 2026) seeks to exchange KV caches by learning adapters that translate sharer KV matrices to the receiver model. However, the adapters are large and expensive to train, and translate individual tokens, which requires the target context to be identical. This is unsuitable for agent communication, where the LLMs have differing context. We introduce Latent Cache Flow (LCF). To address efficiency, we observe that keys and values can be jointly translated and compressed, reducing the adapter to about 4% of C2C's size. To address differing context, we design the adapter to transmit a summary of new information that the target model does not have. Our early experiments show that a pruned 13 MB LCF adapter can be more accurate than C2C at 956 MB in shared-context settings; for different contexts, LCF improves F1 by 7.5% and Exact Match by 23% while 8.5 times faster than text-based communication.

Maximillian Rossi, Prajwal Raghunath, Eugene Wu

Columbia University

KV Caching KV-Cache Compression Multi-Agent LLMs

Ask PDF

Apr 14, 2026

When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration

Multi-agent LLM systems are moving beyond discrete-token messages toward richer relays that preserve internal state. Recent work such as LatentMAS transmits full key-value (KV) caches between agents but pays a high memory and communication cost. We adapt KV-cache eviction to this setting and introduce Orthogonal BackFill (OBF), which injects a low-rank residual from the discarded KV states back into the retained ones, orthogonal to what is already kept. With only

9.9\%

20.2\%

of the prompt KV retained, compressed relay cuts bandwidth by

4.7\times

and GPU memory by

8\%

at under

5\%

wall-clock overhead, and stays close to full relay in accuracy across nine benchmarks, ahead of it on several. OBF matches or improves over headwise eviction on all nine, and its gain is proportional to the accuracy gap eviction opens against full relay (

r{=}0.78

across three model scales), so it gives back part of what eviction takes. Code is available at https://github.com/markli404/When-Less-Latent-Leads-to-Better-Relay.

Yiping Li, Zhiyu An, Wan Du

Department of Computer Science and Engineering, University of California, Merced, Merced, CA 95343

LLM Compression KV Caching KV-Cache Compression Multi-Agent Communication Multi-Agent LLMs

Ask PDF

Oct 27, 2026

Selective and Hybrid-State Communication for Latent Multi-Agent Systems

Efficient communication remains a major bottleneck in large language model multiagent systems. Latent communication reduces repeated natural-language exchange by transferring internal model states between agents, but these transfers remain costly and are typically designed around conventional KV-cache representations. We address this problem from two complementary perspectives. We introduce SelectKV, a boundary-aware method that selectively communicates KV states while protecting inherited context, and HyLaMAS, which extends latent collaboration to hybrid linear-attention models by communicating both KV caches and recurrent states through a shared workspace. Across six benchmarks, SelectKV reduces communicated KV positions and logical KV payload by 9.7-9.9%, with accuracy unchanged on two benchmarks, within one percentage point on three, and six points lower on HumanEval+. Across Qwen3.5 model scales, HyLaMAS reduces decoded output tokens by 53-80% and achieves 1.16-2.78x speedups relative to text-based multi-agent communication. Together, these independently evaluated methods represent two complementary approaches to efficient latent-state communication across conventional Transformer and hybrid model architectures.

Miles Liao, Aaron Smieszek, Yuhao Ge

Independent Researcher

Multi-Agent Communication Multi-Agent LLMs

Ask PDF

Abstract

Multi-Agent Communication

Explore similar work

Latent Cache Flow: Model-to-Model Communication Without Text

When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration

Selective and Hybrid-State Communication for Latent Multi-Agent Systems