cs.LGOct 10, 2026

Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families

Authors: Abdulrahman Alabdulkareem, Irene Tenison, Yang Zhang, Yada Zhu, Boris Katz, Dan Gutfreund

Organizations: MIT CSAIL · MIT-IBM Computing Research Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM multi-agent systems split work across agents that communicate through text. An agent summarizes its findings, another agent receives the text. This constrains communication to discrete tokens and costs redundant compute, limiting bandwidth and capabilities of multi-agent systems. Recent works demonstrate KV-cache transfer between LLMs but they either require the LLMs to be within the same family or cannot function across context, thus requiring the receiver to prefill text. We present SwaT (Swarm Telepathy), the first cross-family KV-cache transfer method that allows prefill-free communication in multi-family multi-agent systems. In an agent-to-agent communication setting, combining SwaT with recent advances in KV-cache compaction beats text-to-text communication in both accuracy and speed, with a mean of 40%40\% higher accuracy and a mean of 4×4\times faster wall-clock time. Experiments show SwaT achieves high performance on 17 benchmarks across 12 LLMs and 6 families. Additionally, given a sender Vision-Language Model (VLM), SwaT is able to transfer the vision capabilities to a text-only receiver LLM exceeding text-to-text communication despite SwaT being trained on text-only data. SwaT keeps both sender and receiver models frozen without fine-tuning and only requires <50M<50M tokens and <1<1 GPU-day to train.

Figures & tables

Figure 1. Communication between two LLM agents from different families. Top: communicating in words is slow and the receiver only gets the gist. Bottom: \swat translates the sender’s latent state (KV-cache) to the receiver’s latent space. Combined with equal-budget cache compression, \swat is on average 40% more accurate than text summarization, at 4×4\times the speed. Left two thirds, two rows. Top row, labelled Text Communication: a blue square-headed robot labelled Family A with a neutral face speaks a text bubble reading The summary is, and an orange dome-headed robot labelled Family B with a neutral face ends up with a grey thought cloud reading I get the gist. Bottom row, labelled SwaT (ours): both robots smile; the blue robot's thought cloud of blue circles, labelled latent state, passes through a green SwaT box and becomes a full thought cloud of orange squares next to the orange robot. Right third, a small scoreboard with columns speed and performance: the words row has an hourglass and two check marks plus an empty slot; the SwaT row has a lightning bolt marked 4 times and three check marks.
PaperDateDifferentCross-Skip
SizesFamilyPrefill
Closed Form ( Heo et al., 2026 )Aug 2026✓✗✓
CacheBridge ( Qu et al., 2026 )Sep 2026✓✗✓
KV Alignment ( Dery et al., 2026 )Jan 2026✓✗✓
Dense Latent ( Chen et al., 2026 )Jun 2026✓✗✓
LCF ( Rossi et al., 2026 )May 2026✓✗✗
Table 1. Cross-model KV transfer. \swat is the first method for cross-family KV-cache transfer with no receiver prefill (i.e., the receiver does not read the text).
Figure 2. \swat between a frozen sender agent ( S\mathcal{S} ) and receiver agent ( R\mathcal{R} ). Left: S\mathcal{S} prefills the document dd once. \swat has BB blocks where block bb cross-attends to S\mathcal{S} ’s KV-cache for the layer group Sb\mathcal{S}_{b} and outputs R\mathcal{R} ’s KV-cache for the layer group Rb\mathcal{R}_{b} . Only \swat is trained where underlining marks learned parameters or modules. Right: block bb maps hbh_{b} to hb+1h_{b+1} through causal self-attention on hbh_{b} , then causal cross-attention on a linear projection of Sb\mathcal{S}_{b} (RoPE removed), then an FFN. Positions and the causal mask in the cross-attention module are aligned across the tokenizers as described in section 3.2 . The output of bb is linearly projected to Rb\mathcal{R}_{b} with receiver RoPE applied to the keys. Left: three towers drawn with abstract depth. Blue Sender Agent (S) with L_S layers shows groups S_0, S_b and S_B separated by vertical dots; a small green SwaT box with B blocks and about 300M parameters, trained in under 6 GPU-hours on under 50M tokens, marked with a flame, shows blocks 0, b and B with b highlighted; orange Receiver Agent (R) with L_R layers shows groups R_0, R_b and R_B. Snowflakes mark both agents as frozen. Arrows connect each sender group to its block and each block to its receiver group. The context enters S as S's tokens and SwaT as R's embeddings; R reads an optional query or nothing. Dashed lines zoom from block b into a green panel on the right: K and V of the sender layers in S_b, K de-RoPE'd, are concatenated and pass through a LayerNorm-plus-linear block into the causal cross-attention of a residual stream with causal self-attention, cross-attention and FFN; The residual stream goes from h_b to h_{b+1}. Underlining identifies learned modules; memory normalization has no learned parameters. An output head of LayerNorm, linear, RMS calibration and receiver RoPE writes the keys and values of the receiver layers in R_b into Agent R.
 In-domain (9)OOD (8) 
Sender ReceiverShort AccContext AccContext F1Short AccContext AccContext F1Mean
Small→\boldsymbol{\rightarrow}Large       
Qwen 3 4B→\rightarrowLlama 3 8B100.6%100.2%110.9%88.7%91.6%98.7%98.5%
Llama 3 8B→\rightarrowQwen 3 14B89.6%98.2%116.1%86.5%98.9%103.5%97.5%
Ministral 3 8B→\rightarrowLlama 3.2 11B104.2%97.5%108.5%89.7%92.8%99.0%99.4%
Gemma 3 4B→\rightarrowGranite 3.3 8B95.4%93.6%159.8%82.0%95.0%131.4%108.5%
Table 2. Zero-shot retention (%, score\swat/scorereceiver\text{score}_{\text{\swat{}}}/\text{score}_{\text{receiver}} ) across 17 benchmarks.
Figure 3. Comparing \swat (ours) with AM ( Zweiger et al., 2026 ) against text-to-text communication ( T2T ) across three model–pairs, five budgets, and seven datasets. \swat + AM exceeds T2T in all metrics (task score, sender GPU time, receiver GPU TTFT). Description
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 16.2  ×\,\times compression≈\approx 12.7  ×\,\times≈\approx 17.2  ×\,\times
Score 34.253.1+18.9 ±\pm 4.637.258.6+21.4 ±\pm 4.529.849.0+19.3 ±\pm 5.5
Tokens 117113−- 3.1 ±\pm 2.1%161149−- 7.4 ±\pm 3.3%114110−- 3.8 ±\pm 2.2%
S\mathcal{S} GPU (ms) 3,6081,527×\times 2.4 ±\pm 0.43,6751,897×\times 1.9 ±\pm 0.27,3292,094×\times 3.5 ±\pm 0.6
R\mathcal{R} TTFT (ms) 9.23.8×\times 2.4 ±\pm 0.221.87.3×\times 3.0 ±\pm 0.36.73.2×\times 2.1 ±\pm 0.1
Table 3. Performance of \swat vs T2T over three model–pairs, five budgets, and the mean over seven datasets (QuALITY, QASPER, HotpotQA, MuSiQue, SQuAD, NarrativeQA, LongHealth). All dataset samples are (d,q)(d,q) : document dd is given to the sender agent ( S\mathcal{S} ) and question qq to the receiver agent ( R\mathcal{R} ). T2T : S\mathcal{S} summarizes dd , R\mathcal{R} prefills the summary along with qq and proceeds to answer. \swat +AM: the KV-cache of dd from S\mathcal{S} is transferred ( \swat ) and compacted (AM) then given to R\mathcal{R} , which prefills only qq and proceeds to answer. Budgets are set by the T2T summary from S\mathcal{S} under five different summary prompts, and \swat +AM is compressed to a similar budget. Metrics: task score (accuracy or F1), tokens communicated between the agents (Tokens), GPU time of S\mathcal{S} per request ( S\mathcal{S} GPU), and time to first token of R\mathcal{R} per request ( R\mathcal{R} TTFT), both in milliseconds. Change is reported ±\pm bootstrap standard error. Green: \swat is statistically better than T2T (95% bootstrap CI).
MethodVQAv2TQAOKVQAGQAVWAvg.
Gemma 3 4B →\rightarrow Granite 3.3 8B
Sender45.3 ∗53.4 ∗22.0 ∗31.4 ∗19.034.2 ±\pm .4 ∗
T2T33.8 ∗32.0 ∗15.7 ∗16.3 ∗7.7 ∗21.1 ±\pm .4 ∗
\swat59.561.828.739.518.341.6 ±\pm .4
Gemma 3 4B →\rightarrow Llama 3 8B
Sender45.3 ∗53.4 ∗22.0 ∗31.4 ∗19.034.2 ±\pm .4 ∗
Table 4. Test accuracy (%) of 3 VLM →\rightarrow text-only LLM pairs on 5 VQA benchmarks (TQA == TallyQA, VW == VizWiz). ±\pm is bootstrap standard error. Sender : VLM answers from the image and question. T2T : the receiver LLM answers from the sender VLM’s caption. \swat : the sender VLM’s KV-cache is transferred to the text-only receiver. \swat is significantly better than results marked with ∗ (95% paired bootstrap).
CMMLU subjectQwenLlama\swat Q →\rightarrow L
3 4B3 8BAcc.Change
Arts90.055.688.8+59.6%
Security study83.755.677.0+38.7%
Concept. physics83.745.673.5+61.2%
Prof. psychology82.856.077.6+38.5%
All 67 subjects69.751.265.1+27.1%
Table 5. Zero-shot specialized knowledge transfer from Qwen 3 4B to Llama 3 8B on CMMLU (Chinese MMLU; accuracy, %). Although our training data contained only English text, \swat transfers most of Qwen’s advantage on foreign text never seen during training. Change is relative to Llama 3 8B.
 Qwen 3 4B →\rightarrow Llama 3 8BLlama 3 8B →\rightarrow Qwen 3 14B
 FLOPs (TFLOPs)Latency (ms)FLOPs (TFLOPs)Latency (ms)
Seq LenRe-prefill\swatRatioRe-prefill\swatSpeedupRe-prefill\swatRatioRe-prefill\swatSpeedup
640.890.04×\times 24.726.02.1×\times 12.31.690.04×\times 45.343.52.2×\times 20.2
2563.590.15×\times 24.431.72.6×\times 12.06.790.15×\times 44.747.82.7×\times 18.0
1K14.570.63×\times 23.252.95.7×\times 0 9.327.490.65×\times 42.595.85.7×\times 16.7
4K61.573.13×\times 19.7209.921.7×\times 0 9.7115.113.20×\times 35.9394.122.3×\times 17.7
Table 6. Receiver re-prefill vs \swat FLOP count and latency.
Sender →\rightarrow ReceiverModeClosed form ( Heo et al., 2026 )Dense latent ( Chen et al., 2026 )\swat (ours)
Qwen 3 4B →\rightarrow Qwen 3 8BSame Family59.9 ±\pm 0.273.5 ±\pm 0.172.3 ±\pm 0.2
Qwen 3 4B →\rightarrow Qwen 2 7BCross Gen.41.0 ±\pm 0.547.3 ±\pm 0.268.1 ±\pm 0.1
Qwen 3 4B →\rightarrow Llama 3 8BCross Family30.5 ±\pm 2.243.5 ±\pm 0.363.5 ±\pm 0.1
Table 7. Mean (%, F1) over 17 benchmarks ( ±\pm SE over 3 training seeds). Other methods are provided cross-tokenizer-aware alignment to function across model families. \swat exceeds others by ≥\geq 20 points in the cross-generation/family cases.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

 ShortContext
PairRowARC-C AccHSwag AccWGrande AccMMLU AccGSM8K AccQuALITY AccQASPER F1Hotpot F1MuSiQue F1
Small →\boldsymbol{\rightarrow} Large
Qwen 3 4BReceiver80.677.259.262.275.264.149.067.434.6
→\rightarrow Llama 3 8B\swat85.283.658.862.067.664.251.170.642.6
 Retention (%)105.7108.399.399.789.9100.2104.4104.8123.4
Llama 3 8BReceiver92.691.469.473.289.880.149.173.143.4
Appendix
Table 8. In-domain (9): per-benchmark receiver score with its own context, \swat score, and zero-shot retention (italic; \swat over receiver). Scores in percent. HSwag = HellaSwag; WGrande = WinoGrande; Hotpot = HotpotQA.
 ShortContext
PairRowMedQA AccOpenBookQA AccMATH-500 AccBoolQ AccRACE AccLongHealth AccSQuAD F1NarrQA F1
Small →\boldsymbol{\rightarrow} Large
Qwen 3 4BReceiver60.076.028.483.781.366.888.974.2
→\rightarrow Llama 3 8B\swat54.879.220.079.876.457.087.573.4
 Retention (%)91.3104.270.495.394.085.498.498.9
Llama 3 8BReceiver64.490.276.689.189.472.086.972.7
Appendix
Table 9. OOD (8): per-benchmark receiver score with its own context, \swat score, and zero-shot retention (italic; \swat over receiver). Scores in percent.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 10  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 9  ×\,\times ( Psum,1P_{\mathrm{sum},1} )≈\approx 10  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 44.662.8+18.247.267.9+20.637.953.8+15.9
Tokens 144132−- 8.2%170154−- 9.6%151138−- 8.3%
S\mathcal{S} GPU (ms) 4,731997×\times 4.73,6871,438×\times 2.69,4521,336×\times 7.1
R\mathcal{R} TTFT (ms) 9.83.6×\times 2.721.86.5×\times 3.47.22.9×\times 2.5
Appendix
Table 10. Experiment 2, HotpotQA. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 58  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 26  ×\,\times ( Psum,3P_{\mathrm{sum},3} )≈\approx 39  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 28.737.2+8.539.047.8+8.830.840.5+9.8
Tokens 130129−- 0.5%308303−- 1.5%203202−- 0.4%
S\mathcal{S} GPU (ms) 4,2993,410×\times 1.37,4814,840×\times 1.513,7915,567×\times 2.5
R\mathcal{R} TTFT (ms) 11.75.9×\times 2.038.012.0×\times 3.211.25.1×\times 2.2
Appendix
Table 11. Experiment 2, LongHealth. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 14  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 10  ×\,\times ( Psum,1P_{\mathrm{sum},1} )≈\approx 15  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 11.824.0+12.211.832.6+20.88.220.0+11.7
Tokens 168168−- 0.2%254248−- 2.3%171166−- 3.4%
S\mathcal{S} GPU (ms) 5,2871,568×\times 3.45,7222,290×\times 2.510,6072,441×\times 4.3
R\mathcal{R} TTFT (ms) 9.93.7×\times 2.727.06.6×\times 4.17.82.9×\times 2.6
Appendix
Table 12. Experiment 2, MuSiQue. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 7  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 7  ×\,\times ( Psum,1P_{\mathrm{sum},1} )≈\approx 9  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 29.265.6+36.429.866.7+36.919.153.1+34.0
Tokens 108103−- 4.9%115103−- 10.7%8480−- 4.3%
S\mathcal{S} GPU (ms) 3,2861,085×\times 3.02,8061,023×\times 2.75,5901,083×\times 5.2
R\mathcal{R} TTFT (ms) 8.33.0×\times 2.816.85.9×\times 2.85.22.6×\times 2.0
Appendix
Table 13. Experiment 2, NarrativeQA. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 41  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 27  ×\,\times ( Psum,1P_{\mathrm{sum},1} )≈\approx 41  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 21.734.3+12.626.536.3+9.823.031.9+8.9
Tokens 106106+0.2%165164−- 0.5%108108−- 0.2%
S\mathcal{S} GPU (ms) 3,2692,125×\times 1.53,8502,781×\times 1.46,9533,337×\times 2.1
R\mathcal{R} TTFT (ms) 8.02.9×\times 2.819.85.5×\times 3.65.42.4×\times 2.2
Appendix
Table 14. Experiment 2, QASPER. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 42  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 30  ×\,\times ( Psum,1P_{\mathrm{sum},1} )≈\approx 52  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 51.659.2+7.657.871.8+13.953.661.6+8.0
Tokens 132128−- 2.5%184180−- 2.0%106104−- 2.3%
S\mathcal{S} GPU (ms) 3,7211,476×\times 2.54,1742,085×\times 2.07,2292,208×\times 3.3
R\mathcal{R} TTFT (ms) 11.86.0×\times 2.027.311.6×\times 2.38.24.9×\times 1.7
Appendix
Table 15. Experiment 2, QuALITY. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
  Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4B
MetricBudgetT2T\swatChangeT2T\swatChangeT2T\swatChange
 Lowest≈\approx 3  ×\,\times compression ( Psum,1P_{\mathrm{sum},1} )≈\approx 4  ×\,\times ( Psum,1P_{\mathrm{sum},1} )≈\approx 4  ×\,\times ( Psum,1P_{\mathrm{sum},1} )
Score 51.688.7+37.148.487.5+39.135.882.4+46.6
Tokens 6460−- 5.5%6047−- 22.9%5047−- 7.2%
S\mathcal{S} GPU (ms) 1,8521,068×\times 1.71,272935×\times 1.42,9231,220×\times 2.4
R\mathcal{R} TTFT (ms) 6.23.1×\times 2.011.55.9×\times 1.94.02.5×\times 1.6
Appendix
Table 16. Experiment 2, SQuAD. T2T : sender summarizes to receiver. \swat +AM: the sender’s cache rendered and compressed with Attention Matching. Budgets are determined by the sender agent in T2T given one of five summary prompts and sorted from lowest to highest per pair. Change: score difference in points, relative token difference, and speedup.
 Qwen3-4B →\rightarrow Llama3-8BLlama3-8B →\rightarrow Qwen3-14BGemma3-12B →\rightarrow Qwen3-4BAll pairs
MetricT2TSwaTChangeT2TSwaTChangeT2TSwaTChangeT2TSwaTChange
Score38.854.8+41.2%39.259.1+50.6%34.152.8+54.9%37.355.5+48.8%
Score Δ\Delta (pts)42.957.8+14.942.961.9+19.038.056.2+18.241.358.6+17.4
Tokens293254−- 13.1%269243−- 9.7%309269−- 13.1%290255−- 12.0%
S\mathcal{S} GPU (ms)8,7522,150×\times 4.16,1982,360×\times 2.619,7612,888×\times 6.810,2342,447×\times 4.2
R\mathcal{R} TTFT (ms)18.04.0×\times 4.531.87.5×\times 4.314.13.4×\times 4.220.14.7×\times 4.3
Appendix
Table 17. Geometric mean performance of SwaT over three sender–receiver pairs, seven datasets and five budgets. T2T and SwaT columns are geometric means over all datasets and budgets; Change is their ratio. Score Δ\Delta is the arithmetic mean score difference in points.
Figure 4. Scatter plot for all individual cells in tables 10 , 11 , 12 , 13 , 14 , 15 and 16 . We note that points in the tokens plot are close to the diagonal by construction as the two methods were picked to have close to matching budgets. Description
AblationΔ\Delta \swat
\swat00
\swat (w/ c2c ( Fu et al., 2026 ) token alignment)−3.20-3.20
Headwise MLP−14.71-14.71
Phase 1 only−36.97-36.97
Phase 2 only−36.96-36.96
Dense loss only−55.11-55.11
Appendix
Table 18. Ablations of our \swat architecture. Each method is compared against the average of two independently trained \swat models. Differences are in percentage points on the mean benchmark score (accuracy or F1). Metrics calculated over 13 benchmarks with the pair Qwen3-4B →\rightarrow Llama-3-8B.
MethodSeq. lenEMF1
Baselines
Qwen 3 4B28K20.436.75
Llama 3 8B truncate8K5.413.32
Llama 3 8B AM20\mathrm{AM}_{20}5.6K21.234.95
\swat
Qwen →AM20\rightarrow^{\mathrm{AM}_{20}} Llama5.6K24.841.87
Appendix
Table 19. Interleaved HotpotQA (500 questions; EM/F1 in %). Sequence lengths is number of text tokens or KV-cache tokens.
MethodVQAv2TQAOKVQAGQAVWAvg.
Gemma 3 4B →\rightarrow Gemma 3 4B
Sender45.3 ∗53.4 ∗22.031.4 ∗19.034.2 ±\pm .4 ∗
\swat53.856.622.435.319.037.4 ±\pm .4
Gemma 3 4B →\rightarrow Granite 3.3 8B
Sender45.3 ∗53.4 ∗22.0 ∗31.4 ∗19.034.2 ±\pm .4 ∗
T2T33.8 ∗32.0 ∗15.7 ∗16.3 ∗7.7 ∗21.1 ±\pm .4 ∗
Appendix
Table 20. Similar to table 4 . Test accuracy (%) for multiple pairs including a control where sender and receiver are the same VLM (Gemma 3 4B →\rightarrow Gemma 3 4B pair).
MethodDocVQAInfoVQAAvg.
Gemma 3 4B →\rightarrow Gemma 3 4B
Sender73.341.657.4 ±\pm .8
\swat40.926.833.8 ±\pm .8
Gemma 3 4B →\rightarrow Granite 3.3 8B
Sender73.341.657.4 ±\pm .8
T2T39.3 ∗25.5 ∗32.4 ±\pm .8 ∗
Appendix
Table 21. Similar to table 4 . Test accuracy (%) for multiple pairs on DocVQA and InfoVQA, which depend on dense text and fine detail in the image.
More in this topic

Multi-Agent Communication

Multi-agent communication studies how autonomous agents exchange information to coordinate decisions, share knowledge, and act together under constraints.

See all 380 papers

Explore similar work

CardsList
Sep 26, 2026

Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs

Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundancy, but prefill-free transfer across model families must handle differences in tokenization, model depth, and KV representations. To address these issues, we propose HeteroFold, a prefill-free cross-family KV cache transfer method that keeps both the sender and receiver frozen. HeteroFold aligns model structures, maps the sender cache into the receiver space, and calibrates it to preserve receiver behavior. Across six transfer directions, HeteroFold achieves the best cache-transfer performance on all four long-context benchmarks and most short-context settings. It also matches text-based communication on the multi-agent benchmark. At 32K context length, Llama-3.1-8B→\rightarrowMinistral-3-14B transfer is 10.7×10.7\times faster than Native Prefill and 1.181.18--1.47×1.47\times faster than the state-of-the-art prefill-free baselines, Dense Latent and KV Ridge. These results show that HeteroFold enables efficient cross-family KV reuse without receiver prefill.

Vincent-Daniel Yun, Woosang Lim, Haneul Yoo +3

University of Southern California, USA · Seoul National University, Republic of Korea · New York University, USA

Multi-Agent LLM SystemsLLM Inference AccelerationMulti-LLM Collaboration
AskPDF
May 19, 2026

Latent Cache Flow: Model-to-Model Communication Without Text

LLM agents today communicate via text, which incurs considerable latency and information loss due to the need to autoregressively decode the sharer model's state and encode at the receiver model. Recent work such as Cache-to-Cache (C2C; Fu et al., 2026) seeks to exchange KV caches by learning adapters that translate sharer KV matrices to the receiver model. However, the adapters are large and expensive to train, and translate individual tokens, which requires the target context to be identical. This is unsuitable for agent communication, where the LLMs have differing context. We introduce Latent Cache Flow (LCF). To address efficiency, we observe that keys and values can be jointly translated and compressed, reducing the adapter to about 4% of C2C's size. To address differing context, we design the adapter to transmit a summary of new information that the target model does not have. Our early experiments show that a pruned 13 MB LCF adapter can be more accurate than C2C at 956 MB in shared-context settings; for different contexts, LCF improves F1 by 7.5% and Exact Match by 23% while 8.5 times faster than text-based communication.

Maximillian Rossi, Prajwal Raghunath, Eugene Wu

Columbia University

KV CachingKV-Cache CompressionMulti-Agent LLMs
AskPDF
Jun 11, 2026

See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

Language-model agents build internal representations of the information they observe and the reasoning they perform. Sharing these representations offers a way to communicate both source information and reasoning across agents. For agents built from different models, this requires aligning their representations while preserving information useful to the receiver. We study this problem through KV-cache communication, examining how an agent uses internal states shared by other agents, with or without direct access to the information that other agents observed. A controlled self-communication study shows that cache pruning causes substantially greater degradation when the receiving agent lacks access to that information. We use this finding to guide dense cross-model cache alignment, combining positional disentanglement and KV-group transformations with reconstruction followed by generation training. Across six directed Qwen3 pairs, aligned caches improve in-domain accuracy over text communication when both agents observe the same input, with fewer estimated inference FLOPs. Experiments with three-agent document sharing and Mistral-to-Qwen transfer further demonstrate that aligned caches can carry information across both multiple separate observations and different model families.

Siyi Chen, Xiaoyan Zhang, Meng Wu +7

University of Michigan · NVIDIA · University of Pennsylvania +2

Multi-Agent CommunicationMulti-Agent LLMs
AskPDF