LatComm: Reusable Context Compression for Latent Communication between Large Language Models

Huawei Lin; Jinting Liu; Tan Yu; Weijie Zhao

cs.CLOct 19, 2026

LatComm: Reusable Context Compression for Latent Communication between Large Language Models

Authors: Huawei Lin, Jinting Liu, Tan Yu, Weijie Zhao

Organizations: Rochester Institute of Technology · NVIDIA

Abstract

Systems that combine several large language models (LLMs) often use one context for many questions. When the context is passed between models as text, the receiving model processes all of its tokens again for every question. Avoiding this cost requires a context representation that is compact, reusable across questions, and readable by a different LLM. Existing compression and latent communication methods each provide only some of these properties. We introduce LatComm, a Latent Communication interface that provides all three by separating encoding from reading: the context is encoded once without the question, and the receiver selects what to read for each question. A frozen context encoder LLM compresses the context into a fixed-size packet and sends it to a different frozen receiver LLM, which reads the packet through cross-attention, with gates computed from its prompt. On 12 multiple-choice question-answering datasets, LatComm improves average accuracy over single-vector baselines by up to 10.2 points. On QuALITY, it reduces the receiver's input by 57x and answers the questions of an article 3x faster.

Figures & tables

Figure 1: Text passing vs. latent passing. (a) The receiver reads the context again for every question. (b) The context is encoded once and reused. (c, d) Measured cost on QuALITY.

Method	Query- independent	Fixed-budget compression	Cross-LLM communication	Bridge-only training
LLMLingua-2 ( Pan et al., 2024 )	✓	✓	✗	✗
AutoCompressor ( Chevalier et al., 2023 )	✓	✗	✗	✗
ICAE ( Ge et al., 2024 )	✓	✓	✗	✗
xRAG ( Cheng et al., 2024 )	✓	✓	✗	✓
CEPE ( Yen et al., 2024 )	✓	✗	✗	✗
DroidSpeak ( Liu et al., 2024a )	✓	✗	✗	✗

Table 1: Comparison of context compression and latent communication. Query-independent (reusable): the representation is produced without the question. Fixed-budget compression (compact): its size does not depend on the input length. Cross-LLM communication: latent representations pass from one LLM to a different LLM. Bridge-only training: only an interface between two frozen LLMs is trained.

Figure 2: LatComm overview. (a) The encoder reads the context; the receiver reads a task prompt. LatComm connects selected layers and adds context information after each paired receiver block. Both LLMs remain frozen. (b) For one layer pair, a context compressor produces reusable slots $Z_{i}$ . The gates compare a summary of the receiver’s prompt states with the compressed slots. The reader combines the gated slot values and adds the result to the receiver state.

Figure 3: Two-stage training of LatComm . (a) In Stage 1, the receiver gets an instruction to reproduce the context $x$ and can read it only from the packet $\mathcal{Z}_{\phi}(x)$ . The reconstruction loss $\mathcal{L}_{\mathrm{rec}}$ trains the compressors, readers, and gates. (b) In Stage 2, the receiver gets the question with its answer choices, and the task loss $\mathcal{L}_{\mathrm{task}}$ on the answer $y$ trains only the readers and gates. The compressors are fixed, so the packet of each training context is computed once and stored. Both LLMs stay frozen in both stages.

Method	Compute per question	One-time compute	Stored size	Cross- LLM
Text passing	$L_{T}(n_{E}{+}n_{T})^{2}$	–	$n_{E}$	✓
KV cache	$L_{T}\,n_{T}(n_{E}{+}n_{T})$	$L_{T}\,n_{E}^{2}$	$L_{T}\,n_{E}\,d_{T}$	✗
LatComm	$L_{T}\,n_{T}^{2}+P\,n_{T}S$	$L_{E}\,n_{E}^{2}$	$P\,S\,d$	✓

Table 2: Orders of growth for three ways to reuse a context of

n_{E}

tokens for questions with

n_{T}

-token prompts.

L_{E}

and

L_{T}

are the numbers of encoder and receiver layers. LatComm uses

P

layer pairs with

S

slots of width

d

. The compute columns treat hidden sizes as constants and assume full self-attention in every layer.

Full text
		Task datasets (%)													Held-out datasets (%)
Method	Encoder	CLOTH	Cosmos	HSwag	LogiQA	MCScr.	MuTual	OBQA	QuAL.	ReClor	SciQ	SIQA	SWAG	Avg	RACE	DREAM	QuAIL	Avg
Raw-text		45.3	40.8	37.0	31.0	82.3	48.0	68.0	44.9	38.6	97.5	48.2	35.6	51.43	66.1	70.9	58.2	65.06
No context
Query-only		25.0	44.8	28.9	26.0	62.6	32.3	53.6	32.2	27.8	77.1	37.7	27.3	39.60	39.1	36.2	34.0	36.41
Constant packet		44.2	63.2	68.9	37.5	75.6	71.9	68.8	39.5	41.8	88.1	57.6	65.8	60.25	45.5	41.2	43.5	43.37

Table 3: Main results with the Qwen3.5-0.8B receiver: accuracy on each dataset. Raw-text and Query-only are the untrained receiver with and without the context in its prompt, and Constant packet trains the readers and gates with one fixed packet. Avg is Task Avg or Held-out Avg. Bold marks the best compressed-context method per column.

Context encoder	Receiver	$P$	$S$	$d$	Packet
Same model
Qwen3.5-0.8B	Qwen3.5-0.8B	24	32	256	384 KiB
Larger encoder
Qwen3.5-4B	Qwen3.5-0.8B	24	32	256	384 KiB
Qwen3.5-9B	Qwen3.5-0.8B	24	32	256	384 KiB
Cross-family

Table 4: Model pairs with the default packet.

P

is the number of layer pairs,

S

the number of slots per pair, and

d

the slot width. The packet stores

P\times S\times d

values in bfloat16, which takes

2PSd

bytes.

Full text
		Task datasets (%)													Held-out datasets (%)
Method	Encoder	CLOTH	Cosmos	HSwag	LogiQA	MCScr.	MuTual	OBQA	QuAL.	ReClor	SciQ	SIQA	SWAG	Avg	RACE	DREAM	QuAIL	Avg
Raw-text		25.0	25.4	25.0	22.0	50.9	23.6	28.0	25.6	24.4	24.8	33.5	24.6	27.73	25.9	34.7	25.2	28.63
No context
Query-only		25.4	25.5	25.7	25.8	48.9	23.8	26.4	25.9	25.8	23.5	33.7	25.2	27.98	26.4	34.6	25.2	28.74
Constant packet		38.2	59.3	67.9	28.4	71.8	59.9	56.8	35.5	28.2	82.0	54.8	65.7	54.04	39.0	41.6	43.3	41.30

Table 5: Results with the Gemma-3-1B-pt receiver: accuracy on each dataset. Raw-text and Query-only are the untrained receiver with and without the context in its prompt, and both are near chance for this receiver. Avg is Task Avg or Held-out Avg. Bold marks the best compressed-context method per column.

Encoder	Receiver	Constant packet	Embedding	LatComm
Same model
Qwen3.5-0.8B	Qwen3.5-0.8B	60.25	61.98	64.91
Larger encoder
Qwen3.5-4B	Qwen3.5-0.8B	60.25	61.98	65.43
Qwen3.5-9B	Qwen3.5-0.8B	60.25	61.98	65.80
Cross-family

Table 6: Task Avg (%) of six model pairs and two trained references with the same receiver. Embedding uses BGE-Multilingual-Gemma2. Each receiver’s Constant packet was trained with its 4B, 9B, or Granite pair.

Quantity	Raw-text	LatComm
Per question
Receiver input tokens	5,915	103
Receiver time (ms)	511	131
Receiver peak memory (GiB)	0.87	0.06
Per context
Encoding time (s)	–	0.71

Table 7: Measured cost on the QuALITY test set (115 articles, 2,086 questions) with the Qwen3.5-4B encoder and the Qwen3.5-0.8B receiver. Times are measured on one H100 GPU at batch size 1. Peak memory is the GPU memory that one call adds to the loaded models.

Figure 4: Reuse cost on QuALITY. (a) Receiver time per question for Raw-text, five LatComm settings, and Query-only. The hatched part of a bar is the time added by the readers and gates. (b) Encoding time per context for three context encoders.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Stage 1: context reconstruction
Dataset	Train examples	Train context tokens	Avg. context	Eval
WikiText-103	446,282	100,278,346	224.7	1,057
FineWeb-Edu	1,553,718	1,033,473,862	665.2	13,926
Stage 1 total	2,000,000	1,133,752,208	566.9	14,983
Stage 2: task supervision
CLOTH	76,660	34,320,807	447.7	11,516

Appendix

Table 8: Data for both training stages and the held-out datasets. Train excludes the development holdout. Tokens count untruncated context text once per training example with the pinned Qwen3.5-0.8B tokenizer, excluding instructions, choices, targets, and special tokens. Reconstruction supervision contains the same text plus one EOS per passage (1,135,752,208 target tokens). Avg. context is the mean training context length in tokens, rounded to one decimal; total-row means are weighted by example count. For the held-out datasets, it is the mean context length of the test questions. Eval is the reconstruction test set, the fixed public multiple-choice reporting suite, or the held-out test set.

Data
Setting	Stage 1	Stage 2
Training examples	2,000,000	282,441
Epochs	1	3
Updates	5,209	2,208
Batch size	384	384
Optimization

Appendix

Table 9: Training hyperparameters of the default configuration. Stage 1 trains the compressors, readers, and gates, and Stage 2 trains only the readers and gates.

Encoder
			Reconstruction
Setting	Packet	Size	Loss	Acc.	Task Avg
Qwen3.5-0.8B	$24\times 32\times 256$	384 KiB	1.83	56.3	64.91
Qwen3.5-4B (default)	$24\times 32\times 256$	384 KiB	1.24	68.5	65.43
Qwen3.5-9B	$24\times 32\times 256$	384 KiB	0.95	75.1	65.80
Slots per pair

Appendix

Table 10: Ablation of the context encoder and the packet settings with the Qwen3.5-0.8B receiver. Each group changes one setting of the default configuration. Packet gives

P\times S\times d

: layer pairs, slots per pair, and slot width. Size is the stored size. Reconstruction gives the loss and the token accuracy (%) on 256 held-out passages at the end of Stage 1. Task Avg (%) is measured after Stage 2.

Figure 5: Training loss of (a) Stage 1 and (b) Stage 2 against the number of context tokens processed, for the configurations of Table 10 . Each column changes one setting, and the solid orange line is the default configuration. The token count is the number of processed examples times the mean context length of the training set. The curves are running means over 50 updates in Stage 1 and 100 updates in Stage 2. Dotted lines mark the ends of the first and second epochs of Stage 2.

Same trained readers and gates
	Task datasets (%)													Held-out datasets (%)
Packet given to the receiver	CLOTH	Cosmos	HSwag	LogiQA	MCScr.	MuTual	OBQA	QuAL.	ReClor	SciQ	SIQA	SWAG	Avg	RACE	DREAM	QuAIL	Avg
Own packet	52.2	61.8	73.4	37.9	80.8	73.9	79.6	43.2	49.8	92.3	66.5	73.2	65.39	52.3	52.5	44.6	49.82
Packet of another context	30.7	48.1	50.3	28.1	63.4	43.8	51.2	31.7	30.6	77.3	45.4	46.4	45.59	37.8	38.5	32.5	36.28
No packet	25.0	44.8	28.9	26.0	62.6	32.3	53.6	32.2	27.8	77.1	37.7	27.3	39.60	39.1	36.2	34.0	36.41
Readers and gates trained with one fixed packet

Appendix

Table 11: Packet controls for the default configuration: accuracy on each dataset. The first three rows use the same trained readers and gates and change only the packet. The packet of another context comes from a different test question. The three rows come from separate evaluation runs, so the first row differs slightly from Table 3 . Columns and abbreviations follow Table 3 .

Figure 6: Accuracy on the test questions that the Qwen3.5-4B encoder answers correctly or wrongly from the full text. Embedding uses BGE-Gemma2.

Figure 7: Receiver time per question against the context length on QuALITY. Each point is one article.

Figure 8: Gate values of the default pair on 5,386 test questions. (a) Mean over the 32 slots of each layer pair. The band covers the 10th to 90th percentile of the questions. (b) Mean gate of each slot. The slots of a layer pair are sorted from the highest mean gate to the lowest.

Figure 9: Packets of the default pair at each layer pair. (a) Mean cosine similarity between two slots of one packet. (b) Mean cosine similarity between the packets of two contexts from the same dataset or from different datasets. (c) Balanced accuracy of identifying the source dataset of a context from its packet.

LLM Context Compression

See all 52 papers

Draft-KV: Learning Useful Latent Communication Between Language Models
Sep 28, 2026Linquan Wu, Shichang Meng, Tianxiang Jiang +7Knowledge Augmentation for Language Models Multi-LLM Collaboration
City University of Hong Kong · University of Science and Technology of China · University of Electronic Science and Technology of China +3
Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compression
May 17, 2026Natalia Trukhina, Vadim VashkelisLLM Compression LLM Context Compression
Embedded Intelligence Lab (EMILAB)
LatentPress: Context Compression Beyond Text and Vision
Sep 1, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +3LLM Inference Acceleration LLM Context Compression
Iowa State University · Cornell University

Sep 28, 2026J/K move · Enter open · S save

Draft-KV: Learning Useful Latent Communication Between Language Models

Linquan Wu, Shichang Meng, Tianxiang Jiang, Haoyu Yang+6

City University of Hong Kong · University of Science and Technology of China · University of Electronic Science and Technology of China+3

Knowledge Augmentation for Language Models Multi-LLM Collaboration

Open paper AskPDF arXiv

Latent communication passes internal states between language models instead of decoded text, but higher receiver accuracy does not show that the receiver used the message content. Across five method-dataset pairs, replacing each message with one from an unrelated question changes accuracy by at most 0.60 points, even when communication adds 15.44 points over the receiver alone. Thus the interface can supply the gain while making the sharer dispensable. Draft-KV instead sends the key-value states formed while the sharer drafts an answer to the current question. Linear projections place these states in a side memory read through a gated attention branch, and progressive training moves from message reconstruction to answer supervision under a guard on harm from mismatched messages. Both models remain frozen and the interface trains 1.05M parameters, 348x fewer than C2C. With a Qwen3-8B sharer, a frozen Qwen2.5-0.5B-Instruct receiver reaches 78.04% on MMLU-Redux, versus 37.45% alone and 36.40% with reassigned messages. At fixed interface size, scaling the sharer from 0.6B to 8B raises accuracy from 46.11% to 78.04%; communication also transfers to held-out tasks and can exceed both models when each holds different evidence.

Abstract

LLM Context Compression

Explore similar work

Draft-KV: Learning Useful Latent Communication Between Language Models

Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compression

LatentPress: Context Compression Beyond Text and Vision