ACCT: Asymmetric Cross-LLM Context Transfer

Yuzhe Liu; Haolin Tian; Tonghan Wang

cs.AIOct 30, 2026

ACCT: Asymmetric Cross-LLM Context Transfer

Authors: Yuzhe Liu, Haolin Tian, Tonghan Wang

Organizations: Tsinghua University, Beijing, China

Abstract

In real-world deployments, large language model (LLM) systems are increasingly evolving from single-model invocation toward heterogeneous deployments in which different models are dynamically invoked for inference. When the same context is reused across models, however, each newly selected target model typically has to process it again, repeatedly incurring substantial computational costs. Therefore, we propose ACCT (Asymmetric Cross-LLM Context Transfer), which targets KV cache transfer between models of different scales within the same model family. ACCT transforms the KV cache produced by a source model into sparse KV states that can be directly consumed by a target model, thereby enabling cross-model reuse of context computation and decoupling context computation from downstream inference. ACCT consists of three lightweight components: the Cartographer identifies query-relevant context tokens for each target layer and KV head; the Allocator determines how many context tokens each target layer and KV head retains under a shared total KV budget; and the Compiler transforms selected source KV into sparse target KV. The target model can therefore perform inference using the transferred KV without reprocessing the original context. We evaluate ACCT in two complementary settings comprising five general-capability benchmarks and six highly context-dependent tasks. Experiments show that ACCT matches or exceeds the target model's original inference performance even when reducing the KV cache size by 75%. At a context length of approximately 19K tokens, ACCT's KV conversion achieves a 6.5-21.2 times speedup over target context prefill. These results demonstrate the feasibility of transferring context computation across models under a limited KV budget.

Figures & tables

Figure 1: ACCT combines Cartographer selection, Allocator budgeting, and Compiler conversion to transfer source KV into sparse target KV for inference without context re-encoding.

Figure 2: Context-state construction latency across six transfer directions. Target prefill encodes the complete context with the target model, whereas ACCT converts already computed source states. The comparison shows the efficiency gain from avoiding target-side context re-encoding, especially for longer contexts.

		General capability					Context-dependent tasks
Model pair	Method	Hella Swag	Wino grande	MMLU	PIQA	Wiki Text2	Search QA	Trivia QA	BoolQ	RACE	WikiHop	PubMed QA
		Acc.	Acc.	Acc.	Acc.	PPL $\downarrow$	F1	F1	Acc.	Acc.	Acc.	Acc.
Qwen3 $4\mathrm{B}\!\to\!8\mathrm{B}$	Target Full	39.70	58.50	59.70	73.20	32.91	46.65	50.48	86.30	87.60	49.60	48.80
	ACCT	33.20	51.50	65.70	73.10	25.94	67.07	66.31	61.90	63.50	36.50	40.20
	Relative	$0.84\times$	$0.88\times$	$1.10\times$	$1.00\times$	$1.27\times$	$1.44\times$	$1.31\times$	$0.72\times$	$0.72\times$	$0.74\times$	$0.82\times$
Qwen3 $8\mathrm{B}\!\to\!4\mathrm{B}$	Target Full	37.10	51.00	63.60	69.20	81.29	38.56	37.68	84.00	87.10	45.80	34.00

Table 1: Task performance of ACCT at

4\times

compression versus Target Full, which uses the target model’s full context KV. Even with 4

\times

compression (75% reduction in the global cache usage) and up to 95.3% reduction in prefilling time, our method can outperform full target KV on several tasks. F1 and accuracy are percentages (higher is better); PPL is lower-is-better. Bold marks improvements over Target Full. Relative is ACCT/Target Full for F1 and accuracy, and Target Full/ACCT for PPL.

Model pair	Compression	Hella Swag	Wino grande	MMLU	PIQA	Wiki Text2	Search QA	Trivia QA	BoolQ	RACE	WikiHop	PubMed QA
		Acc.	Acc.	Acc.	Acc.	PPL $\downarrow$	F1	F1	Acc.	Acc.	Acc.	Acc.
Qwen3 $4\mathrm{B}\!\to\!8\mathrm{B}$	$1\times$	38.60	51.40	55.80	70.90	23.26	38.54	47.03	77.80	69.40	35.10	45.60
	$2\times$	36.80	53.10	62.70	73.20	22.16	67.59	74.92	70.30	71.10	39.10	27.60
	$4\times$	33.20	51.50	65.70	73.10	25.94	67.07	66.31	61.90	63.50	36.50	40.20
Qwen3 $8\mathrm{B}\!\to\!4\mathrm{B}$	$1\times$	37.40	50.50	44.70	69.30	60.74	12.54	14.11	68.00	56.00	4.80	48.40
	$2\times$	32.80	52.50	53.60	70.10	142.09	61.54	57.65	62.60	66.80	36.10	55.20

Table 2: Task performance of bidirectional Qwen3 4B

\leftrightarrow

8B transfer at

1\times

and

2\times

compression, compared with the

4\times

setting used in the previous experiments. ACCT supports target inference at different compression ratios and outperforms full retention on multiple tasks with fewer KV states.

Model pair	Method	Hella Swag	Wino grande	MMLU	PIQA	Wiki Text2	Search QA	Trivia QA	BoolQ	RACE	WikiHop	PubMed QA
		Acc.	Acc.	Acc.	Acc.	PPL $\downarrow$	F1	F1	Acc.	Acc.	Acc.	Acc.
Qwen3 $4\mathrm{B}\!\to\!8\mathrm{B}$	ACCT	33.20	51.50	65.70	73.10	25.94	67.07	66.31	61.90	63.50	36.50	40.20
	Random	31.70	50.80	27.00	61.20	68.42	11.09	20.12	38.90	27.30	15.30	51.60
	Uniform	35.20	50.60	62.80	72.40	25.24	41.07	42.72	62.90	64.20	34.10	22.80
	Bridge-only	35.10	51.10	41.70	70.20	35.24	44.15	50.13	63.10	61.70	33.80	15.40
	Native KV	38.00	57.40	54.30	73.20	37.19	59.94	59.58	67.20	75.30	44.50	36.20

Table 3: Component ablations for Qwen3 transfer at

4\times

compression. Random replaces learned selection with random selection, Uniform replaces learned allocation with equal allocation, Bridge-only removes the correction procedure, and Native KV uses target-native states at the same positions. Cartographer, allocator, compiler all contribute significantly to the performance of ACCT.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Cartographer	Allocator	Compiler residual
Training examples	5,000	5,000	500
Epochs	1	1	4
Global batch	16	4	4
Optimizer updates	313	1,250	500
Optimizer	AdamW	AdamW	AdamW
Learning rate	$10^{-4}$	$10^{-2}$	$10^{-4}$

Appendix

Table 4: Recorded training schedules.

Pair	$D_{S}$	$(L_{T},H_{T})$	$(w_{M},d_{M},c_{M})$	$(w_{W},m_{W},D)$
Qwen3 4B $\to$ 8B	2560	(36,8)	(416,112,56)	(416,28,4)
Qwen3 8B $\to$ 4B	4096	(36,8)	(512,128,64)	(512,32,4)
Llama 3B $\to$ 8B	3072	(32,8)	(448,112,56)	(448,28,4)
Llama 8B $\to$ 3B	4096	(28,8)	(512,128,64)	(512,32,4)
Qwen2.5 7B $\to$ 32B	3584	(64,8)	(480,128,64)	(480,32,4)
Qwen2.5 32B $\to$ 7B	5120	(28,4)	(576,144,72)	(576,36,5)

Appendix

Table 5: Recorded geometry for six directions; D is the shared encoder/decoder depth.

Pair	Cartographer	Allocator	Compiler	Total
Qwen3 4B $\to$ 8B	1,338,768	288	61,151,168	62,490,224
Qwen3 8B $\to$ 4B	2,474,112	288	80,917,504	83,391,904
Llama 3B $\to$ 8B	1,665,424	256	62,676,608	64,342,288
Llama 8B $\to$ 3B	2,473,600	224	70,398,976	72,872,800
Qwen2.5 7B $\to$ 32B	2,081,440	512	108,981,440	111,063,392
Qwen2.5 32B $\to$ 7B	3,424,048	112	71,890,624	75,314,784

Appendix

Table 6: Exact interface parameter counts; Compiler includes the fitted, frozen Bridge.

Pair	Bridge	Residual	Gradient-optimized total
Qwen3 4B $\to$ 8B	9,510,912	51,640,256	52,979,312
Qwen3 8B $\to$ 4B	9,510,912	71,406,592	73,880,992
Llama 3B $\to$ 8B	8,454,144	54,222,464	55,888,144
Llama 8B $\to$ 3B	7,397,376	63,001,600	65,475,424
Qwen2.5 7B $\to$ 32B	16,908,288	92,073,152	94,155,104
Qwen2.5 32B $\to$ 7B	3,698,688	68,191,936	71,616,096

Appendix

Table 7: Compiler decomposition and gradient-optimized parameters of the entire interface.

Group	Dataset	Examples	Metric
General	HellaSwag ( Zellers et al., 2019 )	1,000	Accuracy
General	Winogrande ( Sakaguchi et al., 2020 )	1,000	Accuracy
General	MMLU ( Hendrycks et al., 2021 )	1,000	Accuracy
General	PIQA ( Bisk et al., 2020 )	1,000	Accuracy
General	WikiText2 ( Merity et al., 2017 )	500	PPL
Context	SearchQA ( Dunn et al., 2017 )	1,000	Token F1

Appendix

Table 8: The fixed 10,000-example evaluation panel.

Pair	General retention	Context retention	Wins over Full
Qwen3 4B $\to$ 8B	101.7%	95.9%	4/11
Qwen3 8B $\to$ 4B	94.7%	106.6%	4/11
Llama 3B $\to$ 8B	64.7%	77.5%	2/11
Llama 8B $\to$ 3B	67.2%	74.7%	2/11
Qwen2.5 7B $\to$ 32B	66.8%	77.5%	1/11
Qwen2.5 32B $\to$ 7B	75.7%	63.4%	0/11

Appendix

Table 9: Descriptive retention and task wins derived from the recorded main-table entries.

		General capability					Context-dependent tasks
Model pair	Method	Hella Swag	Wino grande	MMLU	PIQA	Wiki Text2	Search QA	Trivia QA	BoolQ	RACE	WikiHop	PubMed QA
		Acc.	Acc.	Acc.	Acc.	PPL $\downarrow$	F1	F1	Acc.	Acc.	Acc.	Acc.
Qwen3 $0.6\mathrm{B}\!\to\!4\mathrm{B}$	Target Full	37.10	51.00	63.60	69.20	81.29	38.56	37.68	84.00	87.10	45.80	34.00
Qwen3 $0.6\mathrm{B}\!\to\!4\mathrm{B}$	ACCT	34.60	49.70	60.70	65.40	46.66	44.29	45.63	68.10	53.20	32.60	49.00
Qwen3 $0.6\mathrm{B}\!\to\!8\mathrm{B}$	Target Full	39.70	58.50	59.70	73.20	32.91	46.65	50.48	86.30	87.60	49.60	48.80
Qwen3 $0.6\mathrm{B}\!\to\!8\mathrm{B}$	ACCT	33.50	50.40	59.00	65.30	38.10	40.29	41.70	62.80	53.90	28.20	43.40

Appendix

Table 10: Additional Qwen3 results on 11 datasets (F1/accuracy in %; PPL

\downarrow

), with bold marking ACCT improvements over the corresponding Target Full.

Context length $L$	Warm-up runs	Timed runs
$L<4096$	2	3
$4096\leq L<8192$	1	3
$L\geq 8192$	1	2

Appendix

Table 11: Warm-up and timed repetitions for each stage.

Direction	$L$	Cart.	Alloc.	Comp.	Convert	Source	Target	Speedup
$4\mathrm{B}\!\to\!8\mathrm{B}$	1001	7.0	7.4	12.8	27.2	69.4	91.1	$3.35\times$
	1024	7.0	7.4	12.9	27.3	69.4	91.9	$3.37\times$
	1729	7.0	7.9	17.0	31.9	106.1	159.0	$4.98\times$
	1918	7.1	7.4	18.1	32.6	131.6	165.3	$5.07\times$
	4126	15.6	7.5	30.6	53.7	270.2	363.9	$6.78\times$
	8158	20.3	7.8	53.3	81.4	533.7	729.8	$8.97\times$

Appendix

Table 12: Qwen3 stage latencies (ms) and state-conversion speedups by context length.

Direction	$L$	Source	Cart.	Alloc.	Comp.	Convert	Target	Speedup
$3\mathrm{B}\!\to\!8\mathrm{B}$	995	42.6	6.2	7.4	12.3	25.9	86.4	$3.34\times$
	996	43.8	6.2	7.4	12.3	25.9	86.4	$3.34\times$
	1551	64.5	6.2	7.4	15.3	28.9	123.8	$4.28\times$
	1900	88.2	6.2	7.4	17.1	30.7	153.1	$4.99\times$
	3989	176.3	9.9	7.4	28.3	45.6	319.9	$7.02\times$
	7848	339.0	17.6	7.4	49.4	74.4	656.8	$8.83\times$

Appendix

Table 13: Llama stage latencies (ms) and state-conversion speedups by context length.

Direction	$L$	Source	Cart.	Alloc.	Comp.	Convert	Target	Speedup
$7\mathrm{B}\!\to\!32\mathrm{B}$	1001	85.7	10.7	8.0	17.2	35.9	308.5	$8.59\times$
	1024	86.4	10.8	8.3	17.3	36.4	310.9	$8.54\times$
	1729	127.3	11.0	8.2	24.4	43.6	552.9	$12.68\times$
	1918	162.2	10.9	8.3	26.2	45.4	584.1	$12.87\times$
	4126	323.6	25.9	8.3	48.6	82.8	1324.1	$15.99\times$
	8158	625.1	34.0	8.4	89.1	131.5	2592.0	$19.71\times$

Appendix

Table 14: Qwen2.5 stage latencies (ms) and state-conversion speedups by context length.

Direction	$L$	Source	Transfer	Total	Target	Speedup
Qwen3 $4\mathrm{B}\!\to\!8\mathrm{B}$	1001	69.4	36.4	105.8	91.1	$0.86\times$
	19070	1496.7	219.6	1716.3	1993.4	$1.16\times$
Llama $3\mathrm{B}\!\to\!8\mathrm{B}$	995	42.6	33.6	76.2	86.4	$1.13\times$
	19047	995.8	202.9	1198.7	1837.5	$1.53\times$
Qwen2.5 $7\mathrm{B}\!\to\!32\mathrm{B}$	1001	85.7	52.3	138.0	308.5	$2.24\times$
	19070	1607.3	397.4	2004.7	6706.3	$3.35\times$

Appendix

Table 15: Target-state construction latency (ms), including source encoding, on the shortest and longest contexts.

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 77 papers

Aug 31, 2026

A Universal Context-Reuse Layer for Cross-Model KV Sharing

Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study cross-model KV sharing, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B

\rightarrow

Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59% to 34.48%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B

\rightarrow

Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B

\rightarrow

Qwen2.5-7B setting, cross-family handoff achieves 44.0% accuracy compared with 45.7% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate context mobility as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.

Yi Li, Dongming Jiang, Yi Zhao +1

University of Texas at Dallas

LLM Inference KV Caching LLM Inference Acceleration KV-Cache Reuse KV-Cache Transfer

Ask PDF

May 9, 2026

ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

Efficient long-context inference in Large Language Models (LLMs) is severely constrained by the Key-Value (KV) cache memory wall, yet existing pruning methods force a choice between low-latency heuristics that sacrifice precision and high-precision reconstruction methods that incur prohibitive prefilling overhead. To bridge this scoring-cost--accuracy gap, we propose ProxyKV, a cross-model proxy pruning framework that offloads importance scoring to a lightweight intra-family Small-Model Proxy executed asynchronously to the Large-Model Target. To bridge the architectural gap between heterogeneous models, we design the HybridAxialMapper, which disentangles temporal feature extraction from cross-head alignment, together with a Multi-Granularity Hybrid Loss that shifts the learning objective from rigid regression to relative ranking consistency. Across the Llama-3.1, Qwen-2.5, and Qwen-3 families spanning targets from 7B up to 32B parameters on LongBench, SCBench, and RULER, ProxyKV matches KVZip on aggregate (recovering

\sim$$98.7\%

of its mean accuracy) while delivering up to a

3.21\times

prefilling speedup on Llama-3.1-8B (dual-GPU;

\sim$$1.5\times

shared single-GPU) and sustaining the speedup at contexts up to 170k tokens on Qwen-2.5-7B.

Junjie Li, Jiong Lou, Jie Li

Shanghai Jiao Tong University

LLM Pruning KV Caching Long-Context Language Model Inference KV-Cache Eviction LLM Inference Acceleration KV-Cache Pruning

Ask PDF

Sep 26, 2026

Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs

Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundancy, but prefill-free transfer across model families must handle differences in tokenization, model depth, and KV representations. To address these issues, we propose HeteroFold, a prefill-free cross-family KV cache transfer method that keeps both the sender and receiver frozen. HeteroFold aligns model structures, maps the sender cache into the receiver space, and calibrates it to preserve receiver behavior. Across six transfer directions, HeteroFold achieves the best cache-transfer performance on all four long-context benchmarks and most short-context settings. It also matches text-based communication on the multi-agent benchmark. At 32K context length, Llama-3.1-8B

\rightarrow

Ministral-3-14B transfer is

10.7\times

faster than Native Prefill and

1.18

1.47\times

faster than the state-of-the-art prefill-free baselines, Dense Latent and KV Ridge. These results show that HeteroFold enables efficient cross-family KV reuse without receiver prefill.

Vincent-Daniel Yun, Woosang Lim, Haneul Yoo +3

University of Southern California, USA · Seoul National University, Republic of Korea · New York University, USA

Multi-Agent LLM Systems LLM Inference Acceleration Multi-LLM Collaboration KV-Cache Transfer

Ask PDF

Abstract

KV-Cache Reuse

Explore similar work

A Universal Context-Reuse Layer for Cross-Model KV Sharing

ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs