Accelerating Cross-Model LLM Inference via Attention Map Reuse

Zhan Jin; Yuze Luo; Bin Cui; Xupeng Miao

cs.LGOct 10, 2026

Accelerating Cross-Model LLM Inference via Attention Map Reuse

Authors: Zhan Jin, Yuze Luo, Bin Cui, Xupeng Miao

Organizations: School of Computer Science & Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University · Key Lab of High Confidence Software Technologies (MOE), Peking University

Abstract

LLM-augmented data workflows often involve multiple heterogeneous models operating over substantial shared context, such as schemas, tool specifications, interaction history, and intermediate results. Although prefix caching can reduce prefilling overhead, existing approaches are largely limited to reuse within the same model because KV states are tightly coupled to model-specific architectures. We present MapFlow, a system for efficient heterogeneous LLM serving via cross-model attention map reuse. Instead of reusing KV states, MapFlow reuses attention maps, which capture token dependency patterns and are more transferable across models. To enable efficient and accurate reuse, we combine partial recomputation with lightweight linear translation across layers and heads. To make attention reuse practical for long contexts, we further introduce block-level sparse compression and a fused retrieval kernel. Experiments show that MapFlow reduces prefilling latency and improves end-to-end serving efficiency while preserving generation quality across heterogeneous model pairs and multi-turn routing workloads. These results suggest that attention maps provide an effective reusable state for heterogeneous LLM serving, especially in prefix-heavy data workflows. Our code is available: https://github.com/PKU-DAIR/MapFlow.

Figures & tables

Figure 1. Illustration of attention mechanism, prefill phase and decode phase.

Figure 2. Illustration of cross-model prefix cache framework in multi-LLM router system.

Notation	Description
$S,R$	Sender and Receiver models
$L_{M},H_{M}$	Number of layers and heads for model $M\in\{S,R\}$
$j,i$	Layer index ( $1\leq j\leq L_{M}$ ) and head index ( $1\leq i\leq H_{M}$ )
$r$	Number of recomputed prefix layers in $R$
$A_{i}^{j}(M)$	Attention map of the $i$ -th head in the $j$ -th layer of model $M$
$\hat{A}_{i}^{j}(M)$	Reused (approximated) attention map for $R$

Table 1. Summary of notations.

Figure 3. Illustration of attention map reuse. Each circle represents a computational module; purple circles signify modules requiring computation, while gray circles indicate inactive modules. The blue rectangles correspond to their respective caches.

Figure 4. TV-distance heatmaps for linear fits between every sender–receiver layer pair in Qwen3-8B $\rightarrow$ 1.7B. The x-axis is the sender-layer index, and the y-axis is the receiver-layer index. Entry $(i,j)$ is the TV distance between the attention map fitted from sender layer $i$ and the original attention map of receiver layer $j$ ; lower values indicate a better fit.

Figure 5. Overview of the data flow mechanism in MapFlow , illustrating the workflow of partial recomputation and linear attention translation.

Figure 7. MapFlow reduces prefill latency while maintaining generation quality.

Model pair	Full prefill	FlexPrefill	IAM	Minference	MapFlow
Qwen3 8B	0.305 /	0.288 /	0.165 /	0.133 /	0.288 /
$\rightarrow$ 1.7B	0.5726	0.5001	0.5285	0.4232	0.4928
Qwen3 14B	0.526 /	0.380 /	0.204 /	0.259 /	0.392 /
$\rightarrow$ 8B	1.8604	1.6026	1.7171	1.3409	1.5767
Ministral3 8B	0.454 /	0.318 /	0.344 /	0.176 /	0.348 /
$\rightarrow$ 3B	1.2043	1.0530	1.1116	0.8638	1.0434

Table 2. F1 / TTFT (s) results on RULER-20K.

Figure 8. MapFlow achieves higher end-to-end throughput in multi-turn conversation routing.

Figure 9. CDF comparison of the distribution of TTFT when switching from large to small models. The y-axis represents the cumulative probability.

Token Range	top_p = 0.7	top_p = 0.8	top_p = 0.9
0 $\sim$ 4k	1.23	1.80	3.11
4k $\sim$ 8k	1.84	3.09	6.35
8k $\sim$ 12k	2.82	4.87	10.07
12k $\sim$ 16k	3.35	5.86	12.29
16k $\sim$ 20k	2.99	5.53	12.94
20k $\sim$ 24k	3.12	5.52	12.84

Table 3. Memory Footprint Increments by Top_p (GiB)

Model	Full attention	$p=0.9$	$p=0.85$	$p=0.8$
Qwen3 1.7B	0.296	0.293	0.286	0.092
Ministral3 3B	0.365	0.418	0.282	0.172
Qwen2.5 7B	0.076	0.055	0.025	0.008
Llama3.1 8B	0.531	0.506	0.306	0.193
Phi-4-Mini	0.389	0.325	0.060	0.020

Table 4. Generation quality under different sparsification ratios (top-

p

thresholds) on RULER (20k context)

Receiver model	Full prefill	MapFlow (top- $p$ =0.9)	FlexPrefill
Qwen3 1.7B	0.432	0.335	0.218
Qwen3 8B	0.464	0.432	0.226
Ministral3 3B	0.367	0.403	0.146
Ministral3 8B	0.513	0.461	0.223

Table 5. Cross-dataset transfer results on 2WikiMultihopQA (F1 score). Weights were trained on HotpotQA and evaluated zero-shot on 2WikiMultihopQA.

Figure 10. MapFlow achieves speedup across varying transition probabilities

p

. The x-axis represents the probability of transitioning from a large model to a small model, while the y-axis represents the P90 TTFT in seconds.

Figure 11. Speedup for single agentic workflow for Ministral model family. Horizontal axis: Prompt length / large model decoding length / medium model decoding length / small model decoding length .

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 216 papers

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
Aug 4, 2026Taekyung Heo, Rasoul Shafipour, Ritchie Zhao +6KV Caching LLM Inference Acceleration
NVIDIA
A Universal Context-Reuse Layer for Cross-Model KV Sharing
Aug 31, 2026Yi Li, Dongming Jiang, Yi Zhao +1LLM Inference KV Caching
University of Texas at Dallas
RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving
Oct 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +6LLM Serving LLM Inference Scheduling
HKUST

Aug 4, 2026J/K move · Enter open · S save

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub+5

NVIDIA

KV Caching LLM Inference Acceleration KV-Cache Reuse

Open paper AskPDF arXiv

Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

Abstract

LLM Serving

Explore similar work

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

A Universal Context-Reuse Layer for Cross-Model KV Sharing

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving