cs.CLSep 26, 2026

KV-Lingo: Learning KV-Cache Translators with Distillation

Authors: Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi, Pierre Ablin

Organizations: Apple

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatible representations. This makes it costly to switch models over a shared context: although the context has already been processed by one model, the incoming model must process it again to build its own cache. We introduce KV-Lingo, a method for translating the KV cache of a source model into one that can be read by a target model. KV-Lingo consists of a collection of linear maps, typically one per layer of the target model, that are applied independently on all tokens' key and value representations. We train these maps using distillation, minimising the divergence between the target model's predictions from its native cache and those from the translated cache. We consider several model pairs spanning multiple sizes and architectures, training one translator per pair on a generic text corpus. The resulting translators preserve strong downstream performance in both small-to-large and large-to-small transfers. Since a switch then costs a linear map and a single decoding step instead of a prefill, replacing re-prefill with cache translation reduces the time to first token after a model switch by 9.6x already on a 64-token prompt for Qwen models on an Apple M3 Ultra, and by up to 29x at 32k context length on an H100. These gains make KV-Lingo particularly useful for dynamic model routing: a context can be processed by one model and handed off to another only when needed, without re-prefilling the shared prefix. We finally show that KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in our multi-turn evaluations.

Figures & tables

Figure 1: KV-Lingo translates a KV cache between two frozen models , so a conversation can switch model without re-prefilling its context. Top: the small model SS prefills the prompt, and the large model LL decodes the answer from the translated cache (K^,V^)(\widehat{K},\widehat{V}) , dashed. Bottom: the same from LL to SS , with a second translator. Each translator (flame, the only trained part) is a set of linear maps, one per target layer for keys and one for values, applied token by token, so its cost grows linearly with the prefix length where re-prefilling grows quadratically.
Figure 2: Layer-to-layer cache correspondence. Linear CKA between every layer of the large (rows) and small (columns) model, on post- kk -norm keys over 1616 k tokens; shared colour scale, dashed line at equal relative depth. Left: pretrained Qwen3 Instruct pair of equal depth. Middle: 1.71.7 B/ 0.60.6 B pair pretrained from scratch, sharing no checkpoint. Right: pretrained Gemma-3 pair differing in depth and key/value width. Appendix B.1 adds every other pair we measured, including two randomly initialised controls.
Figure 3: Cache translation scores , reasoning-off, mean of MMLU-Pro, ARC-C , RULER, LongMemEval, CoQA, IFEval, XQuAD, RepLiQA and MT-Bench-101. (a) Shape-preserving , Qwen3-0.6B/1.7B: a distilled head-wise linear translator suffices. (b) Cross-layer , Qwen3-0.6B/8B; head-mixing KV-Lingo at ν=2\nu=2 uses the R2R^{2} assignment. ν=1\nu=1 for MLPs.
Figure 4: Score against time to first token, long context, S→LS\to L . Each colour is one long-context evaluation task. Open squares: Qwen3-8B decoding from its own cache, timed as one full prefill. Filled markers: Qwen3-8B reading a translated cache, timed as the translation of an already computed source cache plus one decoding step (Appendix Y ). Left : shape-preserving pair. Right : cross-layer pair at ν=2\nu=2 . Throughput is the same for all methods.
Figure 5: Multi-turn model switching on CoQA. Qwen3- 44 B ↔\leftrightarrow Qwen3- 88 B alternate over 10 CoQA turns, reasoning off; panels differ in which model answers turn 11 . Both strategies use the same translators. The naive strategy collapses; KV-Lingo stays level with full re-prefill through turn 1010 . Translated conditions: mean over three seeds; error bars: standard error over evaluation items (Appendix E.1 ).
Figure 6: Switch cost of KV-lingo. (a) Speedup of the switch on an H100, the target served by vLLM: its time to first token after re-prefilling, divided by that after translation plus one decoding step. (b) The terms of that ratio for the switch into Qwen3-1.7B. (c) The same terms on an Apple M3 Ultra (MLX), from Qwen3-4B into Qwen3-30B-A3B, plus the 4B’s own prefill.
Appendix figures & tables69 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Diagonal contrast of the per-layer CKA grid, every pair. Unweighted linear CKA between source layer ℓ\ell and target layer ℓ′\ell^{\prime} , summarised as the gap between the diagonal ( ℓ=ℓ′\ell=\ell^{\prime} ) and the off-diagonal mean; the parenthesised number is the mean offset of the per-row argmax from the diagonal, in layers. The released equal-depth Qwen3 pairs, Base and Instruct alike, have a strong on-diagonal peak at offset 00 , which licenses the identity assignment. The from-scratch, cross-layer and Gemma-3 pairs keep about half that contrast and, except for the from-scratch pair, sit near the relative-depth diagonal; they are the pairs for which the source layer is selected rather than assumed. Random initialisation gives essentially no structure.
PairsideRlin2R^{2}_{\mathrm{lin}}Rnl2R^{2}_{\mathrm{nl}}nonlin. extraρ\rho
Qwen3 8B/4BK0.8150.8150.8250.8250.0100.0100.9890.989
V0.6930.6930.7080.7080.0150.0150.9800.980
Qwen3 1.7B/0.6BK0.7660.7660.7940.7940.0280.0280.9650.965
V0.6580.6580.6970.6970.0390.0390.9450.945
from scratch 1.7B/0.6BK0.7520.7520.8050.8050.0530.0530.9350.935
V0.6510.6510.7240.7240.0730.0730.9030.903
Appendix
Table 1: Ablation: the impact of translating with an MLP instead of a linear map. Per-layer linear fit of the target’s cache from the source’s, averaged over layers, on held-out tokens ( L→SL\to S ; pre-normalisation capture point, which leaves values unchanged; d=1024d=1024 per side). Rlin2R^{2}_{\mathrm{lin}} is the no-bias least-squares fit, the estimator of the closed-form translator; Rnl2R^{2}_{\mathrm{nl}} a two-layer MLP on the same data; ρ=Rlin2/Rnl2\rho=R^{2}_{\mathrm{lin}}/R^{2}_{\mathrm{nl}} the fraction of recoverable signal the linear map already captures. The S→LS\to L direction agrees to within 0.0180.018 on keys and 0.0370.037 on values.
   R2R^{2} , layer average
PairDir.sideno biasaffine
Qwen3 8B/4BL→SL\to SK0.8150.8150.8150.815
V0.6930.6930.6930.693
S→LS\to LK0.8080.8080.8080.808
V0.6760.6760.6760.676
Qwen3 1.7B/0.6BL→SL\to SK0.7660.7660.7660.766
Appendix
Table 2: Ablation: the impact of adding an intercept to the translators. We compare the per-layer fit of linear translators with and without bias, at the pre-normalisation capture point. The two R2R^{2} columns are layer averages on held-out tokens. On both Qwen3 pairs the bias is worth at most 0.0010.001 of R2R^{2} at any single layer, an order of magnitude below what the MLP finds (Table 1 ). The from-scratch pair has more room for an offset: its S→LS\to L keys gain 0.0120.012 on average.
Modeleffective rank (first →\to mid →\to last)top-1 variance fraction (first →\to last)
Qwen3-1.7B, trained59→15→17759\to 15\to 1770.31→0.100.31\to 0.10
Qwen3-1.7B, random init365→583→558365\to 583\to 5580.04→0.010.04\to 0.01
Qwen3-8B, trained24→48→13524\to 48\to 1350.52→0.100.52\to 0.10
Qwen3-8B, random init401→821→830401\to 821\to 8300.04→0.000.04\to 0.00
from-scratch 1.7B, trained13→113→15713\to 113\to 1570.51→0.100.51\to 0.10
Appendix
Table 3: Effective rank of the per-layer cache covariance (of d=1024d=1024 ), at the first layer, mid-stack and last layer, with the variance fraction carried by the top direction. Trained caches are low-rank and anisotropic; randomly initialised models of the same architecture are neither, which is the null for Figure 2 .
Figure 8: Architecture and training of KV-Lingo translators. Both rows start from the same prefix x1:nx_{1:n} and end at a distribution over the same continuation y1:my_{1:m} ; Stage 2 of § 3.2 asks the two to agree. Top: the source prefills x1:n−1x_{1:n-1} , the translator TT maps that cache into the target’s format, and one forward pass of the target on the held-back token xnx_{n} completes the cache and emits x^n+1\hat{x}_{n+1} . Bottom: the target prefills the whole prefix itself. The cache is the only thing that differs between the two rows. Source and target are frozen; the KL is backpropagated through the target into TT , which carries all the trainable parameters.
BenchmarkAbilityPrefixnsamplesn_{\textbf{samples}}Reasoning (budget)Metric
Nemotron chatLM1010 – 1616 kheld-outoff/on ( −- )cont. PPL, KL
MMLU-Proknowledge MC≤3\leq 3 k1,4001{,}400off/on ( 11 k/ 88 k)accuracy (macro)
ARC-Challengescience MC≤3\leq 3 k1,1721{,}172off/on ( 11 k/ 88 k)accuracy
GSM8Karithmetic≤1\leq 1 k256256on ( 22 k)EM flexible
IFEvalconstraints≤2\leq 2 k541541off ( 11 k)prompt/instr. acc.
CoQAgrounded QA≤1\leq 1 k256256off ( 128128 )F1, gold-in-ans.
Appendix
Table 4: The evaluation suite. Reasoning is the mode the reported numbers are scored in, with the generation budget (max new tokens) per mode in parentheses; teacher-forced rows need none. Prefix is the translated prefix length in tokens: a range or ladder where the benchmark fixes it, a cap where it does not. The language-modelling row is scored on held-out conversations from the training mixture.
  head-mixinghead-wise
PairDirectionν=1\nu{=}1 , depthν=1\nu{=}1 , selectedν=2\nu{=}2ν=4\nu{=}4ν=1\nu{=}1ν=2\nu{=}2ν=4\nu{=}4ν=8\nu{=}8
Qwen3-0.6B/1.7BL→SL\to S3×10−53{\times}10^{-5}  10−410^{-4}   
Qwen3-0.6B/1.7BS→LS\to L10−410^{-4}  10−310^{-3}   
Qwen3-4B/8BL→SL\to S3×10−53{\times}10^{-5}  10−410^{-4}   
Qwen3-4B/8BS→LS\to L3×10−53{\times}10^{-5}  3×10−43{\times}10^{-4}   
Qwen3-0.6B/8BL→SL\to S10−410^{-4}3×10−43{\times}10^{-4}10−410^{-4}3×10−53{\times}10^{-5}3×10−33{\times}10^{-3}10−310^{-3}3×10−43{\times}10^{-4}3×10−43{\times}10^{-4}
Appendix
Table 5: Selected stage-2 learning rates of the linear translators, by pair, direction and layer assignment. On the shape-preserving pairs every target layer reads its counterpart, so the two ν=1\nu=1 columns share one rate; on the cross-layer pairs ν\nu is the number of source layers per target layer. Depth : relative-depth match. Selected : the R2R^{2} assignment on Qwen3, which at ν=1\nu=1 coincides with greedy match up to the ridge penalty, and greedy match on Gemma. The head-mixing columns at ν≥2\nu\geq 2 use the R2R^{2} assignment and the head-wise columns greedy match, as in Appendix H . The head-mixing rates at ν≥2\nu\geq 2 were selected at seed 5656 , all others at seed 4242 . The MLP rates are given in the text, and those of the MoE pair in Appendix N .
Figure 9: Realised prefix length of the long-context instruct training mixture. Histogram over 240240 probe samples, split by reasoning status, on log-spaced bins. Both statuses span the short unpacked prompts and the packed window alike, so a downstream reasoning-on/off comparison is not also a comparison of prefix lengths.
 shape-preserving Qwen3cross-layer Qwen3Gemma-3Gemma-4
Benchmarkseed sdeval seseed sdeval seseed sdeval seseed sdeval se
MMLU-Pro0.0100.0100.0120.0120.0160.0160.0120.0120.0160.0160.0130.0130.0110.0110.0130.013
ARC-C0.0060.0060.0090.0090.0140.0140.0120.0120.0230.0230.0120.0120.0050.0050.0100.010
GSM8K0.0120.0120.0260.0260.0210.0210.0260.0260.0120.0120.0220.022——
IFEval0.0080.0080.0200.0200.0120.0120.0210.0210.0100.0100.0210.021——
CoQA0.0130.0130.0250.0250.0360.0360.0240.0240.0550.0550.0270.027——
Appendix
Table 6: Seed spread against evaluation noise. For each family trained at three seeds, seed sd is the standard deviation of a cell’s three per-seed scores and eval se the standard error over evaluation items of one seed’s score, each the median over the family’s cells (translators, directions and reasoning modes) on that benchmark. RULER pools its per-rung cells, LongMemEval is the mean over its four rungs, and judge ratings are rescaled to [0,1][0,1] . —: not evaluated for that family.
PairDirectionAssignmentseed 42423-seed meanseed 4343seed 4444
Qwen3-0.6B/8BL→SL\to Sdepth0.371±0.0070.371_{\pm 0.007}0.351±0.0610.351_{\pm 0.061}0.2820.2820.4000.400
Qwen3-0.6B/8BL→SL\to Sgreedy0.360±0.0070.360_{\pm 0.007}0.408±0.0420.408_{\pm 0.042}0.4390.4390.4240.424
Qwen3-0.6B/8BS→LS\to Ldepth0.408±0.0070.408_{\pm 0.007}0.410±0.0030.410_{\pm 0.003}0.4110.4110.4130.413
Qwen3-0.6B/8BS→LS\to Lgreedy0.435±0.0070.435_{\pm 0.007}0.441±0.0050.441_{\pm 0.005}0.4420.4420.4460.446
Qwen3-1.7B/4BL→SL\to Sdepth0.458±0.0070.458_{\pm 0.007}0.461±0.0130.461_{\pm 0.013}0.4760.4760.4510.451
Qwen3-1.7B/4BL→SL\to Sgreedy0.478±0.0070.478_{\pm 0.007}0.492±0.0130.492_{\pm 0.013}0.5040.5040.4930.493
Appendix
Table 7: Seed 4242 against the three-seed mean, for the families reported at seed 4242 . One benchmark-set mean per translator, reasoning-off: the four-benchmark mean of the cross-layer figures (MMLU-Pro, ARC-C, RULER, LongMemEval), the six-benchmark mean of Figure 25 , and the mean over the eleven rows of Table 14 , with judge ratings rescaled to [0,1][0,1] . Seed 42 is the score the paper reports, ±\pm its standard error over evaluation items; 3-seed mean is ±\pm the seed standard deviation.
Figure 10: Shape-preserving Qwen3 pairs, reasoning-off, both directions. Trained translators are the mean over three seeds; error bars: standard error over evaluation items (Appendix E.1 ).
Figure 11: Shape-preserving Qwen3 pairs, reasoning-on, both directions. Trained translators are the mean over three seeds; error bars: standard error over evaluation items (Appendix E.1 ).
  nativetranslated
    closedKV-LingoKV-LingoKV-LingoMLP
 BenchmarksmalllargeformH-M, KLH-M, mixed losshead-wisehead-wise
Qwen3-0.6B/1.7BMMLU-Pro0.234±0.0110.234_{\pm 0.011}0.379±0.0130.379_{\pm 0.013}0.210±0.0110.210_{\pm 0.011}0.251‾±0.010\underline{0.251}_{\pm 0.010}0.279‾±0.010\underline{0.279}_{\pm 0.010}0.262‾±0.010\underline{0.262}_{\pm 0.010}0.283‾±0.010\underline{\mathbf{0.283}}_{\pm 0.010}
ARC-C0.568±0.0140.568_{\pm 0.014}0.770±0.0120.770_{\pm 0.012}0.640‾±0.014\underline{0.640}_{\pm 0.014}0.660‾±0.012\underline{0.660}_{\pm 0.012}0.694‾±0.012\underline{0.694}_{\pm 0.012}0.659‾±0.012\underline{0.659}_{\pm 0.012}0.701‾±0.012\underline{\mathbf{0.701}}_{\pm 0.012}
IFEval0.580±0.0210.580_{\pm 0.021}0.662±0.0200.662_{\pm 0.020}0.416±0.0210.416_{\pm 0.021}0.530±0.0190.530_{\pm 0.019}0.476±0.0190.476_{\pm 0.019}0.515±0.0190.515_{\pm 0.019}0.534±0.020\mathbf{0.534}_{\pm 0.020}
CoQA0.475±0.0260.475_{\pm 0.026}0.672±0.0260.672_{\pm 0.026}0.291±0.0220.291_{\pm 0.022}0.476‾±0.024\underline{0.476}_{\pm 0.024}0.516‾±0.023\underline{\mathbf{0.516}}_{\pm 0.023}0.446±0.0230.446_{\pm 0.023}0.446±0.0240.446_{\pm 0.024}
Appendix
Table 8: Large to small evaluation suite, Qwen3 pairs with the same shape, reasoning-off. The large model prefills, the small one decodes from the translated cache. Trained translators are the mean over three seeds; the closed-form fit is a single solve. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model. Bold : the best translator on that row. The head-wise MLP adapter of Chen et al. (2026) is shown at the width that matches our map in parameters and FLOPs (hidden dimension is 4×4\times the input dimension). Pre-normalisation capture throughout. Metrics follow Table 4 : accuracy for MMLU-Pro (macro over 1414 categories), ARC-C and LongBench-v2 (untruncated subset), prompt-level strict accuracy for IFEval, a 11 – 1010 judge score for MT-Bench-101, token-F1 for CoQA and for XQuAD (pooled at equal nn over its eleven languages), graded fact coverage on the answerable split for RepLiQA, recall averaged over 44 k– 3232 k for RULER, and judged accuracy averaged over the four dilution rungs for LongMemEval. H-M means head-mixing. The mixed loss is an ablation, presented in Appendix U . —: not run.
  nativetranslated
    closedKV-LingoKV-LingoKV-LingoMLP
 BenchmarksmalllargeformH-M, KLH-M, mixed losshead-wisehead-wise
Qwen3-0.6B/1.7BMMLU-Pro0.234±0.0110.234_{\pm 0.011}0.379±0.0130.379_{\pm 0.013}0.191±0.0100.191_{\pm 0.010}0.246‾±0.009\underline{0.246}_{\pm 0.009}0.234±0.0090.234_{\pm 0.009}0.202±0.0080.202_{\pm 0.008}0.265‾±0.010\underline{\mathbf{0.265}}_{\pm 0.010}
ARC-C0.568±0.0140.568_{\pm 0.014}0.770±0.0120.770_{\pm 0.012}0.497±0.0150.497_{\pm 0.015}0.509±0.0120.509_{\pm 0.012}0.488±0.0120.488_{\pm 0.012}0.471±0.0120.471_{\pm 0.012}0.560±0.013\mathbf{0.560}_{\pm 0.013}
IFEval0.575±0.0210.575_{\pm 0.021}0.664±0.0200.664_{\pm 0.020}0.346±0.0200.346_{\pm 0.020}0.539±0.0190.539_{\pm 0.019}0.502±0.0190.502_{\pm 0.019}0.529±0.0180.529_{\pm 0.018}0.553±0.020\mathbf{0.553}_{\pm 0.020}
CoQA0.475±0.0260.475_{\pm 0.026}0.672±0.0260.672_{\pm 0.026}0.144±0.0180.144_{\pm 0.018}0.546‾±0.024\underline{0.546}_{\pm 0.024}0.568‾±0.024\underline{\mathbf{0.568}}_{\pm 0.024}0.475±0.0220.475_{\pm 0.022}0.555‾±0.025\underline{0.555}_{\pm 0.025}
Appendix
Table 9: Small to large evaluation suite, Qwen3 pairs with the same shape, reasoning-off. The small model prefills, the large one decodes from the translated cache. Trained translators are the mean over three seeds; the closed-form fit is a single solve. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model by more than one standard error. Bold : the best translator on that row. The head-wise MLP adapter of Chen et al. (2026) is shown at the width that matches our map in parameters and FLOPs (hidden dimension is 4×4\times the input dimension). Pre-normalisation capture throughout. Metrics follow Table 4 : accuracy for MMLU-Pro (macro over 1414 categories), ARC-C and LongBench-v2 (untruncated subset), prompt-level strict accuracy for IFEval, a 11 – 1010 judge score for MT-Bench-101, token-F1 for CoQA and for XQuAD (pooled at equal nn over its eleven languages), graded fact coverage on the answerable split for RepLiQA, recall averaged over 44 k– 3232 k for RULER, and judged accuracy averaged over the four dilution rungs for LongMemEval. H-M means head-mixing. The mixed loss is an ablation, presented in Appendix U . On the 0.6B/1.7B pair the head-wise MLP’s MT-Bench-101 cell is the mean of two seeds: the third returns a score outside the judge’s 11 – 1010 scale, a judge-parser bug that read “1996” as a rating. —: not run.
Figure 12: Qwen3-4B/8B, reasoning-off, average scores of MMLU-Pro, ARC-C, IFEval, CoQA, XQuAD, MT-Bench-101, RULER, LongMemEval, LongBench-v2. The head-mixing MLP, computed only in the small to large setting, performs on par with the head-wise MLP. The head-mixing MLP is a single seed (seed 42), the other trained translators the mean over three seeds; error bars: standard error over evaluation items (Appendix E.1 ).
Figure 13: Shape-preserving small Qwen3 pair, reasoning-on, both directions. Average score over MMLU-Pro, ARC-C, GSM8K, RepLiQA, MT-Bench-101 and LongMemEval.
Figure 14: Per-head structure of the closed-form map, S→LS\to L . Share of each source head’s map norm sent to each target head, for key-to-key (top) and value-to-value (bottom) blocks, averaged over layers; rows sum to one, and white is chance ( 1/81/8 ). diag. : mean of the diagonal. Released Qwen3-0.6B/1.7B and 4B/8B against a 0.6B/1.7B pair we pretrained from scratch (Appendix V ), from independent initialisations or with the small model sliced out of the large one.
Figure 15: Every translator on Qwen3-8B/0.6B , both directions, reasoning-off. Each marker is the mean score over MMLU-Pro, ARC-C, RULER, LongMemEval, CoQA, IFEval, XQuAD, RepLiQA and MT-Bench-101, weighted equally, with RULER’s four context budgets averaged first and judge scores rescaled to [0,1][0,1] ; LongBench-v2 is left out, as it was run in S→LS\to L only. Black and grey lines are Qwen3-8B and Qwen3-0.6B on their own cache. Triangles are closed-form fits, stars KV-Lingo after self-distillation, circles the two MLP translators of Appendix E ; filled markers mix heads, open ones are head-wise. The head-mixing linear translators use the R2R^{2} assignment, their closed form at ν=1\nu=1 ; the head-wise linear translators and the head-mixing MLP use greedy match, the head-wise MLP relative depth. Head-mixing KV-Lingo at ν=4\nu=4 matches the two MLPs in parameters; head-wise KV-Lingo would need ν=32\nu=32 , so ν=8\nu=8 stands in at a quarter of their count. Head-mixing KV-Lingo at ν=1\nu=1 is the mean over three seeds, at ν=4\nu=4 seed 56, and every other trained translator seed 42; no error bars.
Figure 16: Every translator on Qwen3-4B/1.7B , both directions, reasoning-off. Everything as in Figure 15 , for the other cross-layer pair.
Figure 17: The relative-depth assignment Eℓ\mathcal{E}_{\ell} of § 3.1 , drawn for the unequal-depth Qwen3-0.6B/8B pair in both directions at ν=4\nu=4 source layers per target layer. A filled cell (ℓ′,ℓ)(\ell^{\prime},\ell) means that target layer ℓ\ell reads source layer ℓ′\ell^{\prime} , through its own matrices TKℓ′→ℓ, TVℓ′→ℓT_{K}^{\ell^{\prime}\to\ell},\,T_{V}^{\ell^{\prime}\to\ell} ; each row therefore carries exactly ν\nu filled cells, and each panel 2νNtgt2\nu N_{\mathrm{tgt}} matrices in total. The dashed line is the exact relative-depth counterpart, c(ℓ)=ℓ(Nsrc−1)/(Ntgt−1)c(\ell)=\ell(N_{\mathrm{src}}-1)/(N_{\mathrm{tgt}}-1) with layers indexed from 00 as in the plot, on which the window is centred everywhere except at the two ends of the stack, where it shifts inward rather than truncating so that every target layer still reads ν\nu distinct source layers. Going up in depth ( S→LS\to L , right) the band is steeper than the diagonal and consecutive target layers reuse source layers; going down ( L→SL\to S , left) it is shallower and consecutive windows slide faster than one layer at a time.
 nativeclosed formself-distilled
BenchmarksmalllargedepthR2R^{2}depthR2R^{2}
L→SL\to S , Qwen3-8B →\to Qwen3-0.6B
MMLU-Pro0.234±0.0110.234_{\pm 0.011}0.615±0.0130.615_{\pm 0.013}0.011±0.0030.011_{\pm 0.003}0.104±0.0080.104_{\pm 0.008}0.205±0.0110.205_{\pm 0.011}0.208±0.011\mathbf{0.208}_{\pm 0.011}
ARC-C0.568±0.0140.568_{\pm 0.014}0.937±0.0070.937_{\pm 0.007}0.000±0.0000.000_{\pm 0.000}0.028±0.0050.028_{\pm 0.005}0.650‾±0.014\underline{0.650}_{\pm 0.014}0.659‾±0.014\underline{\mathbf{0.659}}_{\pm 0.014}
RULER0.879±0.0060.879_{\pm 0.006}0.998±0.0010.998_{\pm 0.001}0.000±0.0000.000_{\pm 0.000}0.001±0.0010.001_{\pm 0.001}0.323±0.0100.323_{\pm 0.010}0.327±0.010\mathbf{0.327}_{\pm 0.010}
LongBench-v20.276±0.0420.276_{\pm 0.042}0.431±0.0460.431_{\pm 0.046}————
Appendix
Table 10: Cross-layer cache translation on two Qwen3 pairs , 2828 against 3636 layers, both directions. Each target layer reads a single source layer, chosen either by relative depth or by the R2R^{2} of its best single-layer fit; each assignment is reported as its closed-form fit and after self-distillation, the latter at seed 42. ±\pm : standard error over evaluation items (Appendix E.1 ). The two anchor columns are the same measurement for both directions of a pair, so they are shown even where that direction’s translator cells are not. Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Five further S→LS\to L cells clear the small model on the value, by +0.009+0.009 to +0.026+0.026 , but by less than their own standard error, and are left unmarked. Bold : the best of the four translators on that row. A dash marks a LongBench-v2 cell left out because the decoding model is at the chance floor. Reasoning-off throughout; Table 12 gives reasoning-on LongMemEval and Table 11 the full RULER ladder. MMLU-Pro is the macro-average over its 1414 categories and ARC-C the micro-average, both lenient generate-and-parse; RULER is recall averaged over 44 k– 1616 k; LongBench-v2 is accuracy on its untruncated subset; LongMemEval is judge accuracy pooled over its oracle, 88 k, 1616 k and 3232 k rungs.
 reasoning-offreasoning-on
ContextsmalllargedepthR2R^{2}smalllargedepthR2R^{2}
L→SL\to S , Qwen3-8B →\to Qwen3-0.6B
44 k0.964±0.0080.964_{\pm 0.008}0.998±0.0030.998_{\pm 0.003}0.361±0.0170.361_{\pm 0.017}0.395±0.017\mathbf{0.395}_{\pm 0.017}0.819±0.0130.819_{\pm 0.013}0.992±0.0040.992_{\pm 0.004}0.372±0.018\mathbf{0.372}_{\pm 0.018}0.335±0.0170.335_{\pm 0.017}
88 k0.855±0.0120.855_{\pm 0.012}0.998±0.0030.998_{\pm 0.003}0.352±0.017\mathbf{0.352}_{\pm 0.017}0.333±0.0160.333_{\pm 0.016}0.737±0.0080.737_{\pm 0.008}0.966±0.0080.966_{\pm 0.008}0.348±0.017\mathbf{0.348}_{\pm 0.017}0.307±0.0170.307_{\pm 0.017}
1616 k0.819±0.0120.819_{\pm 0.012}0.999±0.0010.999_{\pm 0.001}0.257±0.017\mathbf{0.257}_{\pm 0.017}0.254±0.0160.254_{\pm 0.016}0.708±0.0110.708_{\pm 0.011}0.983±0.0060.983_{\pm 0.006}0.246±0.0170.246_{\pm 0.017}0.274±0.017\mathbf{0.274}_{\pm 0.017}
3232 k0.717±0.0130.717_{\pm 0.013}0.996±0.0030.996_{\pm 0.003}0.150±0.0150.150_{\pm 0.015}0.161±0.013\mathbf{0.161}_{\pm 0.013}0.637±0.0140.637_{\pm 0.014}0.986±0.0060.986_{\pm 0.006}0.155±0.015\mathbf{0.155}_{\pm 0.015}0.149±0.0140.149_{\pm 0.014}
Appendix
Table 11: RULER recall by context length , four cross-layer Qwen3 cells, both reasoning modes. small and large are the two models of the pair decoding their own caches; depth and R2R^{2} are the two self-distilled translators, at seed 42. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error; no cell of this table qualifies. Bold : the better of the two assignments on that row, within each reasoning mode. The translators are fitted at up to 1616 k tokens, so 3232 k is extrapolation. The closed-form fits score 0.0000.000 – 0.0130.013 everywhere in this table and are omitted.
 nativeclosed formself-distilled
CellsmalllargedepthR2R^{2}depthR2R^{2}
Qwen3-8B →\to Qwen3-0.6B0.383±0.0190.383_{\pm 0.019}0.763±0.0170.763_{\pm 0.017}0.000±0.0000.000_{\pm 0.000}0.000±0.0000.000_{\pm 0.000}0.373±0.019\mathbf{0.373}_{\pm 0.019}0.343±0.0180.343_{\pm 0.018}
Qwen3-4B →\to Qwen3-1.7B0.531±0.0190.531_{\pm 0.019}0.723±0.0170.723_{\pm 0.017}0.000±0.0000.000_{\pm 0.000}0.000±0.0000.000_{\pm 0.000}0.453±0.019\mathbf{0.453}_{\pm 0.019}0.453±0.0180.453_{\pm 0.018}
Qwen3-0.6B →\to Qwen3-8B0.383±0.0190.383_{\pm 0.019}0.763±0.0170.763_{\pm 0.017}0.006±0.0020.006_{\pm 0.002}0.004±0.0020.004_{\pm 0.002}0.506‾±0.020\underline{0.506}_{\pm 0.020}0.531‾±0.019\underline{\mathbf{0.531}}_{\pm 0.019}
Qwen3-1.7B →\to Qwen3-4B0.531±0.0190.531_{\pm 0.019}0.723±0.0170.723_{\pm 0.017}0.002±0.0010.002_{\pm 0.001}0.003±0.0010.003_{\pm 0.001}0.566‾±0.019\underline{0.566}_{\pm 0.019}0.585‾±0.019\underline{\mathbf{0.585}}_{\pm 0.019}
Appendix
Table 12: LongMemEval with reasoning on , four cross-layer Qwen3 cells. Judge accuracy pooled over the oracle, 88 k, 1616 k and 3232 k rungs, on the 389389 questions common to all four ( n=1556n=1556 question-rung pairs). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best of the four translators on that row. Self-distilled columns are seed 42. ±\pm : standard error over evaluation items (Appendix E.1 ).
Figure 18: Layer assignment against ν\nu , Qwen3-8B/0.6B , both directions, head-mixing KV-Lingo, reasoning-off. Markers and native lines as in Figure 20 ; each marker is the mean score over MMLU-Pro, ARC-C, RULER and LongMemEval, weighted equally, with RULER’s four context budgets averaged first. Within each group of ν\nu , the ticks are relative-depth match, the R2R^{2} assignment (the ν\nu source layers with the highest single-layer R2R^{2} , chosen separately for keys and values) and greedy match. At ν=1\nu=1 greedy match reduces to the R2R^{2} assignment up to the ridge penalty, so it has no separate translator there. Every point is a single seed, seed 42 at ν=1\nu=1 and seed 56 at ν≥2\nu\geq 2 ; error bars: standard error over evaluation items (Appendix E.1 ).
Figure 19: Layer assignment against ν\nu , Qwen3-4B/1.7B , both directions, head-mixing KV-Lingo, reasoning-off. Everything as in Figure 18 , for the other cross-layer pair.
Figure 20: Head-mixing against head-wise KV-Lingo over ν\nu , Qwen3-8B/0.6B, both directions, reasoning-off. Each marker is the mean score over the nine benchmarks of Figure 15 , weighted equally. Black and grey lines are Qwen3-8B and Qwen3-0.6B on their own cache. Filled stars are the head-mixing translator with the R2R^{2} assignment, open stars the head-wise one with greedy match. Every marker is a single seed: head-mixing at seed 42 for ν=1\nu=1 and seed 56 for ν≥2\nu\geq 2 , head-wise at seed 42.
Figure 21: Cache translation with Qwen3-0.6B/8B, reasoning-on. Average over MMLU-Pro, ARC-C, GSM8K, RepLiQA, MT-Bench-101 and LongMemEval. Every point is a single seed, seed 42 except the head-mixing translator at ν=4\nu=4 (seed 56); error bars: standard error over evaluation items (Appendix E.1 ).
Figure 22: KV-Lingo score against time to first token, Qwen3- 44 B/Qwen3- 1.71.7 B, R2R^{2} assignment, reasoning-off. RULER above and LongMemEval below, sharing an xx window: xx is the mean time to first token over the eval’s own prefix lengths, yy the eval’s own metric, and colour the context budget, light at the shortest rung to dark at 3232 k. Filled stars are the translated cache at ν=2,4,8\nu=2,4,8 source layers per target layer, growing with ν\nu , each a single seed (seed 56), charged the translation and one decoding step (Appendix Y ); open markers are the decoding model on its own cache, the only native baseline each panel carries. Error bars: standard error over evaluation items (Appendix E.1 ).
Figure 23: RULER recall against the number of source layers per target layer ν\nu , the four cross-layer Qwen3 cells, head-mixing KV-Lingo with the R2R^{2} assignment, reasoning-off. Colour is the context budget, light at 44 k to dark at 3232 k. The black and grey hairlines are the large and the small model on their own cache, one line per budget. Every point is a single seed, seed 42 at ν=1\nu=1 and seed 56 at ν∈{2,4,8}\nu\in\{2,4,8\} , and the shaded band carries the standard error over evaluation items (Appendix E.1 ).
 nativetranslated, R2R^{2} assignment
Benchmarksmalllargeν=1\nu=1ν=2\nu=2ν=4\nu=4ν=8\nu=8
L→SL\to S , Qwen3-8B →\to Qwen3-0.6B
MMLU-Pro0.234±0.0110.234_{\pm 0.011}0.616±0.0130.616_{\pm 0.013}0.208±0.0110.208_{\pm 0.011}0.367‾±0.012\underline{\mathbf{0.367}}_{\pm 0.012}0.274‾±0.012\underline{0.274}_{\pm 0.012}0.320‾±0.012\underline{0.320}_{\pm 0.012}
ARC-C0.564±0.0140.564_{\pm 0.014}0.939±0.0070.939_{\pm 0.007}0.659‾±0.014\underline{0.659}_{\pm 0.014}0.838‾±0.011\underline{\mathbf{0.838}}_{\pm 0.011}0.826‾±0.011\underline{0.826}_{\pm 0.011}0.832‾±0.011\underline{0.832}_{\pm 0.011}
RULER0.834±0.0060.834_{\pm 0.006}0.997±0.0010.997_{\pm 0.001}0.286±0.0080.286_{\pm 0.008}0.431±0.0090.431_{\pm 0.009}0.453±0.0090.453_{\pm 0.009}0.576±0.008\mathbf{0.576}_{\pm 0.008}
LongMemEval0.333±0.0190.333_{\pm 0.019}0.652±0.0190.652_{\pm 0.019}0.289±0.0190.289_{\pm 0.019}0.337‾±0.020\underline{0.337}_{\pm 0.020}0.368‾±0.020\underline{0.368}_{\pm 0.020}0.392‾±0.021\underline{\mathbf{0.392}}_{\pm 0.021}
Appendix
Table 13: Sweeping the number of source layers per target layer on the four cross-layer Qwen3 cells of Table 10 : ν∈{1,2,4,8}\nu\in\{1,2,4,8\} , the R2R^{2} assignment, self-distilled, reasoning-off. The ν=1\nu=1 column is the seed-42 run of Table 10 ; each wider map is a single seed, seed 56, the only seed trained. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best ν\nu on that row. MMLU-Pro is the macro-average over its 1414 categories and ARC-C the micro-average, both lenient generate-and-parse; RULER is recall averaged over 44 k– 3232 k; LongMemEval is judge accuracy pooled over its oracle, 88 k, 1616 k and 3232 k rungs.
Figure 24: Cache translation with KV-Lingo (head-mixing) with the pair Gemma-3-4B/12B-it, both directions, with ν∈{1,2,4}\nu\in\{1,2,4\} source layers per target layer. Averaged score over MMLU-Pro, ARC-C, LongBench-v2, GSM8K, CoQA, IFEval, XQuAD, RULER at 44 k, 88 k and 1616 k, and RepLiQA. MT-Bench-101 is left out of the average; Tables 14 and 15 report it. LongBench-v2 is kept in the average although Table 14 omits it, since its native anchors sit near chance at this pair’s 1616 k budget. The ν=1\nu=1 closed-form fits and self-distilled depth map are those of Table 14 ; every other self-distilled point is a single map from Table 15 . Every self-distilled point, the ν=1\nu=1 depth map included, is seed 42; error bars: standard error over evaluation items (Appendix E.1 ).
 nativeclosed formself-distilled
Benchmarksmalllargedepthgreedydepthgreedy
L→SL\to S , Gemma-3-4B/12B-it
MMLU-Pro0.400±0.0130.400_{\pm 0.013}0.584±0.0130.584_{\pm 0.013}0.315±0.0120.315_{\pm 0.012}0.198±0.0110.198_{\pm 0.011}0.429‾±0.013\underline{\mathbf{0.429}}_{\pm 0.013}0.391±0.0130.391_{\pm 0.013}
ARC-C0.791±0.0120.791_{\pm 0.012}0.933±0.0070.933_{\pm 0.007}0.665±0.0140.665_{\pm 0.014}0.682±0.0140.682_{\pm 0.014}0.830‾±0.011\underline{\mathbf{0.830}}_{\pm 0.011}0.773±0.0120.773_{\pm 0.012}
GSM8K0.871±0.0210.871_{\pm 0.021}0.883±0.0200.883_{\pm 0.020}0.738±0.0280.738_{\pm 0.028}0.680±0.0290.680_{\pm 0.029}0.855±0.0220.855_{\pm 0.022}0.871±0.021\mathbf{0.871}_{\pm 0.021}
CoQA0.607±0.0270.607_{\pm 0.027}0.657±0.0260.657_{\pm 0.026}0.644‾±0.026\underline{0.644}_{\pm 0.026}0.608‾±0.027\underline{0.608}_{\pm 0.027}0.541±0.0270.541_{\pm 0.027}0.706‾±0.025\underline{\mathbf{0.706}}_{\pm 0.025}
Appendix
Table 14: Cross-layer cache translation on the Gemma-3-4B/12B-it pair , 3434 against 4848 layers and a KV width of 10241024 against 20482048 , both directions. One rectangular map covers the stack; each target layer reads a single source layer chosen inside its own attention group, either by relative depth or greedy match, and each assignment is reported as its closed-form fit and after self-distillation, the latter seed 42. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best of the four translators on that row. MMLU-Pro and ARC-C are lenient generate-and-parse accuracy, GSM8K flexible exact match, CoQA and XQuAD token-F1, IFEval prompt-level strict accuracy, RepLiQA judged accuracy on the answerable questions, MT-Bench-101 a 11 – 1010 judge score, and RULER is recall at each of three prefix lengths. Gemma-3’s chat template carries no reasoning switch, so the suite has a single mode rather than the reasoning-on/off pair reported elsewhere. Three points to note. The 4B outscores the 12B on RepLiQA, so on that row the small model is the harder anchor. On the S→LS\to L MT-Bench-101 row both maps print 9.039.03 , but only the depth map clears the 4B’s 8.8838.883 by one standard error ( 9.033−0.148=8.8849.033-0.148=8.884 , against 9.031−0.153=8.8789.031-0.153=8.878 ). And the two blocks’ native anchors come from byte-identical generations that on IFEval alone score differently in the third decimal ( 0.7150.715 against 0.7170.717 for the 4B, 0.7620.762 against 0.7650.765 for the 12B), because IFEval’s language-detection check was unseeded; the block’s own pair is the one its rows are read against. LongBench-v2 is omitted: at the 1638416384 -token budget this pair was evaluated at, 94%94\% of its items are truncated and its L→SL\to S native anchor sits half a standard deviation above the four-way chance floor, so no delta against native is readable from it.
 nativedepthgreedy
Benchmarksmalllargeν=2\nu=2ν=4\nu=4ν=1\nu=1ν=2\nu=2ν=4\nu=4
L→SL\to S , Gemma-3-4B/12B-it
MMLU-Pro0.400±0.0130.400_{\pm 0.013}0.584±0.0130.584_{\pm 0.013}0.431‾±0.013\underline{\mathbf{0.431}}_{\pm 0.013}0.429‾±0.013\underline{0.429}_{\pm 0.013}0.416‾±0.013\underline{0.416}_{\pm 0.013}0.412‾±0.013\underline{0.412}_{\pm 0.013}0.429‾±0.013\underline{0.429}_{\pm 0.013}
ARC-C0.791±0.0120.791_{\pm 0.012}0.933±0.0070.933_{\pm 0.007}0.846‾±0.011\underline{\mathbf{0.846}}_{\pm 0.011}0.838‾±0.011\underline{0.838}_{\pm 0.011}0.832‾±0.011\underline{0.832}_{\pm 0.011}0.828‾±0.011\underline{0.828}_{\pm 0.011}0.841‾±0.011\underline{0.841}_{\pm 0.011}
GSM8K0.871±0.0210.871_{\pm 0.021}0.883±0.0200.883_{\pm 0.020}0.844±0.0230.844_{\pm 0.023}0.867±0.0210.867_{\pm 0.021}0.863±0.0220.863_{\pm 0.022}0.863±0.0220.863_{\pm 0.022}0.902‾±0.019\underline{\mathbf{0.902}}_{\pm 0.019}
CoQA0.607±0.0270.607_{\pm 0.027}0.657±0.0260.657_{\pm 0.026}0.563±0.0280.563_{\pm 0.028}0.556±0.0280.556_{\pm 0.028}0.661‾±0.026\underline{0.661}_{\pm 0.026}0.689‾±0.025\underline{\mathbf{0.689}}_{\pm 0.025}0.686‾±0.025\underline{0.686}_{\pm 0.025}
Appendix
Table 15: Wider maps on the Gemma-3-4B/12B-it pair : each target layer reads ν\nu source layers inside its own attention group, both directions, self-distilled. depth extends the relative-depth assignment of Table 14 to ν\nu neighbouring layers; greedy picks the ν\nu source layers by orthogonal matching pursuit on the cache reconstruction, and at ν=1\nu=1 is therefore a different assignment from the greedy match of Table 14 . Each cell is a single map (seed 42), trained at the learning rate that won its own cell’s sweep. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best of the five maps on that row. Native anchors, metrics and eval sets are those of Table 14 . No closed-form fits were evaluated at ν>1\nu>1 .
Figure 25: Cache translation with KV-Lingo on the pair Gemma-4-E2B/E4B-it, both directions, for the relative-depth (“depth”) and greedy within-group assignments, each as its closed-form fit (triangles) and after self-distillation (stars, seed 42); error bars: standard error over evaluation items (Appendix E.1 ). Averaged score over MMLU-Pro, ARC-C, LongBench-v2, XQuAD, RULER (mean over 44 k– 1616 k) and MT-Bench-101, all reasoning-off, with judge scores rescaled to [0,1][0,1] . Per-benchmark scores are in Table 16 , which reports MT-Bench-101 and LongBench-v2 reasoning-on instead.
 nativeclosed formself-distilled
Benchmarksmalllargedepthgreedydepthgreedy
L→SL\to S , Gemma-4-E2B/E4B-it
MMLU-Pro0.468±0.0130.468_{\pm 0.013}0.576±0.0130.576_{\pm 0.013}0.346±0.0120.346_{\pm 0.012}0.354±0.0130.354_{\pm 0.013}0.481‾±0.013\underline{\mathbf{0.481}}_{\pm 0.013}0.474‾±0.013\underline{0.474}_{\pm 0.013}
ARC-C0.897±0.0090.897_{\pm 0.009}0.939±0.0070.939_{\pm 0.007}0.872±0.0100.872_{\pm 0.010}0.879±0.0100.879_{\pm 0.010}0.912‾±0.008\underline{\mathbf{0.912}}_{\pm 0.008}0.905‾±0.009\underline{0.905}_{\pm 0.009}
XQuAD0.470±0.0090.470_{\pm 0.009}0.546±0.0090.546_{\pm 0.009}0.298±0.0100.298_{\pm 0.010}0.296±0.0100.296_{\pm 0.010}0.488‾±0.010\underline{0.488}_{\pm 0.010}0.544‾±0.010\underline{\mathbf{0.544}}_{\pm 0.010}
MT-Bench-1019.45±0.139.45_{\pm 0.13}9.35±0.159.35_{\pm 0.15}8.94±0.198.94_{\pm 0.19}8.74±0.218.74_{\pm 0.21}9.25±0.169.25_{\pm 0.16}9.36±0.12\mathbf{9.36}_{\pm 0.12}
Appendix
Table 16: Grouped cache translation on the Gemma-4-E2B/E4B-it pair. Both native anchors and two within-group source-layer assignments (relative-depth and greedy), each as its closed-form fit and after self-distillation. Trained cells are seed 42. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best of the four translators on that row. Reasoning-off, except MT-Bench-101 and LongBench-v2, which are reasoning-on (with reasoning off both native anchors fall below LongBench-v2’s 25%25\% chance floor). MMLU-Pro and ARC-C are lenient generate-and-parse accuracy; XQuAD is token-F1 over the eleven languages; MT-Bench-101 is a 11 – 1010 judge score; RULER is recall averaged over 44 k– 1616 k; LongBench-v2 is accuracy. Note that E2B outscores E4B on MT-Bench-101, so on that row the small model is the harder anchor.
Figure 26: Cache translation across a tokenizer boundary, on one span of text. The two models segment Summarise this unbelievably long prefix differently; the vertical bars stand at the character offsets where both of them end a token, which delimit the minimal groups. (a) Inside a group, the source tokens are spread evenly over the target ones, and the translated keys of a source token are rotated at the position of the target token it is assigned to: a position can carry two translated tokens, another none. (b) Stage 1 regresses the keys and values of a source token onto the average of those of the target tokens of its group, which is the assigned token itself in a group holding one token on each side. Boilerplate from the two chat templates is prefilled by the target. The source row is the segmentation Qwen3 actually produces; the target row stands in for a SentencePiece one.
  nativetranslated
 Benchmarksmalllargeself-distilled
L→SL\to SMMLU-Pro0.318±0.0120.318_{\pm 0.012}0.629±0.0130.629_{\pm 0.013}0.406‾±0.013\underline{0.406}_{\pm 0.013}
ARC-C0.725±0.0130.725_{\pm 0.013}0.937±0.0070.937_{\pm 0.007}0.833‾±0.011\underline{0.833}_{\pm 0.011}
RepLiQA0.754±0.0190.754_{\pm 0.019}0.795±0.0180.795_{\pm 0.018}0.740±0.0190.740_{\pm 0.019}
LongMemEval0.542±0.0220.542_{\pm 0.022}0.702±0.0210.702_{\pm 0.021}0.455±0.0220.455_{\pm 0.022}
GSM8K0.516±0.0310.516_{\pm 0.031}0.922±0.0170.922_{\pm 0.017}0.387±0.0300.387_{\pm 0.030}
Appendix
Table 17: Cache translation across a tokenizer boundary , Qwen3-8B ↔\leftrightarrow Mistral-7B-Instruct-v0.3. The small model is Mistral-7B-Instruct-v0.3 and the large one Qwen3-8B, each shown decoding from its own cache; both blocks are scored on the same evaluation sets, so the two anchor columns repeat. The translator combines the three assigned source layers with one rectangular map per source layer ( 201201 M parameters), trained at a single seed. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. MMLU-Pro and ARC-C are lenient generate-and-parse accuracy at a 20482048 -token budget, GSM8K flexible exact match, IFEval prompt-level strict accuracy, and XQuAD token-F1 pooled over its eleven languages; RepLiQA is judged correctness over the whole 512512 -item set, answerable and unanswerable together; LongMemEval is judged accuracy averaged over its oracle, 88 k and 1616 k rungs. The mean row weights the seven benchmarks equally.
Figure 27: Cache translation across a tokenizer boundary, Qwen3-8B ↔\leftrightarrow Mistral-7B-Instruct-v0.3. Mean over the seven benchmarks of Table 17 , both directions, a single map per direction. Error bars: standard error over evaluation items (Appendix E.1 ).
Figure 28: MLA latent translation, Qwen3- 44 B/ 88 B converted with CARE. Mean over the benchmarks of Table 18 other than IFEval, both directions. Triangles are the closed-form fit, stars the self-distilled KV-Lingo map, a single seed. Error bars: standard error over evaluation items (Appendix E.1 ).
  nativetranslated
 Benchmarksmalllargeclosed formKV-Lingo
 head-mixinghead-mixing
L→SL\to SMMLU-Pro0.306±0.0120.306_{\pm 0.012}0.351±0.0120.351_{\pm 0.012}0.291±0.012\mathbf{0.291}_{\pm 0.012}0.258±0.0110.258_{\pm 0.011}
ARC-C0.759±0.0130.759_{\pm 0.013}0.819±0.0110.819_{\pm 0.011}0.785‾±0.012\underline{\mathbf{0.785}}_{\pm 0.012}0.769‾±0.012\underline{0.769}_{\pm 0.012}
XQuAD0.505±0.0080.505_{\pm 0.008}0.508±0.0080.508_{\pm 0.008}0.381±0.0080.381_{\pm 0.008}0.440±0.008\mathbf{0.440}_{\pm 0.008}
CoQA0.639±0.0270.639_{\pm 0.027}0.567±0.0280.567_{\pm 0.028}0.519±0.0280.519_{\pm 0.028}0.577±0.028\mathbf{0.577}_{\pm 0.028}
Appendix
Table 18: Performance of KV-Lingo on an MLA pair (small: 4B, large: 8B), compared to the performance of each model equipped with its own cache. One map per cell (seed 42). ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the better of the two translators on that row. IFEval is prompt-level strict accuracy, and the mean row weights the six benchmarks equally. On the mean, KV-Lingo lands about 0.040.04 below both native models, whose own means differ by only 0.0040.004 , so in both directions the switch costs some quality. The closed-form fit beats the self-distilled translator on MMLU-Pro and ARC-C in L→SL\to S and on ARC-C in S→LS\to L .
  continuation KLaccuracy −- own native
  closed formself-distilledMMLU-ProARC-C
RecoveryDir.latentK/VlatentK/VlatentK/VlatentK/V
6060 kL→SL\to S2.6132.6130.1170.1170.1030.1030.0230.023+0.011±0.013+0.011_{\pm 0.013}+0.004±0.013+0.004_{\pm 0.013}+0.023±0.014+0.023_{\pm 0.014}+0.045±0.010+0.045_{\pm 0.010}
6060 kS→LS\to L0.9770.9770.0980.0980.1360.1360.0210.021−0.095±0.012-0.095_{\pm 0.012}−0.086±0.013-0.086_{\pm 0.013}−0.090±0.014-0.090_{\pm 0.014}−0.048±0.010-0.048_{\pm 0.010}
120120 kL→SL\to S3.0333.0330.1170.1170.1520.1520.0210.021−0.049±0.013-0.049_{\pm 0.013}+0.047±0.013+0.047_{\pm 0.013}+0.010±0.013+0.010_{\pm 0.013}+0.053±0.010+0.053_{\pm 0.010}
120120 kS→LS\to L1.6511.6510.1030.1030.1090.1090.0210.021−0.059±0.013-0.059_{\pm 0.013}−0.087±0.013-0.087_{\pm 0.013}−0.081±0.012-0.081_{\pm 0.012}−0.061±0.010-0.061_{\pm 0.010}
Appendix
Table 19: Translating latents against translating the keys and values they compress. One pair, Qwen3- 44 B/ 88 B, converted to MLA and, as the baseline, left as it is; both versions given the same recovery fine-tune at two budgets, and their translators fit by the same recipe on the same data, so architecture is the only difference. Continuation KL is the held-out KL to the target’s own next-token distribution, the quantity stage 2 minimises, reported for the closed-form fit and after self-distillation. Accuracy −- own native is the translated cache’s generate-and-parse accuracy minus the target’s own, so 00 is a lossless translation and positive means the translation helped. One seed per cell (seed 42). ±\pm : standard error over evaluation items, paired between the translated and the native run (Appendix E.1 ); the KLs carry none, as per-sequence KLs were not stored. Because the KL is measured against each target’s own cache and the target differs between the two directions, the columns are commensurable in meaning rather than against a fixed reference, but the closed-form pair, which involves no optimiser, separates the two versions the same way and by more.
 S→LS\to L (target 1.7B)L→SL\to S (target 0.6B)
 ARC-CMMLU-ProcomputeARC-CMMLU-Procompute
Target, own cache0.770±0.0120.770_{\pm 0.012}0.379±0.0130.379_{\pm 0.013}1.001.000.568±0.0140.568_{\pm 0.014}0.234±0.0110.234_{\pm 0.011}1.001.00
Source alone0.568±0.0140.568_{\pm 0.014}0.234±0.0110.234_{\pm 0.011}—0.770±0.0120.770_{\pm 0.012}0.379±0.0130.379_{\pm 0.013}—
DroidSpeak, verbatim ( j=0j=0 )0.000±0.0000.000_{\pm 0.000}0.000±0.0000.000_{\pm 0.000}0.000.000.000±0.0000.000_{\pm 0.000}0.000±0.0000.000_{\pm 0.000}0.000.00
DroidSpeak, profiler0.770±0.0120.770_{\pm 0.012}0.381±0.0130.381_{\pm 0.013}0.930.930.568±0.0140.568_{\pm 0.014}0.234±0.0110.234_{\pm 0.011}1.001.00
 ( j=26j=26 )( j=28j=28 )
Appendix
Table 20: DroidSpeak against KV-Lingo, Qwen3-0.6B/1.7B, reasoning-off. ARC-C ( 11721172 items) and MMLU-Pro ( 14001400 items) accuracy. Compute is the target-side cost before decoding starts, in analytic MACs per token as a fraction of the target’s own prefill. In L→SL\to S the profiler selects all 2828 layers, so DroidSpeak coincides with the target’s own cache by construction. KV-Lingo is seed 4242 . ±\pm : standard error over evaluation items (Appendix E.1 ); the L→SL\to S verbatim cell is the j=0j=0 point of Table 21 , on its subsample.
 S→LS\to LL→SL\to S
jjARC-CMMLU-ProARC-CMMLU-Pro
00.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}
44.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}
88.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}.000±.000.000_{\pm.000}
1212.018±.007.018_{\pm.007}.005±.003.005_{\pm.003}.000±.000.000_{\pm.000}.002±.002.002_{\pm.002}
1616.075±.013.075_{\pm.013}.021±.007.021_{\pm.007}.000±.000.000_{\pm.000}.002±.002.002_{\pm.002}
Appendix
Table 21: DroidSpeak accuracy against the recompute boundary jj . The target recomputes layers 00 to j−1j-1 and reuses the source’s cache above. Subsampled to 400400 ARC-C and 420420 MMLU-Pro items; the last row is the target’s own cache on the same items. ±\pm : standard error over evaluation items (Appendix E.1 ).
 reasoning-offreasoning-on
 own cachetranslatedown cachetranslated
Benchmark4BMoE4B →\to MoEMoE →\to 4B4BMoE4B →\to MoEMoE →\to 4B
MMLU-Pro0.5480.5480.6740.6740.5250.5250.3680.3680.6070.6070.7360.7360.6060.6060.4370.437
ARC-C0.9180.9180.9480.9480.9160.9160.8490.8490.9380.9380.9700.9700.959‾\underline{0.959}0.9050.905
GSM8K0.9140.9140.9300.9300.8980.8980.8630.8630.8200.8200.8590.8590.863‾\underline{0.863}0.836‾\underline{0.836}
IFEval0.7710.7710.8390.8390.7450.7450.6320.6320.7580.7580.8450.8450.795‾\underline{0.795}0.7060.706
Appendix
Table 22: Qwen3-4B and Qwen3-30B-A3B: the whole evaluation suite, both directions and both reasoning on and off. 4B and MoE decode from their own caches; 4B →\to MoE is the MoE decoding from the translated cache of the 4B, and MoE →\to 4B the 4B decoding from the translated cache of the MoE. Underline : the translated cache beats the 4B on its own cache. Metrics as in Table 8 , with GSM8K scored by flexible exact match, and RULER and LongMemEval given per context length rather than averaged. The eight benchmarks above the middle rule have prefixes of at most 33 k tokens. The translators are trained on prefixes of up to 1616 k tokens, so 3232 k is extrapolation.
Figure 29: Dense →\to MoE mean scores, by task group and reasoning mode. Qwen3-4B ( Dense , grey) and Qwen3-30B-A3B ( MoE , black) decode from their own caches; the stars are the MoE decoding from the KV-Lingo translation of the dense model’s cache. Short, judged averages the two judge-scored short-context benchmarks, RepLiQA and MT-Bench-101, with the 11 – 1010 judge ratings rescaled to 00 – 11 ; short, other averages the six others, MMLU-Pro, ARC-C, GSM8K, IFEval, CoQA and XQuAD; long averages RULER, LongMemEval and LongBench-v2, with RULER and LongMemEval first averaged over their context lengths. Per-benchmark scores are in Table 22 .
   held-out KL at step 
TranslatorParametersLearning rate00100010002500250050005000Time
Qwen3-4B →\to Qwen3-30B-A3B
linear151151 M10−610^{-6}0.2070.2070.07490.07490.06890.06890.06590.06593.83.8 h
  3×10−63{\times}10^{-6}0.2070.2070.08460.08460.07060.07060.0634\mathbf{0.0634}3.83.8 h
  10−510^{-5}0.2070.2070.11540.11540.08940.08940.07420.07423.83.8 h
linear ++ MLP1.561.56 B10−610^{-6}0.2070.2070.07680.07680.06900.06900.06540.06546.96.9 h
Appendix
Table 23: Linear against non-linear translators , Qwen3-4B and Qwen3-30B-A3B, both directions. Held-out KL of equation 3.3 during stage 2, one seed per run; step 00 is the closed-form fit, shared by every run of a direction. Time is the wall-clock of stage 2 on 88 B200 GPUs. The 4B →\to MoE MLP at 3×10−53{\times}10^{-5} was stopped at step 20002000 , at 0.1500.150 against 0.0730.073 for the linear map at 3×10−63{\times}10^{-6} . In the reverse direction, the MLP’s best learning rate is the larger of the two it was run at, so its optimum is not bracketed. Bold : the lowest final KL of each direction.
Prompt tokens646412812825625651251210241024
Qwen3-4B prefill (ms)67.967.994.994.9140.4140.4248.7248.7393.7393.7
Qwen3-30B-A3B prefill (ms)151.8151.8188.6188.6215.8215.8287.9287.9421.4421.4
translation, fp32 (ms)1.81.83.13.15.75.711.011.017.217.2
translation, bf16 (ms)1.71.72.72.74.74.77.37.313.113.1
Qwen3-4B decoding (tokens/s)70.170.170.070.069.669.668.668.667.967.9
Qwen3-30B-A3B decoding (tokens/s)71.571.571.371.370.770.769.669.668.968.9
Appendix
Table 24: Latency on an Apple M3 Ultra , using MLX in bf16 , batch size 11 . Decoding speed is averaged over 256256 tokens generated after the prompt. In a switch, the MoE’s first token comes after the fp32 translation of the 4B’s cache and one decoding step of the MoE; as a prefill accelerator, after the 4B’s prefill as well. Speedup divides the MoE’s prefill by that time.
 pre- kk -normpost- kk -norm
Source →\to targetR2R^{2}MMLU-ProARC-CR2R^{2}MMLU-ProARC-C
Qwen3-1.7B →\to 0.6B0.8050.8050.257\mathbf{0.257}0.671\mathbf{0.671}0.852\mathbf{0.852}0.2020.2020.5910.591
Qwen3-0.6B →\to 1.7B0.7970.7970.218\mathbf{0.218}0.544\mathbf{0.544}0.801\mathbf{0.801}0.0790.0790.2620.262
Qwen3-8B →\to 4B0.8490.8490.541\mathbf{0.541}0.930\mathbf{0.930}0.882\mathbf{0.882}0.4690.4690.8590.859
Qwen3-4B →\to 8B0.8470.8470.526\mathbf{0.526}0.9120.9120.867\mathbf{0.867}0.4970.4970.914\mathbf{0.914}
Appendix
Table 25: The MSE objective is not always aligned with downstream performance: the example of pre- versus post- kk -norm. Closed-form translators fitted before and after the target’s kk -norm, on both Qwen3 pairs in both directions. R2R^{2} is measured against the target’s own cache on held-out prefixes of Nemotron-SFT-Instruction-Following-Chat-v2. Accuracy is that of the target decoding from that cache. Better capture point in bold. Post-norm translators have a better R2R^{2} fit, but worse downstream performance: the MSE objective is not always aligned with downstream performance.
fitting normoff-the-shelf KLafter stage 2
Frobenius (uniform)0.1750.1750.01950.0195
sink ×2\times 20.1340.1340.01960.0196
sink ×4\times 40.1030.1030.01940.0194
sink ×8\times 80.0850.0850.01950.0195
sink ×16\times 160.0780.0780.01960.0196
sink ×32\times 320.0760.0760.01980.0198
Appendix
Table 26: Fitting norms on the Qwen3-1.7B-Base/0.6B-Base pair , L→SL\to S , short prefixes. Left: the closed-form translator on its own, next-token KL against native decoding. Right: the same maps after stage 2, run to convergence ( 1515 k steps, one seed, norms paired on a common validation batch). The two columns are measured on different continuation lengths and should be read down, not across.
Figure 30: Stage 2 from two of the eight closed-form initialisations. Held-out continuation KL against native decoding over 1515 k self-distillation steps; Qwen3-1.7B-Base →\to Qwen3-0.6B-Base, L→SL\to S , one seed, both runs paired on a common validation batch. F0 is the Frobenius row of Table 26 and A-mass the attention-mass one; the five sink initialisations track F0 and the attention-Fisher one tracks A-mass, so only the two endpoints are drawn and the table carries the rest. (a) shows the full run, (b) zooms the tail, where the attention-weighted initialisation holds about 0.00080.0008 below the level the Frobenius run settles into.
Figure 31: Reasoning off against reasoning on, both shape-preserving Qwen3 pairs. Mean score over four equally weighted benchmark families (MMLU-Pro, ARC-C, MT-Bench-101 rescaled from its 11 – 1010 rating, and RULER over 44 / 88 / 1616 k), with reasoning off (open markers) and on (filled), so the segment joining a pair of markers is that condition’s reasoning boost. A translated cache is decoded by the small model in L→SL\to S and by the large one in S→LS\to L , and is therefore read against grey and black respectively. Translated caches are the mean over three seeds. Error bars, in both reasoning modes and on the native models too, are the standard error over evaluation items (Appendix E.1 ).
Figure 32: KV-Lingo multi-turn protocol. LL and SS alternate over a shared context. Each row is the cache line of the model that decodes at that turn, and each block one turn’s span of it: a block labelled LL or SS holds the cache that model wrote itself, a block labelled TT the translation of a span the other model wrote. At each switch, the outgoing model’s line is offloaded, and the incoming model’s line is extended by translating only the span written since it last decoded; here, at turn 44 , SS decodes from its resident line while LL ’s line waits off the accelerator. No span passes through more than one translator.
Figure 33: Multi-turn model switching on QuAC. Span F1 against turn index, same pair and protocol as Figure 5 , 256256 frozen conversations truncated to depth 88 . Full re-prefill is exact and costs a native prefill over the whole history at every switch; KV-Lingo translates only the span the target’s cache lacks. Translated conditions are the mean over three seeds; error bars, on every condition including re-prefill, are the standard error over evaluation items (Appendix E.1 ).
Figure 34: Multi-turn model switching on MultiChallenge. Judged pass rate against conversation length, same pair and protocol as Figure 5 . The history is pre-written and fixed, so every condition reaches the final prompt over a byte-identical sequence and the two models’ caches split the tokens roughly evenly, unlike CoQA and QuAC, where the passage sits in the turn- 11 block. Bins are disjoint item sets of the size annotated at the foot of each panel, so the depth axis varies across items rather than within one. Translated conditions are the mean over three seeds; error bars, on every condition including re-prefill, are the standard error over evaluation items (Appendix E.1 ).
evaluationcellspost- kk -norm worse inmean Δ\Delta
RULER, reasoning-off ( 44 – 3232 k)16161414−0.027±0.002-0.027_{\pm 0.002}
RULER, reasoning-on ( 44 – 3232 k)16161212−0.026±0.005-0.026_{\pm 0.005}
LongMemEval, reasoning-on, four rungs16161212−0.010±0.005-0.010_{\pm 0.005}
LongMemEval, reasoning-on, multi-turn at 1616 k4444−0.027±0.010-0.027_{\pm 0.010}
LongBench-v2, reasoning-off4433−0.010±0.007-0.010_{\pm 0.007}
RepLiQA, reasoning-on, judged accuracy and token-F18844−0.004±0.004-0.004_{\pm 0.004}
Appendix
Table 27: Post- kk -norm minus pre- kk -norm capture, per evaluation. Self-distilled translator, over the two Qwen3 pairs in both directions. A cell is one pair and direction and, for long-context evaluations, one context length, scored on that evaluation’s own metric. ±\pm : standard error of the mean difference over evaluation items, accounting for the items the cells of a row share (Appendix E.1 ).
 nativetranslated
Benchmarksmalllargepre- kk -normpost-RoPE
L→SL\to S , Qwen3-0.6B/1.7B
MMLU-Pro0.234±0.0110.234_{\pm 0.011}0.379±0.0130.379_{\pm 0.013}0.219±0.0110.219_{\pm 0.011}0.249‾±0.011\underline{\mathbf{0.249}}_{\pm 0.011}
ARC-C0.568±0.0140.568_{\pm 0.014}0.770±0.0120.770_{\pm 0.012}0.616‾±0.014\underline{0.616}_{\pm 0.014}0.660‾±0.014\underline{\mathbf{0.660}}_{\pm 0.014}
GSM8K0.641±0.0300.641_{\pm 0.030}0.746±0.0270.746_{\pm 0.027}0.695‾±0.029\underline{\mathbf{0.695}}_{\pm 0.029}0.641±0.0300.641_{\pm 0.030}
CoQA0.475±0.0260.475_{\pm 0.026}0.672±0.0260.672_{\pm 0.026}0.469±0.026\mathbf{0.469}_{\pm 0.026}0.386±0.0260.386_{\pm 0.026}
Appendix
Table 28: Pre- kk -norm against post-RoPE capture on the evaluation suite , translators fitted at a 1616 k training prefix cap, seed 4242 (post-RoPE’s only seed), reasoning-off except GSM8K. ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : beats the small model, in S→LS\to L by more than one standard error. Bold : the better capture point.
Figure 35: Where the two capture points diverge: extrapolation past the training prefix cap. RULER NIAH substring recall against evaluation context budget on the Qwen3- 0.60.6 B/ 1.71.7 B pair, both directions. One panel per training prefix cap; the dotted line marks that cap, so everything to its right is extrapolation. Both curves are the same self-distilled translator fitted at the two capture points, pre- kk -norm averaged over three seeds and post-RoPE at seed 4242 , its only seed; error bars are the standard error over evaluation items (Appendix E.1 ). Post-RoPE matches pre- kk -norm within the cap and falls away beyond it, to near-zero recall at 3232 k for the 22 k and 88 k caps, while pre- kk -norm degrades smoothly across the whole range.
 nativeclosed formKV-Lingomixed
Benchmarksmalllargewholecontextwholecontextwholecontext
L→SL\to S , Qwen3- 44 B/ 88 B
RULER, 44 k1.000±0.0001.000_{\pm 0.000}0.998±0.0030.998_{\pm 0.003}0.994±0.0040.994_{\pm 0.004}0.823±0.0110.823_{\pm 0.011}0.998±0.003\mathbf{0.998}_{\pm 0.003}0.998±0.001\mathbf{0.998}_{\pm 0.001}0.998±0.002\mathbf{0.998}_{\pm 0.002}0.996±0.0020.996_{\pm 0.002}
RULER, 88 k1.000±0.0001.000_{\pm 0.000}0.998±0.0030.998_{\pm 0.003}0.989±0.0050.989_{\pm 0.005}0.706±0.0160.706_{\pm 0.016}1.000±0.000\mathbf{1.000}_{\pm 0.000}1.000±0.000\mathbf{1.000}_{\pm 0.000}1.000±0.000\mathbf{1.000}_{\pm 0.000}0.999±0.0010.999_{\pm 0.001}
RULER, 1616 k0.998±0.0030.998_{\pm 0.003}0.999±0.0010.999_{\pm 0.001}0.960±0.0090.960_{\pm 0.009}0.350±0.0170.350_{\pm 0.017}0.997±0.001\mathbf{0.997}_{\pm 0.001}0.992±0.0040.992_{\pm 0.004}0.995±0.0020.995_{\pm 0.002}0.983±0.0040.983_{\pm 0.004}
RULER, 3232 k0.995±0.0030.995_{\pm 0.003}0.996±0.0030.996_{\pm 0.003}0.897±0.0100.897_{\pm 0.010}0.168±0.0110.168_{\pm 0.011}0.972±0.0070.972_{\pm 0.007}0.949±0.0090.949_{\pm 0.009}0.977±0.007\mathbf{0.977}_{\pm 0.007}0.966±0.0070.966_{\pm 0.007}
Appendix
Table 29: Translating the context alone instead of the whole prompt. The four evaluation sets that carry a document followed by a question about it, on the Qwen3- 44 B/ 88 B pair, reasoning-off, both directions. whole is the protocol used everywhere else in the paper: the source prefills the entire rendered prompt and the map translates all of it. context translates the document alone, and the target prefills the question, the assistant header and the empty reasoning block on top of the translated cache before answering. The two protocols are scored on the same items, so they pair per item. Translated cells are the mean over three seeds; ±\pm : standard error over evaluation items (Appendix E.1 ). mixed adds a cross-entropy term to the self-distillation loss (Appendix U ). Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best translator on that row, across both protocols. RULER is substring recall, LongBench-v2 accuracy on the untruncated subset ( n=116n=116 ), LongMemEval and RepLiQA judged accuracy.
 nativeclosed formKV-Lingo
Benchmarksmalllargetextprompttraceprompttrace
MMLU-Pro0.319±0.0120.319_{\pm 0.012}0.473±0.0130.473_{\pm 0.013}0.508±0.0130.508_{\pm 0.013}0.001±0.0010.001_{\pm 0.001}0.335‾±0.012\underline{0.335}_{\pm 0.012}0.260±0.0090.260_{\pm 0.009}0.473‾±0.013\underline{\mathbf{0.473}}_{\pm 0.013}
ARC-C0.690±0.0140.690_{\pm 0.014}0.873±0.0100.873_{\pm 0.010}0.875±0.0100.875_{\pm 0.010}0.009±0.0030.009_{\pm 0.003}0.666±0.0140.666_{\pm 0.014}0.726‾±0.011\underline{0.726}_{\pm 0.011}0.870‾±0.010\underline{\mathbf{0.870}}_{\pm 0.010}
GSM8K0.641±0.0300.641_{\pm 0.030}0.746±0.0270.746_{\pm 0.027}0.848±0.0230.848_{\pm 0.023}0.031±0.0110.031_{\pm 0.011}0.297±0.0290.297_{\pm 0.029}0.686‾±0.025\underline{0.686}_{\pm 0.025}0.862‾±0.020\underline{\mathbf{0.862}}_{\pm 0.020}
Appendix
Table 30: Handing the cache over after the reasoning trace rather than at the prompt. Qwen3- 1.71.7 B →\to Qwen3- 0.60.6 B, reasoning-on. Under prompt the source prefills the prompt, the map translates it, and the target does its own thinking and answering, the protocol used everywhere else in the paper. Under trace the source also generates its chain of thought on its own cache, the map translates the prompt and the trace together, and the target only writes the answer that follows the closing reasoning marker. text is the reference for the second protocol: the same target answering from its own prefill of prompt ++ trace, which is what an errorless map would hand it. Translated cells are the mean over three seeds; ±\pm : standard error over evaluation items (Appendix E.1 ). Underline : the translated cache beats the small model; this pair is read L→SL\to S throughout, so no standard-error margin applies. Bold : the best translator on that row, across both switch points. MMLU-Pro and ARC-C are letter accuracy with a parse failure counted incorrect, GSM8K flexible exact match. On MMLU-Pro, the trace variants are helped by a format effect worth about 66 accuracy points (asked only to commit an answer, the target emits a parseable one about twice as reliably as either model reasoning and concluding in one turn), so matching the source there is not a sign of better reasoning.
Figure 36: The same translator and recipe on the released Qwen3- 0.60.6 B/ 1.71.7 B-Base pair and on a 0.60.6 B/ 1.71.7 B pair pretrained from scratch. Both panels report absolute scores. Each pair’s two native models are short horizontal lines (black the larger, grey the smaller) with the band between them shaded, and the cost of translation is the distance from a marker to its own target’s line (grey for L→SL\to S , black for S→LS\to L ). Triangles are the closed-form translator (stage 1), circles the self-distilled one (stage 2). (a) Continuation perplexity on Nemotron-CC, which the translators are fitted on, and on WikiText; the axis is logarithmic, so equal vertical distances are equal log-perplexity gaps in both pairs. (b) Accuracy averaged over the six zero-shot tasks; the released pair’s closed-form translator was not run on this suite. Every self-distilled translator is a single run (seed 4242 ). Error bars in (b) are the standard error over evaluation items (Appendix E.1 ), binomial from the aggregate accuracies since only aggregate scores were stored; (a) has none, as per-document perplexities were not stored.
  nativetranslated
 BenchmarksmalllargeKV-Lingohead-wise MLPhead-wise MLP
 1.0×1.0\times4.0×4.0\times
L→SL\to SBoolQ0.523±0.0110.523_{\pm 0.011}0.496±0.0110.496_{\pm 0.011}0.547‾±0.011\underline{\mathbf{0.547}}_{\pm 0.011}0.510±0.0110.510_{\pm 0.011}0.519±0.0110.519_{\pm 0.011}
HellaSwag0.457±0.0110.457_{\pm 0.011}0.484±0.0110.484_{\pm 0.011}0.448±0.011\mathbf{0.448}_{\pm 0.011}0.439±0.0110.439_{\pm 0.011}0.443±0.0110.443_{\pm 0.011}
PIQA0.691±0.0110.691_{\pm 0.011}0.708±0.0110.708_{\pm 0.011}0.685±0.011\mathbf{0.685}_{\pm 0.011}0.677±0.0110.677_{\pm 0.011}0.683±0.0110.683_{\pm 0.011}
ARC-E0.578±0.0110.578_{\pm 0.011}0.636±0.0110.636_{\pm 0.011}0.557±0.011\mathbf{0.557}_{\pm 0.011}0.509±0.0110.509_{\pm 0.011}0.529±0.0110.529_{\pm 0.011}
Appendix
Table 31: KV-Lingo against the head-wise MLP translator on a pair with unaligned heads. The from-scratch 0.60.6 B/ 1.71.7 B pair of Appendix V , whose two models start from independent initialisations. KV-Lingo against the head-wise gated MLP of Chen et al. (2026) at matched parameters and FLOPs ( 59.059.0 M against our 58.758.7 M) and at 4×4\times that budget ( 235.9235.9 M), all trained with the same two-stage recipe and 50005000 self-distillation steps. Underline : the translated cache beats the small model, and in S→LS\to L by more than one standard error. Bold : the best translator on that row. BoolQ and WinoGrande are accuracy, HellaSwag, PIQA and the two ARC splits length-normalised accuracy; zero-shot, 20002000 examples per task, one map per cell. ±\pm : standard error over evaluation items (Appendix E.1 ), binomial from the aggregate accuracy, since per-item outcomes were not stored for these runs. On BoolQ the large model scores below the small one.
   KV-Lingo (ours)head-wise MLP
 Corpusnative PPLclosed formCEself-distilled1.0×1.0\times4.0×4.0\times
L→SL\to SNemotron12.5112.510.2360.2360.0480.0480.031\mathbf{0.031}0.0850.0850.0690.069
WikiText23.2023.200.4440.4440.1060.1060.067\mathbf{0.067}0.2290.2290.1550.155
S→LS\to LNemotron10.9910.990.3030.3030.0600.0600.043\mathbf{0.043}0.0740.0740.0650.065
WikiText19.5119.510.5580.5580.1380.1380.101\mathbf{0.101}0.1810.1810.1700.170
Appendix
Table 32: Continuation perplexity on the from-scratch pair , for the translators of Table 31 , plus KV-Lingo’s closed-form initialisation and KV-Lingo trained with cross-entropy (CE) instead of self-distillation. The last five columns are the gap ln⁡(translated PPL)−ln⁡(native PPL)\ln(\text{translated PPL})-\ln(\text{native PPL}) in nats, so zero is native and lower is better; native PPL is the target’s own continuation perplexity on that corpus. A 6464 -token continuation after a 256256 -token prefix, 128128 documents, one map per cell. No standard errors are shown: per-document perplexities were not stored. Bold : the lowest gap on that row.
   nativetranslated
  Benchmarksmalllargeclosed formKV-Lingo
Qwen3 0.60.6 B/ 1.71.7 BL→SL\to SARC-C0.441±0.0150.441_{\pm 0.015}0.550±0.0150.550_{\pm 0.015}0.429±0.014\mathbf{0.429}_{\pm 0.014}0.423±0.0140.423_{\pm 0.014}
SQuAD-v10.8260.8260.8810.8810.7850.7850.821\mathbf{0.821}
BoolQ0.720±0.0100.720_{\pm 0.010}0.799±0.0090.799_{\pm 0.009}0.783‾±0.009\underline{\mathbf{0.783}}_{\pm 0.009}0.772‾±0.009\underline{0.772}_{\pm 0.009}
WinoGrande0.597±0.0140.597_{\pm 0.014}0.658±0.0130.658_{\pm 0.013}0.597±0.0140.597_{\pm 0.014}0.611‾±0.014\underline{\mathbf{0.611}}_{\pm 0.014}
S→LS\to LARC-C0.441±0.0150.441_{\pm 0.015}0.550±0.0150.550_{\pm 0.015}0.472‾±0.015\underline{0.472}_{\pm 0.015}0.493‾±0.015\underline{\mathbf{0.493}}_{\pm 0.015}
Appendix
Table 33: Cache translation on two Qwen3-Base pairs. The released Qwen3- 0.60.6 B/ 1.71.7 B-Base and Qwen3- 44 B/ 88 B-Base pairs, both directions, with KV-Lingo, the identity layer assignment and the recipe of § 4.1 . KV-Lingo cells are the mean over three seeds; the closed-form fit is a single solve. ±\pm : standard error over evaluation items (Appendix E.1 ), binomial from the aggregate accuracy since only aggregate scores were stored for these runs (for KV-Lingo, taken at the three-seed mean); SQuAD-v1 token-F1 is not binary, so its cells carry none. Base models have no chat template, so both pairs are fitted on plain text and every task is run 55 -shot on at most 20002000 examples. Underline : the translated cache beats the small model, in S→LS\to L by more than one standard error. Bold : the better translator on that row. ARC-C is length-normalised accuracy, SQuAD-v1 token-F1, BoolQ and WinoGrande accuracy. The native columns repeat within a pair. On the 44 B/ 88 B L→SL\to S SQuAD-v1 row, the underlined 0.9060.906 is 0.90630.9063 against the small model’s 0.90570.9057 .
  NemotronWikiTextCodeBooksarXiv
Qwen3 0.60.6 B/ 1.71.7 BL→SL\to S+0.03+0.03+3.23+3.23+4.51+4.51+2.18+2.18+2.74+2.74
S→LS\to L+2.58+2.58+6.50+6.50+5.92+5.92+5.50+5.50+5.07+5.07
Qwen3 44 B/ 88 BL→SL\to S+0.28+0.28+1.41+1.41+0.62+0.62+1.20+1.20+1.08+1.08
S→LS\to L+1.04+1.04+0.36+0.36+1.51+1.51+2.57+2.57+2.03+2.03
Appendix
Table 34: The Base-pair translators off their fitting distribution. Relative continuation-perplexity gap (translated−native)/native(\text{translated}-\text{native})/\text{native} , in percent, for the self-distilled translators of Table 33 on Nemotron-CC, which they are fitted on, and four corpora they never see. Mean over three seeds; a 256256 -token continuation after a 256256 -token prefix, 128128 documents per corpus; zero is native and lower is better. No standard errors are shown: per-document perplexities were not stored. The gap is relative because absolute perplexity varies six-fold across corpora ( 2.42.4 on code to 14.714.7 on books). The worst of the twenty cells costs 6.5%6.5\% and the median 2.1%2.1\% , so the translators do not rely on their fitting corpus. The closed-form fit was not run on this suite.
  validation KLΔln⁡PPL\Delta\ln\mathrm{PPL}BoolQ
PairDirectionsplitjointNemotronWikiTextsplitjoint
0.60.6 B/ 1.71.7 BL→SL\to S0.02020.02020.01750.0175+0.0037+0.0037+0.0076+0.00760.7400.7400.7290.729
 S→LS\to L0.02650.02650.02570.0257+0.0012+0.0012+0.0039+0.00390.7160.7160.7090.709
44 B/ 88 BL→SL\to S0.01430.01430.01200.0120+0.0013+0.0013+0.0045+0.00450.8180.8180.8070.807
 S→LS\to L0.01750.01750.01640.0164+0.0018+0.0018+0.0009+0.00090.8310.8310.8440.844
Appendix
Table 35: Split against joint maps on the Qwen3 Base pairs , after self-distillation, one seed per map. Validation KL is the forward KL from the target decoding from its own cache to the target decoding from the translated cache, in nats per token over 20482048 Nemotron continuation tokens. Δln⁡PPL\Delta\ln\mathrm{PPL} is the split map’s continuation log-perplexity minus the joint map’s, in nats, over 128128 documents per corpus with a 256256 -token prefix and a 256256 -token continuation. BoolQ is 00 -shot accuracy on 20002000 items.
Figure 37: XQuAD token-F1 by language, both translation directions, Qwen3-4B/8B. Horizontal bars indicate the small model’s and large model’s scores, with the band between them shaded. Both directions are then overlaid: L→SL\to S in blue, S→LS\to L in orange. Stars are the score of KV-Lingo translators, averaged over three seeds; triangles are the closed-form baseline. Error bars, on the native scores and the closed-form fits too, are the standard error over evaluation items (Appendix E.1 ). These translators have been fitted on English-only data (from Nemotron-SFT-Instruction-Following-Chat-v2), so English, left of the dotted line, is their training language and the other ten are not. Reasoning-off; 256256 questions per language. The distilled translators outperform the small model in most of the languages, showing striking OOD generalisation.
 into the smaller modelinto the larger model
nntranslatestepre-prefilltranslatestepre-prefill
Qwen3-0.6B/1.7B
204820483.53.529.229.229.929.93.53.529.029.030.030.0
409640964.34.329.429.434.034.04.34.329.029.049.649.6
819281927.97.929.429.475.775.77.97.929.629.6107.0107.0
163841638415.215.229.429.4189.2189.215.515.529.729.7250.9250.9
Appendix
Table 36: Components of the switch cost on an H100 , the two shape-preserving Qwen3 pairs, Hugging Face Transformers ( bf16 , batch 11 , both paths on the same stack): translation, one decoding step of the target, and its re-prefill, in ms, every term measured at every length. The translation times enter the speedups of Figure 6 a. This harness spends a fixed 2929 to 4040 ms on every decoding step, against 1.81.8 to 8.58.5 ms with vLLM (Table 37 ), so ratios taken within it understate the gain.
Prompt tokens22 k44 k88 k1616 k3232 k
Qwen3-0.6B3030 / 1.81.83838 / 1.91.95858 / 2.12.1123123 / 2.42.4405405 / 2.92.9
Qwen3-1.7B3131 / 2.62.63737 / 2.62.66767 / 2.82.8145145 / 3.13.1506506 / 3.63.6
Qwen3-4B4444 / 4.64.65454 / 4.74.7111111 / 4.94.9275275 / 5.35.310651065 / 6.06.0
Qwen3-8B6161 / 7.07.07272 / 7.17.1150150 / 7.37.3543543 / 7.77.713841384 / 8.58.5
Gemma-3-4B4141 / 4.74.75151 / 4.74.79292 / 4.74.7179179 / 4.74.7561561 / 4.74.7
Gemma-3-12B9090 / 11.111.1101101 / 11.111.1345345 / 10.910.9681681 / 11.111.114511451 / 11.411.4
Appendix
Table 37: Serving with vLLM on an H100 , bf16 , one request at a time: median time to first token / median time per output token, in ms, over 1616 requests per prompt length, each generating 128128 tokens. Time to first token is measured at the client. The last row is Gemma-3-12B given the 44 KV heads of Gemma-3-4B, with random weights, which times its decoding step against a cache of the 4B’s width under a memory budget.
  decoding throughputpeak memory (GB)
contextbatchfull (tokens/s)narrowednarrowed ++ translatorfull →\to narrowed ++ translator
88 k116262 – 6565+9+9 – 10%10\%−5%-5\%9.9→9.19.9\to 9.1
88 k88160160 – 171171+33+33 – 37%37\%+22+22 – 25%25\%21.6→14.921.6\to 14.9
3232 k115353+16+16 – 20%20\%+1+1 – 5%5\%13.9→11.013.9\to 11.0
3232 k88110110 – 115115+44+44 – 49%49\%+36+36 – 40%40\%50.2→29.450.2\to 29.4
Appendix
Table 38: Decoding under a memory budget , Gemma-3-12B in 44 -bit on Apple M3 Ultra (MLX), random weights and cache. The full cache has 88 KV heads, the narrowed one 44 , the width of Gemma-3-4B, at unchanged depth; the translator is a random per-head map applied to the query and to the attention output. Each cell generates 128128 tokens, best of 33 ; ranges span two machines. Throughput changes are relative to the full cache. Peak memory is that of the whole process during decoding (weights, cache and temporaries).
More in this topic

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,101 papers

Explore similar work

CardsList
Jul 31, 2026

Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models

Heterogeneous Large Language Model (LLM) systems increasingly rely on shared contexts, retrieved evidence, and multi-agent dialogue histories, yet their internal key-value (KV) caches remain model-specific and cannot be reused across architectures. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi-model reasoning and long-context generation. We propose Mixture-of-Translators(MoT), a cache translation framework that maps context KV caches from a source LLM into the cache space of a target LLM. Unlike prior approaches that depend on a single projection path or global shared latent space, MoT uses multiple translator modules to capture diverse source--target mappings. To further reduce residual translation error, we introduce a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory. We reveal two competing failure modes in cache translation: propagated translation shift from early injection and last-state shift from late injection. MoT addresses them through translator mixtures and target-side correction. Across homogeneous and heterogeneous translations among Qwen2.5, GPT-2, and OPT models, MoT preserves downstream QA performance, including Qwen2.5-7B-scale translation with 51.0% average closed-set QA accuracy and 0.43 average extractive QA F1. In practical case studies, MoT enables quality-preserving memory reuse for multi-agent reasoning and retains 96.3% of direct-context quality in long-context cache-augmented generation, demonstrating scalable KV cache reuse across heterogeneous LLMs.

Jin-woo Lee, Minkyung Song, Junghyun Oh +4

Chungnam National University · KISTI

Memory-Augmented Language ModelsKV CachingLLM Inference Acceleration
AskPDF
Aug 4, 2026

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

Taekyung Heo, Rasoul Shafipour, Ritchie Zhao +6

NVIDIA

KV CachingLLM Inference AccelerationKV-Cache Reuse
AskPDF
Aug 31, 2026

A Universal Context-Reuse Layer for Cross-Model KV Sharing

Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study cross-model KV sharing, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B →\rightarrow Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59% to 34.48%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B →\rightarrow Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B →\rightarrow Qwen2.5-7B setting, cross-family handoff achieves 44.0% accuracy compared with 45.7% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate context mobility as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.

Yi Li, Dongming Jiang, Yi Zhao +1

University of Texas at Dallas

LLM InferenceKV CachingLLM Inference AccelerationKV-Cache Reuse
AskPDF