Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is significant and heavily impacts serving costs. This work proposes to lessen these memory requirements. While recent work has largely addressed KV cache reduction via compression and eviction along the temporal axis, we argue that the depth dimension offers an orthogonal and robust avenue for optimization. Although prior research suggests that a full cache for every layer is redundant, implementing cross-layer cache sharing remains a practical challenge; existing methods typically suffer from reduced throughput or increased time-to-first-token. In this paper, we demonstrate that dropping a layer's cache offers efficient optimization without information loss. We propose a simple training approach: random cross-layer attention. During training, layers randomly choose to attend either to their own KV states or those of a preceding layer. This stochastic process adapts the model to be robust to various depth-wise cache sharing strategies, ensuring flexibility for unknown hardware constraints at deployment time. Our evaluations show that applying this scheme during pre-training or fine-tuning enables depth-wise cache sharing for various model families. Furthermore, for larger models in data-constrained settings, this approach is suggestive of a regularization-like effect, frequently preserving or improving performance while significantly reducing the cache's memory footprint.
Figures & tables
Figure 1: Language models are often described as extremely efficient databases, able to store training data in their parameters. The KV cache however doesn’t reflect that, and a single book may require as much space as the model itself. Memory footprint per token can be obtained as follows for transformer decoders: 2× #layers × #kv_heads × #head_dim ×2 , where the leftmost factor of 2 is due to storing both K and V representations, while the rightmost factor of 2 is due to each floating point value requiring two bytes, assuming FP16 precision.
Figure 2: Performance comparison under varying cache retention rates for base models and their counterparts trained with random cross-layer attention. We report F1 scores for Question-Answer evaluations under different levels of cache retention for standard and R-CLA fine-tuned models. R-CLA avoids much of the degradation base models incur upon cache sharing, and preserves or improves performance at full cache.
Figure 3: Cache size and prediction performance trade-offs for different models.
Figure 4: Behavior of a model implementing R-CLA. During training , each layer decides whether to attend to KV states obtained from the past layer’s hidden states, or from a randomly chosen layer in the past, so each batch observes a different cache sharing strategy. During testing , a fixed deterministic cache sharing scheme is employed, and the same group of layers always share the same KV cache.
Figure 5: Cache size vs. evaluation loss for Qwen3-1.7B. R-CLA models (red) use the full 28 -layer architecture with depth-wise cache sharing, while baselines (blue) are shallower Transformers with the same cache footprint. Deeper models with shared cache consistently outperform shallower models, showing that cache sharing preserves the benefit of depth.
p
0.00
0.25
0.50
0.60
0.75
Eval Loss
2.424
2.428
2.441
2.446
2.461
Table 1: Pre-training evaluation loss for Qwen3-1.7B trained with R-CLA at varying cross-layer attention probability p . Training remains stable across all values, with loss increasing by less than 2% even at p=0.75 .
Llama-3.1-8B
Mistral-7B
Qwen3-8B
Dataset
Retention
Base
R-CLA
Δ%
Base
R-CLA
Δ%
Base
R-CLA
Δ%
HotpotQA
100%
0.203
0.306
+51.1
0.215
0.242
+12.4
0.233
0.357
+53.1
50%
0.171
0.305
+78.7
0.162
0.211
+30.0
0.085
0.314
+267.8
25%
0.080
0.237
+196.2
0.031
0.162
+421.3
0.011
0.098
+826.6
MSMarco
100%
0.301
0.318
+5.6
0.310
0.316
+1.7
0.291
0.329
+13.0
50%
0.226
0.324
+43.2
0.236
0.300
+26.9
0.112
0.227
+102.4
Table 2: Effect of R-CLA on F1 Score for models fine-tuned for Question-Answer. We compare the baseline standard self-attention with R-CLA trained under p=0.6 . Evaluation is performed across three depth-wise cache retention levels.
Dataset
Retention
Method
R-CLA
CLA@2
CLA@4
RD-CLA@2
RD-CLA@4
HotpotQA
100%
0.306
0.192
0.331
0.223
0.167
50%
0.305
0.149
0.284
0.179
0.134
25%
0.237
0.089
0.172
0.100
0.071
MSMarco
100%
0.318
0.282
0.303
0.323
0.300
50%
0.324
0.184
0.159
0.247
0.211
Table 3: Ablations on top of Llama-3.1-8B. We compare R-CLA ( p=0.6 ) against deterministic CLA fixed at every 2nd or 4th layer, and their randomly applied counterparts (RD-CLA, p=0.6 ).
Input
Group
Peak Mem.
KV Cache
TTFT
Throughput
Length
Size
(MB)
(MB)
(ms)
(tok/s)
2,048
1
16,573
306
66
32.9
4
16,357
77
63
43.0
8,192
1
19,319
1,170
297
34.0
4
18,455
293
286
41.6
16,384
1
22,981
2,322
713
34.2
Table 4: Inference efficiency across input lengths. Architecture matches Qwen3-8B (36 layers, 8 KV heads, bfloat16) on a single 80GB GPU. Group size g=1 is the standard baseline; g=4 shares every 4 consecutive layers. Time to first token (TTFT) measures prefill latency.
Batch
Group
Peak Mem.
KV Cache
TTFT
Throughput
Size
Size
(MB)
(MB)
(ms)
(tok/s)
2
1
22,973
2,322
588
33.3
4
21,245
580
573
40.3
4
1
30,281
4,643
1,201
22.7
4
26,825
1,161
1,159
25.9
8
1
44,897
9,287
2,395
12.8
Table 5: Batch size scaling at 8,192-token context. Same architecture as Table 4 . At batch size 16, the baseline ( g=1 ) exceeds GPU memory, while cache sharing ( g=4 ) completes successfully.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training loss as a function of tokens processed for Qwen3-1.7B with varying KV cache sharing ratios.
Figure 7: Train loss (left) and eval loss (right) over fine-tuning steps for Base (blue) vs. R-CLA (red) on Qwen3-8B (top) and Llama-3.1-8B (bottom). R-CLA shows slower learning throughout, consistent with a regularization-like effect from stochastic KV routing.
Figure 8: F1 score vs. cache retention (%) for Llama-3.1-8B across five QA tasks. R-CLA (red, solid) maintains performance across retention levels, while CLA@ k variants (dashed) degrade outside their trained retention level. Data from Table 3 .
Dataset
Model
Retention
F1
Exact Match (EM)
ROUGE-L
Base
R-CLA
Δ%
Base
R-CLA
Δ%
Base
R-CLA
Δ%
HotpotQA
Llama-3.1-8B
100%
0.203
0.306
+51.1%
0.128
0.221
+72.3%
0.206
0.307
+49.0%
50%
0.171
0.305
+78.7%
0.102
0.216
+112.6%
0.173
0.305
+75.8%
25%
0.080
0.237
+196.2%
0.035
0.153
+339.6%
0.088
0.240
+171.6%
Mistral-7B
100%
0.215
0.242
+12.4%
0.122
0.141
+15.8%
0.216
0.243
+12.6%
50%
0.162
0.211
+30.0%
0.081
0.106
+30.5%
0.165
0.212
+28.6%
Appendix
Table 6: We report Base, R-CLA ( p=0.6 ), and relative improvement ( Δ% ) for F1, Exact Match (EM), and ROUGE-L across three cache retention levels.