Scaling depth is a key route to stronger large language models, but Pre-LN architectures often suffer from depth-wise information dilution, limiting the effective use of deep layers. While cross-layer connectivity has proven effective at mitigating this problem, existing designs usually introduce additional computation or memory access. We revisit cross-layer KV sharing as an efficient form of such connectivity. Beyond cache compression, each KV reuse edge creates a direct gradient path from deep layers to source KV projections, turning KV sharing into an implicit structural regularizer for KV representation geometry. We provide a systematic analysis of this mechanism, showing that proper sharing topologies sharpen key retrieval directions while enriching the value content subspace, whereas existing adjacent or global-anchor designs do not fully exploit this effect. Motivated by this, we propose DepthGraft, a hierarchical KV-sharing architecture that reconstructs deep-layer KV from stride-aligned shallow-middle source pairs. DepthGraft distributes non-local gradient flow across source layers, while preserving prefill early completion and reducing KV cache by roughly one third. Experiments on dense and MoE models from 0.5B to 16B parameters show consistent downstream improvements under both GQA and MLA. Notably, on the 16B MoE model, DepthGraft achieves a 7.0% absolute improvement on MMLU-Pro, along with gains of 6.6% on CMMLU and 6.4% on C3. These results validate the potential of DepthGraft as a scalable design for large-scale pretraining. The project page is available at https://github.com/jianke0604/DepthGraft.
Figures & tables
Figure 1: Results on the 16B MoE model. Left two: DepthGraft demonstrates a consistent advantage throughout training. Right two: DepthGraft reduces prefill time (TTFT) and KV cache usage by approximately one-third, while incurring almost no loss in decoding speed.
Figure 2: Evolution of KV representation geometry during training, visualized via effective rank (averaged over all layers).
Figure 3: (a) Adjacent-layer sharing (CLA/MLKV) does not reduce K erank below the baseline. (b) Global-anchor sharing concentrates gradient mass on anchor layers.
Model
Method
Knowledge & Understanding
Reasoning & Mathematics
Code
Chinese
Avg.
MMLU
M-Pro
LAMB.
ARC-C
SIQA
GSM8K
HEval
MBPP
LCBench
C-Eval
CMMLU
C3
0.5B
GQA
29.37
3.43
43.46
25.09
36.08
1.36
4.27
3.60
1.22
30.81
29.07
26.74
19.54
DepthGraft
30.07
4.86
43.66
28.18
34.19
1.97
6.71
5.20
0.52
29.99
30.35
27.73
20.29
1B
GQA
30.35
7.86
52.58
31.27
35.26
6.67
9.15
9.40
2.09
31.45
30.23
31.67
23.17
DepthGraft
32.70
8.57
54.17
27.84
38.08
6.07
9.76
12.60
2.61
31.43
30.52
33.92
24.02
1.5B
GQA
34.81
11.14
54.52
36.43
43.81
14.10
12.80
16.20
3.48
34.64
33.56
44.79
28.36
Table 1: Dense scaling results of baseline and DepthGraft on 0.5B–1.5B models. We report accuracy or pass rate on each benchmark and average performance over all benchmarks. The best result within each model size is shown in bold .
Model
Method
Knowledge & Understanding
Reasoning & Mathematics
Code
Chinese
Avg.
MMLU
M-Pro
LAMB.
ARC-C
SIQA
GSM8K
HEval
MBPP
LCBench
C-Eval
CMMLU
C3
3B A0.5B
GQA
31.44
7.43
51.42
26.46
35.62
6.22
12.80
14.40
1.22
32.29
28.19
28.78
23.02
DepthGraft
32.32
6.29
52.39
32.30
37.36
6.82
13.41
13.00
2.26
32.51
28.82
31.41
24.07
9B A1.5B
GQA
44.18
20.43
62.36
48.11
47.54
26.99
23.17
32.20
7.65
40.00
36.69
55.12
37.04
DepthGraft
46.53
19.71
63.48
52.92
53.58
28.81
22.56
29.60
8.17
42.65
41.26
62.63
39.33
9B A1.5B
MLA
43.30
18.29
63.48
46.74
47.75
27.52
22.56
29.60
4.00
37.85
35.02
55.95
36.01
Table 2: MoE scaling results of baseline models and DepthGraft on 3B–16B models. We report accuracy or pass rate on each benchmark and average performance over all benchmarks. The best result within each model setting is shown in bold .
Model
Method
TTFT (ms)
TPOT (ms)
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
9B MLA
Baseline
52.22
89.05
161.75
314.09
611.19
3.39
3.54
3.84
4.09
4.59
DepthGraft
43.82
69.89
122.77
233.31
447.51
3.42
3.57
3.87
4.13
4.62
16B GQA
Baseline
102.26
198.39
422.94
1000.15
2655.90
5.07
5.25
5.60
6.35
7.79
DepthGraft
84.44
151.13
313.31
726.11
1891.80
5.12
5.30
5.64
6.40
7.84
Table 3: Long-context inference efficiency of baseline models and DepthGraft. We report time-to-first-token (TTFT) and time-per-output-token (TPOT) from 8K to 128K context lengths.
Table 4: Ablation study on the 16B A3B MoE model. We decompose DepthGraft along two axes: cross-layer gradient connectivity and stage layout. Best results within each group are in bold .
Method
Type
MMLU
M-Pro
CMMLU
C3
C-Eval
GSM8K
HEval
Avg.
GQA Baseline
–
51.72
23.86
40.68
70.74
47.60
39.50
22.56
42.38
CLA / MLKV
Adjacent
48.81
20.14
40.64
58.25
40.50
37.91
20.12
38.05
YOCO
Global-1
50.28
27.57
43.87
67.73
46.59
42.91
24.39
43.33
FusedKV
Global-2
53.86
27.71
46.48
70.90
50.66
44.88
25.00
45.64
DepthGraft
Hierarchical
54.21
30.86
47.29
77.15
50.00
45.34
28.05
47.56
Table 5: Comparison with prior KV sharing methods on the 16B A3B MoE model. Type indicates the sharing topology. Best results are in bold .
Length
Method
NIAH
VT
FWE
QA
Avg.
s1
s2
s3
mk1
mk3
mv
mq
4K
Baseline
100
100
75.0
100
99.0
90.5
95.5
43.2
80.7
84.0
86.8
DepthGraft
100
100
100
100
99.0
98.3
98.5
81.6
84.3
86.0
94.8
8K
Baseline
100
100
70.0
97.0
96.0
58.3
95.3
44.0
65.0
66.0
79.2
DepthGraft
100
100
100
94.0
100
98.0
94.5
80.0
78.7
61.0
90.6
16K
Baseline
100
100
82.0
96.0
93.0
48.3
91.5
45.6
66.0
63.0
78.5
Table 6: Long-context performance on RULER after context extension. NIAH denotes needle-in-a-haystack retrieval; VT, FWE, and QA denote variable tracking, frequent-word extraction, and SQuAD-based question answering.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Parameter
0.5B
1B
1.5B
Training
lr schedule
WSD
WSD
WSD
lr
8.0×10−4
6.0×10−4
6.0×10−4
warmup steps
2,000
2,000
2,000
decay ratio
0.2
0.2
0.2
AdamW- β
(0.9, 0.95)
(0.9, 0.95)
(0.9, 0.95)
weight decay
0.1
0.1
0.1
Appendix
Table 7: Hyperparameters and model configurations for dense-model pretraining.
Group
Parameter
3B A0.5B
9B A1.5B
9B A1.5B
16B A3B
GQA
GQA
MLA
GQA
Training
lr schedule
WSD
WSD
WSD
WS
lr
4.82×10−4
4.82×10−4
4.82×10−4
4.2×10−4
warmup steps
2,000
2,000
2,000
750
decay ratio
0.2
0.2
0.2
–
AdamW- β
(0.9, 0.95)
(0.9, 0.95)
(0.9, 0.95)
(0.9, 0.95)
Appendix
Table 8: Hyperparameters and model configurations for MoE-model pretraining.
Figure 4: DepthGraft workflow. The shallow and middle stages compute and cache KV normally, while deep-stage KV is reconstructed from stride-aligned shallow–middle source pairs. The deep stage therefore does not materialize independent KV cache, while gradients from deep-layer attention flow back to the source KV projections through the reconstruction paths.
Method
MMLU
M-Pro
LAMB.
ARC-C
SIQA
GSM8K
HEval
MBPP
LCBench
C-Eval
CMMLU
C3
Avg.
Baseline + SFT
53.72
35.00
60.77
67.01
59.83
52.08
37.80
35.80
14.96
49.46
43.47
72.33
48.52
DepthGraft + SFT
56.84
37.57
62.28
71.82
62.54
55.04
43.29
42.80
14.96
52.03
47.28
77.59
52.00
Appendix
Table 9: Downstream performance after SFT on the 16B A3B models. Both models use the same fine-tuning recipe.
Method
MMLU
M-Pro
LAMB.
ARC-C
SIQA
GSM8K
HEval
MBPP
LCBench
C-Eval
CMMLU
C3
Avg.
SWA baseline
37.05
11.71
60.26
21.62
43.67
16.38
21.34
22.20
3.13
34.34
34.32
31.45
28.12
SWA + DepthGraft
38.25
13.57
61.43
33.33
40.31
19.50
18.90
23.00
2.57
33.57
36.54
35.63
29.72
Appendix
Table 10: Compatibility with SWA on an 8B MoE model. The two variants share the same attention-mask pattern and training recipe.
Figure 5: Per-layer V effective rank during training for the 9B GQA baseline, FusedKV, and DepthGraft. FusedKV improves the anchor layer but leaves nearby non-anchor layers close to the baseline, whereas DepthGraft yields consistent rank elevation across the shown source layers due to distributed stride-aligned connectivity.
Figure 6: Average attention score assigned to the first token over training steps. DepthGraft consistently reduces first-token attention concentration compared with the baseline on both GQA and MLA models.
Figure 7: Training loss curves across representative model scales and attention variants. DepthGraft consistently improves training loss, but the gain is modest relative to its downstream benchmark improvements, supporting its interpretation as a KV-geometry regularizer.
Layer
K src1
K src2
V src1
V src2
10
0.6905
1.1955
2.1655
0.7594
11
0.4738
1.1936
1.6323
1.4713
12
0.5894
1.2603
2.4572
1.2290
13
0.7178
1.0534
2.6295
0.8892
14
0.7208
0.9557
1.8670
0.6671
Appendix
Table 11: Final reconstruction-coefficient norm relative to initialization. Src1 denotes the shallow source and src2 denotes the middle source. Values larger than 1 indicate increased reliance on the corresponding source after training.
Depth band
PP rank 0
PP rank 1
PP rank 2
PP rank 3
Shallow sources
0, 1, 2
3, 4, 5
6, 7, 8
9, 10, 11
Middle sources
12, 13, 14
15, 16, 17
18, 19, 20
21, 22, 23
Deep reuse
24, 25, 26
27, 28, 29
30, 31, 32
33, 34, 35
Appendix
Table 12: A balanced PP/VPP-compatible layer placement for a 36-layer model with PP size 4. Each stride-aligned source–reuse triple is placed on the same physical PP rank.