We investigate a fundamental structural question in Grouped Query Attention (GQA) transformers: do the layers most sensitive to task correctness coincide with the layers where positional encoding adaptation has the greatest leverage? We term this the co-localization hypothesis and test it on Llama 3.1 8B, a 32-layer GQA model with a 4:1 query-to-key-value head ratio. We introduce \LSLORA, which restricts LoRA adaptation to layers identified via a novel correctness-differential hidden-state metric, and GARFA (GQA-Aware RoPE Frequency Adaptation), which attaches 8 learnable per-KV-head scalar multipliers to each targeted layer. Contrary to the co-localization hypothesis, we discover strong anti-localization: task-sensitive layers concentrate in the late network (ℓ∈{23-31}) while RoPE-influential layers dominate the early network (ℓ∈{0-9}), yielding Spearman rs=−0.735 (p=1.66×10−6). Despite this anti-localization, a 4-way cross-layer ablation shows that applying both interventions to the sensitivity-identified layers outperforms all alternative configurations by 4-16 percentage points across six diverse benchmarks (MMLU, GPQA, HumanEval+, MATH, MGSM, ARC), approaching Claude 3.5 Haiku on HumanEval+ (67.1% vs. 68.3%) at $100 total compute cost.
Figures & tables
Component
Params
%
Llama 3.1 8B (base, frozen)
8,030M
—
LS-LoRA ( r=64 , 10 layers)
42,598,400
0.53%
GARFA (8 heads, 10 layers)
80
<0.001%
Total trainable
42,598,480
0.53%
Table 1: Trainable parameter counts for the full method.
Dataset
Domain
Size
Magicoder-OSS-75K [ wei2024magicoder ]
Code
75,000
CodeAlpaca-20K [ codealpaca ]
Code
20,000
MetaMathQA-30K [ yu2024metamath ]
Math
30,000
OpenHermes-2.5 [ teknium2023openhermes ]
General
20,000
Total
145,000
Table 2: Training dataset composition.
Exp.
LoRA
GARFA
Role
A
L∗
L∗
Co-localized (ours)
B
L∗
Lr
LoRA alone?
C
Lr
L∗
GARFA alone?
D
Lr
Lr
Random control
Table 3: Cross-layer ablation design.
Figure 1: Scatter plot of sensitivity δℓ vs. RoPE influence ρℓ per layer. Green: co-localized (layer 0 only). Orange: sensitive-only (23–31). Blue: RoPE-influential only (1–9). Gray: neither top-10. Dashed: linear fit. Spearman rs=−0.735 ( p<10−5 ).
Figure 2: Dual-axis layer profile: sensitivity δℓ (orange, left axis) and RoPE influence ρℓ (blue dashed, right axis) vs. layer index. The profiles are anti-correlated: sensitivity rises through late layers while RoPE influence concentrates in early layers. Green band marks layer 0, the sole co-localized layer.
Model
MMLU
GPQA
HumanEval+
MATH
MGSM
ARC
Avg.
Llama 3.1 8B (baseline)
68.88
35.49
68.90
31.46
45.67
56.66
51.18
Exp D: Lr + Lr (control)
64.01
26.56
51.22
22.87
34.04
52.65
41.89
Exp C: Lr + L∗
67.18
31.70
60.37
28.12
39.78
53.16
46.72
Exp B: L∗ + Lr
67.21
34.82
60.37
25.04
43.45
56.23
47.85
Exp A: L∗ + L∗ (ours)
68.53
34.82
67.07
28.95
43.60
56.14
49.85
Claude 3.5 Haiku (target)
71.70
33.30
68.30
41.30
75.90
89.20
63.28
Table 4: Main results across six benchmarks. Bold green = best fine-tuned model per column. Avg. = macro-average. All scores are percentages.
Benchmark
Δ (pp)
HumanEval+
+15.8
MGSM
+9.6
GPQA
+8.3
MATH
+6.1
MMLU
+4.5
ARC
+3.5
Table 5: Exp A minus Exp D (pp). All six benchmarks are positive.
Figure 3: Cross-layer ablation: grouped bars per benchmark. Experiment A consistently leads Experiments B, C, and D. The A > B ≈ C > D pattern holds across all six benchmarks.
Figure 4: Exp A minus Exp D per benchmark (pp). All six values are positive, confirming universal improvement of co-localized over random layer selection.
1DAIMLD, Moscow, Russia · 2St. Petersburg Department of the Steklov Institute of Mathematics, St. Petersburg, Russia · 3St. Petersburg State University