LLM post-training combines supervised fine-tuning (SFT), a mode-covering forward-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective. Frequency-weighted likelihood training leaves a well-known signature: anisotropy, in which a few residual channels carry disproportionately large activations. Anisotropy is widely documented and usually treated as a defect, yet its function and its interaction with post-training remain unclear. We first analyze it. A label-free outlier rule isolates about 5% of residual channels that are essential for language modeling: removing them raises perplexity from 10 to over 106, versus 35 for count-matched random channels. Yet they barely distinguish correct from incorrect reasoning. SFT reshapes them, whereas RL leaves them largely intact and adapts the complementary channels. These channels therefore form the model's coherence substrate, and reasoning adaptation happens elsewhere. We then exploit this. SphereGate learns one bounded gain per residual channel on a frozen backbone. Its activation-weighted gradients provably limit movement of high-energy coherence channels and leave the remaining channels free. With 0.1M trainable parameters, SphereGate outperforms parameter-efficient baselines by 2.0--7.3 points on MATH-500 across Qwen2.5 (0.5B--7B) and Llama-3-8B, is comparable or exceeds full-model GRPO. Anisotropy is not a defect but a division of labor that post-training can exploit.
Figures & tables
Figure 1: Anisotropy and changes during post-training. (a) Illustration of representations spread evenly across directions (isotropic) or concentrated around a preferred direction (anisotropic). The channel sketch highlights components whose average activations are unusually far from zero. (b) Top: these selected channels form a small group across the displayed layers. Middle: starting from the same pretrained model, supervised fine-tuning (SFT) changes which channels belong to this group more than direct reward-based training with GRPO in the compared runs. Bottom: in a separate GRPO run starting from SFT, the remaining channels change more relative to their starting activation levels. Shading shows uncertainty estimated by repeatedly resampling evaluation problems.
Figure 2: Coherence and reasoning representations. (a) Correct and incorrect rollouts have higher subspace overlap in coherence channels than in their complement. (b) SFT produces larger coherence-channel membership changes than direct GRPO relative to the same pretrained model. (c) During SFT-initialized GRPO, complementary reasoning channels undergo larger baseline-relative activation changes than coherence channels. (d) Hidden-state cosine similarity between non-anisotropic diagonal-gate GRPO and full-model GRPO, both initialized from the same pretrained checkpoint, across decoder blocks 1–28. Shading denotes pointwise 95% problem-bootstrap confidence intervals.
Figure 3: The General Framework for Our Proposed Method, SphereGate.
Method
Qwen2.5-0.5B-It
Qwen2.5-3B-It
Qwen2.5-7B-It
Llama-3-8B-It
Base Model
32.5 ± 0.82
64.5 ± 0.71
70.0 ± 0.80
33.5 ± 0.76
Full GRPO
41.6 ± 0.72
69.2 ± 0.60
77.2 ± 0.61
37.4 ± 0.65
IA 3
31.7 ± 1.02
63.0 ± 0.88
68.6 ± 0.94
32.4 ± 0.93
TinyLoRA
33.5 ± 0.74
68.0 ± 0.63
75.0 ± 0.58
35.0 ± 0.72
LoRA-XS
35.0 ± 0.68
67.0 ± 0.57
77.0 ± 0.52
36.0 ± 0.66
Subnetwork
34.5 ± 0.91
67.0 ± 0.76
73.0 ± 0.70
34.0 ± 0.84
Table 1: Performance on MATH-500 under GRPO using greedy decoding. All values are absolute accuracy scores on a 0 – 100 scale. Standard deviations are estimated across three training seed checkpoints.
Method
AMC23
Minerva Math
Qwen2.5-7B-It
Llama-3-8B-It
Qwen2.5-7B-It
Llama-3-8B-It
Base Model
51.9 ± 1.38
8.8 ± 0.47
33.66 ± 0.72
12.39 ± 0.61
Full GRPO
53.6 ± 1.00
10.2 ± 0.34
36.1 ± 0.70
14.1 ± 0.70
IA 3
50.8 ± 1.47
8.4 ± 0.52
32.90 ± 0.81
11.80 ± 0.78
LoRA-XS
52.8 ± 1.11
9.2 ± 0.39
34.85 ± 0.61
12.34 ± 0.74
TinyLoRA
52.2 ± 1.27
9.4 ± 0.22
34.73 ± 0.58
11.08 ± 0.86
Table 2: Mathematical reasoning performance on AMC23 and Minerva Math using Avg@16. Standard deviations are estimated across three training seed checkpoints.
Method
AIME24
OlympiadBench
Base Model
16.67
41.5
Full GRPO
26.67
44.5
IA 3
16.67
40.2
LoRA-XS
20.00
43.0
TinyLoRA
16.67
43.0
Subnetwork
16.67
42.0
Table 3: Transfer performance on AIME24 and OlympiadBench for Qwen2.5-7B-Math-Instruct. AIME24 and OlympiadBench is evaluated using maj@16, All values are percentages.
(a) Restricted-gain training: Qwen2.5-3B-Instruct
Gate variant
MATH-500 greedy (%)
Baseline (un trained)
64.5
SphereGate (all channels)
70.0
Anisotropic-only gate
65.1
Random Channel gate
66.9
Matched Complement-only gate
67.5
Table 4: Complementary channel ablations. (a) MATH-500 accuracy after GRPO with different trainable channel groups; higher is better. Excluded gains remain at one, so their activations are not removed. (b) WikiText-2 perplexity under inference-time activation removal; lower is better. Anisotropic and matched-complement conditions remove the same number of coordinates in every decoder block. Matched-complement perplexities are averaged across five random-mask seeds. Full-test perplexities use BF16 inference and FP64 loss accumulation. Norm restoration is an ablation control, not part of the SphereGate training operator.
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.