GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training

Jianing Qi; Hao Tang; Zhigang Zhu

cs.LGAug 26, 2026

GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training

Authors: Jianing Qi, Hao Tang, Zhigang Zhu

Organizations: CUNY Graduate Center · The City College of New York

Abstract

We study how post-training changes the weights of Large Language Models (LLMs) relative to their pretrained weights. Across 12 post-training chains with supervised fine-tuning (SFT) and reinforcement learning (RL), we express each weight update in the pretrained matrix's singular value decomposition (SVD) frame. This decomposition separates the changes of three geometrically distinct components: diagonal values, which reshapes singular values; off-diagonal values, which rotates the coupling between pretrained input and output directions; and null-space values, which routes outside the matrix's original nonzero SVD core. On a math evaluation suite, we find that removing the diagonal component usually preserves most of the gains from post-training. These results suggest that post-training gains are carried primarily by reconfiguring and extending pretrained pathways rather than by substantially changing singular values of pre-trained models.

Figures & tables

Panel of Figure 1 — Figure 1: Left: the leading singular values remain visually almost identical before and after post-training. We plot three chains with large spectral changes. Right: Across nearly all tested transitions, removing the diagonal reproduces the full post-trained model with trivial degradation with the exception of C1.

Figure 2: Ablation scores relative to base and full model. Each bar is one transition, sorted by full score. The gray bar marks the base model score; the black bar marks the full post-trained model score. Colored markers show the performance of each isolated component: diagonal only (pink), off-diagonal only (green), null space only (orange), and OffNull (purple). Red names correspond to SFT models, and blue names are RL models.

Panel of Figure 3 — Figure 3: Left: off vs null accuracy. Right: interaction of off and null

Figure 4: (A) Normalized Frobenius norms (log scale) of the SFT, incremental RL, and cumulative updates. SFT dominates in all chains. (B) Off-diagonal cosine similarity between SFT and incremental RL rotations. Diamonds show the median over matrices and bars show bootstrap 95% confidence intervals. Significance is measured against a matrix- and chain-matched isotropic null for the same median statistic.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Chain	Base Model (HF identifier)	Size	Pipeline
C1	Qwen/Qwen2.5-Math-1.5B [ Yang et al., 2024 ]	1.5B	SFT $\to$ GRPO ( $\times$ 2)
C2	Qwen/Qwen3-8B-Base [ Yang et al., 2025 ]	8B	SFT $\to$ GPPO
C3	Qwen/Qwen3-14B [ Yang et al., 2025 ]	14B	SFT $\to$ GRPO
C4	Qwen/Qwen3-8B-Base [ Yang et al., 2025 ]	8B	GRPO (direct)
C5	mistralai/Mistral-7B-v0.1 [ Jiang et al., 2023 ]	7B	SFT $\to$ DPO
C6	Qwen/Qwen2.5-Math-7B [ Yang et al., 2024 ]	7B	SFT $\to$ DPO / PPO † / RAFT †

Appendix

Table 1: Chain inventory. C6 and C10 share the same pretrained base (Qwen2.5-Math-7B) but use different post-training pipelines.

\dagger

indicates training directly from the pretrained base rather than from the preceding SFT checkpoint.

Chain	Family	Trained Model	Size	From	Method	Domain
C1	Qwen 2.5	DeepSeek-R1-Distill-Qwen-1.5B [ Guo et al., 2025 ]	1.5B	Qwen2.5-Math-1.5B	SFT	Reasoning
C1	Qwen 2.5	DeepScaleR-1.5B-Preview [ Luo et al., 2025 ]	1.5B	DeepSeek-R1-Distill-Qwen-1.5B	GRPO	Math
C1	Qwen 2.5	Nemotron-Reasoning-Qwen-1.5B [ Liu et al., 2025 ]	1.5B	DeepSeek-R1-Distill-Qwen-1.5B	GRPO	Math
C2	Qwen 3	Klear-Reasoner-8B-SFT [ Su et al., 2026 ]	8B	Qwen3-8B-Base	SFT	Reasoning
C2	Qwen 3	Klear-Reasoner-8B [ Su et al., 2026 ]	8B	Klear-Reasoner-8B-SFT	GPPO	Reasoning
C3	Qwen 3	UniReason-Qwen3-14B-SFT [ Huan et al., 2025 ]	14B	Qwen3-14B	SFT	Reasoning

Appendix

Table 2: Complete inventory of incremental transitions. Each row is one training step producing the listed model. “From” gives the full name of the immediate predecessor model.

\dagger

marks custom checkpoints not publicly released on HuggingFace. Domain indicates the training objective: Math = mathematical reasoning, General = general instruction following, Reasoning = broad reasoning / thinking, Agent = agentic tool use, Metacog. = metacognitive self-reflection.

Chain	Transition	$r_{\Delta}$	SV cos	$\delta_{\max}$
Post-SFT follow-up transitions (12 transitions)
C6 (Qwen2.5-7B)	SFT $\to$ DPO	0.0003	1.000000 $\pm$ 0	0.01 $\pm$ 0.01%
C8 (LLaMA-3-8B)	SFT $\to$ PPO	0.0010	1.000000 $\pm$ 0	0.01 $\pm$ 0.01%
C1 (Qwen2.5-1.5B)	SFT $\to$ GRPO-A	0.0010	1.000000 $\pm$ 0	0.01 $\pm$ 0.01%
C10 (Eurus-2-7B)	SFT $\to$ PRIME	0.0010	1.000000 $\pm$ 0	0.02 $\pm$ 0.01%
C2 (Qwen3-8B)	SFT $\to$ GPPO	0.0020	1.000000 $\pm$ 0	0.02 $\pm$ 0.01%

Appendix

Table 3: Spectral invariance of weight matrices across 29 transitions: incremental post-SFT updates, from-base RL, and cumulative SFT, sorted by

r_{\Delta}

within each group. SV cosine and

\delta_{\max}

report mean

\pm

std across all weight matrices in the transition.

Transition	$\mathrm{SR}_{0}$	$\mathrm{SR}_{\Delta}$	$\mathrm{SR}_{1}$	$\Delta$ %	$\alpha$
SFT (base $\to$ SFT)
C1	109.8	82.3	109.2	$-0.60$	$-0.007$
C2	167.7	277.5	165.2	$-1.49$	$+0.017$
C3	183.1	197.2	183.5	$+0.17$	$-0.002$
C5	181.0	19.7	181.2	$+0.07$	$+0.001$
C6	176.6	10.7	176.3	$-0.15$	$-0.001$

Appendix

Table 4: Stable rank and additivity for all 42 transitions.

\mathrm{SR}_{0}

\mathrm{SR}_{\Delta}

\mathrm{SR}_{1}

: stable rank of

\mathbf{W}_{0}

\Delta\mathbf{W}

\mathbf{W}_{1}

;

\alpha

: additivity ratio.

Chain	Base $\to$ SFT model	Base	Full	Diag	Off	Null	OffNull
C1	Qwen2.5-1.5B $\to$ DS-R1-Distill	31.3	49.2	6.4 (13%)	31.3 (64%)	18.5 (37%)	28.9 (59%)
C2	Qwen3-8B $\to$ KlearReasoner-SFT	21.2	60.4	14.0 (23%)	9.5 (16%)	9.7 (16%)	56.1 (93%)
C3	Qwen3-14B $\to$ UniReason-SFT	17.2	49.1	28.0 (57%)	54.3 (111%)	55.7 (113%)	45.8 (93%)
C5	Mistral-7B $\to$ mistral-sft-beta	3.6	5.7	1.9 (33%)	5.8 (102%)	6.2 (110%)	5.7 (100%)
C6	Qwen2.5-Math-7B $\to$ RLHFlow-SFT	35.8	43.5	38.8 (89%)	49.7 (114%)	48.0 (110%)	42.1 (97%)
C7	OLMo-2-7B $\to$ SFT	5.0	8.6	7.9 (92%)	9.0 (105%)	9.8 (114%)	7.5 (88%)

Appendix

Table 5: Causal decomposition: SFT transitions (base

\to

SFT). Accuracy (%) averaged over five math benchmarks; retention relative to Full in parentheses.

Chain	RL model	Alg.	Base	Full	Diag	Off	Null	OffNull
C1	DeepScaleR	GRPO	49.2	51.8	41.0 (79%)	43.8 (85%)	51.7 (100%)	54.4 (105%)
C1	Nemotron	GRPO	49.2	61.3	40.3 (66%)	57.6 (94%)	59.2 (97%)	62.7 (102%)
C2	KlearReasoner	GPPO	60.4	71.7	58.6 (82%)	69.8 (97%)	69.0 (96%)	72.6 (101%)
C3	UniReason	GRPO	49.1	66.5	75.1 (113%)	64.2 (96%)	65.9 (99%)	65.0 (98%)
C5	zephyr-beta	DPO	5.7	3.1	5.2 (168%)	5.3 (171%)	4.7 (153%)	3.7 (120%)
C6	DPO	DPO	43.5	49.7	41.2 (83%)	45.0 (90%)	48.5 (98%)	49.8 (100%)

Appendix

Table 6: Causal decomposition: RL transitions (SFT

\to

RL or base

\to

RL). Same format as Table 5 .

Chain	Transition	$f_{\mathrm{diag}}$	$f_{\mathrm{off}}$	$f_{\mathrm{null}}$
SFT (base $\to$ SFT)
C1	Qwen2.5-1.5B $\to$ DS-R1-Distill	0.204	0.288	0.508
C2	Qwen3-8B $\to$ KlearReasoner-SFT	0.149	0.456	0.397
C3	Qwen3-14B $\to$ UniReason-SFT	0.011	0.504	0.487
C5	Mistral-7B $\to$ mistral-sft-beta	0.000	0.508	0.492
C6	Qwen2.5-Math-7B $\to$ RLHFlow-SFT	0.000	0.453	0.548

Appendix

Table 7: Mean energy fraction of each

\Delta\mathbf{W}

component, averaged over all layers and module types. Fractions sum to 1 by construction.

Figure 5: Cross method comparison: same base, different training. (A) Frobenius norms of the two updates (log scale), with the smaller to larger ratio annotated. (B) Off-diagonal cosine similarity. The matched isotropic null accounts for each pair’s matrix shapes and the aggregation statistic used in the figure. All pairs are positively aligned beyond this null. The sole exception is C3, where independently trained SFT and RL produce nearly identical rotations ( $\cos=0.98$ ) and closely matched update magnitudes.

Transition	Full	Remove diag.	Remove random	$\Delta_{\mathrm{diag}}$	$\Delta_{\mathrm{rand}}$
C1 SFT	49.2	28.9	44.2	$-20.3$	$-5.0$
C2 SFT	60.4	56.1	45.2	$-4.3$	$-15.2$

Appendix

Table 8: Removing the learned diagonal component compared with removing a random component of the same Frobenius norm within the pretrained SVD core. Random-removal results are averaged over three seeds.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Understanding and Exploiting Anisotropy in Post-Training
Sep 26, 2026Samyak Jha, Harshvardhan Saini, Yizhen Liao +2LLM Post-Training Representation Geometry in Language Models
Indian Institute of Technology Dhanbad · National University of Singapore
Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
Apr 1, 2026Yuhang Li, Donghyun Lee, Ruokai Yin +1Low-Rank Approximation LLM Compression
Yale University · University of Sounthern California
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China

Sep 26, 2026J/K move · Enter open · S save

Understanding and Exploiting Anisotropy in Post-Training

Samyak Jha, Harshvardhan Saini, Yizhen Liao, Yiming Tang+1

Indian Institute of Technology Dhanbad · National University of Singapore

LLM Post-Training Representation Geometry in Language Models RL for Language Model Reasoning LLM Reasoning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

LLM post-training combines supervised fine-tuning (SFT), a mode-covering forward-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective. Frequency-weighted likelihood training leaves a well-known signature: anisotropy, in which a few residual channels carry disproportionately large activations. Anisotropy is widely documented and usually treated as a defect, yet its function and its interaction with post-training remain unclear. We first analyze it. A label-free outlier rule isolates about 5% of residual channels that are essential for language modeling: removing them raises perplexity from 10 to over

10^6

, versus 35 for count-matched random channels. Yet they barely distinguish correct from incorrect reasoning. SFT reshapes them, whereas RL leaves them largely intact and adapts the complementary channels. These channels therefore form the model's coherence substrate, and reasoning adaptation happens elsewhere. We then exploit this. SphereGate learns one bounded gain per residual channel on a frozen backbone. Its activation-weighted gradients provably limit movement of high-energy coherence channels and leave the remaining channels free. With 0.1M trainable parameters, SphereGate outperforms parameter-efficient baselines by 2.0--7.3 points on MATH-500 across Qwen2.5 (0.5B--7B) and Llama-3-8B, is comparable or exceeds full-model GRPO. Anisotropy is not a defect but a division of labor that post-training can exploit.

Abstract

LLM Post-Training

Explore similar work

Understanding and Exploiting Anisotropy in Post-Training

Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning