Spectral Rank and Step-Direction Dynamics in Language-Model Post-Training
Organizations: Department of Electrical Engineering & Computer Science, University of Missouri-Columbia, MO, USA
Abstract
Language-model post-training modifies weights through cumulative updates and successive displacements, yet existing studies report seemingly conflicting spectral properties: nearly full-rank updates and low stable-rank changes. We show these observations are compatible and characterize update directions during training. For singular values following with , we prove that stable rank becomes negligible relative to fixed-energy rank as dimension increases. Full singular value decompositions of selected OLMo-2-7B updates confirm this separation: three proximal policy optimization (PPO) attention updates have stable ranks of 9.6-18.3 but require 532-1,222 singular components to capture 80% of their energy. We next compute consecutive checkpoint-displacement cosines from full-model inner products. Across nine publicly released reinforcement learning from verifiable rewards (RLVR) stages, all 131 adjacent pairs have negative cosines on their native checkpoint grids, including PPO and Group Relative Policy Optimization (GRPO) trajectories. Supervised fine-tuning comparisons across four model families show configuration-dependent alignment, including negative late-training cosines for Mistral and FP32 Llama. Finally, a controlled OLMo-2-1B study with three paired seeds and a 2,048-token response cap compares group-relative advantages against a learned critic with generalized advantage estimation (GAE). At matched 60-rollout windows, late-training mean cosines are +0.259 and +0.012, respectively; the paired critic-minus-group difference is -0.247 (seed-level 95% interval [-0.358, -0.137]). These findings distinguish spectral concentration, energy coverage, and trajectory direction, and demonstrate that the advantage system influences directional alignment under matched training conditions.
Figures & tables
| Matrix class | Rows | Columns | Count |
|---|---|---|---|
| Attention query, key, value, output | 4,096 | 4,096 | 128 |
| MLP up and gate | 11,008 | 4,096 | 64 |
| MLP down | 4,096 | 11,008 | 32 |
| Token embedding and language-model head | 100,352 | 4,096 | 2 |
| PPO | SFT | |||||
|---|---|---|---|---|---|---|
| Matrix | ||||||
| Query 0 | 9.58 | 0.790 | 532 | 1.10 | 0.987 | 1 |
| MLP up 0 | 115.60 | 0.368 | 2,077 | 356.95 | 0.289 | 2,221 |
| Query 16 | 18.14 | 0.536 | 1,222 | 1.48 | 0.848 | 255 |
| MLP down 16 | 58.78 | 0.345 | 2,120 | 792.32 | 0.277 | 2,250 |
| Query 31 | 18.27 | 0.564 | 1,149 | 1.99 | 0.779 | 599 |
| Public stage | Released method | Width | Pairs | Mean cosine |
|---|---|---|---|---|
| OLMo-2 7B | PPO | 60 | 5 | |
| OLMo-2 13B RLVR1 | PPO | 60 | 12 | |
| OLMo-2 13B RLVR2 | PPO | 60 | 12 | |
| OLMo-2 13B final | PPO | 60 | 12 | |
| OLMo-2 1B RLVR1 | GRPO | 200 | 12 | |
| OLMo-2 1B final | GRPO | 200 | 12 |
| Trajectory | Width and unit | |||
|---|---|---|---|---|
| Public 7B PPO | 60 release steps | (5) | (4) | (3) |
| 7B SFT seed 42 | 500 trainer steps | (9) | (8) | (7) |
| Seed | Group-relative | Critic–GAE | Difference |
|---|---|---|---|
| 42 | |||
| 43 | |||
| 44 | |||
| Mean | |||
| 95% interval |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Paired seeds | 42, 43, 44 |
| Rollout batches | 360 per run |
| Prompts and responses | 8 prompts, 8 responses per prompt |
| Prompt cap and response cap | 2,048 tokens each |
| Sampling | Temperature 1, top- 1, no top- restriction |
| Actor update passes | 4 per rollout batch |
| Arm | Seed | Final correct | Final EOS | Training cap hit |
|---|---|---|---|---|
| Group-relative | 42 | 16/32 | 32/32 | 1.102% |
| Group-relative | 43 | 18/32 | 32/32 | 1.189% |
| Group-relative | 44 | 20/32 | 32/32 | 0.786% |
| Critic–GAE | 42 | 19/32 | 27/32 | 0.291% |
| Critic–GAE | 43 | 18/32 | 28/32 | 0.269% |
| Critic–GAE | 44 | 18/32 | 29/32 | 0.213% |
| Arm | Seed | Generated tokens |
|---|---|---|
| Group-relative | 42 | 5,286,865 |
| Critic–GAE | 42 | 5,832,426 |
| Group-relative | 43 | 4,779,950 |
| Critic–GAE | 43 | 5,861,391 |
| Group-relative | 44 | 5,072,345 |
| Critic–GAE | 44 | 4,776,819 |
| Width | Group-relative seeds | Critic–GAE seeds | Paired 95% interval |
|---|---|---|---|
| 10 | , , | , , | |
| 20 | , , | , , | |
| 40 | , , | , , | |
| 60 | , , | , , |
| Trajectory | Separation 1 | Separation 2 | Separation 3 |
|---|---|---|---|
| Public 7B PPO | |||
| 7B SFT seed 42 | |||
| Group-relative seed 42 | |||
| Group-relative seed 43 | |||
| Group-relative seed 44 | |||
| Critic–GAE seed 42 |
| Stage | Minimum coefficient | Maximum coefficient |
|---|---|---|
| OLMo-2 7B | ||
| OLMo-2 13B RLVR1 | ||
| OLMo-2 13B RLVR2 | ||
| OLMo-2 13B final | ||
| OLMo-2 1B RLVR1 | ||
| OLMo-2 1B final |
| Model | Rows | Columns | Count |
| OLMo-2 1B final | 100,352 | 2,048 | 2 |
| 2,048 | 8,192 | 16 | |
| 8,192 | 2,048 | 32 | |
| 2,048 | 2,048 | 64 | |
| OLMo-2 13B final | 100,352 | 5,120 | 2 |
| 5,120 | 13,824 | 40 |
| Run | Matrix | Relative stable-rank error | Energy error | Original angle | Full-SVD angle |
|---|---|---|---|---|---|
| PPO | Query 0 | 3.9e-07 | 0.00514 | 0.31404 | 0.29726 |
| PPO | MLP up 0 | 1.5e-06 | 0.00939 | 0.09487 | 0.07653 |
| PPO | Query 16 | 5.7e-07 | 0.00818 | 0.09691 | 0.05868 |
| PPO | MLP down 16 | 4.3e-07 | 0.01029 | 0.08623 | 0.08062 |
| PPO | Query 31 | 1e-07 | 0.00761 | 0.13848 | 0.13671 |
| PPO | MLP up 31 | 1.7e-07 | 0.01044 | 0.13122 | 0.17695 |
| Model | Precision | Seed | Mean cosine | Positive | Negative |
|---|---|---|---|---|---|
| OLMo-2-7B | BF16 | 42 | 9/9 | 0/9 | |
| OLMo-2-7B | BF16 | 43 | 9/9 | 0/9 | |
| OLMo-2-7B | BF16 | 44 | 9/9 | 0/9 | |
| Llama-3.2-3B | BF16 | 42 | 9/9 | 0/9 | |
| Llama-3.2-3B | BF16 | 43 | 9/9 | 0/9 | |
| Llama-3.2-3B | BF16 | 44 | 9/9 | 0/9 |
| Model | Precision | Seed | Sep. 2 | Sep. 3 | Width 1,000 | Width 1,500 |
|---|---|---|---|---|---|---|
| Llama-3.2-3B | BF16 | 42 | ||||
| Llama-3.2-3B | BF16 | 43 | ||||
| Llama-3.2-3B | BF16 | 44 | ||||
| Qwen2.5-3B | BF16 | 42 | ||||
| Mistral-7B-v0.3 | BF16 | 42 | ||||
| Mistral-7B-v0.3 | BF16 | 43 |
| Seed 42 | Seed 43 | |||
|---|---|---|---|---|
| Endpoint | BF16 | FP32 | BF16 | FP32 |
| 1000 | ||||
| 1500 | ||||
| 2000 | ||||
| 2500 | ||||
| 3000 | ||||