Do Orthogonality Constraints Help Language-Model Training Beyond Sphere Optimization?
Abstract
A growing body of work reports benefits from orthogonality constraints in fine-tuning large language models (LLMs). These results raise an open question: do orthogonality constraints also help LLM pre-training? We conduct a systematic empirical study of decoder-only LLM pre-training with models ranging from 360M to 3.1B parameters, trained on nearly 200B tokens. We examine fourteen variants of orthogonality training: some constrain the weight matrix itself, while others use orthogonal factors to reparameterize it. Because a strictly orthogonal square matrix loses half of its degrees of freedom, we also study block-wise orthogonality, where every rows form a row-orthonormal (Stiefel) matrix and the block size is a hyperparameter. Among block sizes ranging from 1 to the shorter side of the matrix, no used throughout the network reliably outperforms , and large blocks are clearly worse; the one placement that helps, symmetric positive-definite factors on the query and key projections, loses its small gain as models grow. At , the constraint reduces to sphere optimization, with each row direction on its own unit sphere, which by itself already improves on tuned Muon. The same two findings hold for a diffusion transformer trained on ImageNet, and after fine-tuning, downstream accuracy does not separate the parameterizations. Our conclusion is that, across the realizations we tried, orthogonality between rows does not help language-model pre-training beyond sphere optimization. Since this appears to conflict with several prior results, we re-examine five prior methods in pre-training and fine-tuning that report orthogonality improvements. In each case the reported gain is attributable to numerical precision, training horizon, optimizer realization, or per-row norm control rather than to the orthogonality the methods impose.
Figures & tables
| Input: , momentum , gradient , learning rate , scale . | ||
|---|---|---|
| 1: | , | Muon direction |
| 2: | tangent step | |
| 3: | polynomial retraction | |
| Trained on C4 | Trained on FPO-mix | |||||
| Method | 360M | 360M | 1.3B | 3.1B | 360M | 1.3B |
| Muon | 2.475 | 2.372 | 2.197 | 2.036 | 2.682 | 2.413 |
| MuonH (fixed matrix norm) | 2.469 | 2.362 | 2.191 | 2.031 | 2.677 | 2.407 |
| OrthMuon (sphere + gain) | 2.462 | 2.356 | 2.184 | 2.024 | 2.671 | 2.400 |
| OrthMuon | 2.463 | 2.358 | – | – | – | – |
| OrthMuon | 2.467 | 2.360 | 2.186 | 2.025 | 2.675 | – |
| Realization at | 360M | 1.3B | 3.1B |
|---|---|---|---|
| Per-row gain | +5.2 | +1.9 | +0.8 |
| Per-block gain | +6.5 | – | – |
| SPD gain | |||
| XXT gain | +0.2 | – | – |
| Projected DGram | +18.8 | – | – |
| Manifold Muon | +35.5 | +27.6 | – |
| SPD factor on | 360M, | 360M, | 1.3B, |
|---|---|---|---|
| Query and key | |||
| Query only | |||
| Key only | |||
| Value only | |||
| Output only | |||
| MLP gate only |
| Method | |||
|---|---|---|---|
| Forward-normalized Muon | 5.5 | – | – |
| MuonH (fixed matrix norm) | 6.7 | 10.8 | 9.8 |
| , no gain | 7.2 | 10.1 | 7.1 |
| , row gain (total) | 13.1 | 17.7 | 15.7 |
| Study | Comparison | Finding |
|---|---|---|
| POET, Pion | fp32 AdamW state; longer training | POET: 31 to 9 milli-loss with fp32 state; behind AdamW at 6B and 30.1B tokens. Pion: behind bfloat16 and fp32 AdamW at 1.3B. |
| Different Layers | DGram projection versus reparameterization; module assignment | Projection trails Muon, , and the reparameterization. Neither DGram in MLP nor per-row in attention improves on . |
| StelLA | Sphere-constrained factor columns | Sphere gains 2.1 points over Euclidean; Stiefel then changes accuracy by . |
| PSOFT | Row-sphere transform in the same selected subspace | Row-sphere scores 0.5 points above the implemented transform at validation-selected rates; the ordering reverses with learning rate. |
| (a) POET, 130M, 0.2B tokens | |||
| Storage | AdamW | POET | |
| bfloat16 | 3.943 | 3.912 | |
| fp32 | 3.793 | 3.785 | |
| (b) Pion, 1.3B, 51B tokens, context 4096 | |||
| AdamW storage | AdamW | Pion | |
| bfloat16 | 2.346 | 2.361 | |
Appendix figures & tables52 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Budget | Steps | Train tokens |
|---|---|---|---|
| 360M | 13,732 | 14.399B | |
| 360M | 27,464 | 28.798B | |
| 360M | 52,800 | 55.365B | |
| 1.3B | 48,584 | 50.944B | |
| 1.3B | 97,168 | 101.888B | |
| 1.3B | 188,739 | 197.907B |
| Model | Layers | Q heads | KV heads | Head dim. | Parameters | Attention | ||
|---|---|---|---|---|---|---|---|---|
| 360M | 24 | 1024 | 3072 | 16 | 16 | 64 | 359.97M | MHA |
| 1.3B | 24 | 2048 | 6144 | 32 | 16 | 64 | 1.274B | GQA ( ) |
| 3.1B | 32 | 2816 | 8448 | 22 | 11 | 128 | 3.135B | GQA ( ) |
| Input: blocks and gain factors or , ; momentum ; gradient ; learning rate ; step ; correction steps , 50 evenly spaced steps. | ||
|---|---|---|
| 1: | , | Nesterov, |
| 2: | whole matrix | |
| 3: | for in parallel do | |
| 4: | tangent step | |
| 5: | (fp32) | Gram error |
| 6: | Horner form | |
| 360M | 1.3B | |||
|---|---|---|---|---|
| Method | k tokens/s | Change | k tokens/s | Change |
| Muon | 2134 | – | 870.3 | – |
| OrthMuon | 2095 | 856.2 | ||
| OrthMuon | 2095 | – | – | |
| OrthMuon | 2107 | 871.0 | ||
| Variant | Results | |
| OrthMuon with block-Stiefel directions; left factor in | ||
| 1 | Per-row gain | Table 1 , Figure 1 |
| 2 | One scalar gain per block | Table 26 |
| 3 | SPD | Tables 26 , 28 |
| 4 | Tables 26 , 28 | |
| 5 | , updated with Muon | Table 26 |
| Budget | Contrast | Mean | 95% interval | Worse seeds |
|---|---|---|---|---|
| +0.81 | 2/3 | |||
| +5.19 | 3/3 | |||
| +0.87 | 2/3 | |||
| +5.09 | 3/3 | |||
| +1.10 | 3/3 | |||
| +3.71 | 3/3 |
| Budget | Seed 42 | Seed 1234 | Seed 5678 | Mean SD | |
|---|---|---|---|---|---|
| 1 | 2.4616 | 2.4631 | 2.4617 | 2.4621 0.0009 | |
| 8 | 2.4637 | 2.4613 | 2.4639 | 2.4629 0.0014 | |
| 32 | 2.4660 | 2.4679 | 2.4681 | 2.4673 0.0012 | |
| 1 | 2.3999 | 2.4001 | 2.4006 | 2.4002 0.0004 | |
| 8 | 2.4018 | 2.3995 | 2.4018 | 2.4011 0.0013 | |
| 32 | 2.4043 | 2.4057 | 2.4058 | 2.4053 0.0008 |
| drop per | Extra steps | ||||
|---|---|---|---|---|---|
| Budget | Per-row | step doubling | needs | ||
| 2.4621 | 2.4673 | 62.0 ( ) | 6.0% | ||
| 2.4002 | 2.4053 | 46.4 ( ) | – | ||
| 2.3564 | 2.3601 | – | 5.7% | ||
| 2.3200 | 2.3224 | 34.5 ( ) | 5.0% |
| Method | |||
|---|---|---|---|
| Tuned Muon | 2.475 | 2.418 | 2.372 |
| MuonH | 2.469 | 2.407 | 2.362 |
| , no gain | 2.468 | 2.408 | 2.365 |
| , row gain | 2.462 | 2.400 | 2.356 |
| Component | |||
|---|---|---|---|
| Unit-row direction constraint, no gain | 7.2 | 10.1 | 7.1 |
| Learned row magnitude | 5.9 | 7.5 | 8.6 |
| Total improvement | 13.1 | 17.7 | 15.7 |
| Model | Budget | Muon | MuonH | Muon | MuonH | |
|---|---|---|---|---|---|---|
| 1.3B | 2.1970 | 2.1911 | 2.1842 | |||
| 1.3B | 2.1524 | 2.1464 | 2.1393 | |||
| 1.3B | 2.1198 | 2.1141 | 2.1079 | |||
| 3.1B | 2.0362 | 2.0307 | 2.0242 |
| Model | / gain | Loss | |
|---|---|---|---|
| 1.3B | , per row | 2.1842 | – |
| 1.3B | , per row | 2.1861 | |
| 1.3B | , SPD | 2.1832 | |
| 3.1B | , per row | 2.0242 | – |
| 3.1B | , per row | 2.0249 | |
| 3.1B | , SPD | 2.0239 |
| Method | LR | Final C4 loss |
|---|---|---|
| MuonH | 0.00625 | 2.7014 |
| MuonH | 0.0125 | 2.6759 |
| MuonH | 0.025 | 2.7140 |
| OrthMuon | 0.001 | 2.7075 |
| OrthMuon | 0.002 | 2.6695 |
| OrthMuon | 0.004 | 2.6818 |
| Method | Mean loss SD | |
|---|---|---|
| OrthMuon | 3 | 2.6709 0.0014 |
| OrthMuon , per-row gain | 3 | 2.6753 0.0004 |
| MuonH | 3 | 2.6771 0.0011 |
| Muon | 3 | 2.6824 0.0013 |
| Budget | Method | Directional LR | Final loss |
|---|---|---|---|
| 0.002 | 2.6709 | ||
| SPD | 0.001 | 2.7057 | |
| SPD | 0.002 | 2.6705 | |
| SPD | 0.004 | 2.6806 | |
| 0.002 | 2.6078 | ||
| SPD | 0.001 | 2.6264 |
| Method | Selected LR / WD | Mean loss SD | |
|---|---|---|---|
| OrthMuon | 0.001 | 3 | 2.4004 0.0005 |
| MuonH | 0.008 | 3 | 2.4071 0.0001 |
| Muon | 0.002 / 0.3 | 3 | 2.4127 0.0003 |
| Natural-language-heavy | Code/math/LaTeX-rich | ||||||
|---|---|---|---|---|---|---|---|
| Method | C4 | FPO mix | DCLM | OLM-Wiki | StarCoder | ProofPile | OLM-arXiv |
| AdamW | 2.5511 | 3.0992 | 3.0984 | 3.2295 | 3.0594 | 3.9805 | 2.8222 |
| Muon | 2.4752 | 3.0203 | 3.0208 | 3.1395 | 2.9839 | 3.7714 | 2.7372 |
| MuonH | 2.4686 | 3.0197 | 3.0158 | 3.1452 | 3.0407 | 3.8348 | 2.7597 |
| , no gain | 2.4681 | 3.0162 | 3.0161 | 3.1461 | 2.9789 | 3.8657 | 2.7456 |
| 2.4621 | 3.0113 | 3.0070 | 3.1254 | 3.0315 | 3.8587 | 2.7449 | |
| Natural-language-heavy | Code/math/LaTeX-rich | ||||||
|---|---|---|---|---|---|---|---|
| Method | C4 | FPO mix | DCLM | OLM-Wiki | StarCoder | ProofPile | OLM-arXiv |
| Muon | 2.3721 | 2.9399 | 2.9390 | 3.0258 | 2.9310 | 3.7282 | 2.6618 |
| MuonH | 2.3623 | 2.9340 | 2.9353 | 3.0572 | 2.8948 | 3.7337 | 2.6993 |
| , no gain | 2.3650 | 2.9286 | 2.9345 | 3.0460 | 2.8328 | 3.7094 | 2.6817 |
| 2.3564 | 2.9266 | 2.9284 | 3.0390 | 2.8769 | 3.7481 | 2.6554 | |
| , per row | 2.3575 | 2.9266 | 2.9322 | 2.9809 | 2.8194 | 3.7563 | 2.6390 |
| Natural-language-heavy | Code/math/LaTeX-rich | ||||||
| Method | C4 | FPO mix | DCLM | OLM-Wiki | StarCoder | ProofPile | OLM-arXiv |
| AdamW | |||||||
| Muon | |||||||
| MuonH | |||||||
| , no gain | |||||||
| Corpus | range | range | ||
|---|---|---|---|---|
| C4 | 1.6 | – | 1.5 | – |
| FPO mix | 8.4 | 6.8 | ||
| DCLM | 9.4 | 7.9 | ||
| OLM-Wiki | 26.1 | 20.9 | ||
| StarCoder | 78.2 | 106.3 | ||
| Realization | Loss | ||
|---|---|---|---|
| 360M; | |||
| Stiefel, per-block gain | 3 | 2.4686 | +6.5 |
| SPD | 3 | 2.4619 | |
| XXT | 3 | 2.4624 | +0.2 |
| XXT + diagonal | 3 | 2.4614 | |
| Projected DGram | 3 | 2.4810 | +18.8 |
| Parameterization | Gain granularity | Loss | vs. | |
|---|---|---|---|---|
| per row | 3 | 2.4621 | 0.0000 | |
| , scalar | per block | 3 | 2.4686 | +0.0065 |
| , SPD | full matrix | 3 | 2.4619 | |
| , XXT | full matrix | 3 | 2.4624 | +0.0002 |
| , XXT + diagonal | full matrix | 3 | 2.4614 | |
| , symmetric-square + Muon | full matrix | 3 | 2.4626 | +0.0005 |
| Left factor | Form | Loss | |
|---|---|---|---|
| Per-row | 2.4673 | +5.2 | |
| SPD | 2.4619 | ||
| SPD + diagonal | 2.4614 | ||
| XXT | 2.4624 | +0.2 | |
| Matrix exponential | , | 2.4629 | +0.8 |
| Indefinite symmetric | 2.4635 | +1.3 |
| Seed | Final C4 loss |
|---|---|
| 42 | 2.1832 |
| 1234 | 2.1834 |
| 5678 | 2.1832 |
| Mean SD | 2.1832 0.0001 |
| Per-row gain | SPD gain | Per-row gain | SPD gain | |
|---|---|---|---|---|
| 128 | 2.4814 | 2.4697 | 2.4174 | 2.4066 |
| 256 | 2.4993 | 2.4792 | 2.4345 | 2.4154 |
| 1024 | 2.6172 | 2.5465 | 2.5524 | 2.4731 |
| Per-row gain | SPD gain | |||
|---|---|---|---|---|
| 128 | 2.1924 | 2.1886 | +8.2 | +4.3 |
| 256 | 2.1989 | 2.1968 | +14.7 | +12.5 |
| 1024 | 2.2411 | 2.2578 † | +56.9 | +73.5 |
| Budget | Gain | Manifold Muon | OrthMuon same | ||
|---|---|---|---|---|---|
| 32 | per row | 2.4977 | 2.4673 | 2.4621 | |
| 32 | SPD | 2.4947 | 2.4619 | 2.4621 | |
| 256 | per row | 2.5064 | 2.4993 | 2.4621 | |
| 1024 | per row | 2.6041 | 2.6172 | 2.4621 | |
| 32 | per row | 2.4280 | 2.4053 | 2.4002 | |
| 32 | SPD | 2.4260 | 2.3996 | 2.4002 |
| Gain | Manifold Muon | OrthMuon same | vs. ( ) | |
|---|---|---|---|---|
| 32 | per row | 2.2118 | 2.1861 | +27.6 |
| 32 | SPD | 2.2110 | 2.1832 | +26.8 |
| 256 | per row | 2.2076 | 2.1989 | +23.4 |
| 1024 | per row | 2.2468 | 2.2411 | +62.6 |
| Arm | Mean | p95 | Stable rank | Condition number | |
|---|---|---|---|---|---|
| Random unit rows | 0.025 | – | 1.34 | 23.9 | 1.38 |
| Muon | 0.032 | 0.080 | 1.385 | 20.4 | 1.61 |
| MuonH | 0.034 | 0.085 | 1.385 | 20.5 | 1.61 |
| 0.030 | 0.076 | 1.380 | 21.6 | 1.54 | |
| , gain LR 0.0048 | 0.030 | 0.076 | 1.377 | 19.2 | 1.74 |
| Per-row | 1.000 | 25.3 | 1.30 |
| Muon | SPD | |||||
|---|---|---|---|---|---|---|
| Matrix | Mean | p95 | Mean | p95 | Mean | p95 |
| Query | 0.052 | 0.162 | 0.047 | 0.138 | 0.105 | 0.328 |
| Key | 0.055 | 0.159 | 0.047 | 0.134 | 0.111 | 0.324 |
| Value | 0.045 | 0.149 | 0.031 | 0.086 | 0.048 | 0.132 |
| Attention output | 0.028 | 0.071 | 0.028 | 0.068 | 0.047 | 0.116 |
| MLP gate | 0.026 | 0.065 | 0.026 | 0.065 | 0.044 | 0.112 |
| Window | Arm | Mean | Stable rank | Condition number | |
|---|---|---|---|---|---|
| 128 | Random unit rows | 0.025 | 1.80 | 71.1 | 2.04 |
| Muon | 0.030 | 2.00 | 58.5 | 2.47 | |
| 0.028 | 1.96 | 61.7 | 2.35 | ||
| Per-row | 0.021 | 1.72 | 69.0 | 2.14 | |
| SPD | 0.035 | 2.46 | 48.3 | 2.88 | |
| 1024 | Random unit rows | 0.025 | 3.94 | 260 | – |
| LR | WD 0.1 | WD 0.3 | WD 1.0 |
|---|---|---|---|
| 0.002 | 2.5140 | 2.4879 | 2.4907 |
| 0.004 | 2.5010 | 2.4808 | 2.5055 |
| 0.008 | 2.8647 | 2.5073 | 2.5461 |
| Block size | Selected LR | Final loss |
|---|---|---|
| 32 | 0.002 | 2.5445 |
| 128 | 0.002 | 2.5581 |
| 1024 | 0.004 | 2.6881 |
| MLP replaced | Attention replaced | |||
|---|---|---|---|---|
| Replacement | Loss | Loss | ||
| Muon | 2.4712 | +10.8 | 2.4679 | +7.5 |
| MuonH | 2.4617 | +1.4 | 2.4679 | +7.5 |
| Per-row | 2.4630 | +2.6 | 2.4635 | +3.2 |
| SPD | 2.4619 | +1.6 | 2.4582 | |
| Projected DGram | 2.4730 | +12.6 | 2.4707 | +10.3 |
| SPD factor on | Seed 42 | Seed 1234 | Seed 5678 | Mean | |
| 360M, | |||||
| None (reference) | 2.4608 | 2.4595 | 2.4608 | 2.4604 | – |
| Query and key | 2.4556 | 2.4557 | 2.4561 | 2.4558 | |
| Query | 2.4580 | 2.4583 | 2.4583 | 2.4582 | |
| Key | 2.4581 | 2.4580 | 2.4582 | 2.4581 | |
| Value | 2.4609 | 2.4607 | 2.4623 | 2.4613 | |
| Arm | Max logit | Median head max | Max | Max-logit range |
|---|---|---|---|---|
| Reference | 112.7 | 9.5 | 78.3 | 56–222 |
| Block size 64 | 110.6 | 10.0 | 144.8 | 42–214 |
| Block size 128 | 1976.0 | 10.7 | 402.2 | 307–5170 |
| Model, budget | Mean 95% half-width | Seeds 42 / 1234 / 5678 |
|---|---|---|
| 360M, | / / | |
| 360M, | / / | |
| 360M, | / / | |
| 1.3B, | / / | |
| 3.1B, | / / |
| Variant | Final loss | |
|---|---|---|
| Default OrthMuon | 3 | 2.4621 |
| Direct row normalization | 3 | 2.4622 |
| Exponential gain | 3 | 2.4625 |
| Tangent gradient before momentum | 3 | 2.4637 |
| Tangent + transported momentum | 3 | 2.4628 |
| Block-Frobenius tangent + row gain | 3 | 2.4641 |
| Family | Learning rate (validation loss) | Selected |
|---|---|---|
| AdamW | 0.0001 (0.7578), 0.0002 (0.7562), 0.0004 (0.7561), 0.0008 (0.7575) | 0.0004 |
| MuonW | 0.0005 (0.7535), 0.001 (0.7532), 0.002 (0.7544), 0.004 (0.7573) | 0.001 |
| OrthAdam | 0.0001 (0.7576), 0.0002 (0.7554), 0.0004 (0.7562) | 0.0002 |
| OrthMuon | 0.000125 (0.7537), 0.00025 (0.7525), 0.0005 (0.7540) | 0.00025 |
| Method | 500K EMA validation loss | FID-50K | IS |
|---|---|---|---|
| AdamW | 0.75623 | 25.50 | 56.69 |
| OrthAdam | 0.75574 | 24.60 | 59.51 |
| OrthAdam query–key-only | 0.75607 | 25.28 | 58.67 |
| MuonW | 0.75321 | 21.55 | 65.35 |
| OrthMuon | 0.75242 | 20.81 | 67.54 |
| OrthMuon query–key-only | 0.75264 | 21.06 | 67.01 |
| Difference | validation loss | FID | IS |
|---|---|---|---|
| OrthAdam AdamW | |||
| OrthMuon MuonW | |||
| OrthAdam query–key | |||
| OrthMuon query–key |
| Arm | Seeds | Diverged | Query–key scale at 480K |
|---|---|---|---|
| AdamW | 3 | 1 | 59–629 |
| OrthAdam | 3 | 1 | 63–140 |
| OrthAdam query–key-only | 3 | 0 | 3.5–6.0 |
| OrthAdam | 2 | 0 | 6.2–8.8 |
| OrthAdam SPD | 1 | 0 | 73 |
| MuonW | 1 | 0 | 457 |
| Arm | Own optimizer | AdamW | AdamW own |
| 70.92 | 72.72 | ||
| Per-row | 71.20 | 72.93 | |
| SPD | 69.65 | 72.63 | |
| MuonH | 68.56 | 71.66 | |
| Muon | 69.82 | 72.06 | |
| All- , gain LR 0.0048 | 71.45 | 73.03 |
| Arm control | Own optimizer | AdamW |
|---|---|---|
| Per-row | ||
| SPD | ||
| MuonH | ||
| Muon | ||
| Query–key SPD all- | ||
| Query–key SPD all- |
| Tokens | POET | OrthAdam | AdamW | POET–AdamW |
|---|---|---|---|---|
| 0.59B | 3.8418 | 4.2115 | 4.2121 | |
| 2.0B | 3.1019 | 3.1130 | 3.1625 | |
| 6.0B | 2.8430 | 2.8023 | 2.8350 | |
| 30.1B | 2.6660 | 2.6207 | 2.6471 |
| Storage | AdamW | POET | POET–AdamW |
|---|---|---|---|
| Full bfloat16 (released) | 3.9427 | 3.9115 | |
| fp32 | 3.7933 | 3.7846 |
| Arm | Final loss | vs. bfloat16 AdamW |
|---|---|---|
| AdamW, fp32 | 2.2655 | |
| AdamW, bfloat16 (released) | 2.3458 | 0 |
| Pion, paper configuration, Pion LR 0.00025 | 2.5786 | |
| Pion, paper configuration, Pion LR 0.0005 | 2.4175 | |
| Pion, paper configuration, Pion LR 0.001 | 2.3609 | |
| Pion, paper configuration, Pion LR 0.002 | 2.3741 |
| Method | Accuracy (mean SD) | vs. Sphere |
|---|---|---|
| Euclidean | 84.537 0.197 | |
| Euclidean + factor decay | 86.408 0.041 | |
| Product of spheres | 86.653 0.190 | 0 |
| Canonical Stiefel | 86.564 0.159 |
| Method | Seed 42 | Seed 43 | Seed 44 | Seed 45 | Mean SD |
|---|---|---|---|---|---|
| Euclidean | 84.405 | 84.814 | 84.387 | 84.543 | 84.537 0.197 |
| Euclidean + U/V decay (1.0) | 86.468 | 86.399 | 86.385 | 86.380 | 86.408 0.041 |
| Product of spheres | 86.514 | 86.616 | 86.553 | 86.930 | 86.653 0.190 |
| Canonical Stiefel | 86.508 | 86.536 | 86.422 | 86.791 | 86.564 0.159 |
| Direct-polar Stiefel ∗ | 86.641 | – | – | – | – |
| Method | |||
|---|---|---|---|
| Euclidean, no decay | 84.602 | 84.596 | 84.777 |
| Product of spheres | 84.835 | 86.650 | 86.136 |
| Canonical Stiefel | 84.835 | 86.690 | 86.256 |
| U/V weight decay | Accuracy |
|---|---|
| 0 | 84.596 |
| 0.01 | 84.542 |
| 0.03 | 85.187 |
| 0.1 | 85.148 |
| 0.3 | 86.250 |
| 1.0 | 86.732 |
| Latent transformation | LR | Seed 42 | Seed 43 | Seed 44 | Mean |
|---|---|---|---|---|---|
| PSOFT (Cayley/Neumann) | 81.371 | 81.488 | 80.933 | 81.264 | |
| 80.503 | 80.444 | 80.694 | 80.547 | ||
| Dense | 81.047 | 80.984 | 80.828 | 80.953 | |
| 81.439 | 81.237 | 80.936 | 81.204 | ||
| Row-sphere | 81.035 | 81.287 | 80.891 | 81.071 | |
| 81.264 | 81.475 | 80.829 | 81.189 |
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 376 papers