PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

Senmiao Wang; Tiantian Fang; Haoran Zhang; Yushun Zhang; Kunxiang Zhao; Alex Schwing; Ruoyu Sun

cs.LGJun 4, 2026

PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

Authors: Senmiao Wang, Tiantian Fang, Haoran Zhang, Yushun Zhang, Kunxiang Zhao, Alex Schwing, Ruoyu Sun

Organizations: The Chinese University of Hong Kong, Shenzhen, China · Google LLC, Mountain View, CA, United States · Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen2026 Research Institute of Big Data · University of Illinois at Urbana-Champaign, Urbana, IL, United States

Abstract

We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at https://github.com/Empath-aln/PC-layer.

Explore similar work

FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
May 19, 2026Yequan Zhao, Ruijie Zhang, Liyan Tan +3Parameter-Efficient Fine-Tuning Low-Rank Adaptation
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
May 12, 2026Kexuan Shi, Hanxuan Li, Zeju Qiu +3Spectral Preconditioning Large Language Model Pretraining
PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
Jul 8, 2026Yazdan Jamshidi, Alexey ShvetsHeterogeneous Pruning Scores Large Language Model Compression

May 19, 2026 · cs.LGJ/K move · Enter open · S save

FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

Yequan Zhao, Ruijie Zhang, Liyan Tan, Niall Moran+2

University of California at Santa Barbara · Amazon Lab126

Both full fine-tuning (Full FT) and parameter-efficient fine-tuning methods such as LoRA introduce weight updates without accounting for the spectral structure established during pretraining. As a result, noisy gradients from limited fine-tuning data can perturb robust pretrained features. We identify spectral preconditioning as the missing ingredient: reparameterizing each weight matrix through its full-rank singular value decomposition (SVD) and freezing one singular basis constrains updates to the pretrained column space, yielding a preconditioned optimization scheme that outperforms unconstrained Full FT at the same trainable parameter count. Building on this insight, we propose FuRA (Full-Rank Adaptation), an efficient full-rank adaptation framework based on a block tensor-train factorization W = LSR, where the large core L is fixed to the pretrained block-wise SVD basis, while only the compact core R and the block-wise singular values S are optimized. This design simultaneously provides full-rank spectral preconditioning, preserves full-rank update expressivity, and achieves parameter, memory, and step-time efficiency comparable to LoRA. FuRA consistently outperforms Full FT across multiple settings, including LLM fine-tuning (+1.37 on LLaMA-3-8B commonsense reasoning), LLM reinforcement learning for mathematical reasoning, and visual instruction tuning for VLMs. Furthermore, the 4-bit quantized variant, QFuRA, also surpasses QLoRA. Code is available at https://github.com/olokevin/FuRA-NIPS

Parameter-Efficient Fine-Tuning Low-Rank Adaptation Spectral Preconditioning Singular Value Decomposition

Open paper Ask about it PDF

Abstract

Explore similar work

FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning