GateTrim: Adaptive-Rank Quantization for Large Language Models

Hongwei Ren; Jincheng Ren; Yuchuan Zhou; Jingchun Ling

cs.LGOct 31, 2026

GateTrim: Adaptive-Rank Quantization for Large Language Models

Authors: Hongwei Ren, Jincheng Ren, Yuchuan Zhou, Jingchun Ling

Abstract

We introduce GateTrim, an adaptive-rank post-training quantization method for large language models that learns matrix-specific ranks through gate optimization and threshold-based rank selection. GateTrim standardizes the low-rank factors of a quantized low-rank--residual representation, learns a separate gate vector for each target weight matrix, and uses a shared threshold to automatically determine its retained rank. The resulting ranks can differ across matrices, and the compact quantized representations are subsequently reconstructed. We evaluate GateTrim under W4A8 on Qwen3-4B/8B/14B, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct. Across all five models, we observe a counterintuitive finding: increasing the threshold from $τ=0.3$ to $τ=0.7$ reduces the mean retained rank by 46.60%--67.89% while simultaneously lowering WikiText-2 perplexity by 0.59%--5.17%. Threshold 0.7 achieves the lowest mean retained rank and final perplexity among the three tested thresholds, 0.3, 0.5, and 0.7, and reduces measured target-export size by 1.50%--4.25% relative to 0.3. GateTrim uses an $8\times512$ -token fitting pool and a separate development pool of the same size. Its Qwen3-4B/8B/14B perplexities at $τ=0.7$ are numerically below TwinQuant's published W4A8 values, reported with a $128\times2048$ -token calibration pool, in a cross-study comparison. On Qwen3-8B, the threshold-0.7 configuration already has the lowest perplexity before reconstruction under both tested initialization seeds; reconstruction improves it further without increasing retained rank or export size.

Figures & tables

Figure 1: GateTrim quantization with learned matrix-specific ranks. Factors are standardized by row (U) and column (V) before quantization; continuous gates determine each matrix’s retained rank under a shared threshold. Hatching marks unselected components in a schematic example. The dense residual retains its dimensions. S1 updates normalization statistics, while S2 and S3 reuse frozen S1 statistics; S3 fixes the selected rank.

Figure 2: The complete three-threshold sweep. Axes are panel-specific; points compare configurations within a model. Lines connect measured settings and do not represent interpolation experiments.

Figure 3: Retained rank at threshold 0.7 for every target projection. Layer indices are zero-based. All heatmaps use the same 1–128 scale; no missing values are imputed.

Figure 4: Qwen3-8B seed-0 PPL at the slice, q0, and post endpoints for each threshold. Lines connect three discrete processing states; they are not optimizer convergence traces. Within each threshold, target dimensions and measured export bytes are unchanged. Table 4 gives the complete values.

Figure 5: Development NMSE before (q0) and after (post) reconstruction for all 252 Qwen3-8B target matrices at threshold 0.7, shown separately for initialization seeds 0 and 1. The dashed diagonal marks equality; points below it have lower NMSE after reconstruction. Both axes use logarithmic scales.

Figure 6: Exact decomposition of measured target-file bytes at threshold 0.7. The denominator excludes non-target model parameters; it is not complete-checkpoint storage. Detailed code, scale, and metadata accounting is retained in the authors’ records.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Retained ranks across all target matrices at each tested threshold, initialization seed 0. Boxes span the 25th–75th percentiles, center lines show medians, whiskers extend to observed values within 1.5 interquartile ranges, and circles show the remaining observations. Each model uses the same 1–128 rank scale. These distributions describe matrix-to-matrix variation, not uncertainty across seeds. Values are taken from existing post-reconstruction exports; retained indices are fixed during reconstruction.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 416 papers

Oct 30, 2026

Quantization of Large Models with Weight Outliers and Sparsity

Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contribution decomposes the original trained weights into a quantized matrix, a low-rank component, and a sparse matrix that accounts for outlier rows in the weights. Due to analytical intractability and the discrete structure of the decomposition, a simulated annealing search is employed to explore candidate decompositions while avoiding poor local minima. Numerical tests demonstrate the effectiveness of this approach across a broad suite of large language models, including Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, and Qwen2.5 7B, evaluated on natural language generation benchmarks using 4-bit NF4 and FP4 quantizers. The novel method can markedly outperform direct quantization while incurring minimal memory overhead.

Leonidas Tsigkounakis, Abraham Jaeger Mountain, Yilang Zhang +1

Department of ECE, University of Minnesota, Minneapolis, MN 55455 · ML Research, Morgan Stanley, New York, NY 10019

LLM Quantization Neural Network Quantization Post-Training Quantization Weight-Only Quantization

Ask PDF

Mar 30, 2026

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

We present PolarQuant, a post-training weight quantization method for large language models (LLMs) that exploits the distributional structure of neural network weights to achieve near-lossless compression. PolarQuant operates in three stages: (1) block-wise normalization to the unit hypersphere, (2) Walsh-Hadamard rotation to transform coordinates into approximately Gaussian random variables, and (3) quantization with centroids matched to the Gaussian distribution. Our ablation reveals that Hadamard rotation alone accounts for 98% of the quality improvement, reducing Qwen3.5-9B perplexity from 6.90 (absmax Q5) to 6.40 (Delta = +0.03 from FP16), making it practically lossless without any calibration data. Furthermore, PolarQuant functions as an effective preprocessing step for downstream INT4 quantizers: PolarQuant Q5 dequantized and re-quantized by torchao INT4 achieves perplexity 6.56 versus 6.68 for direct absmax INT4, while maintaining 43.1 tok/s throughput at 6.5 GB VRAM. Code and models are publicly available.

Caio Vicentino

LLM Quantization Neural Network Quantization Post-Training Quantization Weight-Only Quantization

Ask PDF

May 30, 2026

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment. However, applying them sequentially poses a problem: PTQ often leaves behind random noise that is spread out (across the model's weights) in a way LoRA can't easily fix, meaning that LoRA ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance. In this paper, we propose ProjQ, a novel framework for constraining quantization noise to the low-rank manifold via orthogonal subspace projection. We derive an efficient alternating algorithm that shapes the quantization noise into a low-rank structure, effectively offloading dominant error components to the subsequent adapter while minimizing the residual error in the orthogonal "uncorrectable" subspace. Our theoretical analysis demonstrates that ProjQ preserves strictly greater model plasticity for downstream tasks compared to standard PTQ. Extensive experiments on LLaMA-2, Qwen2.5 and Qwen3 confirm that ProjQ consistently outperforms existing methods in both quantization error compensation and downstream task fine-tuning, achieving up to

2\times

lower evaluation loss for compensation and matching the performance of standard 4-bit baselines on language modeling tasks with only 3 bits. The code is available on https://github.com/yy9301/ProjQ .

Wenya Yu, Chao Zhang, Li Wang +2

School of Mathematics and Statistics, Central South University, Changsha, China · Department of Computer Science, Central South University, Changsha, China · CRAN, Université de Lorraine, CNRS, Nancy, France +1

LLM Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

Quantization of Large Models with Weight Outliers and Sparsity

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression